As far as I know, the only thing that affects user list integirty is when bots increase the follower account. For instance, before I lost about 200 people, overnight, I saw my numbers go up by about 200 people. So I wsn't surprised when they went.