Two responses come to mind on this. The first is that a higher power would have to exist, much less give a flying fuck about me, to even bother. The second is that there are plenty of people in the world - abusers and the like - who whether they're obvious about it or not actually are trying to destroy your life and make you miserable. It doesn't take a deity to make feeling like that be spot-on.
Edit: also, the world and people are fucked up enough to create a billion oceans full of misery for every damn one of us, without needing any malice at all. The malicious ones are just shit icing on the shit cake.