I feel like this also. I think people sometimes mistake me being reasonably polite and mild mannered for me being truly kind. I'm not thoroughly unkind but I doubt I'm as 'nice' as people might think I am.
I've known truly good people in life who literally spent a good deal of their time helping others with no expectation of reward. I'm certainly not actively good like that.
I suppose I don't like to hurt people. It truly bothers me when I think I have. But then, I suppose if I was that kind, I'd take care not to hurt them at all!
I absolutely retaliate though. While that isn't kind, I actually think that's necessary in life otherwise, people will walk all over others. Especially when they think they're meek enough to let them. I won't tolerate that now and I think it's actually better to stand up to people like that to make them realise they can't just bulldoze everyone in their path.
My perceived 'kindness/ goodness' also leads to even weirder situations though. Where other people can see that I'm not all together nice/ good. Maybe they get annoyed that they think I'm falsely representing myself or, that others have bought into the false representation but, they'll start citing where I've been less than pleasant.
It also goes the same with skill level in my job. I'm not that great. I know I'm not that great. I don't market myself as being so. Yet, sometimes when I've received really lovely compliments, others are quick to point out my failings.
Which is actually pretty hurtful in both instances really. Because you feel like- you're not actually claiming to be kind or talented yourself. You're just quietly living your life. Others have decided to give you a compliment, which has somehow annoyed another set of people who then work at belittling you. So you start to think- I wish they'd never even complimented me to begin with! If it's going to provoke a whole bunch of criticism by someone else! Especially when you weren't actually looking for either.