I'm wondering if it's possible for folks to commit an abuse, but to still not be abusive. Essentially I don't want to overreact and say I'm being abused by someone if they do some abusive actions toward me, but also act normal towards me.
If somebody commits an act of abuse, they are being abusive. That's describing actions in the world, not the state of their internal character.
Without examples, people who habitually commit acts of abuse and then attempt to repair the relationship through normal and pleasant behavior before committing another act of abuse are considered abusers. What you're describing fits that pattern.
There is some gray area in exactly when someone crosses that line, based on what other people perceive and how the victim of an act of abuse feels. It's an issue of fuzzy logic. And we don't really know how to make that line any clearer without resorting to black-and-white thinking.