Yes. Its better to get out of that environment..or somehow to make it stop. It completely messes you up mentally..you said that you are not ok with it..and tried very hard to make him know that. And that kind of environment is not good for the kids either .
Its better to sort this problem first.. coz some uncontrollable problems like this go a long way resulting in more negative things. Its better to do it..before more other complicated problems gets added up to this.. and makes you feel you are completely buried underneath them.it definitely doesn't look healthy without a second thought.. and as you said it seems like.. you are not standing up for yourself and allowing yourself to face all that whats going on.
I have the same problem.. i can't stand up for myself sometimes..be it lack of confidence or feeling of unworthiness i dont know.
But, when you are facing abuse..you can not clearly realise the wrong thing thats going on. Once you are out of it..with a clear mind..Thats when it shocks you that it was clearly abuse.
I saw a woman in real life.. she faced abuse for decades..i mean decades. And did not try to solve it. She just thought everything is going fine at that time..things are very messed up there and her kids are messed up too. He is narcissistic , highly manipulative. I am not trying to make you feel guilty by saying this. But, if you dont solve problems.. they turn out very bad that you can't correct them anymore.. and it ruins lives.
If you alone can't confront him..take someone's help.. but if this is the only problem.. may be its better to find ways to solve it. You can take help from a therpaist or counselor(marriage counselor too if thats a thing there). First share it with any of your family or friends who have known you from a long time..and its better to take help from outside.