It is ok to do what you need to do for you and to maintain your health.
Some may try to guilt you or control you or become nasty when you do not follow what they expect of you.
But, as far as I am concerned, if they do not get that you need space or time, if they insist that you follow their agenda, then that is on them - not you. Ultimately, it seems to me that this is an attempt to control or coerce you according to what they want, and is toally lacking in empathy and understanding that it is ok for you to do what you need to do.
The other thing? It seems hypocritical to me. I am not saying all people do this, but so very many IME who try to force people into what they want have no problem ignoring when others try to do the same to them. Kinda part and parcel of lack of empathy and consideration, and says so very much about who they are - not you.
And that is their problem not yours IMO.
You do you to maintain your own health.
<3