That is what holds me back too. The effect that it would have on others.
It would be so ideal if those who would be affected could just listen with an open mind, say what they want to say, ask what they want to ask and accept and make peace with our decision.
If you truly feel that this is the right decision for you I hope that you can find a way to handle this aspect of choosing to leave. None of us want to hurt those that we love.
Again, as much as you can, the survival instinct is a part of human nature and something people like us wrestle with, try to make sure that this is the right decision for you. (I'm sure you've been over it many times in your mind.)