Imo , that person can never be your friend, they wouldn't be able to even if they wanted. You should just tell them you are not comfortable, obviously, you know that person best, if they are someone who could understand , tell them upfront but if you feel any kind of fear about them, tell them over the phone, I say this , partly because your friend makes me a bit paranoid but because I KNOW how people can react to things, be safe.
You deserve to have friends who are boys that can understand you and don't see you as a mere object; coming from a very conservative country, most boys that didn't take no from an answer here, as in they keep pressuring the girl or even other boys for things that they want, drugs, money, cheating at school or at work, etc; were not very kind in their day to day life , let's just keep it at that, my life philosophy for any type of relationships (that I make once in a blue moon) is: it's not a failed friendship if you are not compatible, it's a bullet dodged.
Most of my fellow boys and men can't comprehend that, same goes with some older women (which I think might be because of the even shittier boys and men which became fathers of times past), in fact I don't think they want to some days, if I choose someone at random, they most likely won't understand it, they have some sort of superiority complex, they could never see themselves rejected in anything and they will NEVER go "Well that person doesn't have much in common with me, it's fine, that friendship wouldn't have lasted" , they'll always go "I'll show them, thinking they are better than me" , it's always some sort of secret agenda their ego makes up and that can be dangerous, I've seen it, I've experienced it and not only boys but girls to can do that shit, it's better to forget about these people, no matter how hard it might be to comprehend that you can't do anything about it.
If you have the option , politely decline the friendship and end on neutral terms while you can still have a civilized conversation, their friend groups aren't much better than them analzying the fact that they barely understand you or your wishes, don't expect to meet other likeminded people around them from that time you accept their behaviour onwards.