Well, considering I haven't met with anyone by myself in well over a decade, I wouldn't really know either way. I think I'd be more comfortable with women, but that's only a guess. Everything has its drawbacks. If I was around a bunch of rowdy guys that would only stress me the hell out, but I guess someone around the same wavelength as me would be alright. Even then, issues of anxiety would probably still crop up to such a point that I'd rather remain isolated.
In a similar way, if I found myself around a woman my age, or younger, then this would, in all likelihood, only make me feel nervous. Even moreso if she were attractive, which would only compound the situation to such a degree that I couldn't really feel comfortable. But again, if we had a lot in common and things didn't get weird, then maybe it'd be alright. It's probably because of the close relationship I have with my mom, but, overall, I guess I'd just feel safer talking/interacting with a woman instead of a man, assuming I had to pick between the two.
At the end of the day, I feel it's true that there are certain topics you can only really talk about candidly with a particular kind of person, regardless of whatever their gender is. It's also true that, for me, I can't really feel comfortable period, unless I happen to already know them. Which itself is a bit of a pickle, since if I'm too anxious to start the process of getting to know anyone, then I'll never reach such a point where I'd actually feel comfortable, leaving me in a predicament of permanent isolation.