Live and let live!
If wearing yoga pants makes some guy happy, I would tell him 100% go for it.
Too many conformist on this plant.
Personally, I love seeing those confident and comfortable enough to "think outside the box" and show some individuality.
Personally, I do my best not to conform, and sometimes I wear those tight fitting yoga type pants called compression tights (looks kind of like the wet suit material that scuba divers wear if one doesn't now what this is). I wear this to the gym especially when I take spin classes.
Sometimes people will look at me twice at the gym, but I don't care. Perhaps, these conformist are judging me and wish they had the guts for self expression? And, I'm middle aged, so society in particular is less forgiving with self expression for me vs younger people. But, I don't give a shit.
But, also I've been able to attract females to myself this way. I'm fit. This makes me stand out. Shows self expression. Shows confidence, etc.
In marketing they say, "differentiate your product". In other words, stand out.