Yeah I understand your point. One reason I've browsed /cgl/ for so long and made sure to lurk as hard as possible in the lolita community is because I don't want to try and make anybody uncomfortable because of who I am. It's completely understandable as a woman that you'd try to be as skeptical as possible towards any man as self-defense.
Honestly, getting into lolita for me was kind of a serendipitous occasion, I realized lolita would be a ridiculously fun fashion to participate in if I did it right, since I was already friends with a few people who were into it and educated me about it. But that was, y'know, me PERSONALLY, and participating in the larger lolita community takes a lot more effort on my part to come off safely and to try and show that I'm a decent human being. But I'm okay with that. If nothing else, I'm pretty confident nowadays that I'm very, very sensitive to being respectful of others in spaces where I'm an outcast otherwise.
If nothing else, it bothers me that a lot of "brolitas" seem to have zero self-awareness of the position they're in, and are just openly creeps without considering the fact that they're in a woman's space and should be more considerate of how they present themselves.