I’ve been thinking a lot about the question posed in your title. I’m a non-binary victim of sexual abuse and sexual harassment, but I have felt that this particular hashtag campaign was created by and for women specifically, and I should respect that by not participating. I don’t feel that giving women (including trans women and non-binary femme/transfeminine people) space to speak amongst themselves silences or ignores the fact that men and non-binary transmasculine people also experience sexual violence.
Victims/survivors of all genders should feel empowered to speak about their experiences if they freely choose to do so. But men are more prone to talk over women and make the conversation all about themselves, as you noted.
Regardless, I appreciate you sharing your perspective. I feel that whoever participates in this particular hashtag, women should be the ones to lead the conversation, because women are, without a doubt, the primary targets of sexual violence.