You're favoring the opinion of one sex over another because of their experiences. In this example, you're favoring woman's opinion over man's opinion because she naturally has more experience with sexual harassment. That's like saying a man's opinion in the business world is more valuable than a woman's because he naturally has more experience with leadership and conflict. If I said the latter, I would be torn to shreds. But it's the same thing you're trying to pull right now.
But "leadership and conflict" aren't natural, essential experiences of being a man. I'm asserting that sexist put-downs are a natural and essential part of being a woman.
If you must have a male corollary, I think there's a case to be made that men suffer from success anxiety--that their self-worth is tied to material success--much more than women (I'm not saying this is "female privilege" or "sexism against men", just that it's something that men are evaluated on much more intensely than women). Would I favour a man's view of male success anxiety over a woman's? Yes, probably.
But things like "leadership" or "conflict" are at best loosely correlated with gender--that doesn't make men automatically experts on it any more than it makes all men CEOs or heads of state just because the majority of those positions are filled by men.
You're favoring the opinion of one sex over another because of their experiences. In this example, you're favoring woman's opinion over man's opinion because she naturally has more experience with sexual harassment. That's like saying a man's opinion in the business world is more valuable than a woman's because he naturally has more experience with leadership and conflict. If I said the latter, I would be torn to shreds. But it's the same thing you're trying to pull right now.