You could like, make sure you challenge other men. That's kind of the issue, it's not that there's too many men who do this to require this response, but that the vast majority of men are more afraid of confronting other men than they are being silent about something they're morally apprehensive about.
Pretty sure that's the like, one of the main things that sexual violence prevention education and research targets is that men are overwhelmingly complicit in rape culture.