You're not complicit just for being a man. You're complicit if you allow the behaviour in your presence. When dudes make offhand comments, call them out, even if it's your buddy and there's plausible deniability (which they use to avoid repercussions, social or otherwise).
Make your buddies feel shame for toeing the line or making shit jokes. If you surround yourself with so many people who don't do that that you don't even have the opportunity to call anyone out, you're already doing a good job.