I think there may be a misunderstanding, or multiple interpretations, of what 'toxic masculinity' refers to.
In the contexts where I've heard it, the term wasn't being used to say that masculinity is toxic /that it should look a certain way / that this is something that's up to men to change. Rather, it refers to the subset of toxic behaviour that is associated with 'masculinity', as a way of organizing discussion. The term isn't meant to ignore or minimize other types of toxic behaviour, it's there when referring to any patterns or solutions that might be in common for that subset of toxicity.
For example, the term "men's mental health" organizes discussion around mental health challenges that men deal with. It doesn't mean that only men deal with mental health issues, nor does it mean that men's mental health issues are more important to deal with than those of other groups.
If society truly wants progress, we should stop policing how masculinity looks and start holding all people accountable for toxic actions, not their gender.
Isn't the first part exactly what 'toxic masculinity' discussions are about, that rigid and harmful gender expectations harm everyone, regardless of where it comes from (men or otherwise)?
As for the second part, I don't think anyone is saying otherwise. We SHOULD hold everyone accountable for toxic actions.