I think the issue was banning for giving votes you didn't agree with, not with sending the DMs. I've sent DMs instead of doing admin actions before, just to open a dialogue, or to give people a chance to push back or explain before I take some kind of action, and that part seems fine. I can't even really articulate why it was that this rubbed people so badly the wrong way, but I think sending the DMs and getting in an extended back and forth did somehow make it worse. Definitely doubling down and banning people (and also DMing them) because their reaction and vote on it wasn't the "correct" and permitted one according to you made it worse.
People can vote. People can react. Setting yourself up as this lord and arbiter of what's right and wrong is always going to make a backlash. If it was me, I would have made a public reply instead of a DM so that other people can weigh in, I would have framed it in terms of "what I allow here" and made sure to clarify the rules on the sidebar instead of framing your point of view as the one that's objectively the right one (which you're still doing here, when you describe calling someone "bro" as "toxic" instead of saying that you personally think it's rude and don't allow it). And then if they still don't agree, you're still within your rights to just say yes okay fine but that's the rules, sorry, and ban them (and then move on yes).
I still think you would have gotten backlash, but framing it in that way would have at least shown you have some awareness that these categories and judgements are just your categories and judgements, and regardless of what the Lemmy software's mod controls have led you to believe, other people are allowed to have their own that are different from yours. If you'd done that I don't think it would have really developed to anything, there might have been one YPTB post about it at worst and then people would have shrugged and moved on with their day.