I do not want people banned for voting activity. How is this even an idea with traction?

I've already explained this, because people value a high-trust community culture. People don't like it when bots, or even individual accounts go to a community and downvote everything on there and keep doing it because they can. It can be corrosive and damaging to new communities viability.

If we are so concerned about the impact of down votes on communities then just remove them from the rank calculations.

That's another option, but this is what the wider fediverse has gone with currently - regarding downvotes as credible when used fairly.

But again, it's up to community moderators here - not instance owners. Some communities won't care, some will only care if its repeated downvote activity with no interaction, and others will be really bad about it.

Yeah, I suppose in my experience - most of the downvoters that caused problems were persistent. You'll never stop downvote noise though. But that also goes for upvote noise.