And? Upvotes increase visibility.
Correct. And communities aspire for visibility. So someone coming into a community to downvote all the posts will potentially be banned for harming that communities visibility.
Why is one bad but the other is acceptable?
Because the point of the Fediverse is to grow, not degrade. Also, it's not downvoting per se that is viewed as inherently harmful - it's bulk downvoting of posts. If someone who doesn’t like metal music enters !metal@lemmy.world and downvotes every post there - and commits themselves to doing it to every new post, this hurts their visibility across the Fediverse. Would you not say its reasonable, in this specific hypothetical, for the community owner to ban a user who does that?
It’s also not an issue in “smaller communities” because there’s not enough content to bury heavily-downvoted posts anyways.
A newly made post that gets downvoted to 0 quickly after being posted is rendered effectively invisible on the fediverse. I’ll give you my own experience: When I was growing !television@lemm.ee before lemm.ee shut down, the community collected about 5 frequent downvoters. None of these accounts ever upvoted anything. They never participated by posting or commenting anything on the community. In a few cases, two of the accounts had no posting history on the fediverse at all. They existed purely to downvote. Whether because they didn’t like television as a topic, or hated news articles being posted - I don’t know. But they would, between them, always downvote posts and sometimes do so early - deprecating their visibility. After I banned them after a time, I noticed an improvement on the community.