But the downside is that people then upvote or downvote based on their preferences, not the community’s.
This is only a downside for niche communites promoting positive topics like equality. Communities about for profit companies, that promote horrible ideas and people, etc are what downvotes are for.
If there was a community called "Nestlé is awesome" I would hope it gets downvoted into oblivion.