Not just fediverse, I think any site that allows "downvotes" has this issue.
Personally, I don't see why the ability to downvote needs to exist. If someone is trolling, ignore it or report it. A troll post with a score of 1 and no comments is better than one with a score of -100 and no comments. The downvotes probably encourages the troll. They know they've upset a bunch of people. All their posts getting no interaction will bore them.
On the other hand, downvotes existing leads to things being hated on for no reason. Someone on asklemmy asks what your favourite pizza topping is and the top comment is pepperoni with a score of 100 and bottom is sardines with a score of -50. You see that and think nobody likes sardines. But what if taking away downvotes changes the scores to 100 pepperoni and 12 sardines. Now sardines isn't looking so bad even though the number of people who like it hasn't changed. What does the downvoting add? It just makes the people who like sardines feel bad. They might end up not contributing in the future and then every answer to asklemmy ends up being identical.