I always hesitate to downvote and have only blocked a small handful of users. Even among the users I've blocked the majority are people whose porn I don't wish to see.
Just because I don't like someone's opinion or attitude about a given subject, I don't feel it's right to throw them away. Even if they're being relatively vile, they may just be having a bad day.
Also, I almost always expand downvoted comments. To me that's a sign saying there may be something interesting there, and there often is. Lemmy isn't as hive-minded as reddit was/is, but there's definitely groupthink and while I know that's human nature, it still bothers me a bit.