You do you:-). But even then there are other things that could have been done instead, e.g. submitting a report to a mod or blocking someone, or leaving a community that is stressing you out. Ofc downvoting is perfectly fine as a choice as well, which will show up in your Attitude score that you are someone that does that, i.e. someone who is "critical" of content that you see on the internet and often interacts with it in that manner. It is up to you and also to the people looking at your profile to determine if you want to consider that "bad" or not, though I sincerely doubt that such a score (of 80) would ever qualify for that term.