I don't expect my votes to be private, but I will vote as if they were. People reading too much into why someone might have voted a certain way and trying to shame them for it is embarrassing, I'm not going to constrain myself to reasons for voting that are immediately legible to anyone via only one bit of information just for their sake. If I want to downvote something with, for instance, a political message I agree with but the person phrased it in a way that goes against my pet peeves, I'm gonna do that, and I think the ideal would be for these numbers not to be taken very personally or seriously.
So basically I'd say it's bad taste to put it in a UI, but only so far as it encourages people to be weird about it, and ultimately that's on them.