If you're concerned about privacy, you should probably know that a lot in the fediverse is very open by design so that other instances can communicate effectively.

That's why everyone can see what you upvote/downvote, even if Lemmy itself doesn't show this to you in its UI.