If you want your votes to be (somewhat) private, probably the best thing to do is to create an alternative account and use that for voting without making any comments/posts.
PieFed also gives you an option so that your votes are only shared within the same instance. So, tools like "lemvotes" wouldn't be able to pick up on your voting history. In that case only the local instance Admins (and maybe moderators) would be able to see your voting history.
From this user it is fairly obvious that it was created recently. They downvoted 49 times in the first few minutes of activity. They then upvoted once a few days later.
https://lemvotes.org/user/PyroRondo@lemmy.world
Based on the first day of activity, it almost does look like this was an account created just for downvoting.