The downvotes don't exist for users on local accounts or anyone on another instance with downvoting disabled. That setting is determined by your home instance.

To put it simply- we can't stop the fediverse from downvoting posts on our instance. We can only remove them from the feed for local users and disable downvoting on the local instance via an admin panel setting. That's essentially what disabling downvoting does at the instance level.

Updating to show you how your post appears to our instance currently: