Right, so we are going to twist ourselves in circles and allow/enable censorship just for using the site, over a single mechanic which doesn't need to impact anything in the first place?
People already fairly and unfairly censored all over the website by community owners for how they use a community. There are unjust bans all the time.
Am I taking crazy pills here? This is such a dumb issue with an easy and obvious solution.
Removing downvotes would also have an impact on how content trends, so its not quite as easy as you're alleging here.
Notably a handful of instances do disable downvotes, like blahaj if I recall.