There is also an argument that expecting everyone to agree with and follow the unwritten rules of 'voting correctly' is just a vocal minority trying to impose their will on all users.

Generally I limit to subs but browse All occasionally to find new communities. Part of this process is upvoting and down voting stuff and seeing which communities has more stuff I want and less stuff I thought was worth downvoting. I then sub or block after that point.

That is how I notice spammed slop as part of blocking a community. No, I won't be going back to remove down votes before blocking just because the community is about spamming slop and the posts fit the sub. Harm isn't a down voting requirement.