If you intentionally show up somewhere you clearly don't want to be just to "boo" people minding their own business, yeah, you should be asked to leave and barred from "participating".
Common courtesy is to not downvote things in /all that you don't like if you're not subscribed. Just ignore it or block the community and move on.
Am I guilty of that? Hell yeah, I am - never said I wasn't. But I do try to be better and not just knee-jerk to the downvote button if I see something I don't like in a community I'm neither subscribed to nor interested in.