So are we assuming the non-masker is trying to be somewhere they've been told they're not welcome like, as a statement of rebellion? Do they not understand that they're not welcome or what they need to change to be welcome? Are they trying to be provocative, or are they trying to gain access to the community for some specific purpose?
"Enforcing this exclusion" is really broad. It has the underlying assumption that the situation is already violent, that there is a thing to enforce against someone who is "disobeying" the thing. Which I'd argue comes from the fact that we've really only lived in police states in modern history. Again, humans are social creatures--we want community and we want to feel included. And we don't consciously do things to each other for no reason. I urge you to think deeply about where the need for "enforcement" comes from, and what we mean by it.