Agreed, but it's still a tough line to draw. Here's a scale:

We've got some problems, and it's probably because of "someone".
Aren't the X suspicious? We should keep an eye on them.
All the X have rights but it's their fault for all our problems.
We should give jobs to everyone before all the X.
We should expel all the X.
We should genocide all the X.

At which point is it ok for the cops to come have a conversation with you?