Because charismatic bad people are very adept at convincing ignorant people that those "others" are, in fact, bad and we all need protection from them. In fact, I think nearly everyone is willing to sign up to harm people in some way - it's just defining who is bad that those political parties f' with.

A minority of really bad people have an out-sized harm on society, and we haven't really figured out a way of preventing that.