It takes a certain baseline amount of social privilege to be able to not give a shit what the haters say. When society writ large decides to hate you (as is implied by the term "social pariah"), then suddenly what people think of you actually matters a lot.