I don't think the privilege of the social contract applies here, because we first have to confirm that they violated the social contract. If they're caught in the act, sure go ahead. But if we start excluding people from the social contract without knowing they are the ones who violated it, it becomes very easy to eject anyone for any reason.
Due process isn't for the benefit of the guilty, but for the safety of the innocent accused.