Freedom of speech just can't be applied selectively, tho.
I mean, for freedom of speech with respect to the State, yeah I'm with you. But if I remember correctly, Chomsky was arguing against antifascists deplatforming fascists, using freedom of speech to justify that opinion, which I think is a weak argument because the whole point of deplatforming fascists is to allow the maximal subset of people to speak.
One day you restrict someone's speech, the next day they are in power and restrict yours.
IMO deplatforming oppressive groups should be considered the maximalist position for freedom of speech. Letting them speak is equivalent to letting them win, which results in a net loss of free speech averaged over the whole community (e.g., LGBTQ people feel less likely to speak freely when homophobes are allowed to freely express LGBTQ-phobic views).