I agree in the context of HP, and I also love Pointy Hat Studio.
But in this context, "muggles" are the people with the social power. Elsewhere in this thread, someone has said "you need psychiatric help if you think magic is real." So sanism and stigma against mental illness are being used to enforce an ideology that denies the cultural experiences of magic users. Those are the actions of the privileged majority.
And our world is more like Pointy Hat than Harry Potter. Because anyone can learn magic. And unlike in Pointy Hat, it's not a secret. Or at least, the secret isn't being kept by us. It's being kept by capitalists. If you want to be a mage, you can just learn magic. I'll even teach you if you want. In fact, here's a link to a spellbook you can read for free: https://usermanual.wiki/Document/PsychonautFieldManualSmall.49336827/view (disclaimer: Archtraitor Bluefluke is a good teacher but has problematic views on chakras and cultural appropriation. Please skip the chapter on chakras, it is a closed tradition. There are plenty of open spells you can learn).
So I don't think "muggles" are being oppressed in our world, and I don't think it's possible for there to exist a meaningful slur against non-mages.