their own spaces
You're making the same conflation as several other people here. A private space can exclude through non-invitation without specific/class exclusion. A conference is not a private space. It is public. By rendering the space public, it creates an equality of people as possible attendants as members of the public. By excluding a generalized group, it discriminates through stereotype, which brings things to the meat of your point.
If you take the whole matter into amorality, there is nothing wrong with ANYTHING the powerful do, and render any argument about dignity of the oppressed meaningless. You have no place left to stand and you lose.
If you argue the powerful are somehow different from other people, you establish a belief in inherent inequality. You have no place from which to claim injustice, and you lose.
If you fight against the powerful without some semblance of reason, you cannot form a cohesive collective, so you will have no power with which to fight them, and you will lose.
Can you name a single historical instance where a dominant group stopped a practice of exclusion because they realized they no longer had the 'transitive legitimacy' to continue it?
Feminism. If each generation of feminists had never made claims to human dignity, there would be no liberation or justice. If they had only focused on stripping the dignity of powerful men, they never would have gotten the support of the rest of their society. Action disrupts the old system but the moral argument is what transforms society into something new. The 'dominant group' isn't the 1% crowd. It's the 90% who they trick into supporting them. The 'powerful' shit themselves at the idea of seeing the majority turned against them. If early feminists hadn't convinced the people around them of the capability and equality of women, it wouldn't have mattered how hard they tried, they would just have been ignored by the majority and snuffed out by the powerful minority. If they hadn't fought to establish a moral norm of equality, all their screams would have been noise fading into the void. Acting like you actually believe in your principles isn't 'disarming' yourself. It's letting the enemy take your rifle so you can take the fort. It's planting the tree so your children can sit in its shade. It's how you get justice rather than get yours.