The racial analogue doesn't fit. The existence of women-only spaces isn't meant to discriminate against men because men are provided comparable alternatives, and they're only discriminatory against trans women because they're not provided an alternative.

Trans women are women, but forcing women to emotionally react to trans women as if they were cis women is not a decision we can or should make for them. Feelings don't care about your facts.

I'll try to come up with some conflicts of interest of the top of my head:

• Women generally prefer restrooms to be gender segregated as a matter of privacy. Trans women who don't have the same eldritch horrors of bodies likely don't fall under the same privacy grouping. Some places have taken the step of making all restrooms unisex but women often don't like it when that happens.
• Lesbian romantic events exist for women to meet other women, and while generally we should treat trans women as women, in no universe can or should we impose that onto people's sexuality.
• Someone who's had a traumatic experience with men and goes to a women-only space, only to discover that someone they've been talking to has been AMAB all along. It could be a healing or re-traumatizing experience, but it shouldn't be imposed upon them as a matter of trans inclusion.
• While trans women aren't necessarily better at sports than cis women, when they are it's reasonable to think their birth sex might play a part (whether right or wrong). That thought alone discourages participation in women's sports.