Sports are a wedge issue used to open the door to further eroding trans rights is the issue, it's not something you can compromise on. The reason sports are even something people even have a problem with is due to the fact that most cis people don't see sex as mutable or trans people as changing their sex, which is the argument behind sports bans, bathroom bans, and other "single sex" spaces like women's shelters or hospital wards. Any of those spaces banning trans people provably leads to more assaults/rapes, or otherwise materially harms trans people. And even if it somehow didn't do that, from a moral standpoint it is still discrimination.

A minority being a minority of the population isn't a reason to ignore those issues, and doing so just opens the door to more discrimination and repression of other groups as well.

All those other issues you listed are also important, but being willing to throw trans people under the bus doesn't help you accomplish them, all it does it hurt trans people. El-sayed, Mamdani, and a bunch of the progressives that have won recently have done so without needing to abandon trans people, and are better off because of it. If you're willing to sacrifice a minority's rights to get elected, all it does is show that you don't have actual principles and makes me believe you'd abandon other causes you otherwise claim to champion the moment it becomes politically inconvenient. And that goes for any minority, not just trans people.

Also, its worth noting that trans women are women, and that includes physically for stuff like sports. Almost every study done on the topic shows that a trans woman athlete has no statistical advantage over a cis woman one.