I do not care for bathroom debates, they're always so overblown and nobody can take any form of compromise. You seem very hung up on extreme edge cases. Every rule in life is going to have some imperfect matching, yet somehow this one is impassible? There's practically nothing in life that isn't going to have some exception where it shouldn't apply.
So is Y chromosomal matching perfect? I never even suggested it should or could be used for bathrooms, but I wouldn't care if it was, because no, no system is. That doesn't mean you should have one shared bathroom for everyone in society, or for anyone to walk in anywhere they decide to. It's such a waste of energy debate.