I don't have an answer for you outside of just saying that labels in any context can limit yourself. Humans like to categorize things, we don't like chaos, but some things don't fit neatly in the boxes we create for them. And that's okay, you can be partially in different boxes.

Attraction is like that - you like what you like, for whatever reasons that maybe you don't even understand. But there's no "invalidation" of anything. It just makes it hard to explain to other people perhaps. But so what.

As a straight male, I don't get why gay men like other men. I understand lesbians, because they like what I like, so there's common ground. Trans people I also have trouble understanding, not because I think they're weird, but because I don't have their frame of reference to relate to. But I'm totally fine with them figuring out how to make their body match their feelings, I get that part, even if I don't share it.

Maybe it's the male aspects of some trans women that are still present as they change? Maybe something else entirely?