I believe its because they don't believe being trans is real, and so they think anyone who is trans is just a pervert cause the only reason they can think anyone would want to be a woman is to perv on women. Its at the end of the day misogyny. Because of this I don't think they even realize trans men even exist, cause it doesnt fit their misogynistic ideas of men and women. To them, women cant be pervs so there is no reason why someone they see as a woman would want to be seen as a man. Even if they do know of trans men, I don't think they see it as problematic (or at least as severely so) because of their misogynism.

However, this is all just my thoughts on it, and also very much a generalization. Even if I am right, it is very likely there are transphobes that this doesnt apply to.