Ok so ethically the whole thing puts doctors in a situation where it’s tough.
A doctor worth their oath will have a hard time ethically, for the same reason your GP should refer you to a psychiatrist for psychiatric meds. The appropriate medical professional needs to be looped in for ethical reasons.
Simply because they are unable to quantify the harm being done, but a therapist can. So in order to uphold their oath they must have proof from a medical professional who can affirm harm is being caused.
Yes, there are assholes who are pushing their transphobia or hatred. But also the other thing.
Now whether or not gender affirming care should be considered “medical” care is another story altogether BUT this is the reality.