But you may have to use them if your body doesn’t switch from androgen to estrogen production as the primary on its own which can be caused by both genetic and/or environmental factors
just to clarify, the body doesn't switch from androgen to estrogen production - what happens is sufficient exogenous estrogen tells the brain there are sufficient sex hormones, and shuts down sex hormone production; if the sex hormones drop low enough, the brain knows to turn production back on - but it will still produce testosterone rather than estrogen ... unfortunately there is no way (currently) to make testes produce female levels of estrogen or to stop producing male levels of testosterone 😞
So the goal is instead to get the body to stop producing male sex hormones by taking enough female sex hormones.
General guidance now is to avoid androgen blockers at first and test testosterone levels to see if your body slows production of them. Androgen blockers have side effects, but without them the changes may take more time.
I think the desire to avoid androgen blockers is likely only relevant for people who opt to try monotherapy - that's a missing component of what you are advocating; typical doses of estrogen that would be taken with anti-androgens will be much lower than necessary for monotherapy, and probably wouldn't be enough on their own to shut down testosterone production.
Also, OP mentioned they have been prescribed anti-androgens and oral estrogen particularly, meaning the dose they were given is probably low, and is in a form that won't work well for monotherapy - so to follow your advice they would probably have to change to injections as a route of administration.
However, you can take anti-androgens and oral estrogen - probably most trans girls in the U.S. take this route, and it's the default way doctors handle transition here (not that I think it's a particularly good way to go about it - it has lots of side effects and less / slower feminization with consistently low E levels, but I think doctors are afraid of monotherapy because they are afraid of high estrogen levels increasing risk of strokes based on some studies done on Premarin; though I don't know why they aren't also afraid of spironolactone causing heart problems, which at least has more evidence behind it than bioidentical estrogen causing strokes).