Well if it was measurable in mg, that would be super bad, since oral estrogen doses for trans women are like 2-4 mg per day. For it to be consumed by men regularly with no feminizing effects, I think it would need to be at least in the picograms/gallon range.

... but then again, i'm not a doctor. I could be wrong about that.

Edit: Turns out it's more than I was expecting. Maybe if you are taking an anti-androgen you might get a noticeable amount of E in your system if you drink multiple liters of milk per day? Not that i'd recommend it, but it is interesting.