most of the world is allergic to milk,

I'm going to need to see some sources to back up that claim.

Edit: this seems to say less than 0,5% of adult population is allergic to milk. https://www.sciencedirect.com/science/article/pii/S1939455122000229

Lactose intolerance is more common.

Experts estimate that about 68 percent of the world’s population has lactose malabsorption.1

https://www.niddk.nih.gov/health-information/digestive-diseases/lactose-intolerance/definition-facts

But that's far from allergy. Allergies are more severe than just bloating, diarrhea and gas.

Human breast milk also has lactose.

i think you mean lactose intolerant, milk allergy is sorta uncommon. most of asians are lactose intolerant due to not evolving with cow milk, but some european people are not lactose intolerant. other ways to get intolerant is through intestinal diseases, like parasites or celiacs, or gi infections. but lactose intolerant do build up as you age too.