I am AFAB - been wearing mens pants for years and I don't have a problem fitting into them. Probably largely depends on individual body shape and even then, taking measurements and matching it to the correct waistband/inseam size usually works out.

My crackpot conspiracy theory is that women's pants pockets are smaller in order to force women to buy and use purses.