People will always find something to complain about, but classes based on objective and quantifiable metrics like weight and height would be more fair and safe than subjective gender identity-based classes, and there would be very little room for debate.

Small guys (and girls too) can always bulk up if they want want to play with the heavyweights. It's the obvious solution to this trivial and mostly imaginary "problem".