It's a bit silly to say they don't reason.

Here's some research where they uncover reasoning circuits in some large transformer models https://github.com/guanzhehong/prop-logic-transformer-circuit ... reasoning has always been just math at th end of the day

And what about alpha go and all that monte carlo tree search stuff, id argue that they can reason far better than humans in these setups

They can't understand something...outside their training data

Idk how easy this is for the the large models, but certainly for classic ML, testing for performance on data outside the training data is 100% part of the development process. (Read up on test/train/validation datasets)

There is also in context learning, which gives rise to zero shot task completion for never trained on tasks.