It seems a lot of people still don't realize that it can't do analysis; it just predicts things that can look a lot like analysis and sometimes it even predicts the correct conclusion. But it's training data will include many cases where a question is posed and responses jump right to the answer without showing any of the underlying work that got there.

So the training isn't "user a asks question, user b considers factors x, y, and z, including how y and z interact complexly with each other, as well as how a subtle complexity of the question means the usual case for these doesn't apply in the same way, and posts this response", which is what an actual intelligent being would need to get from the question to the answer like an expert would.

The training is "user a asks question, user b posts response". That training is more like how to train someone to pretend to be an expert because all the information is just the question/response pair (with no reasoning that wasn't included in the response), and the tone or language used to convey the response.

That's why they always sound so confident, because they are predicting what a confident response would look like, but the actual content of the response is based purely on correlations between the words used in the question and whatever words followed variations of those words used in other questions or stories or wherever the training data came from.

Though even if everyone wrote out their full line of thought for every response, it still wouldn't result in actual intelligence because it's still just correlating words, not applying logic and analysis to them. That's all LLMs do. Hallucinations are fundamental to how they work.