As OP said, LLMs are really good at generating text that is fluid and looks natural to us. So if you want that kind of output, LLMs are the way to go.
Not all LLM prompts ask factual questions and not all of the generated answers need to be correct.
Are poems, songs, stories or movie scripts 'correct'?

I'm totally against shoving LLMs everywhere, but they do have their uses. They are really good at this one thing.

I'm not here defending the practical value of these models. I'm just explaining what they are and what they're not.

Some of them can produce the correct answer. Of we do the test next year and they do better than humans then, isn't it progress?

The AI was asked the same question repeatedly and gave different answers, due to its randomised structure.

People will also often do this (I have, personally), but because our actions seem to be strongly influenced by time-dependent stuff (like sense perception and short-term memory contents), I'd expect you'd need to ask at different times.

It's a Large Language Model. It doesn't "know" anything, doesn't think, and has zero metacognition. It generates language based on patterns and probabilities. Its only goal is to produce linguistically coherent output - not factually correct one.

It gets things right sometimes purely because it was trained on a massive pile of correct information - not because it understands anything it's saying.

So no, it doesn't "guess." It doesn't even know it's answering a question. It just talks.

In people, even animals. In a pile of disorganized bits and bytes in a piece of crap? No.

This gets very murky very fast when you start to think how humans learn and process, we're just meaty pattern matching machines.