Beyond proving hallucinations were inevitable, the OpenAI research revealed that industry evaluation methods actively encouraged the problem. Analysis of popular benchmarks, including GPQA, MMLU-Pro, and SWE-bench, found nine out of 10 major evaluations used binary grading that penalized “I don’t know” responses while rewarding incorrect but confident answers.

“We argue that language models hallucinate because the training and evaluation procedures reward guessing over acknowledging uncertainty,” the researchers wrote.

I just wanna say I called this out nearly a year ago: https://lemmy.zip/comment/13916070

One of these days, the world will no longer reward bullshitters, human or AI. And society will benefit greatly.

I don't know where I read it but sort of said that it to have that much information inside the models it was basically similar to a compression algorithm.

From logic, if we have a lossy compression then its mostly luck if the output is equal to the original. Sometimes it will tip one way and sometimes the other.

A hallucination is something that disagrees with your active inputs (ears, eyes, etc). AIs don't have these active inputs, all they have is the human equivalent of memories. Everything they draw up is a hallucination, literally all of it. It's simply coincidence when a hallucination matches reality.

Is it really surprising that the thing that can only create hallucinations is often wrong? That the thing that can only create hallucinations will continue to be wrong on a regular basis in the future as well?

I don't get why they'd be called hallucinations thought. What LM's do is predict the next word(s). If it hasn't trained on enough data sets, the prediction confidence will be low. Their whole output is a hallucination based on speculation. If they actually don't know the next word order, they'll start spewing nonsense. Though I guess that would only happen if they were forced to generate text indefinitely.. at some point they'd cease making (human) sense.

LMs aren't smart, they don't think, they're not really AI. There aren't errors, there aren't hallucinations, this is by design.

... yes? This has been known since the beginning. Is it news because someone finally convinced Sam Altman?

Neural networks are universal estimators. "The estimate is wrong sometimes!*" is... what estimates are. The chatbot is not an oracle. It's still bizarrely flexible, for a next-word-guesser, and it's right often enough for these fuckups to become a problem.

What bugs me are the people going 'see, it's not reasoning.' As if reasoning means you're never wrong. Humans never misremember, or confidently espouse total nonsense. And we definitely understand brain chemistry and neural networks well enough to say none of these bajillion recurrent operations constitute the process of thinking.

Consciousness can only be explained in terms of unconscious events. Nothing else would be an explanation. So there is some sequence of operations which constitutes a thought. Computer science lets people do math with marbles, or in trinary, or on paper, so it doesn't matter how exactly that work gets done.

Though it's probably not happening here. LLMs are the wrong approach.

Great. With that mathematically settled, can we spend all that money, computing hardware and electricity on something more worthwhile now?

They're still lying. For fucks sake. It's like they impaled you on a pike and just admitted "okay so we did prick you with that needle".

ALL IT DOES IS HALLUCINATE. ALL IT DOES IS HALLUCINATE. ALL IT DOES IS HALLUCINATE. ALL IT DOES IS HALLUCINATE!

SOMETIMES the hallucinations happen to resemble reality. Just because a hallucination happens to look similar to reality does not make it real.

IT IS NOT PERCEIVING REALITY.

EVER!

EVER!

Ever.

The OpenAI research identified three mathematical factors that made hallucinations inevitable: epistemic uncertainty when information appeared rarely in training data, model limitations where tasks exceeded current architectures’ representational capacity, and computational intractability where even superintelligent systems could not solve cryptographically hard problems

Perhaps it's because I'm dumb, but does anyone else feel like there are lots of buzzwords here?

Like, does "cryptographically hard" refer to NP-hard or something??

This may be a mean thing to say, but could parts of this article be generated by an LLM?

As a mathematician dabbling in AI: we told you so

... yes? This has been known since the beginning. Is it news because someone finally convinced Sam Altman?

Neural networks are universal estimators. "The estimate is wrong sometimes!*" is... what estimates are. The chatbot is not an oracle. It's still bizarrely flexible, for a next-word-guesser, and it's right often enough for these fuckups to become a problem.

What bugs me are the people going 'see, it's not reasoning.' As if reasoning means you're never wrong. Humans never misremember, or confidently espouse total nonsense. And we definitely understand brain chemistry and neural networks well enough to say none of these bajillion recurrent operations constitute the process of thinking.

Consciousness can only be explained in terms of unconscious events. Nothing else would be an explanation. So there is some sequence of operations which constitutes a thought. Computer science lets people do math with marbles, or in trinary, or on paper, so it doesn't matter how exactly that work gets done.

Though it's probably not happening here. LLMs are the wrong approach.

sucks when the pesky limits of physical reality get in the way of your AI superintelligences from actually being intelligent.

That's great! Let's use it for healthcare, taxes, education, news and military operations. 👍

What do you mean "admits"? It has been known from the very beginning to everyone who has read about LLMs at least from the output of those very LLMs (the irony, huh?).