I'm just going to rant a bit, because this exemplifies why, I think, LLMs are not just bullshit but a looming public health crisis.

Language is a tool used by humans to express their underlying thoughts.

For most of human evolution, the only entities that could use language were other humans - that is, other beings with minds and thoughts.

In our stories and myths and religions, anything that talked to us like a person - a God, a spirit, a talking animal - was something intelligent, with a mind, to some degree, like ours. And who knows how many religions were started when someone heard what sounded like a voice in the rumble of thunder or the crackling of a burning bush and thought Someone must be talking directly to them?

It's part of the culture of every society. It's baked into our genetics. If something talks to us, we assume it has a mind and is expressing its thoughts to us through language.

And because language is an inexact tool, we instinctively try to build up a theory of mind, to understand what the speaker is actually thinking, what they know and what they believe, as we hold a conversation with them.

But now we have LLMs, which are something entirely new to this planet - technology that flawlessly mimics language without any underlying thought whatsoever.

And if we don't keep that in mind, if we follow our instincts and try to understand what the LLM is actually "thinking", to build a theory of mind for a tool without any mind at all, we necessarily embrace unreason. We're trying to rationalize something with no reasoning behind it. We are convincing ourselves to believe in something that doesn't exist. And then we return to the LLM tool and ask it if we're right about it, and it reinforces our belief.

It's very easy for us to create a fantasy of an AI intelligence speaking to us through chat prompts, because humans are very, very good at rationalizing. And because all LLMs are programmed, to some degree, to generate language the user wants to hear, it's also very easy for us to spiral down into self-reinforcing irrationality, as the LLM-generated text convinces us there's another mind behind those chat prompts, and that mind agrees with you and assures you that you are right and reinforces whatever irrational beliefs you've come up with.

I think this is why we're seeing so much irrationality, and literal mental illness, linked to overuse of LLMs. And why we're probably going to see exponentially more. We didn't evolve for this. It breaks our brains.