I was with you until your second sentence. You're still giving it too much credit. It's a text prediction engine. It uses training data to correlate words with a lot of context. It's not a delusion because it doesn't "know" anything, it has just encoded correlations.

While hallucinations can be the result of bad information included in the training data, due to the nature of how it works, it will always have a set of tokens it deems more likely given any context and will predict those tokens. Now, it's possible that it will predict "I don't know", but only if it decides that that is the most likely response to the prompt. Maybe the context given will correlate closer to a different answer, even with curated training data.

Uncurated training data doesn't help, of course, but this problem won't be resolved by curating the training data better, just improved (assuming your goal is to get it to be more truthful... Which could counter its ability to innovate and work outside of what it knows for sure, so I'm not even sure if "always be truthful" is even a worthwhile goal for AI, especially from the perspective of those who own and control the big ones, who ultimately get the say unless it's taken from them).