Where I've been coming to is, there's value when you only need to 80/20 something AND there is no requirement that the output is checked. The main situation for that is programming assistance with local-run applications that do not require hardware security.

For example, there's an open source local-run server program that I use to play an abandoned service game entirely on my device now (no exposure to the internet), and it was heavily written by Claude. I have no issue with that. There's no issue with security holes because it's not exposed to the internet, and no issues with it hallucinating, because those are solvable bugs that come out during user testing. It did provide functional value because you could slot-machine instruct it to solve a problem and you can easily see if the problem is solved, and still save time.

But nobody is really boxing it in based on the actual limitations of what it can do and its flaws, so yeah, we're here with people blind to its faults and treating it as if it were a sentient assistant and not the probability matrix calculator that it is.

It doesn't really matter whose fault it is, it's going to harm everyone

AI is like starting from the cake and working back to confirm the eggs, the milk, and the flour.

that analogy does not make any sense.

I mentioned information acquisition and thinking are typically not separate processes, but if we act as if they are hypothetically: The cake is the final synthesis of your various information combined with the thought to understand them. LLMs give you output that may look like cake - it follows grammar, structure, and mimics thought. The output is also always a statistically correlative combination of various sources in petabytes of training data - Wikipedia, some random Reddit comment, a conspiracy website, a doctor's blog.

When you get that LLM's "cake," you can decide to eat it, and also decide if it tastes right. You can pick it apart and look at the structure a bit. But it takes a lot of effort to confirm what went into it after it's baked. You may be able to taste when it's made with rat poison, but some poisons taste like food. Was it made with sugar syrup or ethylene glycol? An LLM is just looking at probabilities and associations, and is limited by training data which you (and even the LLM company) cannot typically even reliably trace - those citations it gives you often do not even contain the propositions they're cited for.

Starting from reliable sources in the beginning is akin to starting with sugar, flour, milk. You've verified them, and now you know the cake is cake.