He's bang on. LLMs have no understanding of anything: they are literally just statistical models.

Humans are animals. LLMs randomly generate text based on the corpus they were trained on and the conversation so far, so stochastic parrot is an accurate description.

LLMs don't learn. Humans do. LLMs generate text randomly using a massive matrix. Humans don't; you lied. An LLM is incapable of lying because it has no understanding of truth. It just bullshits convincingly all the time. It's very very good at it, but it's all hallucinated for the LLM, true or false.

Expecting your random word generator to tell you truths is insane. The training measure is "sounds right" not "is right". It passes if it sounds like the other discourse it read. Just like the confident drunk guy at the pub who thinks he knows everything passes of he convinces the other drunk guys at the pub.

Whereas humans learn at school and on the job and the training measure is "your teacher or supervisor approves". LLMs were not trained on truth or accuracy. Trusting in them and treating them as equivalent to human intelligence, as you and a whole bunch of other folks do, is profoundly unsound, and soon the necessary price rises to pay for the processing costs (let alone the vast, vast, vast, vast, vast debts on the infrastructure) are going to make most slophouses which jettisoned their human talent go out of business. And very, very few people indeed will be sorry at that point.

Meanwhile LLM slop is shitting in github all day long, every day, and shitting on the internet, and it will eat it's own shit and produce crappier shit.

Your analogies don't change the truth, and that is that LLMs don't know the difference between sounds correct and is correct any more than MAGA voters know the difference between sounds good to me and is good for me.

Language models are not databases and they are not markov bots (similar function but work directly using statistical word association maps). The big difference is that those things are algorithms someone wrote and can fully comprehend what they do, but machine learning models are large algorithms built by another algorithm processing training data. There is much more uncertainty about what is going on under the hood.

There is also great uncertainty about what concepts like understanding or thinking might mean in computer science terms. The main thing we can really know is that ultimately a human mind is a computer, which means that understanding and thinking have some yet unknown mathematical representation, and therefore a comparison can be made. We should eventually be able to quantify whether or to what extent a given algorithm thinks. But you said in another comment that you don't believe minds can be represented mathematically; this should mean that such comparisons would be apples to oranges, but you're making them anyway for some reason, and implying they have predictive power for the limitations of LLMs.

Certainly they do have limitations, at least individually and possibly as a technology. There are things given models are bad at, there are things they initially seem to be able to do well as humans but fail in different ways that suggest over-reliance on pattern matching. But these have been determined empirically through testing. The idea that they are "just statistical models" and this knowledge can be used to say what is impossible for them from philosophical first principles keeps getting repeated but has never worked in practice. The reality is that no one knows enough to say for sure where the line is.

geohot

Now there's a name I haven't heard in a long time. George Hotz was the guy who first jailbroke iOS and the PlayStation 3 and made the towelroot exploit for early versions of Android, before legal threats drove him out of the scene.

A machine learning model is a computer program. It is fundamentally a math equation, which we understand completely.

Lol you couldn't be more wrong about this. One of the most widely commented things about DNNs is that we don't really understand them completely. I don't know how you would miss that if you knew anything about AI at all.

A living brain is not fundamentally a math equation

It is. A very complicated one, sure. Which part of the brain do you think is impossible to simulate with maths?

What's fascinating about this conversation is ... how do people think software used to be made ? With talented and knowledgeable developers who would never "hallucinate" an API or a library function ? With cybersec experts who would never put their user's data in jeopardy ? With performance investigators checking the computational complexity of each function ? Bitch please...

Software engineering is not the kind of mystical cathedral building these people have in mind, it's more like a musty workshop in Pakistan where they make tractor tires with no safety equipment and a cigarette in their mouth. We've been throwing imperfect humans in various states of lucidity at every problem known to man for 30 years but suddenly people start believing that their bog standard CRUD software should be written by monks having attained cosmic godhood.

remember, when you interview for a job and they ask you, "do you have any questions?", you ask;

• has AI ever been used to develop your product?
• what percentage of your product has been written by agenetic AI?
• is the use of AI tracked as a performance indicator?

I am a very lazy man, micromanagement takes so much fucking energy. Hire talented people, give clear and unambiguous guidance and trust them to do the work. It is amazing how easy management can be when you don't get in the way.

Being in leadership is way easier than being an engineer and the pay is better too. Some people really overcomplicate shit.

It's going to be a wonderful time to be a Freelance Senior Developer and above in a few years.

Aah, thanks. I really like the idea of this QUERY method.

Tbf, LLMs may be comparable in complexity to their specific brain.

The mistake has been thinking this implies LLMs can never do X task

As this article points out, an LLM can spit out chunks of regurgitated code that it scraped from the internet, but that does not make the LLM a programmer. The resulting output is an attempt to find an existing pattern in the database which fits with what the user has asked for, but it is not a product of actually understanding the use case for the code. It is just statistical correlation.

So, sure, an LLM can be set up to generate output related to X task. If you can collect and clean data that can be used to train the kind of output you want, it should be able to produce an approximate facsimile of the results you want. Is that valuable for your use case? Maybe.

We're still just talking about what is essentially a complex search function. The statistical model returns results from its database that correlate most closely to your input. That does not mean it returns the right answer. If there is no good correlation, it will still return a result.

As long as you understand that the result you get is just a correlation based on your input and may or may not be relevant to your specific problem, and you are not fooled into believing that the LLM actually understands what you're asking and produced a result by "thinking" about it, then you might be able to use an LLM as an effective tool - to search a large collection of information for something that is relevant(ish) to what you're asking for.

The real mistake has been broad misunderstanding of what LLMs actually do, and trying to use them as general-purpose problem solving tools (or worse, as accurate and reliable sources of information).

Not really.

A machine learning model is a computer program. It is fundamentally a math equation, which we understand completely.

A living brain is not fundamentally a math equation, and is not purely a statistical model, at least not in any empirically demonstrable way. We don't understand completely how it works, but we do know that it's more complex than what you're trying to imply.

The comparison is not valid. Machine learning models are not an equivalent to a biological brain.

Not in my experience.

They have failure modes that are easier for humans to spot.

invalid criticisms like “they are just stochastic parrots”

That's not a criticism per se, it's a description of how they work.

Probably fine if you review the code carefully. And if you're working in a domain that AI is decent at (e.g. web stuff). But even if it wasn't it doesn't mean AI cannot program.

If you’re letting your engineers find uses for it instead of constantly demanding that they generate lengthy “user stories” and decision documents and deferring thinking to agents instead of quickly planning stuff out using their experience then you’re probably quite an outlier by now.

This is true. But it doesn't invalidate any of my points. Humans have unique failure modes too.

This is like saying "but it is just a sophisticated network of neurons. All it can do is transform input signals into output signals. That's what it does."

Let them fail and then scramble to rehire.