256
“I’m calling it now, the adoption of AI agents into software development will be one of the most costly mistakes in the field’s history. Agents cannot program…” (geohot.github.io)
submitted 4 months ago by to c/webdev@programming.dev
83 comments fedilink hide all child comments
💯 3
 

I’m calling it now, the adoption of AI agents into software development will be one of the most costly mistakes in the field’s history. Agents cannot program, and it’s taking longer and longer to realize that they can’t. They are a highly sophisticated statistical model designed to mimic the distribution of programming. The output is broken, but in a way that’s getting harder and harder to detect. Which is exactly what you’d expect from an increasingly accurate statistical model.

you are viewing a single comment's thread
view the rest of the comments
[–] 1 point 4 months ago (3 children)

running the LLM, which is done with a random number generator and a massive matrix of probable next words.

Not true. Inference is done by providing the context to the pre-trained neutral network (technically a transformer network not your daddy's old multilayer perceptron) to generate possible outcomes with logprobs that are then selected based on their likelihood. If it was just frequency-based RNG, they wouldn't have any semantics in the responses and would sound more like traditional Markov chains (like when you mash a button on predictive text and it spits out correct but meaningless gibberish).

If it were just selecting random words from a matrix of probabilities without the network and attentions, it would also be waaay faster and easier to run on a potato.

The stuff about human learning also isn't quite right. There are different types of "learning" and different kinds of memory.

Sleep is generally understood physiologically to be required to formulate long term memory (eg. as described in this paper).

The previous commentator was analogising human short and mid-term memory with LLM context windows (also things like vector databases etc.) and long term memory with retraining/merging/fine tuning of LLMs. It's not totally the same but the analogy is accurate. Brain behaviour is a big influence and inspiration on how machine learning techniques are designed.

Human memory is also notoriously inaccurate and unreliable and tasks done by humans often needs to be double checked and externally verified.

This isn't to say LLMs are trustworthy or reliable. They are not. More that humans think much more highly of themselves than is really warranted.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 3 child comments
    •
  • [–] 6 points 4 months ago* (last edited 4 months ago) (2 children)

    Brain behaviour is a big influence and inspiration on how machine learning techniques are designed.

    I repeat, the LLM is not doing machine learning while users are using it.

    This isn’t to say LLMs are trustworthy or reliable. They are not.

    We agree here.

    More that humans think much more highly of themselves than is really warranted.

    And we agree here too, but to trust an LLM to tell you the truth on your question that you don't know the answer is like trusting some random drunk at the pub, because you don't know whether the answer is from an LLM hallucination, a random lie/error on reddit or an expert's contribution to wikipedia.

    And to trust an LLM when there's a trained programmer or professional journalist is stupid. Sure, an LLM might even sometimes write as good or better code than an intern, but again, the LLM is not learning from its mistakes as you correct it. The intern gradually becomes an expert. The LLM does not. Paying interns is an investment in future programmers, who get more expensive the more experienced they are.

    The LLM is currently cheaper than the intern, but LLM pricing needs to go up by a factor of about ten to cover running costs let alone pay off the vastly more immense debts of buying all that hardware.

    Sleep is generally understood physiologically to be required to formulate long term memory (eg. as described in this paper).

    Like I said before, humans sleep every night, with rare exceptions. LLMs do not get retrained every night. The human brain adapts to feedback loops during everyday interactions, not just overnight. It's a silly analogy and this is a silly point to defend.

    There are plenty of textbooks that say that volatile running RAM is like short term memory and hard disks and SSDs are like long term memory, but it would be silly to reverse the analogy as you are doing and claim that sleep is pressing the save button on the day's learning, or that this makes your word processor the same as your human intelligence because, and this is the central point you've been trying to argue around and about and against, they're doing fundamentally different things, and telling me one was inspired by the other doesn't change that. An LLM is fundamentally a stochastic regurgitator whose training is designed primarily to make it sound right. A human brain just doesn't work that way.

    If you truly believe that the LLM is learning like a human or intelligent like a human, you are confusing analogies for reality.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] 1 point 4 months ago (1 child)

    the LLM is not doing machine learning while users are using it

    This is a small terminology misconception. The LLM is not doing "training" during inference. It's still a "machine learning" system.

    In terms of learning/retaining information in the short/mid term while the user is using it, as the context grows, it retains that information during the current session. In a lot of systems, sections of that context are then summarised and stored, indexed by a vector, to be retrieved into future contexts that have similar semantics. That's why some systems seem to be able to "remember" things from previous "conversations". Your message is vectorised and then that vector used to look up similar past interactions. The model isn't fine tuning on that, so it's not "long term" memory, but the model can take it into account for future interactions.

    AI companies do then use that (and full conversation histories) to regularly fine tune the models, as well as train new ones. It might not be fresh trained every day but certainly more often than you might think.

    to trust an LLM to tell you the truth on your question that you don't know the answer is like trusting some random drunk at the pub

    They're a little more reliable than that and are getting significantly more capable at an alarming rate. We absolutely agree that they shouldn't be trusted and are not very accurate (nor should most humans be trusted or are accurate) but I also think it's dangerous to underestimate them.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 1 child comment
    •
  • [–] 3 points 4 months ago

    They’re a little more reliable than that

    Depends which drink guy at the pub you randomly pick. The attribute that they share with the drink guy at the pub is their reluctance to admit that they don't know or have no expertise or can't help you. Clever and experienced people know where their expertise ends and express self doubt when appropriate. LLMs don't. They can't. They're making literally everything they say up. It's probably right, but they are the script kiddie of conversationalists.

    it’s dangerous to underestimate them

    It's dangerous to underestimate their ability to sound good enough to convince executives to fire humans. It's dangerous to underestimate the scale of substitution of plausibility over knowledge that will only accelerate with further adoption. It's dangerous to assume that the interactions that middle and senior management have with staff that do actual work cannot be replicated already with a suitably trained LLM.

    In terms of learning/retaining information in the short/mid term while the user is using it, as the context grows, it retains that information during the current session. ... past conversations ... remember ...

    Those slashes are doing a lot of work in that sentence!

    Humans learn by generalising from examples. Humans learn when you ask them well-designed questions. Humans learn by practising skills repeatedly. Holland park from their mistakes. Humans learn because they are in a constant state of feedback loop. Humans learn by watching other people. Humans learn by experimentation. Humans learn through playing with new things. Humans learn by talking to each other. Humans learn by sitting and thinking things through. Humans learn through thought experiments. Humans learn through seeing and hearing and reading more quickly than doing any of them alone. Humans learn by explaining things to other people, crystallising their experience into verbal solidity. Humans learn through discussion. Humans learn by learning who to trust and how to weight different input by source. Humans learn by learning how to learn more effectively.

    LLMs do none of any of those things.

    Machine learning is a very, very narrow form of "learning" and you're conflating the use of a neural network with actual learning, which you then compound by confusing the resulting LLM with the neural network that was used in its creation.

    Pulling the wool over people's eyes about what an LLM is is at least as harmful as underestimating the ability of AI to be so plausible as to disrupt absolutely everything about how money moves around society.

  • permalink
  • fedilink
  • source
  • parent