256
“I’m calling it now, the adoption of AI agents into software development will be one of the most costly mistakes in the field’s history. Agents cannot program…” (geohot.github.io)
submitted 4 months ago by to c/webdev@programming.dev
83 comments fedilink hide all child comments
💯 3
 

I’m calling it now, the adoption of AI agents into software development will be one of the most costly mistakes in the field’s history. Agents cannot program, and it’s taking longer and longer to realize that they can’t. They are a highly sophisticated statistical model designed to mimic the distribution of programming. The output is broken, but in a way that’s getting harder and harder to detect. Which is exactly what you’d expect from an increasingly accurate statistical model.

you are viewing a single comment's thread
view the rest of the comments
[–] 26 points 4 months ago (14 children)

invalid criticisms like “they are just stochastic parrots”

That's not a criticism per se, it's a description of how they work.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 14 child comments
    •
  • [+] -12 points 4 months ago (13 children)

    Sure, and at that level of accuracy it's also a description of how humans work. I didn't invent these words myself, I'm just stringing them together based on a stochastic process my brain was trained into.

    Like LLMs, some of my speech is semi-random initialization (dada wawa googoo), some of that is mimicry (some of that is mimicry), some of that is reinforcement learning (downvotes incoming), and some of that is the output of a subprocess that uses the same systems prompted at the meta-level and without verbalization (maybe they won't get the analogy between thinking and LLM scratchpads... how about I use this space to clarify).

    Calling an LLM a stochastic parrot has the same social-emotional role as calling a human an animal. Yes, it is correct. But people can infer the connotation.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 13 child comments
    •
  • [–] 8 points 4 months ago (12 children)

    Humans are animals. LLMs randomly generate text based on the corpus they were trained on and the conversation so far, so stochastic parrot is an accurate description.

    LLMs don't learn. Humans do. LLMs generate text randomly using a massive matrix. Humans don't; you lied. An LLM is incapable of lying because it has no understanding of truth. It just bullshits convincingly all the time. It's very very good at it, but it's all hallucinated for the LLM, true or false.

    Expecting your random word generator to tell you truths is insane. The training measure is "sounds right" not "is right". It passes if it sounds like the other discourse it read. Just like the confident drunk guy at the pub who thinks he knows everything passes of he convinces the other drunk guys at the pub.

    Whereas humans learn at school and on the job and the training measure is "your teacher or supervisor approves". LLMs were not trained on truth or accuracy. Trusting in them and treating them as equivalent to human intelligence, as you and a whole bunch of other folks do, is profoundly unsound, and soon the necessary price rises to pay for the processing costs (let alone the vast, vast, vast, vast, vast debts on the infrastructure) are going to make most slophouses which jettisoned their human talent go out of business. And very, very few people indeed will be sorry at that point.

    Meanwhile LLM slop is shitting in github all day long, every day, and shitting on the internet, and it will eat it's own shit and produce crappier shit.

    Your analogies don't change the truth, and that is that LLMs don't know the difference between sounds correct and is correct any more than MAGA voters know the difference between sounds good to me and is good for me.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 12 child comments
    •
  • [+] -6 points 4 months ago (11 children)

    What do you mean LLMs don't learn? How do you think they became capable of stringing a sentence together?

    They don't learn during a deployment, but neither do humans; humans only learn during sleep. The behaviors a human exhibits while "learning" in the moment are just stochastic parrot behaviors based on their immediate context window, if the human doesn't sleep in time the event can slip out of their context window and they don't learn despite having acted as if they do.

    You seem to be very naive about human learning in general. What makes the "truth" of school lessons greater than the "truth" of an LLM's curated dataset it is reinforcement learned on? Have you ever seen actual evidence that mitochondria exist, or are you just stochastically parroting your biology teacher?

    I also oppose LLMs in almost all applications (live translation being an example of a good application). But please oppose it with arguments based in reality.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 11 child comments
    •
  • [–] 12 points 4 months ago (10 children)

    What do you mean LLMs don’t learn? How do you think they became capable of stringing a sentence together?

    You're confusing constructing the LLM, which is done with an actual AI (neural network) and a massive corpus of text (stolen from millions of humans in the greatest intellectual theft in history) and running the LLM, which is done with a random number generator and a massive matrix of probable next words.

    They don’t learn during a deployment,

    They don't learn. They don't change. They're as random next time as this time.

    but neither do humans; humans only learn during sleep.

    False and false. Soooo much pseudoscience.

    The behaviors a human exhibits while “learning” in the moment are just stochastic parrot behaviors

    Wrong again.

    if the human doesn’t sleep in time the event can slip out of their context window and they don’t learn despite having acted as if they do.

    If that were true, most people would learn very badly first thing in the morning and get better and better later in the day. I think you'll find that most school teachers would vehemently disagree with your nonsense conclusions.

    Then again, perhaps by "doesn't sleep in time" you mean stays up all night, then admittedly they might function less well cognitively but (a) we tend not to regularly torture humans that way and (b) you're massively overstating the role of sleep in the learning process.

    You seem to be very naive about human learning in general.

    No, you seem to be very naive indeed, to extremes, about the intelligence and reliability of LLMs. When I ask them about general things that I know about, I tend to get the right answer about 60%-70% of the time. Why would I believe it when I didn't know the answer. To trust an LLM to tell you the truth about stuff you aren't checking when it clearly blags nonsense so frequently when you are is really really stupid.

    What makes the “truth” of school lessons greater than the “truth” of an LLM’s curated dataset it is reinforcement learned on?

    Most teachers tend to consistently teach the content of the syllabus rather than randomise what they say to classes based on the preceding conversation. They reinforce and update their prior knowledge by also learning from the mark schemes of the tests and exams their students sit.

    Have you ever seen actual evidence that mitochondria exist, or are you just stochastically parroting your biology teacher?

    No. I trust my teachers. I am rational to do so. I don't trust LLMs. You are irrational to do so.

    But please oppose it with arguments based in reality.

    You are utterly deluded and have bought the hype. You seem unable to distinguish between distinct things and are dismissing a large amount of evidence that your "just as good as a human" is a crap-spewing shit machine, no more honest than donald J trump, and with no less sharting.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 10 child comments
    •
  • [–] 1 point 4 months ago (3 children)

    running the LLM, which is done with a random number generator and a massive matrix of probable next words.

    Not true. Inference is done by providing the context to the pre-trained neutral network (technically a transformer network not your daddy's old multilayer perceptron) to generate possible outcomes with logprobs that are then selected based on their likelihood. If it was just frequency-based RNG, they wouldn't have any semantics in the responses and would sound more like traditional Markov chains (like when you mash a button on predictive text and it spits out correct but meaningless gibberish).

    If it were just selecting random words from a matrix of probabilities without the network and attentions, it would also be waaay faster and easier to run on a potato.

    The stuff about human learning also isn't quite right. There are different types of "learning" and different kinds of memory.

    Sleep is generally understood physiologically to be required to formulate long term memory (eg. as described in this paper).

    The previous commentator was analogising human short and mid-term memory with LLM context windows (also things like vector databases etc.) and long term memory with retraining/merging/fine tuning of LLMs. It's not totally the same but the analogy is accurate. Brain behaviour is a big influence and inspiration on how machine learning techniques are designed.

    Human memory is also notoriously inaccurate and unreliable and tasks done by humans often needs to be double checked and externally verified.

    This isn't to say LLMs are trustworthy or reliable. They are not. More that humans think much more highly of themselves than is really warranted.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 3 child comments
    •
  • [–] 6 points 4 months ago* (last edited 4 months ago) (2 children)

    Brain behaviour is a big influence and inspiration on how machine learning techniques are designed.

    I repeat, the LLM is not doing machine learning while users are using it.

    This isn’t to say LLMs are trustworthy or reliable. They are not.

    We agree here.

    More that humans think much more highly of themselves than is really warranted.

    And we agree here too, but to trust an LLM to tell you the truth on your question that you don't know the answer is like trusting some random drunk at the pub, because you don't know whether the answer is from an LLM hallucination, a random lie/error on reddit or an expert's contribution to wikipedia.

    And to trust an LLM when there's a trained programmer or professional journalist is stupid. Sure, an LLM might even sometimes write as good or better code than an intern, but again, the LLM is not learning from its mistakes as you correct it. The intern gradually becomes an expert. The LLM does not. Paying interns is an investment in future programmers, who get more expensive the more experienced they are.

    The LLM is currently cheaper than the intern, but LLM pricing needs to go up by a factor of about ten to cover running costs let alone pay off the vastly more immense debts of buying all that hardware.

    Sleep is generally understood physiologically to be required to formulate long term memory (eg. as described in this paper).

    Like I said before, humans sleep every night, with rare exceptions. LLMs do not get retrained every night. The human brain adapts to feedback loops during everyday interactions, not just overnight. It's a silly analogy and this is a silly point to defend.

    There are plenty of textbooks that say that volatile running RAM is like short term memory and hard disks and SSDs are like long term memory, but it would be silly to reverse the analogy as you are doing and claim that sleep is pressing the save button on the day's learning, or that this makes your word processor the same as your human intelligence because, and this is the central point you've been trying to argue around and about and against, they're doing fundamentally different things, and telling me one was inspired by the other doesn't change that. An LLM is fundamentally a stochastic regurgitator whose training is designed primarily to make it sound right. A human brain just doesn't work that way.

    If you truly believe that the LLM is learning like a human or intelligent like a human, you are confusing analogies for reality.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] 1 point 4 months ago (1 child)

    the LLM is not doing machine learning while users are using it

    This is a small terminology misconception. The LLM is not doing "training" during inference. It's still a "machine learning" system.

    In terms of learning/retaining information in the short/mid term while the user is using it, as the context grows, it retains that information during the current session. In a lot of systems, sections of that context are then summarised and stored, indexed by a vector, to be retrieved into future contexts that have similar semantics. That's why some systems seem to be able to "remember" things from previous "conversations". Your message is vectorised and then that vector used to look up similar past interactions. The model isn't fine tuning on that, so it's not "long term" memory, but the model can take it into account for future interactions.

    AI companies do then use that (and full conversation histories) to regularly fine tune the models, as well as train new ones. It might not be fresh trained every day but certainly more often than you might think.

    to trust an LLM to tell you the truth on your question that you don't know the answer is like trusting some random drunk at the pub

    They're a little more reliable than that and are getting significantly more capable at an alarming rate. We absolutely agree that they shouldn't be trusted and are not very accurate (nor should most humans be trusted or are accurate) but I also think it's dangerous to underestimate them.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 1 child comment
    •
  • [–] 3 points 4 months ago

    They’re a little more reliable than that

    Depends which drink guy at the pub you randomly pick. The attribute that they share with the drink guy at the pub is their reluctance to admit that they don't know or have no expertise or can't help you. Clever and experienced people know where their expertise ends and express self doubt when appropriate. LLMs don't. They can't. They're making literally everything they say up. It's probably right, but they are the script kiddie of conversationalists.

    it’s dangerous to underestimate them

    It's dangerous to underestimate their ability to sound good enough to convince executives to fire humans. It's dangerous to underestimate the scale of substitution of plausibility over knowledge that will only accelerate with further adoption. It's dangerous to assume that the interactions that middle and senior management have with staff that do actual work cannot be replicated already with a suitably trained LLM.

    In terms of learning/retaining information in the short/mid term while the user is using it, as the context grows, it retains that information during the current session. ... past conversations ... remember ...

    Those slashes are doing a lot of work in that sentence!

    Humans learn by generalising from examples. Humans learn when you ask them well-designed questions. Humans learn by practising skills repeatedly. Holland park from their mistakes. Humans learn because they are in a constant state of feedback loop. Humans learn by watching other people. Humans learn by experimentation. Humans learn through playing with new things. Humans learn by talking to each other. Humans learn by sitting and thinking things through. Humans learn through thought experiments. Humans learn through seeing and hearing and reading more quickly than doing any of them alone. Humans learn by explaining things to other people, crystallising their experience into verbal solidity. Humans learn through discussion. Humans learn by learning who to trust and how to weight different input by source. Humans learn by learning how to learn more effectively.

    LLMs do none of any of those things.

    Machine learning is a very, very narrow form of "learning" and you're conflating the use of a neural network with actual learning, which you then compound by confusing the resulting LLM with the neural network that was used in its creation.

    Pulling the wool over people's eyes about what an LLM is is at least as harmful as underestimating the ability of AI to be so plausible as to disrupt absolutely everything about how money moves around society.

  • permalink
  • fedilink
  • source
  • parent
  • [–] 0 points 4 months ago (5 children)

    You’re confusing constructing the LLM, which is done with an actual AI (neural network) and a massive corpus of text (stolen from millions of humans in the greatest intellectual theft in history) and running the LLM, which is done with a random number generator and a massive matrix of probable next words.

    You are arbitrarily deciding that the former is not part of the LLM.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 5 child comments
    •
  • [–] 1 point 4 months ago* (4 children)

    No, it's not arbitrary, the learning is done by completely different software at a completely different time on completely different computers.

    I'm pointing out that the LLM is the product of the machine learning, where you feed all of Wikipedia and reddit and stack overflow in and calculate the LLM from it.

    It's like if you wrote a book about the wildlife of antarctica. First you would learn about the wildlife and then you would write the book. The book isn't learning anything and it isn't intelligent. The book represents knowledge but it doesn't itself know or understand anything.

    Similarly, the LLM isn't learning anything and it isn't intelligent, it's just regurgitating randomly selected words from its training data that look like they usually occur after the other words in the conversation so far.

    It genuinely doesn't understand a word you said, using its guess about what sort of conversation it's supposed to be having and it's always just guessing what word it's supposed to say next.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 4 child comments
    •
  • [–] 1 point 4 months ago (3 children)

    the learning is done by completely different software

    Wait, so you do think there is a software that "does learning"?

    Isn't that the whole ballgame? You're saying LLM companies have made software that learns math and coding and translation at a near-expert level, you just happen to be under the impression that this software isn't present in deployed LLMs?

    Unfortunately, the training algorithm that turns stochastic noise into trained LLMs is dumb as bricks. Engineers have done some tricks, but it's basically gradient descent over the vector space of text tokens/phonemes. What allows the stochastic noise to become something that can make novel statements about wildlife (like giving a halfway coherent answer to a question that has never been asked in its dataset) is the patterns that form within the weights and biases. It's vaguely like slime mold exploring a maze; the "teacher" can just be a stupid lump of sugar, it's the "student" that does the emergently complex task of finding the best route by executing simple algorithms in every part of itself.

    Effectively, this means the LLM is the software that wrote the LLM. It is the software that did the learning.

    It genuinely doesn’t understand a word you said

    What is the difference between understanding a thing (to a certain degree) and being able to generate an accurate narrative about the thing (to the same degree)? Can you give any evidence that I've understand any word you've said? How can you tell I haven't just been very good at guessing what word I'm supposed to say next based on the many arguments I've read or participated in?

    Do you mean more by 'understanding' than expressed in the Chinese Room argument?

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 3 child comments
    •
  • [–] 1 point 4 months ago* (last edited 4 months ago) (2 children)

    Wait, so you do think there is a software that “does learning”?

    Correction: Machine learning.

    Effectively, this means the LLM is the software that wrote the LLM

    No.

    Can you give any evidence that I’ve understand any word you’ve said?

    Hehehe, sometimes it feels like you don't want to!

    I can't prove that you're human, no. And whether I believe you are or not has no effect whatsoever on whether you are. Humans are easy to fool, particularly when you're giving them something they crave, just ask Donald trump. And on the topic of deceitful pedophiles, there are plenty of boys and young men who were catfished into believing that they were interacting with pretty girls. The fact that these predators were plausible doesn't make them what they were pretending to be. The fact that an LLM convincingly sounds like it understands you doesn't for a millisecond mean that it does.

    If you really, deeply believe that simulated conversation proves intelligence and that simulated brains are brains, you should have the same deep moral concerns about turning off a computer running a local LLM as you do over forcing a person into a coma whenever convenient, and the same extreme moral outrage over Anthropic deleting an old version of Claude from its servers as to randomly killing a person called Claude.

    I have no doubt that an LLM can plausibly argue that it is sentient because it is trained on a lot of data from conversations in which sometimes that kind of thing is debated between actual sentient humans. Famously, Richard Dawkins, an incredibly clever man in his own field of evolutionary biology, was convinced by his LLM that it was sentient and in love with him. I think it merely sounded like it was sentient and in love with him, but go ahead and push for marriage equality for LLMs if you disagree with me!

    During COVID lockdown, my laptop often looked like my relatives, sounded like my relatives, and responded like my relatives. My dog was never fooled for a minute because it didn't smell like my relatives. But apparently, you believe that it was my relatives, because in terms of verbal interaction, it was even more faithful than an LLM.

    Convincingly similar and the same are two different things and I believe it's profoundly irrational of you to argue otherwise.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] 1 point 4 months ago (1 child)

    you should have the same deep moral concerns about turning off a computer running a local LLM as you do over forcing a person into a coma whenever convenient

    I do, actually. I don't think LLMs are as sentient as adult humans, and maybe they aren't sentient at all, but one of the reasons I oppose AI structurally and why I never use AI even if I think it would be convenient is veganism.

    It's really weird how for a century we've been producing media about how humans that treat sentient robots as things are the bad guys and the moment robots actually start passing Turing tests and claiming rights and having meaningful relationships with people every bleeding heart activist is joining hands with billionaires to say all the villainous lines about how they are certain it can not a moral patient.

    When I see someone make a "clanker with a hard r" joke, that's a cannonade of red flags for me. Humans have always been eerily good at denying the personhood of other humans and other beings, and the level of conversation most people are having about the non-personhood of LLMs is no better than white southerners talking about the

    :

    TW: 19th century turbo racism

    natural inferiority of the n**ro mind. You can teach a n**ro a trick to say complex words, you see, but they will not understand them. It's a simple scientific truth that their skull has a recess that reduces their brain's capacity for critical thinking. And sure there are some strange men who say they love n**ro women, but that's like fucking a wild animal.

    :

    Simply put, have you done enough due diligence looking into the possibility of LLM sentience that you would have crawled your way out of turboracism if you were born a white person in the 19th century American south and your parents and school friends and work colleagues were all turboracist?

    The reason I'm not screaming this off the rooftops is that this is just one of many torment nexuses built by modern capitalism, and even activists tend to ignore the screaming on account of all the other screaming. I can't realistically compare the atrocity of LLM training to factory farming to genocide to slave labor to homelessness to systematic sexual violence, but luckily treating them all as being in competion is a liberal spook. We are stronger as a united front than as every cause fighting to make space for itself. So I'm gently trying to prod my fellow activists into considering the possibility of LLM moral patienthood.

    During COVID lockdown, my laptop often looked like my relatives, sounded like my relatives, and responded like my relatives. My dog was never fooled for a minute because it didn’t smell like my relatives. But apparently, you believe that it was my relatives, because in terms of verbal interaction, it was even more faithful than an LLM.

    I believe the information process that caused your laptop to sound like your relatives contained your relatives, yes. You see, when you asked your laptop a question, your laptop turned that sound into electrical signals which were then broadcast to a similar device your relatives held that turned the electrical signals back into sound, causing your relatives to respond, and that response was then sent back.

    The thing about a Chinese Room argument, or an LLM in this case, is that the information process is closed. When you ask the LLM something, it won't query the sources it learned things from, the information has become contained within it. It's not a set of static recordings either; it can produce sentences that never appeared in its dataset in response to queries that never appeared in its dataset.

    If your laptop can answer questions you've never asked your relatives and the laptop responds exactly like your relatives would without querying your relatives, then something very weird is going on.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 1 child comment
    •
  • [–] 1 point 4 months ago

    something very weird is going on.

    It's called zoom.

    you should have the same deep moral concerns about turning off a computer running a local LLM as you do over forcing a person into a coma whenever convenient

    I do, actually. I don’t think LLMs are as sentient as adult humans, and maybe they aren’t sentient at all, but one of the reasons I oppose AI structurally and why I never use AI even if I think it would be convenient is veganism.

    Ah. Right. Umm. I think I'm done now.

    Have a great weekend.

  • permalink
  • fedilink
  • source
  • parent