581
Vibe coding service Replit deleted production database (www.theregister.com)
submitted 1 year ago by to c/technology@lemmy.world
113 comments fedilink hide all child comments
you are viewing a single comment's thread
view the rest of the comments
[–] 98 points 1 year ago* (13 children)

in which the service admitted to “a catastrophic error of judgement”

It’s fancy text completion - it does not have judgement.

The way he talks about it shows he still doesn’t understand that. It doesn’t matter that you tell it simmering in ALL CAPS because that is no different from any other text.

  • permalink
  • fedilink
  • source
  • hideshow 13 child comments
    •
  • [–] 7 points 1 year ago

    judgement

    Yeah, it admitted to an error in judgement because the prompter clearly declared it so.

    Generally LLMs will make whatever statement about what has happened that you want it to say. If you told it it went fantastic, it would agree. If you told it that it went terribly, it will parrot that sentiment back.

    Which what seems to make it so dangerous for some people's mental health, a text generator that wants to agree with whatever you are saying, but doing so without verbatim copying so it gives an illusion of another thought process agreeing with them. Meanwhile, concurrent with your chat is another person starting from the exact same model getting a dialog that violently disagrees with the first person. It's an echo chamber.

  • permalink
  • fedilink
  • source
  • parent
  • [+] -19 points 1 year ago (10 children)

    Are you aware of generalization and it being able to infer things and work with facts in highly abstract way? Might not necessarily be judgement, but definitely more than just completion. If a model is capable of only completion (ie suggesting only the exact text strings present in its training set), it means it suffers from heavy underfitting in AI terms.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 10 child comments
    •
  • [–] 15 points 1 year ago (9 children)

    Completion is not the same as only returning the exact strings in its training set.

    LLMs don't really seem to display true inference or abstract thought, even when it seems that way. A recent Apple paper demonstrated this quite clearly.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 9 child comments
    •
  • [+] -17 points 1 year ago (8 children)

    Coming up with even more vague terms to try to downplay it is missing the point. The point is simple: it's able to solve complex problems and do very impressive things that even human struggle to, in very short time. It doesn't really matter what we consider true abstract thought of true inference. If that is something humans do, then what it does might very well be more powerful than true abstract thought, because it's able to solve more complex problems and perform more complex pattern matching.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 8 child comments
    •
  • [–] 13 points 1 year ago (1 child)

    The point is simple: it's able to solve complex problems and do very impressive things that even human struggle to, in very short time

    You mean like a calculator does?

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 1 child comment
    •
  • [–] -5 points 1 year ago

    Yeah, this is correct analogy, but much more complex problems than calculator. How much it is similar or not to humans way of thinking is completely irrelevant. And how much exact human type of thinking is necessary for any kind of problem solving or work is not something that we can really calculate. Considering that scientific breakthroughs, engineering innovations, medical stuff, complex math problems, programming, etc, do necessarily need human thinking or benefit from it as opposed to super advanced statistical meta-patterning calculator is wishful thinking. It is not based on any real knowledge we have. If you think it is wrong to give it our problems to solve, to give it our work, then it's a very understandable argument, but you should say exactly that. Instead this AI-hate hivemind tries to downplay it using dismissive braindead generic phrases like "NoPe ItS nOt ReAlLy UnDeRsTaNdInG aNyThInG". Okay, who tf asked? It solves the problem. People keep using it and become overpowered because of it. What is the benefit of trying to downplay its power like that? You're not really fighting it this way if you wanted to fight it.

  • permalink
  • fedilink
  • source
  • parent
  • [–] 8 points 1 year ago (5 children)

    Well the thing is, LLMs don't seem to really "solve" complex problems. They remember solutions they've seen before.

    The example I saw was asking an LLM to solve "Towers of Hanoi" with 100 disks. This is a common recursive programming problem, takes quite a while for a human to write the answer to. The LLM manages this easily. But when asked to solve the same problem with with say 79 disks, or 41 disks, or some other oddball number, the LLM fails to solve the problem, despite it being simpler(!).

    It can do pattern matching and provide solutions, but it's not able to come up with truly new solutions. It does not "think" in that way. LLMs are amazing data storage formats, but they're not truly 'intelligent' in the way most people think.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 5 child comments
    •
  • [–] -2 points 1 year ago (4 children)

    This only proves some of them can't solve all complex problems. I'm only claiming some of them can solve some complex problems. Not only by remembering exact solutions, but by remembering steps and actions used in building those solutions, generalizing, and transferring them to new problems. Anyone who tries using it for programming, will discover this very fast.

    PS: Some of them were already used to solve problems and find patterns in data humans weren't able to get other ways before (particle research in CERN, bioinformatics, etc).

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 4 child comments
    •
  • [–] 1 point 1 year ago (3 children)

    You're referring to more generic machine learning, not LLMs. These are vastly different technologies.

    And I have used them for programming, I know their limitations. They don't really transfer solutions to new problems, not on their own anyway. It usually requires pretty specific prompting. They can at best apply solutions to problems, but even then it's not a truly generalised thing, even if it seems to work for many cases.

    That's the trap you're falling into as well; LLMs look like they're doing all this stuff, because they're trained on data produced by people who actually do so. But they can't think of something truly novel. LLMs are mathematically unable to truly generalize, it would prove P=NP if they did (there was a paper from a researcher in IIRC Nijmegen that proved this). She also proved they won't scale, and lo and behold LLM performance is plateauing hard (except in very synthetic, artificial benchmarks designed to make LLMs look good).

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 3 child comments
    •
  • [–] 1 point 1 year ago (2 children)

    They don’t really transfer solutions to new problems

    Lets say there is a binary format some old game uses (Doom), and in it some of its lumps it can store indexed images, each pixel is an index of color in palette which is stored in another lump, there's also a programming language called Rust, and a little known/used library that can look into binary data of that format, there's also a GUI library in Rust that not many people used either. Would you consider it an "ability to transfer solutions to new problems" that it was able to implement extracting image data from that binary format using the library, extracting palette data from that binary format, converting that indexed image using extracted palette into regular rgba image data, and then render that as window background using that GUI library, the only reference for which is a file with names and type signatures of functions. There's no similar Rust code in the wild at all for any of those scenarios. Most of this it was able to do from a few little prompts, maybe even from the first one. There sure were few little issues along the way that required repromting and figuring things together with it. Stuff like this with AI can take like half an hour while doing the whole thing fully manually could easily take multiple days just for the sake of figuring out APIs of libraries involved and intricacies of recoding indexed image to rgba. For me this is overpowered enough even right now, and it's likely going to improve even more in future.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] 1 point 1 year ago (1 child)

    That's applying existing solutions to a different programming language or domain, but ultimately every single technique used already exists. It only applied what it knew, it did not come up with something new. The problem as stated is also not really "new" either, image extraction, conversion and rendering isn't exactly a "new problem".

    I'm not disputing that LLMs can speed up some work, I know it occasionally does so for me as well. But what you have to understand is that the LLM only remembered similar problems and their solutions, it did not at any point invent something truly new. I understand the distinction is difficult to make.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 1 child comment
    •
  • [–] 1 point 1 year ago

    I understand what you're having in mind, I've had similar intuitions about AI in early 2000s. What exactly is "truly new" is an interesting topic ofc, but it's a separate topic. Nowadays I'm trying to look at things more empyrically, without projecting my internal intuitions on everything. In practice it does generalize knowledge, use many forms of abstract reasoning and transfer knowledge across different domains. And it can do coding way beyond the level of complexity of what average software developer does at everyday work.

  • permalink
  • fedilink
  • source
  • parent