85
The Decoherence Trap: When AI slop drowns out the signal in an endless sea of noise (maevyn.codes)
submitted 3 weeks ago by to c/programming@programming.dev
7 comments fedilink hide all child comments
 

Decoherence grows relative to the rate of change of a system, and it increases faster the more decoherent a system is, resulting in runaway exponential growth. It is not a new phenomenon, and humans have always countered it through carefully finding ways to do more with less, unifying disparate patterns and models into a cohesive whole. However, the widespread use of AI is currently producing it at a rate that is unsustainable.

The good news is that this rules out the specter of runaway recursive self-improvement, the intelligence explosion that goes parabolic before anyone can react, as decoherence both prevents AI from improving in a bubble and limits the rate at which it can improve outside of it. The bad news is that we are barreling toward a different kind of disaster: Decoherence Collapse.

all 7 comments

sorted by: hot top controversial new old
[–] 9 points 2 weeks ago

Fantastic write-up!

  • permalink
  • fedilink
  • source
    •
    [–] 7 points 2 weeks ago (1 child)

    This is everything I’ve ever wanted to say about AI at my job, plus things I never would’ve thought of, perfectly articulated, well-cited, and emphasized with poignant graphics. I’m floored.

  • permalink
  • fedilink
  • source
  • hideshow 1 child comment
    •
    • [–] 3 points 2 weeks ago

    We are currently living through the golden age of Ai slop.

  • permalink
  • fedilink
  • source
    •
    [–] 1 point 2 weeks ago (2 children)

    This article argues that any conceivable society that holds art competitions will be less capable than a society that does not, because art does not represent real world objects so holding a competition in art trains people to lose track of reality, eventually rendering them unable to do even basic functions as they try to paint instead of walk.

    After all, if you aren't currently looking at a flesh and blood rhino, you can never learn to get better at drawing rhinos, and that shoddy workmanship means you'll never be able to draw a savannah (which has rhinos and many other things), let alone develop the live-updated 3D model you need to walk across one.

    Soon, people who aren't currently looking at rhinos will be unable to drag themselves across the floor, not knowing how to draw the hands and feet that would propel them.

  • permalink
  • fedilink
  • source
  • hideshow 2 child comments
    •
  • [–] 2 points 2 weeks ago (1 child)

    Did you read the article? It's long, but worth it. It describes analytical problem solving and software development.

    Did I miss anything supporting your claim that the article argues about art?

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 1 child comment
    •
  • [–] 1 point 2 weeks ago

    The article makes general assertions about systems. It gives plumbing, software, and machine learning as examples, and I simply add the examples of human learning and human societies.

    An agent specifying the criteria of a test inspired by the real world and then evaluating the performance of a bunch of other agents is equally information-incestuous when those agents are LLMs as when they are humans. If the observations of the article must hold true for LLM training, then they must hold true for humans. If they do not hold true for humans, then they do not prohibit the existence of LLM training programs that likewise avoid the decoherence trap.

    Given the fact that evolution was able to find human creativity and art as forms of cohering enrichment of human performance beyond the limits of the training data of a world without art, I would expect that there is a good chance tens of thousands of engineers and billions of dollars of compute will be able to find some form of abstract expression and learning as a form of cohering enrichment of LLM performance beyond the limits of the training data of a world without that abstract LLM expression.

  • permalink
  • fedilink
  • source
  • parent
    • 5 comments in c/technology@beehaw.org view that post
    [–] 4 points 2 weeks ago

    Really good, it put into words an outline of an idea I had in the back of my brain. Probably the only original, intelligent article written about AI recently.

  • permalink
  • fedilink
  • source
    •
    [–] 2 points 2 weeks ago* (last edited 2 weeks ago)

    I don't think the primary counterargument is tech progression, I think the primary counterargument is that decoherence as defined here, and tech debt in general, are usually a function of humans balancing work time vs deliverable demand (for coding), and making tradeoffs like those adhoc patches vs rebuilds-to-maintain-coherence... but that is not intrinsic to coding, and is much less relevant to an agent swarm who could feasibly decide that a full rebuild using the lessons gleaned about design, but without the actual code debt, is warranted and possible within a short time window.

    If a cyclical process of n iterative updates > rebuild back to coherence > n iterative updates > rebuild back to coherence etc is used, you could mitigate the exponential loss of coherence, provided you set some kind of baseline for what rebuilds produce.

  • permalink
  • fedilink
  • source
    •
    [–] 0 points 2 weeks ago* (last edited 2 weeks ago) (2 children)

    This is interesting but oddly difficult to parse, ironically the writing style has LLM vibes with all the fluffy generic statements and overall lack of density.

  • permalink
  • fedilink
  • source
  • hideshow 2 child comments
    •