223
Google unveils TurboQuant, a new AI memory compression algorithm — and yes, the internet is calling it ‘Pied Piper’ (techcrunch.com)
submitted 7 months ago by to c/technology@lemmy.world
36 comments fedilink hide all child comments
you are viewing a single comment's thread
view the rest of the comments
[–] 47 points 7 months ago (4 children)

TurboQuant, meanwhile, could lead to efficiency gains and systems that require less memory during inference. But it wouldn’t necessarily solve the wider RAM shortages driven by AI, given that it only targets inference memory, not training — the latter of which continues to require massive amounts of RAM.

I didn’t realize the RAM shortage was mostly due to training—I would have thought inference was at least a big a factor.

  • permalink
  • fedilink
  • source
  • hideshow 4 child comments
    •
  • [–] 26 points 7 months ago (3 children)

    Inference is dirt cheap in comparison. Hundreds to thousands of concurrent users can be served by hardware costing in the high-thousands to low-ten-thousands.

    Training those same foundational models is weeks to months of time on tens to hundreds of millions worth of hardware.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 3 child comments
    •
  • [–] 11 points 7 months ago (2 children)

    Yeah—but in theory you only need to train once, while inference costs are ongoing and scale up with usage.

    I guess it’s ultimately a business decision by AI companies to weigh how often retraining is worth the cost.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] 16 points 7 months ago

    Yeah i don't think they ever stop training is the thing. At this point I'd assume they have multiple training pipelines to try different shit out, just queued up to hit the big farms as soon as the last models are done training.

    Resting isn't a thing in capitalism.

  • permalink
  • fedilink
  • source
  • parent
  • [–] 2 points 7 months ago

    Training is constant. None of these models by any of these providers are static. You'll notice that they are releasing new models and new model versions regularly.

    This means that training is happening constantly. It never stops. There's always new shit being trained.

  • permalink
  • fedilink
  • source
  • parent