260
Before Resigning, Jensen Huang Admitted He Was Facing an “Avalanche” of Excess Wafers and Memory, New Report Sheds More Light on the GPU & RAM Crisis (respawnfirst.com)
submitted 1 day ago by to c/technology@lemmy.world
removed 77 comments fedilink hide all child comments
you are viewing a single comment's thread
view the rest of the comments
[–] 1 point 1 day ago* (last edited 1 day ago) (1 child)

It makes sense that general purpose hardware would just be a stopgap for data centers who need to run one very specific workload like LLM prefill at the lowest cost possible. Nvidia’s exact target market who want general purpose hardware for a variety of tasks like running games, Blender, video editors, and some ML models now can’t afford it. Not to worry, though. Even if a B200 sucks at prefill compared to purpose-built hardware, I’m sure a lot of us would happily buy one or two for like $100 each.

  • permalink
  • fedilink
  • source
  • hideshow 1 child comment
    •
  • [–] 4 points 1 day ago*

    An issue with LLM inference is that the software architecture changes significantly, very quickly, and the combination of Nvidia’s software support + some architectural flexibility makes GPUs quite advantageous for that.

    Just as an example, everyone saw KV cache as this existential limiting factor for LLMs. Now, it’s tiny. Another is the old dense model paradigm vs. the almost comically sparse ones coming out. Even matrix-multiply compute could be rendered irrelevant if Bitnet gains traction.

    And that’s not to speak of training.

  • permalink
  • fedilink
  • source
  • parent