423
DeepSeek ditches Nvidia for Huawei chips in V4 launch (cybernews.com)
submitted 6 months ago by to c/technology@lemmy.world
89 comments fedilink hide all child comments
you are viewing a single comment's thread
view the rest of the comments
[–] 2 points 6 months ago* (last edited 6 months ago) (9 children)

Nope! You don't know what you're talking about. At all. But you can have fun running a 1.6 trillion parameter model on CPU at basically 0 tokens per second at scale, MoE or not.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 9 child comments
    •
  • [–] 2 points 6 months ago

    You can actually get kind of acceptable performance on CPU alone, but you need rather specific CPUs, like SPR or newer Intel Xeons. These support AMX, which is almost like a mini tensor core, so you can actually get decent throughput in TFLOPs out of GNR Xeons. Memory bandwidth with max channels is also acceptable, something like ~800 GB/s per socket with maxed out MRDIMMs, which is not too far behind consumer GPUs like 3090 and 4090.

    Not anywhere near the performance of real GPUs of course, and not something acceptable for scale or production workloads, but good enough for local inference.

  • permalink
  • fedilink
  • source
  • parent
  • [–] 2 points 6 months ago* (last edited 6 months ago) (7 children)
  • permalink
  • fedilink
  • source
  • parent
  • hideshow 7 child comments
    •
  • [–] 2 points 6 months ago* (last edited 6 months ago) (6 children)

    You've proved my point that you don't know what you're talking about by blindly linking to the git repo. Couldn't find any source that supports your claim? I wonder why.

    Sure you can serve one request at a time to one patient user at a slow token per second rate, which makes running locally viable, but there is no RAM that has the bandwidth to run this model at scale. Even flash would be incredibly slow on CPU with multiple requests. You'd need the high bandwidth of VRAM and to run across multiple GPUs in a scalable way, it requires extremely high bandwidth interconnects between GPUs.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 6 child comments
    •
  • [–] -3 points 6 months ago (5 children)

    Thank you for proving my point. It can be run on a cpu

    “It’s slow, it’s inefficient” it still runs

    It’s a foundational model just like R1 was.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 5 child comments
    •
  • [–] 3 points 6 months ago* (4 children)

    Yes, you can run it at scale.

    at scale

    Shift those goalposts! We went from "at scale" to "it still runs"

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 4 child comments
    •
  • [–] 2 points 6 months ago (3 children)

    Quote me in full.

    You can run it at scale, on huawei. You can also run it on a cpu

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 3 child comments
    •
  • [–] -1 points 6 months ago* (2 children)

    Quote me in full.

    Okay!

    You can run at scale, on huawei. You can also run it on a cpu

    Yeah, that is absolutely not what you argued.

    Anyway, you've conceded that I'm correct that you cannot run it at scale on a CPU, because running on CPU is too slow and inefficient, and that they instead use GPU hardware like Huawei GPUs to run the model at scale. That's good enough for me!

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •