423
DeepSeek ditches Nvidia for Huawei chips in V4 launch (cybernews.com)
submitted 6 months ago by to c/technology@lemmy.world
89 comments fedilink hide all child comments
you are viewing a single comment's thread
view the rest of the comments
[–] 3 points 6 months ago (3 children)

Yeah, I'd expect KTransformers to add support eventually, especially considering their existing support for previous DeepSeek models. One of the tricky parts is that backends need both FP8 and MXFP4 support. As far as I'm aware no inference engine supports both on CPU at the moment (llama.cpp added fp4 support recently, but doesn't have fp8, while kt-kernel doesn't support fp4 yet).

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 3 child comments
    •
  • [–] 2 points 6 months ago* (2 children)

    Not to speak of the new attention scheme and the (IIRC) MLP changes.

    I’m very much looking forward to ik_llama.cpp implementing it. I don’t think I can quite fit Flash on my rig (hence no Ktransformers for me) but a little quantization of the sparse layers, and it’d be perfect.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] 1 point 6 months ago (1 child)

    Makes sense, even Flash is fairly sizable! KTransformers also has a "llamafile" backend which uses GGUFs, but ik_llama will almost certainly perform better if you're not on a NUMA setup. In my case, I'm using a dual socket motherboard, so KTransformers performs quite a bit better (I think ik_llama hasn't implemented extensive NUMA optimizations quite yet, but sounds like it's coming), though I normally use KTransformers for native FP8 weights.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 1 child comment
    •
  • [–] 2 points 6 months ago

    It is! 143GB last I checked. I'm on 128GB RAM + 3090, 1 NUMA node, so I think it's juuust barely too tight. But it should be perfect with a few of the "sparsest" MoEs quantized.

    If KTransformers supports something like that, I may have to finally check it out, since v4 won't need many esoteric features.

  • permalink
  • fedilink
  • source
  • parent