423
DeepSeek ditches Nvidia for Huawei chips in V4 launch (cybernews.com)
submitted 6 months ago by to c/technology@lemmy.world
89 comments fedilink hide all child comments
you are viewing a single comment's thread
view the rest of the comments
[–] 8 points 6 months ago* (1 child)

Even with a bitnet, it’s almost definitely better to train on a high precision float then refine down to bits.

I would expect bitnet to require more layers for equivalent quality too.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 1 child comment
    •
  • [–] 3 points 6 months ago

    I just meant for mass inference serving.

    Yeah, I haven’t seen much in the way of bitnet training savings yet, like regular old QAT. It does appear that Deepseek is finetuning their MoEs in a 4-bit format now, though.

  • permalink
  • fedilink
  • source
  • parent