88
Company accidentally spent $500 million on Claude AI in one month after forgetting usage limits (techstartups.com)
submitted 5 months ago by to c/Technology@programming.dev
17 comments fedilink hide all child comments
you are viewing a single comment's thread
view the rest of the comments
[–] -3 points 5 months ago (2 children)

These models run on normal computers, and they are giving them away.

Does your company not have computers?

  • permalink
  • fedilink
  • source
  • hideshow 2 child comments
    •
  • [–] 2 points 5 months ago* (1 child)

    Not really. The state of the art models are huge, even the open-weight ones. You really don't want to quantize below 4-bit, and even that's a bit of a stretch... Ideally you'd use at least 8-bit to get good results with these models when used for coding.

    GLM-5.1 needs around 400GB VRAM at 4-bit quantization. Apple aren't making the Mac Studio with 512GB unified RAM any more, so you'd need something like 5 x Nvidia A100 80GB to run a model like this.

    Kimi K2.6 is around the same size.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 1 child comment
    •
  • [–] 1 point 5 months ago

    Distillation works better than quantization, to the point Qwen recently out-benchmarked its 397B model with a 27B model, two months apart. Arguably the only reason to train comically large models is that this is a decent strategy for finding very small models.

  • permalink
  • fedilink
  • source
  • parent