Glad they set a time rather than leaving it open

you are viewing a single comment's thread
view the rest of the comments
[–] 15 points 2 months ago (6 children)

Nope. For 27B a 16GB video card (8 in a pinch) + 16 (32 better) GB RAM will work for reasonable context, if not screamingly fast. Any M series Macbook with 32 Gb+ (24 in a pinch). Hell, there are setups to run even larger models of a nvme drive if you have enough patience.

Smaller models like 8B are fine on a CPU, or even a phone.

  • source
  • parent
  • hideshow 6 child comments
  • [–] 3 points 2 months ago* (last edited 2 months ago) (4 children)

    I don't think you can run 27B on 16GB without degrading its quality. You'd have to drop to lower quantization. Is there some new magic?

    35B A3B runs well on 8 and 16GB GPUs.

  • source
  • parent
  • hideshow 4 child comments
  • [–] [S] 6 points 2 months ago (1 child)

    Everything runs well if one is patient enough 😂

    But yeah, with a smaller context and carefully offloading some of the layers to the CPU, it'll run just not very fast. And definitely using Q4 quants.

  • source
  • parent
  • hideshow 1 child comment
  • [–] 3 points 2 months ago

    Everything runs well if one is patient enough 😂

    I mean, if you're really patient you can even run with CPU only as long as you have enough RAM... My ~decade old computer with DDR3 RAM gets 14 tok/s prompt processing and ~5 tok/s decode on MoE models (e.g. a Qwen 3.6 35B-A3B variant at Q6_K -- my current default model). That pp tok/s is painfully slow... but it still works!

    Same computer, same model but using a discrete GPU that can fit all the weights on the card -- ~900 tok/s prompt processing, ~70 tok/s decode.

  • source
  • parent
  • load more comments (1 reply)