Glad they set a time rather than leaving it open

you are viewing a single comment's thread
view the rest of the comments
[–] 4 points 2 months ago (17 children)

How much does it even cost to run that stuff? Do you need your own private data center?

  • source
  • hideshow 17 child comments
  • [–] 15 points 2 months ago (6 children)

    Nope. For 27B a 16GB video card (8 in a pinch) + 16 (32 better) GB RAM will work for reasonable context, if not screamingly fast. Any M series Macbook with 32 Gb+ (24 in a pinch). Hell, there are setups to run even larger models of a nvme drive if you have enough patience.

    Smaller models like 8B are fine on a CPU, or even a phone.

  • source
  • parent
  • hideshow 6 child comments
  • [–] 3 points 2 months ago* (last edited 2 months ago) (4 children)

    I don't think you can run 27B on 16GB without degrading its quality. You'd have to drop to lower quantization. Is there some new magic?

    35B A3B runs well on 8 and 16GB GPUs.

  • source
  • parent
  • hideshow 4 child comments
  • [–] [S] 6 points 2 months ago (1 child)

    Everything runs well if one is patient enough 😂

    But yeah, with a smaller context and carefully offloading some of the layers to the CPU, it'll run just not very fast. And definitely using Q4 quants.

  • source
  • parent
  • hideshow 1 child comment
  • [–] 3 points 2 months ago

    Everything runs well if one is patient enough 😂

    I mean, if you're really patient you can even run with CPU only as long as you have enough RAM... My ~decade old computer with DDR3 RAM gets 14 tok/s prompt processing and ~5 tok/s decode on MoE models (e.g. a Qwen 3.6 35B-A3B variant at Q6_K -- my current default model). That pp tok/s is painfully slow... but it still works!

    Same computer, same model but using a discrete GPU that can fit all the weights on the card -- ~900 tok/s prompt processing, ~70 tok/s decode.

  • source
  • parent
  • load more comments (1 reply)
  • [–] 8 points 2 months ago (6 children)

    qwen 3.6 27b runs on an AMD AI MAX 395+ fairly well.

    I have one in a laptop with 64gb of ram that I bought before things went too crazy for $2200. Not sure how much one is now.

    Anyway, you don't need a datacenter and that's why I personally think the current AI "boom" is a bubble around a nascent tech that's not really ready for lots of the situations it's been sold for. 🤷‍♂️

  • source
  • parent
  • hideshow 6 child comments
  • [–] 6 points 2 months ago (4 children)

    qwen 3.6 27b runs on an AMD AI MAX 395+ fairly well.

    Try Qwen 3.6 35B-A3B on your hardware instead; it's much faster.

  • source
  • parent
  • hideshow 4 child comments
  • [–] 4 points 2 months ago (2 children)

    27b is much better though...

    And it still gets 15-20 t/s with MTP which is plenty fast.

  • source
  • parent
  • hideshow 2 child comments
  • [–] 3 points 2 months ago (1 child)

    27b is much better though...

    I haven't really found that to be the case for my uses, but getting much faster decode is really useful for me... Have you tried running the MoE model at a higher quant? I usually use Q6_K.

  • source
  • parent
  • hideshow 1 child comment
  • [–] 4 points 2 months ago

    I run both of them at Q8, unquantized KV cache and same settings.

    35b does really well at most tasks but once the dependencies get complicated and the tool calls reach into the hundreds, 35b struggles where 27b still excels.

    So generally I use 27b unless I need it fast, then I swap to 35b or 122b depending on what I need.

  • source
  • parent
  • load more comments (1 reply)
  • [–] 6 points 2 months ago

    I have been running Qwen 3.6 for a while now on a Mac Mini and it's been really nice. It is responding to changes requested in a code review right now and it's doing a pretty good job of it.

  • source
  • parent