you are viewing a single comment's thread
view the rest of the comments
[–] 8 points 2 months ago* (2 children)

You need to use MTP (Multi token prediction) to get acceptable speeds on a Strix Halo. On llama.cpp you just need to add --spec-type draft-mtp --spec-draft-n-max 3 -np 1 to your arguments.

You should see around 15-20 t/s in coding tasks at Q8, which will still be much slower than Qwen3-Coder-Next-UD-Q8 since it's 27b active parameters instead of 3b.

As for Q8 vs BF16, it usually makes barely to no difference in quality. Speed is higher on Q8 though. Still testing with 3.8 since it just released.

  • source
  • parent
  • hideshow 2 child comments