you are viewing a single comment's thread
view the rest of the comments
[–] 4 points 2 months ago* (2 children)

They’re posted on GitHub pretty regularly, here’s one for the 3090/A5000 specifically.

https://github.com/mikecovlee/qwen3.8-27b-24gb-recipe

In particular, I think you want at least the following flags to limit the context length and quantize the Key and Value caches (full context is 16GB alone at full precision KV):

-c 65535 --cache‑type‑k q8_0 --cache‑type‑v q8_0

Using q8_0 for Key and Value cuts that in half. So native full context of 262,144 tokens costs 8 GiB instead of 16, but to fit on one card, something like 65535 would only add 2 GB of VRAM at Q8 KV, respectively. I have also heard good things about using q4_0 for KV, which would cut the context budget by half again, with less than 2% loss.

  • source
  • parent
  • hideshow 2 child comments
  • [–] 1 point 2 months ago (1 child)

    Thanks. I did get it working on the 3090 alone with reduced context and KV cache quantized to 4 bits, but it was getting itself muddled up and going round in circles more. Once I got it up to the full 262K context and 8-bit KV cache, it seemed noticeably less confused and more useful, but it needed the extra 10GB for that.

  • source
  • parent
  • hideshow 1 child comment