you are viewing a single comment's thread
view the rest of the comments
[–] 5 points 2 months ago* (last edited 2 months ago) (6 children)

Wouldn’t Q4 with FP8 KV cache and full context fit on the 24GB card alone? I’m fairly sure 64K fits, if not, and I’m also really curious about loss rates for some of the newer experimental INT4 KV caches.

  • source
  • parent
  • hideshow 6 child comments
  • [–] 2 points 2 months ago* (last edited 2 months ago) (5 children)

    So I heard, but I couldn't get it to fit. I'm really fumbling my way around this stuff so it's possible I'm just not doing it right.

  • source
  • parent
  • hideshow 5 child comments
  • [–] 3 points 2 months ago (4 children)

    What software are you using for your inference server (llama.cpp, ollama, vllm, sglang, etc)? There are quite a few published recipes that should get you where you want to go.

  • source
  • parent
  • hideshow 4 child comments
  • [–] 3 points 2 months ago (3 children)

    It's llama.cpp. If you know where I could find those recipes I'd appreciate a link.

  • source
  • parent
  • hideshow 3 child comments
  • [–] 4 points 2 months ago* (2 children)

    They’re posted on GitHub pretty regularly, here’s one for the 3090/A5000 specifically.

    https://github.com/mikecovlee/qwen3.8-27b-24gb-recipe

    In particular, I think you want at least the following flags to limit the context length and quantize the Key and Value caches (full context is 16GB alone at full precision KV):

    -c 65535 --cache‑type‑k q8_0 --cache‑type‑v q8_0

    Using q8_0 for Key and Value cuts that in half. So native full context of 262,144 tokens costs 8 GiB instead of 16, but to fit on one card, something like 65535 would only add 2 GB of VRAM at Q8 KV, respectively. I have also heard good things about using q4_0 for KV, which would cut the context budget by half again, with less than 2% loss.

  • source
  • parent
  • hideshow 2 child comments
  • [–] 1 point 2 months ago (1 child)

    Thanks. I did get it working on the 3090 alone with reduced context and KV cache quantized to 4 bits, but it was getting itself muddled up and going round in circles more. Once I got it up to the full 262K context and 8-bit KV cache, it seemed noticeably less confused and more useful, but it needed the extra 10GB for that.

  • source
  • parent
  • hideshow 1 child comment