you are viewing a single comment's thread
view the rest of the comments
[–] 3 points 2 months ago (4 children)

What software are you using for your inference server (llama.cpp, ollama, vllm, sglang, etc)? There are quite a few published recipes that should get you where you want to go.

  • source
  • parent
  • hideshow 4 child comments
  • [–] 3 points 2 months ago (3 children)

    It's llama.cpp. If you know where I could find those recipes I'd appreciate a link.

  • source
  • parent
  • hideshow 3 child comments
  • [–] 4 points 2 months ago* (2 children)

    They’re posted on GitHub pretty regularly, here’s one for the 3090/A5000 specifically.

    https://github.com/mikecovlee/qwen3.8-27b-24gb-recipe

    In particular, I think you want at least the following flags to limit the context length and quantize the Key and Value caches (full context is 16GB alone at full precision KV):

    -c 65535 --cache‑type‑k q8_0 --cache‑type‑v q8_0

    Using q8_0 for Key and Value cuts that in half. So native full context of 262,144 tokens costs 8 GiB instead of 16, but to fit on one card, something like 65535 would only add 2 GB of VRAM at Q8 KV, respectively. I have also heard good things about using q4_0 for KV, which would cut the context budget by half again, with less than 2% loss.

  • source
  • parent
  • hideshow 2 child comments
  • [–] 1 point 2 months ago (1 child)

    Thanks. I did get it working on the 3090 alone with reduced context and KV cache quantized to 4 bits, but it was getting itself muddled up and going round in circles more. Once I got it up to the full 262K context and 8-bit KV cache, it seemed noticeably less confused and more useful, but it needed the extra 10GB for that.

  • source
  • parent
  • hideshow 1 child comment