you are viewing a single comment's thread
view the rest of the comments
[–] 5 points 2 months ago* (9 children)

I hooked up a 3090 24GB and a 3080 10GB for a total of 34GB VRAM, and this setup will run the 4-bit quantization with 8-bit KV cache and the full 262K context at a satisfyingly good speed, and it's smart enough to be useful.

  • source
  • hideshow 9 child comments
  • [–] 5 points 2 months ago* (last edited 2 months ago) (6 children)

    Wouldn’t Q4 with FP8 KV cache and full context fit on the 24GB card alone? I’m fairly sure 64K fits, if not, and I’m also really curious about loss rates for some of the newer experimental INT4 KV caches.

  • source
  • parent
  • hideshow 6 child comments
  • [–] 2 points 2 months ago* (last edited 2 months ago) (5 children)

    So I heard, but I couldn't get it to fit. I'm really fumbling my way around this stuff so it's possible I'm just not doing it right.

  • source
  • parent
  • hideshow 5 child comments
  • [–] 3 points 2 months ago (4 children)

    What software are you using for your inference server (llama.cpp, ollama, vllm, sglang, etc)? There are quite a few published recipes that should get you where you want to go.

  • source
  • parent
  • hideshow 4 child comments
  • [–] 3 points 2 months ago (3 children)

    It's llama.cpp. If you know where I could find those recipes I'd appreciate a link.

  • source
  • parent
  • hideshow 3 child comments
  • [–] 4 points 2 months ago* (2 children)

    They’re posted on GitHub pretty regularly, here’s one for the 3090/A5000 specifically.

    https://github.com/mikecovlee/qwen3.8-27b-24gb-recipe

    In particular, I think you want at least the following flags to limit the context length and quantize the Key and Value caches (full context is 16GB alone at full precision KV):

    -c 65535 --cache‑type‑k q8_0 --cache‑type‑v q8_0

    Using q8_0 for Key and Value cuts that in half. So native full context of 262,144 tokens costs 8 GiB instead of 16, but to fit on one card, something like 65535 would only add 2 GB of VRAM at Q8 KV, respectively. I have also heard good things about using q4_0 for KV, which would cut the context budget by half again, with less than 2% loss.

  • source
  • parent
  • hideshow 2 child comments
  • [–] 1 point 2 months ago (1 child)

    Thanks. I did get it working on the 3090 alone with reduced context and KV cache quantized to 4 bits, but it was getting itself muddled up and going round in circles more. Once I got it up to the full 262K context and 8-bit KV cache, it seemed noticeably less confused and more useful, but it needed the extra 10GB for that.

  • source
  • parent
  • hideshow 1 child comment
  • [–] [S] 4 points 2 months ago (1 child)
  • [–] 4 points 2 months ago

    I don't know if I'm doing this right, but this is what llama-bench just gave me:

    >llama-bench -m 'C:\...\models\unsloth\Qwen3.8-27B-GGUF\Qwen3.8-27B-UD-Q4_K_XL.gguf' -p 2048 -n 128
    ggml_cuda_init: found 2 CUDA devices (Total VRAM: 34815 MiB):
      Device 0: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes, VRAM: 24575 MiB
      Device 1: NVIDIA GeForce RTX 3080, compute capability 8.6, VMM: yes, VRAM: 10239 MiB
    load_backend: loaded CUDA backend from F:\llama\llama-b10764-bin-win-cuda-12.4-x64\ggml-cuda.dll
    load_backend: loaded RPC backend from F:\llama\llama-b10764-bin-win-cuda-12.4-x64\ggml-rpc.dll
    load_backend: loaded CPU backend from F:\llama\llama-b10764-bin-win-cuda-12.4-x64\ggml-cpu-haswell.dll
    | model                          |       size |     params | backend    | ngl |            test |                  t/s |
    | ------------------------------ | ---------: | ---------: | ---------- | --: | --------------: | -------------------: |
    | qwen35 27B Q4_K - Medium       |  16.34 GiB |    27.32 B | CUDA       |  -1 |          pp2048 |       1599.03 ± 3.30 |
    | qwen35 27B Q4_K - Medium       |  16.34 GiB |    27.32 B | CUDA       |  -1 |           tg128 |         37.11 ± 0.76 |
    
    build: 3d3d7c818 (10764)
    

    The motherboard is pretty basic (Asus Prime X570-P) so the 3090 is connected to PCIe at 16x speed and the 3080 at 4x speed, which will slow things down a bit. The OS is Windows. I have Linux on the machine too but I haven't tried it under Linux yet.

  • source
  • parent