Not my settings! All credits to "Stainless-Bacon". I did however replicate the setup just now and I think it's worthy of spreading.
The trick is in a targeted offloading of only a specific kind of layers to CPU making it possible to run a higher quant that otherwise would be too slow to be usable.
Remember to compile llama.cpp with GGML_CUDA_FA_ALL_QUANTS=ON
Original settings below. Since I'm using the iGPU for the system I'm slightly tweaking them to use even more VRAM. On my 5060Ti I'm getting prefill 500-600tps (ub at 2048) and tg at ~10tps.
export GGML_CUDA_DISABLE_GRAPHS=1
llama-server
--model Qwen3.6-27B-Q4_K_M_MTP.gguf
--chat-template-file froggeric_fix.jinja
--chat-template-kwargs '{"preserve_thinking": true}'
--jinja
--spec-type draft-mtp
--spec-draft-n-max 2
--fit off
--n-gpu-layers 99
--override-tensor 'blk.(2[0-9]|3[0-9]|4[0-3]).ffn_.*=CPU'
--ctx-size 96000
--batch-size 512
--ubatch-size 512
--cache-type-k q5_0
--cache-type-v q4_1
--parallel 1
--temp 0.60
--top-p 0.95
--top-k 20
--min-p 0.0
--presence-penalty 0.0
--repeat-penalty 1.0
--flash-attn on
--no-mmap
--host 0.0.0.0
Interesting, thanks for sharing. For me I need a bit more context (160k) and I need my KDE desktop running which forces me to stick with Q3 instead of Q4 on my RX 9070 XT. But offloading these specific layers instead of using fit = on nets my a respectable 4 t/s more (20t/s overall). That's pretty usable.
all 5 comments