A P40 with 24GB is ~150€, a V100 (32GB) is ~600€. Both of these fit Qwen3.6 27B (The P40 is about 3x slower though). The V100 even fits 400k context with a Q4 KV-Cache , which means you can have two slots for parallel processing (llama-cpp). You don't even have to use system memory. One of my inference servers is running with 8gb of DDR3 and a 2nd Gen i7, so my old hardware has a good use again.
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
replies: