you are viewing a single comment's thread
view the rest of the comments
[–] 1 point 3 months ago* (last edited 3 months ago) (2 children)

A P40 with 24GB is ~150€, a V100 (32GB) is ~600€. Both of these fit Qwen3.6 27B (The P40 is about 3x slower though). The V100 even fits 400k context with a Q4 KV-Cache , which means you can have two slots for parallel processing (llama-cpp). You don't even have to use system memory. One of my inference servers is running with 8gb of DDR3 and a 2nd Gen i7, so my old hardware has a good use again.

  • source
  • parent
  • hideshow 2 child comments
  • [–] 1 point 3 months ago (1 child)

    thanks, good stuff! sadly, got none of these locally available.

  • source
  • parent
  • hideshow 1 child comment
  • [–] 3 points 3 months ago

    I had to get mine from ebay and wait a couple weeks as they came from China too. Also I recommend having a 3D-Printer and some Blower-Fans on hand as you will either have to buy or print your own fan shroud for these server cards.

  • source
  • parent