I'm new to local llms, i'm running Debian on this machine. Where should I start and what model(s) could I get testing on it?

you are viewing a single comment's thread
view the rest of the comments
[–] 7 points 1 week ago* (1 child)

I have 8GB VRAM and 32GB RAM, mostly using llama.cpp to run Tiel 35b at Q4, it's ok at writing code in an agent as long as it can compile and run tests to iteratively fix its mistakes.

Though there's been new optimized engines, like Strata, that could maybe allow you to run Qwen Flash Next at decent speeds.

There's also Qwen 4 coming out soon that may give some good new options for us VRAM poors. Just stick to MoE models.

  • source
  • hideshow 1 child comment