you are viewing a single comment's thread
view the rest of the comments
[–] 29 points 1 week ago* (last edited 1 week ago)

I started off with Ollama, but liked llama.cpp better because you can download one GGUF file the day it’s released and add it to a source controlled config.ini with parameters to make it work best on your hardware.

I didn’t get serious about local LLMs until I realized I could run Qwen 3.6 35B-A3B on my 3070 with q4 for KV cache and get over 100k context. I found Ollama’s Modelfile approach and model handling in general to be clunky and discouraging important tweaks, but llama.cpp keeps everything simple.

  • source