I'm a noob to local LLMs. I want to use an LLM to create Python and Bash functions from well-defined specs written by me. I've used Claude's Sonnet for this mostly.

I'm a light user of LLMs and almost never hit the request limit on Claude.

It seems like cloud LLMs are in a price war right now and last-gen capabilities are bottoming out in cost? Is that correct?

I pay about $0.12 per kWh, probably going up as more AI data centers get built.

Hardware I have:

  • <16GB VRAM - AMD BC-250 APU - "16 GB total with approximately 12 GiB assigned to GPU UMA and 4 GiB left for the OS" (hardware unlocking changes by the day)
  • 8GB RAM - 11th gen Intel laptop with Xe graphics
  • 8GB RAM - Pixel 7a
  • 32 GB DDR3 - 2nd gen intel - doubt this does anything

Hardware I'm eventually selling:

  • 8GB VRAM RTX 3070 + 32 GB DDR5 + 1TB NVMe SSD - AMD Ryzen 5 7600X CPU - putting this here in case it's substantially better than the BC-250

Among this hardware, I should look for a model that fits on the BC-250?

you are viewing a single comment's thread
view the rest of the comments
[–] 1 point 1 week ago

Rule of thumb that holds up well: model file size + room for context + a bit for the process.

  • 7-8B at 4-bit: ~4-5 GB file, so 8 GB total is comfortable
  • 13B at 4-bit: plan ~12 GB
  • 70B: not a CPU thing, don't bother

The one people forget is context. Cranking num_ctx (or OLLAMA_CONTEXT_LENGTH) up because the model "supports 128k" eats memory fast, and the usual symptom is a model that loads fine and then dies halfway through a long answer.

On CPU, expect roughly 5-15 tok/s for a 4-bit 7-8B on decent cores. Memory bandwidth matters more than core count after a point.

  • source