18
Low to midrange systems (8-32 GB) vs. free cloud tiers (lemmy.frozeninferno.xyz)
submitted 2 months ago by to c/localllama@sh.itjust.works
7 comments fedilink hide all child comments
 

I'm a noob to local LLMs. I want to use an LLM to create Python and Bash functions from well-defined specs written by me. I've used Claude's Sonnet for this mostly.

I'm a light user of LLMs and almost never hit the request limit on Claude.

It seems like cloud LLMs are in a price war right now and last-gen capabilities are bottoming out in cost? Is that correct?

I pay about $0.12 per kWh, probably going up as more AI data centers get built.

Hardware I have:

  • <16GB VRAM - AMD BC-250 APU - "16 GB total with approximately 12 GiB assigned to GPU UMA and 4 GiB left for the OS" (hardware unlocking changes by the day)
  • 8GB RAM - 11th gen Intel laptop with Xe graphics
  • 8GB RAM - Pixel 7a
  • 32 GB DDR3 - 2nd gen intel - doubt this does anything

Hardware I'm eventually selling:

  • 8GB VRAM RTX 3070 + 32 GB DDR5 + 1TB NVMe SSD - AMD Ryzen 5 7600X CPU - putting this here in case it's substantially better than the BC-250

Among this hardware, I should look for a model that fits on the BC-250?

you are viewing a single comment's thread
view the rest of the comments
[–] 9 points 2 months ago* (last edited 2 months ago) (1 child)

8GB VRAM RTX 3070 + 32 GB DDR5 + 1TB NVMe SSD - AMD Ryzen 5 7600X CPU - putting this here in case it’s substantially better than the BC-250

Yes this can run Qwen 3.6 35b-a3b pretty nicely! And they might be releasing an updated version of that soon. Your BC-250 only has 16GB total which is not enough for 35b.

I also have 32GB RAM and 8GB VRAM, my computer is a little slower than yours, see my guide: https://lemmus.org/post/24235317

For the BC-250 you might try smaller models like Ling 3.0 Tiny, Ornith 1.5 9b, or Gemma 4 12b QAT

For your 8GB RAM devices, you can run Gemma 4 e4b QAT, Qwen 3.5 4b, or maybe Ling 3.0 Tiny

I’m a noob to local LLMs.

Use Unsloth Desktop or llama.cpp. Then you can connect Zoo Code to it, that's a VSCode extension which I like for programming with my local LLMs.

  • permalink
  • fedilink
  • source
  • hideshow 1 child comment
    •
  • [–] 2 points 1 month ago

    I can second Ornith! I have an AMD 6650xt 8gb (it's rdna2 so I can't take advantage of rocm and have to use vulkan).

    Ornith is absolutely the best code assist model I've found in the small local space so far. It provides usable output and is really good with tool calling.

    The key for getting a model running well locally is not just the model, but figuring out exactly what arguments you should use with llamacpp to run it well on your machine. Since I'm on a dedicated GPU, I need to make sure the full context and kv cache fit fully in VRAM and morning gets offloaded. Offloading to CPU and RAM is the biggest hindrance to token throughput that I've found.

  • permalink
  • fedilink
  • source
  • parent