llama.cpp built for hip/rocm (since I have AMD GPUs) and a bunch of custom code. One of the first things I did with LLMs was make my own chat client. Most of the useful stuff I get out of LLMs right now involves small focused tasks that I run in a batch with custom scripts (e.g. bulk OCR / image analysis tasks and stuff like that).

Hermes agent (actually two: one running Qwen3.8-27b locally via llama.cop on an RTX 3090 running Ubuntu server with access to my personal stuff as an assistant but no internet access to avoid personal info or secrets egress and another running Deepseek 4.1 Flash 1017/GPT-6.1 Sol - for tougher tasks - served via OpenRouter with full web access but no access to personal info and secrets). They both run in docker containers with limited access to host. I also use squid or caddy reverse proxies to protect against unauthorized web or network access. I am the bridge between them when needed.

Without knowing your hardware, it's impossible to give you advice.

I'm on an 8gb rx7600 with a 5700x CPU with 64gb 3600mts ram.

I use llama.cpp with the rocm driver, then have a custom sandbox for coding and heavily restricted filesystem access, all setup in pi harness, with access to a searxng instance for web search. Using orinth1.5 35b a3b on the iq4_xs unsloth quant.

This gives me the full ability to do anything, from creating reports, coding projects and fairly complex tasks.

I also have stable diffusion setup for image generation with flux 2 klein 4b, which is using qwen4 4b.

I've been playing with small models that can fit on the pitiful 8gb of VRAM on my Radeon RX 7600. So far qwen3.5-9b-artemis has done pretty well at solving merge conflicts on rebases which involves some tool calling. I might try some of the other tooling tuned versions if anyone has any recommendations.

I didn't have much joy with any of the Gemma4 models but that might have been due to API differences.

Ollama running Llama and Gemma, alongside Whisper for speech recognition, running off a 4070 12GB.

Gemma does not work quite right and probably can't stay, just not quite enough memory even when it's the only model loaded.