Without knowing your hardware, it's impossible to give you advice.

I'm on an 8gb rx7600 with a 5700x CPU with 64gb 3600mts ram.

I use llama.cpp with the rocm driver, then have a custom sandbox for coding and heavily restricted filesystem access, all setup in pi harness, with access to a searxng instance for web search. Using orinth1.5 35b a3b on the iq4_xs unsloth quant.

This gives me the full ability to do anything, from creating reports, coding projects and fairly complex tasks.

I also have stable diffusion setup for image generation with flux 2 klein 4b, which is using qwen4 4b.