What tools and projects do you use for your local LLM chats and agents? I'm trying to sift through the mess of LLM harness and tools to find how to really get the most out of local LLMs.

you are viewing a single comment's thread
view the rest of the comments
[–] 3 points 2 weeks ago (2 children)

I'm on an rx7600 too. I have done loads of testing and benchmarking and the best model is orinth1.5 35b a3b. I get around 40tok/s generation from it. I run it with llama.cpp with the rocm driver, and use the iq4_xs unsloth quant. It's based on qwen3.6 35b a3b, but trained on complex thinking and took calling. It's miles ahead of anything else I've tried.

  • source
  • parent
  • hideshow 2 child comments
  • [–] 1 point 2 weeks ago (1 child)

    How big is your system prompt? My 6.5k (which is mostly 5.5k of tool defs) leads to an error message.

  • source
  • parent
  • hideshow 1 child comment
  • [–] 1 point 2 weeks ago

    Not at my pc now, but it's fairly big. I use pi harness, with a searxng instance, plus a custom sandbox and a read only mount point and a full write mount point plus office tools and coding tools. I use a context window of 130k tokens with auto compaction.

  • source
  • parent