19
what is your selfhosted local LLM stack? (lemmy.blahaj.zone)
submitted 2 weeks ago by to c/fosai@lemmy.world
8 comments fedilink hide all child comments
 

What tools and projects do you use for your local LLM chats and agents? I'm trying to sift through the mess of LLM harness and tools to find how to really get the most out of local LLMs.

you are viewing a single comment's thread
view the rest of the comments
[–] 2 points 2 weeks ago (3 children)

I've been playing with small models that can fit on the pitiful 8gb of VRAM on my Radeon RX 7600. So far qwen3.5-9b-artemis has done pretty well at solving merge conflicts on rebases which involves some tool calling. I might try some of the other tooling tuned versions if anyone has any recommendations.

I didn't have much joy with any of the Gemma4 models but that might have been due to API differences.

  • permalink
  • fedilink
  • source
  • hideshow 3 child comments
    •
  • [–] 3 points 2 weeks ago (2 children)

    I'm on an rx7600 too. I have done loads of testing and benchmarking and the best model is orinth1.5 35b a3b. I get around 40tok/s generation from it. I run it with llama.cpp with the rocm driver, and use the iq4_xs unsloth quant. It's based on qwen3.6 35b a3b, but trained on complex thinking and took calling. It's miles ahead of anything else I've tried.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] 1 point 2 weeks ago (1 child)

    How big is your system prompt? My 6.5k (which is mostly 5.5k of tool defs) leads to an error message.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 1 child comment
    •
  • [–] 1 point 2 weeks ago

    Not at my pc now, but it's fairly big. I use pi harness, with a searxng instance, plus a custom sandbox and a read only mount point and a full write mount point plus office tools and coding tools. I use a context window of 130k tokens with auto compaction.

  • permalink
  • fedilink
  • source
  • parent