19
what is your selfhosted local LLM stack? (lemmy.blahaj.zone)
submitted 2 weeks ago by to c/fosai@lemmy.world
8 comments fedilink hide all child comments
 

What tools and projects do you use for your local LLM chats and agents? I'm trying to sift through the mess of LLM harness and tools to find how to really get the most out of local LLMs.

you are viewing a single comment's thread
view the rest of the comments
[–] 1 point 2 weeks ago (1 child)

How big is your system prompt? My 6.5k (which is mostly 5.5k of tool defs) leads to an error message.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 1 child comment
    •
  • [–] 1 point 2 weeks ago

    Not at my pc now, but it's fairly big. I use pi harness, with a searxng instance, plus a custom sandbox and a read only mount point and a full write mount point plus office tools and coding tools. I use a context window of 130k tokens with auto compaction.

  • permalink
  • fedilink
  • source
  • parent