I'm new to local llms, i'm running Debian on this machine. Where should I start and what model(s) could I get testing on it?

you are viewing a single comment's thread
view the rest of the comments
[–] [S] 2 points 1 week ago* (3 children)

Thanks for the very rich reply. I'm a bit reassured that I can do some basic stuff, I'm not a poweruser anyway, if this word is still used. The aim is to not rely on external tools and actors as much as possible for private stuff. My work does give me access to a commercial gpt for work, but I don't want to use it for my private life matters. The learning curve seems a bit higher that I thought, but I saw some tools that could maybe help me there, like ollama or localai.io, any thoughts on these projects?

  • source
  • parent
  • hideshow 3 child comments
  • [–] 5 points 1 week ago* (2 children)

    Friends don't let friends use Ollama

    If you want a gentle start, may I suggest Jan.ai? It's a ride on lawn mower compared to the sports car I suggested...but you need to dip your toe into the water before you go ludicrous speed.

    Once you get a feel for that, I'm going to suggest some MCP tools to you as well. I don't know if Jan finally added MCP servers to their stack (they should have by now!).

    The following are ... ok. Not the best and not what I run with but again, lawn mower before sports car

    https://github.com/ofershap/mcp-server-markdown

    https://github.com/jeffreyrampineda/kiwix-wiki-mcp-server

    https://github.com/modelcontextprotocol/servers/tree/main/src/filesystem

    https://github.com/exa-labs/exa-mcp-server

    https://exa.ai/

    https://www.home-assistant.io/integrations/mcp_server/

    Sorry - again: too much info. Start with Jan. You'll be amazed...and then perplexed...and then frustrated. That's the time to migrate thru to llama.cpp or PolyStrata and start adding those MCP tools I mentioned above.

    https://github.com/VecSzn/PolyStrata

    https://github.com/srware-net/LocalLightChat/

    https://github.com/open-webui/open-webui

    Then we can get freaky :) Oh...we have such sights to show you lol

  • source
  • parent
  • hideshow 2 child comments