I'm new to local llms, i'm running Debian on this machine. Where should I start and what model(s) could I get testing on it?

you are viewing a single comment's thread
view the rest of the comments
[–] 5 points 1 week ago* (2 children)

Friends don't let friends use Ollama

If you want a gentle start, may I suggest Jan.ai? It's a ride on lawn mower compared to the sports car I suggested...but you need to dip your toe into the water before you go ludicrous speed.

Once you get a feel for that, I'm going to suggest some MCP tools to you as well. I don't know if Jan finally added MCP servers to their stack (they should have by now!).

The following are ... ok. Not the best and not what I run with but again, lawn mower before sports car

https://github.com/ofershap/mcp-server-markdown

https://github.com/jeffreyrampineda/kiwix-wiki-mcp-server

https://github.com/modelcontextprotocol/servers/tree/main/src/filesystem

https://github.com/exa-labs/exa-mcp-server

https://exa.ai/

https://www.home-assistant.io/integrations/mcp_server/

Sorry - again: too much info. Start with Jan. You'll be amazed...and then perplexed...and then frustrated. That's the time to migrate thru to llama.cpp or PolyStrata and start adding those MCP tools I mentioned above.

https://github.com/VecSzn/PolyStrata

https://github.com/srware-net/LocalLightChat/

https://github.com/open-webui/open-webui

Then we can get freaky :) Oh...we have such sights to show you lol

  • source
  • parent
  • hideshow 2 child comments