57
Selfhost an LLM (downonthestreet.eu)
submitted 1 year ago by to c/selfhosted@lemmy.world
20 comments fedilink hide all child comments
 

Hi all, i am quite an old fart, so i just recently got excited about self hosting an AI, some LLM...

What i want to do is:

  • chat with it
  • eventually integrate it into other services, where needed

I read about OLLAMA, but it's all unclear to me.

Where do i start, preferably with containers (but "bare metal") is also fine?

(i already have a linux server rig with all the good stuff on it, from immich to forjeio to the arrs and more, reverse proxy, Wireguard and the works, i am looking for input on AI/LLM, what to self host and such, not general selfhosting hints)

you are viewing a single comment's thread
view the rest of the comments
[–] 8 points 1 year ago*

I read about OLLAMA, but it’s all unclear to me.

There's really nothing more to it than the initial instructions tell you. Literally just a "curl -fsSL https://ollama.com/install.sh | sh". Then you're just a "ollama run qwen3:14b" away from having a chat with the model in your terminal.
That's the "chat with it"-part done.

After that you can make it more involved by serving the model via API, manually adding .gguf quantizations (usually smaller or special-purpose modified bootleg versions of big published models) to your Ollama library with a modelcard, ditching Ollama altogether for a different environment or, the big upgrade, giving your chats a shiny frontend in the form of Open-WebUI.

  • permalink
  • fedilink
  • source
    •