57
Selfhost an LLM (downonthestreet.eu)
submitted 1 year ago by to c/selfhosted@lemmy.world
20 comments fedilink hide all child comments
 

Hi all, i am quite an old fart, so i just recently got excited about self hosting an AI, some LLM...

What i want to do is:

  • chat with it
  • eventually integrate it into other services, where needed

I read about OLLAMA, but it's all unclear to me.

Where do i start, preferably with containers (but "bare metal") is also fine?

(i already have a linux server rig with all the good stuff on it, from immich to forjeio to the arrs and more, reverse proxy, Wireguard and the works, i am looking for input on AI/LLM, what to self host and such, not general selfhosting hints)

you are viewing a single comment's thread
view the rest of the comments
[–] 2 points 1 year ago (3 children)

i had to do a particular command to get the AMD GPU properly available in docker. i can't find that if you need

  • permalink
  • fedilink
  • source
  • hideshow 3 child comments
    •
  • [–] 1 point 1 year ago (2 children)

    Is Radeon V with 8 GB HBM worth using today?

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] 2 points 1 year ago* (1 child)

    not for LLMs. I have a 16GB and even what I can fit in there just isn't really enough to be useful. It can still do things and quickly enough, but I can't fit models that large enough to be useful.

    I also don't know if your GPU is compatible with ROCM or not.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 1 child comment
    •