146
What's up, selfhosters? It's self hosting Sunday! (lemmy.nocturnal.garden)
submitted 1 year ago by to c/selfhosted@lemmy.world
72 comments fedilink hide all child comments
 

What’s up, what’s down and what are you not sure about?

Let us know what you set up lately, what kind of problems you currently think about or are running into, what new device you added to your homelab or what interesting service or article you found.

Personally I'm finally reaping the fruits of my labour and enjoy my stable homelab without doing much. One node went down recently and the other took over until I restarted so I was not in a hurry to fix things. Enjoying family time and only running updates that aren't automated (yet). I'm about to dig a bit deeper into logging, probably setting up central log collection like Loki at some point, but not yet.

you are viewing a single comment's thread
view the rest of the comments
[–] 2 points 1 year ago (6 children)

Might want a bigger GPU, I have a 3080ti and the 12gb is pretty limiting in terms of how large a model you can use, or like one thing I was hoping to do was essentially replace Google Assistant/Gemini and can't realistically run a good model and the STT/TTS off the one gpu.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 6 child comments
    •
  • [–] 2 points 1 year ago (5 children)

    Thats why i was considering training my own model if possible. Ive been toying around with kobold.CPP and gpt4all which both have RAG implementations.

    My idea is to essentially chat with documentation and as a separate use case, have it potentially be a AI search engine but locally hosted. I do still prefer to search myself, but fuck man, searches have gotten so bad, and the kobold.CPP web lookup feature was pretty neat IMO.

    So yea you're not wrong, I'm just hoping that if in train it and or give it documentation it can reference when answering, it will be suitable. Mostly AI has been good for me as kind of a rubber ducky when troubleshooting and helping me search for things when I have some specific question and in don't want "top 5 things vaguely related to your question" results.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 5 child comments
    •
  • [–] 2 points 1 year ago* (4 children)

    Interesting, I mainly have used text generation webui which has a search support plug in, kinda nifty to use my searxng instance for it. It's a bit finicky though.

    Another thing to keep in mind then (apologies if this is just repeating info you already know), you'd also want to keep in mind your total potential context size in relation to the model size, since both take up VRAM. Reading search results/pages can eat up a lot

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 4 child comments
    •
  • [–] 2 points 1 year ago* (last edited 1 year ago) (3 children)

    Yea I'm aware but I appreciate the insight :) so far my local ai experience has been lack luster so I'm hoping that training and RAG will make up for the context size at least a little. Ifnit can answer accurately in the first place, it may not need as big of a context window.

    If you haven't tried using RAG in some form, I would recommend giving it a go. Its pretty cool stuff, helps make models answer more accurately based on the documentation you give them though in my case, ive had limited success. Tbh, chatgpt has become my last resort when I just wanna get something done but I don't like using it due to the privacy concerns, not to mention the ethical issues I have with ai training in general from big tech.

    How is searxng BTW? Would you say its good to host or do you use a normal search engine more often? Or do you just use it for the AI search plugin?

    Ive actually been thinking about using it rather than duckduckgo but was also hopeful the search index they are working on would be enough to satisfy my needs, or that a self hosted AI enabled search engine would work well enough when I need it.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 3 child comments
    •
  • [–] 2 points 1 year ago (2 children)

    I've completely replaced my searching with searxng, it is a little slower and ofc if I have an outage or something at home I have to go back to a different search temporarily but overall I like it a lot.

    It was one of the first things I set up last year with my homelab because I am attempting to degoogle a fair amount, the Ai search stuff was just a fun test

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •