40
Hardware for local inference? (lemmy.ca)
submitted 5 months ago by to c/selfhosted@lemmy.world
25 comments fedilink hide all child comments
 

I want to host some LLM's locally and use more advanced models. Since new hardware is out of the question, I think I should be able to pull something off buying some yesteryear equipment on ebay etc. Did anybody attempt such a project? Does it scale horizontally? (I.e. can I connext two boxes to overcome single box slowness?)

you are viewing a single comment's thread
view the rest of the comments
[–] [S] 5 points 5 months ago

thank you folks. Your input gives me a decent starting point. I'll start digging based on info/experiences shared, maybe I can find someone locally selling old GPU with enough ram for cheap

  • permalink
  • fedilink
  • source
    •