441
My home server was a totally justified purchase (lemmy.world)
submitted 2 months ago by to c/programmer_humor@programming.dev
38 comments fedilink hide all child comments
šŸ¤£ 4
 

cross-posted from: https://lemmy.world/post/50893918

you are viewing a single comment's thread
view the rest of the comments
[ā€“] 1 point 2 months ago (3 children)

Yeah I can run any open weight models which leave me enough vram to not crash. But its a bit of a gotcha because you also need enough system ram to load the model. I use it to heavily parallelize training tasks.. Honestly, I need to tinker with it more but I'm pretty annoyed at how ollama has gone deep in the paint as basically being a tool for accessing cloud models.

Someday TM

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 3 child comments
    •
  • [ā€“] 1 point 2 months ago (1 child)

    Why not llama.cpp? Ollama is just a wrapper over it.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 1 child comment
    •
  • [ā€“] 1 point 2 months ago

    Mostly time. I didn't buy the machine to run someone else's models. I bought it to run my own models and the machine has a job to do. Tinkering with self hosting llms, I guess. I appreciate them as models. But my machine has a day job.

  • permalink
  • fedilink
  • source
  • parent
  • [ā€“] 1 point 2 months ago

    Iā€™m pretty annoyed at how ollama has gone deep in the paint as basically being a tool for accessing cloud models.

    Can you expound a little more on what you mean by this?

    I like ollama, but I only really use it to load models and then hit the API. My current issue with them is that they don't seem to support non-text base interaction very well.

  • permalink
  • fedilink
  • source
  • parent