you are viewing a single comment's thread
view the rest of the comments
[–] 3 points 3 months ago*

If you're running stuff overnight on a homelab, you can get away with using very large models as agents in CPU RAM.

Another option is a hybrid approach: farm "dumb" privacy-insensitive calls to an API you can subscribe to for like $40 a year, and let your local LLM do the orchestrating. This is what I do: I run MiMo locally, but also farm calls out to the exact same LLM in the cloud when appropriate... And I don't mind one bit. I'm happy to tip them for open sourcing it in the first place.

  • source
  • parent