you are viewing a single comment's thread
view the rest of the comments
[–] 1 point 14 hours ago (1 child)

Would need a very beefy cluster, and that definitely isn’t free.

  • source
  • parent
  • hideshow 1 child comment
  • [–] 1 point 14 hours ago*

    Well yes, I'm not saying it's a free lunch. The point is a large up-front investment buys you unlimited tokens in perpetuity for no more than the cost of electricity to power the cluster — which can be surprisingly cheap for UMA machines.

    I've swapped off API models entirely and have been able to do a lot more useful work since I don't have to worry about token cost or usage limits. Granted it's slower per task (less tk/s), but overall much more productive, and private.

  • source
  • parent