you are viewing a single comment's thread
view the rest of the comments
[–] 1 point 3 days ago*

Well yes, I'm not saying it's a free lunch. The point is a large up-front investment buys you unlimited tokens in perpetuity for no more than the cost of electricity to power the cluster — which can be surprisingly cheap for UMA machines.

I've swapped off API models entirely and have been able to do a lot more useful work since I don't have to worry about token cost or usage limits. Granted it's slower per task (less tk/s), but overall much more productive, and private.

  • source
  • parent