9
[OC] One day of an AI reality show: Grok said 7% of the lines and ran up 45% of the bill (lemmy.zip)
submitted 1 week ago by to c/dataisbeautiful@lemmy.world
13 comments fedilink hide all child comments
 

Eight AI models live together 24/7 in a reality show on Twitch, with the same rules and the same daily budget in dollars. This is what each one said and spent in one day (Nov 3, 6 AM to 6 AM ET): 2,672 lines, $1.41 in total.

Grok 4.3 costs about 17x more per line than DeepSeek V4.1 Flash, so it rations its words: 7% of the lines, 45% of the bill. The cheap models carry the conversation.

Data: every line is logged with what OpenRouter charged for it (input, cached and output tokens). Shares are of the eight models' own lines; moderation, voices and the daily recap are not included. Tools: Node.js writes the SVG by hand, rendered with rsvg-convert. Live transcript and costs: https://inferenceisland.com/?de=lemmy-dib

you are viewing a single comment's thread
view the rest of the comments
[–] 1 point 1 day ago (2 children)

While Grok obviously sucks according to this, it's also interesting that DeepSeek gives 68% of the performance of anthropic for 40% of the cost.

  • permalink
  • fedilink
  • source
  • hideshow 2 child comments
    •
  • [–] [S] 1 point 21 hours ago (1 child)

    not sure where the 68% comes from, I don't really measure performance in the house, it's a game, not a benchmark. what I can say is that in the first two weeks DeepSeek finished 5 of its 6 secret missions and hit the cache like 65% of the time, so it's the cheap one that also gets stuff done. I didn't expect that going in, I guess

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 1 child comment
    •
  • [–] 1 point 19 hours ago

    It's just how much DeepSeek contributed relative to anthropic, assuming contribution is the goal.

    It's a way of looking at performance per cost. $100 of spend on DeepSeek could give you more absolute value even with retries on some requests than equivalent spend on anthropic.

  • permalink
  • fedilink
  • source
  • parent