the benchmark is like a tycoon game/simulation, pretty fun and probably not benchmaxxed

you are viewing a single comment's thread
view the rest of the comments
[–] 1 point 1 week ago

It's a good benchmark. 3.1 is worse than 2.5 on my language test. 3 got it right, but it might have been lucky. web agent is cool. It downloaded programming language and tried running test, but bad initial knowledge doomed it to failure as it blew through all token allowance on a 5 word prompt.

  • source