810
ChatGPT 'got absolutely wrecked' by Atari 2600 in beginner's chess match — OpenAI's newest model bamboozled by 1970s logic (www.tomshardware.com)
submitted 1 year ago by to c/technology@lemmy.world
200 comments fedilink hide all child comments
you are viewing a single comment's thread
view the rest of the comments
[–] 6 points 1 year ago* (3 children)

In this case it's not even bad prompts, it's a problem domain ChatGPT wasn't designed to be good at. It's like saying modern medicine is clearly bullshit because a doctor loses a basketball game.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 3 child comments
    •
  • [–] 2 points 1 year ago (2 children)

    I imagine the "author" did something like, "Search http://google.scholar.com/ find a publication where AI failed at something and write a paragraph about it."

    It's not even as bad as the article claims.

    Atari isn't great at chess. https://chess.stackexchange.com/questions/24952/how-strong-is-each-level-of-atari-2600s-video-chess
    Random LLMs were nearly as good 2 years ago. https://lmsys.org/blog/2023-05-03-arena/
    LLMs that are actually trained for chess have done much better. https://arxiv.org/abs/2501.17186

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] 1 point 1 year ago (1 child)

    Wouldn't surprise me if an LLM trained on records of chess moves made good chess moves. I just wouldn't expect the deployed version of ChatGPT to generate coherent chess moves based on the general text it's been trained on.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 1 child comment
    •
  • [–] 2 points 1 year ago

    I wouldn't either but that's exactly what lmsys.org found.

    That blog post had ratings between 858 and 1169. Those are slightly higher than the average rating of human users on popular chess sites. Their latest leaderboard shows them doing even better.

    https://lmarena.ai/leaderboard has one of the Gemini models with a rating of 1470. That's pretty good.

  • permalink
  • fedilink
  • source
  • parent