24
ARC-AGI-3 news: In the past two weeks multiple groups have scored at least 99% on the public challenges at relatively low costs. (arcprize.org)
submitted 3 months ago by to c/technology@lemmy.world
5 comments fedilink hide all child comments
 

The top three solutions come from independent researchers. The best solution was built by a group of PhDs and professors, who released a corresponding paper. They all make use of some form of world-model.

I've generally been a skeptic, and I still am, but this news surprised me because I expected ARC-AGI-3 to remain difficult for a long while.

Note that the scores are self-reported and need to be independently verified. The solutions have not been tested against the larger private test set.

Primer on ARC-AGI-3:

ARC-AGI-3 is an interactive reasoning benchmark which challenges AI agents to explore novel environments, acquire goals on the fly, build adaptable world models, and learn continuously.

A 100% score means AI agents can beat every game as efficiently as humans.

Instead of solving static puzzles, agents must learn from experience inside each environment—perceiving what matters, selecting actions, and adapting their strategy without relying on natural-language instructions.

all 5 comments

sorted by: hot top controversial new old
[–] 4 points 3 months ago (1 child)

If the benchmark is solved, then why official arc prize page on X https://xcancel.com/arcprize does not mention it? And instead they post results of models that are not even close to 100%?

  • permalink
  • fedilink
  • source
  • hideshow 1 child comment
    •
  • [–] [S] 5 points 3 months ago*

    ARC Prize maintains multiple tracks around their benchmarks. They have "verified" leaderboards, "community" leaderboards, and they also run the ARC Prize competition.

    They update the "verified" leaderboards when they test raw LLMs without sophisticated harnesses. They seem to update this sporadically and only occasionally do press releases or blog posts about new scores. For example, the latest score from Claude Opus 5 (High) is 30% at $20,000, and they didn't post about that as far as I know. Again, this just the raw LLM without an agentic or world-model harness.

    The ARC Prize competition has a harder set of criteria. Participants have to use smaller, open models with a limited compute budget, with open source code, and of course the solutions are verified by ARC Prize at the end of the competition.

    The "community" leaderboards, which is what this post is about, are self-reported and not verified by ARC Prize. There are no restrictions on what model is used or limitations on compute. So naturally they aren't going to make official news releases about those, unless they decide to verify them at some point.

    The only reason I chose to post this is that the top solutions seem legitimate, with source code released, and two of them have associated papers.

  • permalink
  • fedilink
  • source
  • parent
    • [–] -4 points 3 months ago (2 children)

    Games are where we play, and train our reasoning/problem solving skills. Games contains temporal/spatial problems, paradoxes, misinformation and so on and so on. IF this is genuine, then these systems DO have the capabilities of navigating almost any problem we can throw at it, and we'll have to begin accepting AGI as being reached.

    That will likely surprise all those that claimed that AI could never be intelligent - or similar claims.

    That said, there are facets of life that are not well projected to games. Social skills are mostly in bigger gameworlds, and kindness/cooperation etc are not prioritized very much. Those types of games can oc be found, but most games are about killing and winning on the dead corpses of the other players, and there are a clear 'edge' of games that tries to expand gaming to include other skill-sets (musik-games, 'girl-games', and many other types).

    Sucky violent gaming culture we have, imho..

    Anyway, even with low social skills and a few skills missing, we are getting really close to AGI, and many will have to adjust their attitude towards the new situation. It'll be interesting..

  • permalink
  • fedilink
  • source
  • hideshow 2 child comments
    •
  • [–] [S] 6 points 3 months ago* (last edited 3 months ago)

    It think it's still unwise to talk about these topics in broad terms like AGI and even "intelligence". We still have to pick the capabilities apart to have useful discussions about them. I agree these games are better tests than many benchmarks, but it's also important to note that these solutions use a combination of well-designed deterministic harnesses, as well as LLMs. So it's inaccurate to say that "LLMs have achieved AGI" (not sure if that's what you were getting at). This feels like an important milestone, but we'll have to continue to probe for failure cases in other categories of problems.

    Aside from emotional intelligence, experience, embodiment, etc., these ARC-AGI-3 solutions all rely on the sandbox being a safe environment to fail. They iterate through the problem thousands of times before coming to a final solution. Many real-world human problems cannot be re-tried safely or efficiently.

  • permalink
  • fedilink
  • source
  • parent