-25
GPT-6 Astra reaches and surpasses human-level performance on ARC-AGI-3 with a memory harness, but still at 1000x the cost - ARC Prize Foundation (arcprize.org)
submitted 2 months ago by to c/technology@lemmy.world
9 comments fedilink hide all child comments
 

Summary:

  • GPT-6 Astra scores 62.7% for $26K on ARC-AGI-3 Semi-Private with our Standard harness, and 99.9% for $19K with a Provider Adapter harness.
  • GPT-6 Astra surpasses the human baseline in action efficiency on ARC-AGI-3. It used fewer actions than the median tested human on 96% of levels.
  • A key behavior observed in GPT-6 Astra was its ability to turn unfamiliar environments into compact symbolic world models. It represented game mechanics as logical rules and developed its own domain-specific language shorthand to track state and plan actions.

For a cost comparison, during our controlled testing, human participants were paid $115 per 90-minute session, plus $5 per game completed. Participants attempted approximately nine games per session, roughly $12.78 per attempted game before bonuses.
Most of this fee pays for the participant’s time and willingness to take the test, rather than the energy their brain uses (a closer proxy to compare with AI). If we look at only the brain’s energy, and price it as electricity, the estimate drops to about 0.6 cents per session, or 0.067 cents per game attempted.1

Astra’s results are also a major milestone worth celebrating. From our perspective, Astra represents a noticeable step-function change in frontier model capabilities.

When we launched ARC-AGI-3, we made it clear that saturating the benchmark would not represent “proof of achieving AGI.” Therefore, while we believe Astra represents meaningful progress towards generalization, we are not claiming that it is AGI.

all 9 comments

sorted by: hot top controversial new old
[–] [S] 16 points 2 months ago (3 children)

I guess this is being down-voted just because it feels pro-AI. Believe me I get it, you can look at my anti-AI post history. But I think it's important to also keep an eye out for real signals towards general intelligence, which is why I wanted to share this. I'm not celebrating it, just bringing awareness.

  • permalink
  • fedilink
  • source
  • hideshow 3 child comments
    •
  • [–] 10 points 2 months ago (2 children)

    For as long as it is an LLM I will not call it intelligence for it does not think. It's an approximation or an estimate at best. While capable this is a major hurdle these companies will face to replace skilled workers. We are a generation at least and a ways off from general intelligence, and not algorithmic output for a given input (idempotency).

    I do however agree that we would remain informed about the ever changing world around us; this however, is until reviewed by an unbiased third party in masse a marketing document. I will be taking it with a grain of salt, but will take it as an improvement over the prior versions.

    Now with that out of the way... interesting. I will be watching this progress, but am far less optimistic in this being as capable as they are suggesting. It's more than likely another Mythos hype attempt that is not revolutionary, but a nice addition to existing systems or augments to staff. Now if we assume its 100% as capable as this suggests then we are in for a ride.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] [S] 4 points 2 months ago

    I agree with most of this. I've also said in the past that LLMs cannot think, and I think that's still true for most models. The reason ARC-AGI-3 is interesting is that it was specifically designed to test reasoning, adaptability, novel problem solving, planning, memory, etc. So it was a surprise to me that Astra was able to defeat it so effectively, and that Astra invents algebras for each novel task.

    But I agree we can't trust OpenAI if these results are self-reported, and we may not be able to trust the ARC Prize Foundation fully either. Extraordinary claims require extraordinary evidence, so we need replication, transparency, and proper open science to confirm things.

    I also agree with ARC Prize's conclusion, that there are still capabilities any AI system would need to demonstrate before we can claim a full general intelligence.

  • permalink
  • fedilink
  • source
  • parent
  • [–] 1 point 2 months ago

    Its all semantics. Im curious, can you define what it means to think? Maybe the term is broadening now that we found some kind of similar process that looks like what we imagine thought to be. AI is bad and evil yadda yadda yadda im not defending it i just really like this philosophically or whatever and itsd be cool to hear why you differentiate "thinking" from human thinking

  • permalink
  • fedilink
  • source
  • parent
    • [–] 6 points 2 months ago (1 child)

    Wow they benchmaxxed this pretty quickly. Wasn't this benchmark released only a few months ago with like 0% success rate from all major AI models?

  • permalink
  • fedilink
  • source
  • hideshow 1 child comment
    •
  • [–] [S] 4 points 2 months ago

    Yes, it was released in March. One important point is that these results are based on the semi-private test set. Perhaps it's best to wait until it is measured against the fully private test set, but that might not happen for months.

  • permalink
  • fedilink
  • source
  • parent
    • [–] 3 points 2 months ago* (2 children)

    OP If you don't wanna be downvoted so much you gotta use a more negative framing like this OpenAI announces new slop generator GPT-6 Astra: Unfortunately not any better than the last one

  • permalink
  • fedilink
  • source
  • hideshow 2 child comments
    •
  • [–] [S] 3 points 2 months ago (1 child)

    I'm okay with the downvotes. There's a stronger anti-AI sentiment now than even just a month ago, and on the whole I think that's going to be a good thing.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 1 child comment
    •
  • [–] 4 points 2 months ago

    Same, im very interested in the technology, ive followed Machine Learning stuff since before most of the kids using ai to cheat in school were born. But the way things are going, the corpos in control of the cutting edge are making such abhorrent and misanthropic decisions that anything but pure hatred for the use of their tools is unacceptable. I struggle to justify using even local models, i just feel icky now.

  • permalink
  • fedilink
  • source
  • parent
    • 6 comments in c/futurology@futurology.today view that post
    [–] [S] 2 points 2 months ago (4 children)

    I guess this is being down-voted just because it feels pro-AI. Believe me I get it, you can look at my anti-AI post history. But I think it's important to also keep an eye out for real signals towards general intelligence, which is why I wanted to share this. I'm not celebrating it, just bringing awareness.

  • permalink
  • fedilink
  • source
  • hideshow 4 child comments
    •
  • [–] 2 points 2 months ago (3 children)

    I mean, I downvoted because these benchmarks are gamed and only serve as promotion material. Unless there are any other content of high-significance here, I don't believe this post should rank higher on any feed.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 3 child comments
    •
  • [–] [S] 2 points 2 months ago (2 children)

    Yes, I'm aware many benchmarks are easily gamed and rely mostly on knowledge acquisition rather than true reasoning. I wouldn't post other benchmark results either.

    However, ARC-AGI-3 is unique in that it tests systems using dynamic, original game environments where the AI has to learn each game from scratch. Importantly, the score is based on action efficiency, not simply beating each game.

    So this benchmark is a bigger signal than most. It should still be judged with skepticism, certainly and doesn't necessarily indicate a fully general intelligence, but it seems like a step-change to me.

    I'd encourage you to read more about the benchmark, including the technical paper, if you're interested.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] 2 points 2 months ago* (1 child)

    I know about each benchmark, but there are infinite ways to game these benchmarks, even ARC.

    How many attempts? How is the final score picked, best, average, worse?

    Are tools allowed? Is the model cheating with a tool handcrafted for solving ARC AGI?

    Also noticing the score is on the "semi-private" benchmark, if thats even a thing.

    The private one has a slower score.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 1 child comment
    •
  • [–] [S] 2 points 2 months ago

    All fair questions. I can imagine OpenAI would cheat in any way they can, but I suppose I'm putting some trust in ARC Prize, or specifically Francois Chollet. I'd be happy to be wrong about this because the implications of it being real are worrying (and I try not to add to the hype/doom). Time will tell I suppose, but I still think it's worth sharing compared to other benchmarks.

  • permalink
  • fedilink
  • source
  • parent
    • [–] 1 point 2 months ago

    They game benchmarks on these because they are desperate to keep funding going. They are so far from profitable

  • permalink
  • fedilink
  • source
    •