We passed the period of exponential LLM growth as of mid-2025 and now the gains are just incremental, which has given open models a chance to rapidly close the gap. Now the narrative is that the “dangerous” open LLMs need to get kneecapped while the “safer” closed LLMs continue with a healthy lead.

@return2ozma it would be so nice if they just advertised these things as tools, not wolrd ending devices...

Article is paywalled (account walled?) so I can’t read the whole thing because I’m not making an account to read one article.

But I think it’s reckless to dismiss all “AI doomsaying” as advertising / hype for AI companies. SOME of it is absolutely that. Some of it is real concern from experts that should not be ignored.

It's true that tech companies have an incentive to lean into a narrative about the risk of a rogue superintelligence and publish misleading exaggerations about how imminent that risk is, but that isn't the same thing as having a strong case that no such risk exists. The article's case for this seems pretty strange to me:

Obviously, we must plaster a huge advisory label on talk of “goals.” Unlike an organism, a large language model (LLM) doesn’t have internal goals any more than a thermostat or a motion detector does. An LLM is just a fixed function that iterates whenever it is used. It has no interiority, no subjectivity, no active persistence over time; any goals it did have would be the human purposes reflected passively in its functional design. At best, ascribing goals to the AI itself is a case of researchers adopting the “intentional stance” because it predicts well in a range of tested circumstances. We can thus, with economy, speak of the “loss function” used in the training of large language models wherein they are programmed to bring the gap between their prediction and the correct answer as close to zero as possible, as a “goal.”

In that case, however, what we are describing as a goal is just an observed regular output of a certain engineering; it is not a real cause. In the Bostrom sense, however, these ascribed goals somehow become real causes, capable of generating new goals.

For the purposes of whether something is apocalyptically dangerous, what it is actually able to do seems like it should matter a lot more than whether the same words used for humans can apply philosophically.

Since large language models can’t “hack” anything, merely plausibly continue a tokenized sequence, they were combined with a coding harness — a software infrastructure layer that allows a large language model to act as an autonomous coding agent — that was specifically optimized for completing cyberexploitation tasks.

A LLM with a coding harness can write and modify a coding harness. It's fair to point to the huggingface hack example as an example of OpenAI giving an exaggerated impression of how dangerous and capable their product is, but you can't pretend that the capabilities of a LLM based agent as a whole system are not the real goalposts here. A LLM plus its harness being able to produce another LLM plus its harness that will do very dangerous and consequential things, that's what such a risk could look like. The question is how good it is at building on itself without human assistance, and whether there are fundamental limitations that may prevent it from ever getting much better at it. Whether such limitations exist and what they are is not a settled issue, and so civilization level risk can't rationally be taken out of consideration.