Neither AlphaGo, not AlphaFold are LLMs though.

Also the mathematical proofs you're talking about seem to be mostly stolen by OpenAI from scientists that interacted with their chatbot before publishing their research.

Computers have "come up with" solutions to problems that eluded human-only problem-solving for decades. That's literally what computers are supposed to be for. That proves nothing about the novelty of AI beyond a narrow range of incremental problem-based solving which, again, we've used computers for for decades.

So you don’t understand how the technology you’re defending functions then?

Yeah this does make me wonder: even now, models need to be constantly trained with new programming languages, libraries,etc. They can ingest new documentation during a session, but they often refuse to believe it. Ex: "the project uses .Net 10, but thats clearly a typo- the current version of .Net is 8". So then if the bubble collapses and training budget drops, the models capability would suffer drastically

Not the author of the blog post, but let me adress that anyways.

I agree that for some purposes, what generative "AI" produces is functionally the same as a novelty. And tautologically, even random output of a machine can contain data that humans may interpret as a "new thought" or "new knowledge" of some kind. But "Knowledge" is an intrinsically social construct and no technology can enhance it on its own, in the greater scheme of things.

Re the first point, speaking as a mathematician: The proofs generated by "AI" thus far are mostly hardly comprehensible and on their own don't actually enhance our knowledge in a meaningful way. We first have to dig through the mud. Only because 15 million dollars or whatever worth of tokens could synthesise a smooth blow up of Navier--Stokes doesn't mean we've actually learned something new and as fas as I've been told (or as far as people have made sense of the mechanics) there were no genuinly new ideas involved.

Re AlphaGo: Not in the scope of "AI" as defined in the article. Given a strict ruleset and given enough computational power, sure you can train a neural network to develop new strategies by just pure chance. And it will "learn" what a good strategy is by objective criteria like winning in simulated games etc.

Human creativity is not devalued by saying that "LLMs are simply not capable of producing new knowledge" because knowledge exists only in our minds.