You've misunderstood many things in those two sentences.

AI can't write programs, at least not complex programs. The programs / functions it can write well are the ones that are the ones that are very well represented in the training data -- i.e. ultra simple functions or programs that have been written and re-written millions of times. What it can't do is anything truly innovative. In addition, it can't follow directions, it has no understanding of what it's doing, it doesn't understand the problem, it doesn't understand its solution to the problem.

The only thing LLMs are able to do is create a believable simulation of what the solution to the problem might look like. Sometimes, if you're lucky, the simulation is realistic enough that the output actually works as a function or program. But, the more complex the problem, or the more distant from the training data, the less it's able to simulate something realistic.

So, rather than building a ladder where the rungs turn into propellers, it's building one where the higher the ladder gets, the less the rungs actually look like rungs.

the problem is that ai's are trained on programs that humans have written. At best the llm architectures it creates will be similar to the state of the art that humans have created at that point.

however, even more important than the architecture of an ai model is the training data that it is trained on. If we start including ai-generated programs in this data, we will quickly observe model collapse: performance of models tend to get worse as more ai-generated data is included in the training data.

rather than AIs generating ever smarter new AIs, the more likely result is that we can't scrape new quality datasets as they've all been contaminated with llm-generated data that will only reduce model performance