the problem is that ai's are trained on programs that humans have written. At best the llm architectures it creates will be similar to the state of the art that humans have created at that point.

however, even more important than the architecture of an ai model is the training data that it is trained on. If we start including ai-generated programs in this data, we will quickly observe model collapse: performance of models tend to get worse as more ai-generated data is included in the training data.

rather than AIs generating ever smarter new AIs, the more likely result is that we can't scrape new quality datasets as they've all been contaminated with llm-generated data that will only reduce model performance