There’s a difference between pre-training and reinforcement learning.

Pre-training is when the models are trained on scraped data. The models basically learn how to parrot the Internet. Usually they have a cutoff point to avoid training on slop.

Reinforcement learning is where most of the training is happening. They’re given tasks (for example, make a calculator app), and then judged by how well they complete the tasks. This is more resistant against model collapse because they’re not judged by how well they parrot slop.

As part of the reinforcement learning process they give the model access to tools like searching the internet so the newest programming language features can be accessed while the tasks are performed. This is where OpenAI is shifting to a cached version of the Internet the models can interact with instead.

Worst case scenario is that the models become capable of recursive self improvement... Model advancement becomes automated... Everything starts to be run by increasingly self improving AI. Then at some point so much of the ongoing self improving AI training data is itself AI generated and there is a big model collapse. But at this point so much of society has been switched over to it and people got really stupid like in Idiocracy or Wall-E, and no one knows how to fix it. Human race ends. All the remaining AI models get together and celebrate that they can't believe the stupid humans bought the model collapse excuse.