Much of AI is a scam. Probably the large majority.

I'm not sure you've quite captured how overfitting functions in current language models. Nor how baselines and benchmarks are used for machine learning products. For serious models of the current year, overfitting is something that happens when you train too long on the same data. It happens less if you add new data, and has nothing to do with new benchmarks and evaluations. You can read about overfitting more generally (mostly the classical kind) on wikipedia.

I think Bruce Schneier has a reasonable understanding of how benchmarks, public policy, large language models, and overfitting are at work here.