If AI actually adds value, it should be trivial to demonstrate that value-add in a way that passes scientific rigor.

The underlying problem is that we don't have a good way to measure code value. Software quality is most closely coporable to a weird combination of scientific paper, mechanical diagnostic, and toy instructuon. And we don't have good ways to measure those, either.

There was apparently one study from Stanford:

https://medium.com/@manusf08/does-ai-really-boost-developer-productivity-a-stanford-study-of-100-000-devs-has-answers-4f64c64ebe97

Note that the headline is misleading -- stanford apparently trainded an AI model to "rate code" in a way that agreed with some of their staff and then ran that on a bunch of projects. The "good at simple and new, bad at complex and old" matches my intuition, but isn't really a stronger test than counting minutes spent in a project or dollars spent on programming with or without AI.

And all AI output is slop. It's just that for some things slop is good enough.

Which really should be an argument more for discarding those things than boiling oceans to generate more of them, but capitalism loves doing wasteful things