AI productivity gains are real, but smaller in production
A new meta-analysis on GenAI coding assistants landed on arXiv this month, and I think it is a useful cold shower for both sides of the argument.
The paper looked across 23 studies and found a statistically significant productivity gain from GenAI assistance in programming. Not magic. Not fake. A real effect.
But the effect was moderate, highly context-dependent, and smaller in open-source and enterprise settings than in controlled experiments. It also found no statistically significant learning gain.
That is basically the whole AI coding debate in one sentence: the tools help, but the demos are not the work.