Anthropic Internal Study: AI Coding Tools Cut Comprehension 17% with No Statistically Significant Speed Gain
Anthropic Research·high signal
Anthropic published a randomized controlled trial of 52 junior engineers learning a new Python library (Trio): the AI-assisted group scored 50% on comprehension tests versus 67% for the manual group, a 17-point gap most pronounced in debugging. The AI group finished ~2 minutes faster but the difference was not statistically significant. Critical nuance: developers who used AI for conceptual inquiry scored 65%+, while those who delegated code generation scored below 40% — the how-you-use-it split matters more than whether you use it.