Fetching from the wire…
01
02
03
04
05
06
07
08
Public story · 2026-08-24 · high
A crossover study of 34 reviewers found AI assistance hurt requirements-smell detection with no time savings to show for it.
Why now: It's the first controlled measurement of this specific cost I've come across.
Researchers put 34 participants through a spec review task twice, once with LLM support and once without, so each person served as their own control. The task was catching requirements smells, vague or conflicting language buried in a spec that costs teams real time to unwind later. Get that step wrong and every downstream design and test built on the spec inherits the ambiguity.
A Bayesian regression fit separately for each outcome found LLM assistance lowered smell detection accuracy in the crossover study. Classification accuracy and task duration showed no significant effect either way.
That's the opposite of the pitch. Most AI-assisted review tools sell on speed, catch more, catch it faster, free up reviewers for harder judgment calls. This study measured one narrow slice of that claim, smell detection in written specs. The speed gain didn't show up, and accuracy went the wrong direction.
I've treated an LLM pass over code or doc review as a free extra set of eyes, on the assumption it can only add coverage. This study puts a number on that assumption, and it's not the number the pitch expects. The paper doesn't say why accuracy dropped. It doesn't distinguish reviewers deferring to the model's suggestions from reviewers just getting lulled by having a second opinion in the loop.
Each link below shares sources, entities, or timing with this story.
Simon Willison released LLM / Shared entity: LLM / Earlier coverage / Tension
Linked by a graph relationship (Simon Willison released LLM); both cover LLM; earlier LLM coverage from 2026-08-16.
Linked by a graph relationship (Simon Willison released LLM); both cover LLM; earlier LLM coverage from 2026-07-27.
Linked by a graph relationship (Simon Willison released LLM); both cover LLM; earlier LLM coverage from 2026-06-19.
Linked by a graph relationship (Simon Willison released LLM); both cover LLM; earlier LLM coverage from 2026-06-18.
Simon Willison released LLM / Shared entity: LLM / Earlier coverage
Linked by a graph relationship (Simon Willison released LLM); both cover LLM; earlier LLM coverage from 2026-08-21.
Linked by a graph relationship (Simon Willison released LLM); both cover LLM; earlier LLM coverage from 2026-08-17.
Linked by a graph relationship (Simon Willison released LLM); both cover LLM; earlier LLM coverage from 2026-08-12.
Linked by a graph relationship (Simon Willison released LLM); both cover LLM; earlier LLM coverage from 2026-08-07.