Study Finds CLAUDE.md Files Don't Move Agent Correctness

A study ran 288 gold-evaluated runs across 17 tasks from three repositories and found persistent context files don't move agent correctness, per arXiv 2607.27250.

Teams write and maintain CLAUDE.md and AGENTS.md files on the assumption they raise pass rates. The paper equivalence-tested that effect and bounded it at 10 to 15 percentage points, small enough to sit inside an agent's normal run-to-run noise.

The researchers tested Claude Code and Codex with AGENTS.md or CLAUDE.md present, absent, or varied, holding everything else constant. When agents failed, the paper traces it to wrong design choices, wrong pattern selection, and imprecise code wiring.

An agent didn't fail because it didn't know a repo used FastAPI. It failed because it wired dependency injection wrong.

The paper also explains why earlier studies on this contradicted each other. Task difficulty correlates at Spearman 0.75 between the two agents on borderline tasks. A small study sampling near that boundary could get a noisy result depending on which way it leaned.

A related study, SWE-NFI, reaches the same conclusion from a different angle. Built from 188 tasks drawn from merged Python pull requests, it scored agents against 92 executable rules. The best agent hit 70.0% functional correctness but scored 0.0 to 1.3 on structural improvement against a human reference of 1.5.

Prose describing a repo's conventions is what got tested, and it didn't move the numbers. A worked example, exact imports, exact session handling, exact error type, gives an agent something to copy instead of something to understand.