A read-only memory check nearly doubled an agent's pass rate

A memory curator that can check facts but can't act on them nearly doubled a coding agent's pass rate, per a study built on a GitHub Copilot SDK setup. The curator ran asynchronously with read-only tools, comparing candidate memories against the current state of the environment before letting anything get saved.

Stale memory is expensive for agents that rely on it to skip redundant work. When nobody checks a saved memory against what's actually true anymore, the agent trusts something wrong and burns extra steps recovering from it. That's the cost this setup targeted directly.

The numbers back it up. Pass rate on CLBench rose to 73% from 39%, according to a new study on memory verification in coding agents. Queries per question fell to 4.7 from 8.8. Task-agent cost dropped to $1.68 from $3.38, roughly half.

The effect held across all 18 comparisons the paper ran, on both Sonnet 4.6 and Opus 4.7. Same direction, every time, regardless of which model did the work.

The curator's job wasn't to generate smarter memories. It rejected the ones that no longer matched reality before they got written down. Read access to the environment, paired with a hard no on write access, was enough to do that.

The paper doesn't say how this holds up outside CLBench, and it doesn't report the curator's false-positive rate. A curator that rejects too aggressively would throw away memories that were still good, which would eat back into the same pass rate it's supposed to protect.