Fetching from the wire…
01
02
03
04
05
06
07
08
Public story · 2026-07-10 · high
The twelve runs disagreed most on what counted as valid evidence, the step you'd most want fixed.
Why now: The 2026-07-10 coverage flags it because a 158-for-158 completion score is exactly the number that gets repeated without the caveat buried in the same paper.
A new skill turns each claim in an ML paper into a target agents must reproduce with logged evidence, per an arXiv paper.
All twelve agent workspaces cleared the completion gate, matching every one of 158 targets with full report coverage. That's the number people will repeat. It's also hiding how differently each run got there.
The paper found the runs diverged on the details that decide whether a replication counts. How they split each paper into targets. How closely their numbers matched the originals.
They diverged too in elapsed time and in how many intermediate executions got swapped out mid-run. Most of all, they diverged in the rules used to decide what evidence was good enough to accept.
That last point is the one that matters. A completion gate is only as trustworthy as the standard behind it. If two runs both hit 100 percent using different evidence-acceptance rules, the gate is measuring self-consistency, not replication.
Each link below shares sources, entities, or timing with this story.
Shared entity: Coding / Same source domain / Shared topic / Earlier coverage
Both cover Coding; reported by the same outlet (arxiv.org); overlapping topics (agent, claim, coding).
Shared entity: Coding / Shared topic / Earlier coverage
Both cover Coding; overlapping topics (agent, coding, doesn); earlier Coding coverage from 2026-05-09.
Same source domain / Shared topic / Tension
Reported by the same outlet (arxiv.org); overlapping topics (agent, coding, doesn); pushes against this story (against).