Coding Agents Can Replicate ML Papers — But Not the Same Way Twice

arXiv·medium signal

Hans and Bilionis (arXiv 2607.02134, July 2) introduce Paper-replication, a coding-agent skill that turns each claim in a scientific ML paper into a target with recorded evidence. All twelve workspaces passed the completion gate and all 158 recorded targets were matched with report coverage — the headline result. The buried finding is more interesting: repeated runs differ in how papers are divided into targets, in numerical fidelity, in elapsed time, in how many intermediate executions get replaced, and in the rules used to accept evidence. Passing the gate does not mean the agent did the same work; nondeterminism lives in the evidence-acceptance criteria, which is exactly where you'd least want it.