One Coding Run Flips Up to 43.6% of AI Research Verdicts

A new paper names a specific failure in automated AI research: the implementation lottery, per arXiv 2607.26587. That's what happens when an automated system builds one implementation of an idea and scores it, crediting the score to the idea itself. Anyone running agent-driven experiment loops that promote or kill ideas on a single score inherits the problem.

The researchers ran 312 assignments across 13 tabular-data tasks using two coding-agent setups. Implementation variance, how much scores swing across different code for the same idea, beat same-artifact rerun variance by more than 5x in one setup. In the other setup, it beat rerun variance by more than 10x.

The ranking damage follows. Pick the winner from one implementation draw and it disagrees with the other two draws' average in 25.6% of decisions in one setup. In the other setup, the disagreement rate hits 43.6%, per the paper.

Any system that crowns an idea after a single coding-agent run is grading the code, not the concept. The paper's own method compares one draw against the mean of the other two, precisely because a single draw swings the verdict by double digits.