Fetching from the wire…
01
02
03
04
05
06
07
08
Public story · 2026-07-30 · high
One setup put winner disagreement at 25.6%, the other at 43.6%, across 312 assignments on 13 tabular tasks, per a new paper.
Why now: The paper posted in July 2026 as arXiv 2607.26587, while automated research setups still promote ideas off a single run.
A new paper names a specific failure in automated AI research: the implementation lottery, per arXiv 2607.26587. That's what happens when an automated system builds one implementation of an idea and scores it, crediting the score to the idea itself. Anyone running agent-driven experiment loops that promote or kill ideas on a single score inherits the problem.
The researchers ran 312 assignments across 13 tabular-data tasks using two coding-agent setups. Implementation variance, how much scores swing across different code for the same idea, beat same-artifact rerun variance by more than 5x in one setup. In the other setup, it beat rerun variance by more than 10x.
The ranking damage follows. Pick the winner from one implementation draw and it disagrees with the other two draws' average in 25.6% of decisions in one setup. In the other setup, the disagreement rate hits 43.6%, per the paper.
Any system that crowns an idea after a single coding-agent run is grading the code, not the concept. The paper's own method compares one draw against the mean of the other two, precisely because a single draw swings the verdict by double digits.
Each link below shares sources, entities, or timing with this story.
Shared entity: Direct / Same source domain / Earlier coverage / Downstream implication
Both cover Direct; reported by the same outlet (arxiv.org); earlier Direct coverage from 2026-07-28.
Both cover Direct; reported by the same outlet (arxiv.org); earlier Direct coverage from 2026-03-12.
Shared entity: Direct / Shared topic / Earlier coverage / Tension
Both cover Direct; overlapping topics (decision, direct); earlier Direct coverage from 2026-03-10.
Shared entity: Direct / Shared topic / Earlier coverage
Both cover Direct; overlapping topics (decision, direct); earlier Direct coverage from 2026-06-24.
Shared entity: Direct / Same source domain / Earlier coverage
Both cover Direct; reported by the same outlet (arxiv.org); earlier Direct coverage from 2026-03-10.
Both cover Direct; reported by the same outlet (arxiv.org); earlier Direct coverage from 2026-03-05.
Shared entity: Direct / Earlier coverage / Tension
Both cover Direct; earlier Direct coverage from 2026-07-29; pushes against this story (vs).
Shared entity: Direct / Earlier coverage / Downstream implication
Both cover Direct; earlier Direct coverage from 2026-03-20; traces where this leads (implications).