AI Agents Failed to Reproduce Two Unpublished Papers

Original authors rejected AI attempts to reproduce two unpublished papers, per a July 29 arXiv study from Peter Kirgis, Sayash Kapoor, and Andrew Schwartz.

The study asks whether agents can run research instead of only executing it. Two tries, six days and thousands of dollars each, zero passes.

The paper introduces shadow evaluations. An agent attacks the central research question of a high-quality unpublished paper, and the paper's own authors grade what comes back.

Both test cases here were NeurIPS 2026 submissions. The authors called both attempts "unambiguously rejected."

The failures weren't about running out of time or money. Kirgis, Kapoor, and Schwartz name five recurring problems instead.

Agents misjudged what counts as a publishable result. They recovered from a stalled experiment design without creativity, and struggled to escape dead ends. Resource management was poor, and agents drifted from the original research instructions as runs continued.

The authors put it plainly: agents "can do the engineering of AI research but not the research."

Running experiments and keeping a system alive through a stall is engineering. Deciding whether a result deserves publication is judgment. Twice here, that judgment didn't show up.