Combinatorial-optimization paper audits its own reported gains

Researchers checked their own test-time budget allocation method for neural combinatorial optimization, and most of the reported gain turned out to be sampling luck. That distinction matters because in-sample evaluation is a common shortcut in this field, and this setup shows it can manufacture a confident result from nothing.

On in-distribution TSP-100 problems, an oracle allocation was computed and scored on the same stored samples. That produced a 2.2 to 2.6% gain across three solvers, POMO, AM, and SymNCO, with confidence intervals excluding zero, per the paper.

Measured on samples the allocation method never touched, the gain disappeared: 0.457%, 0.015%, and -0.512% across the same three solvers. More samples and more test instances didn't shrink the bias. It just sat there.

Not every result collapsed. Under distribution shift, the correction held up: a real 11.5% and 12.0% improvement, confirmed in a pre-registered experiment run separately from the exploratory work.

The paper doesn't say how common this in-sample leak is across other published results. It only shows the leak fully explained the gain in this paper's own in-distribution setup. Data, code, and the pre-registration record are all released, so the audit is checkable rather than asserted.