Stop evaluating model routing by replaying logs: only 3% of replayed agent states are still valid, and replay mispredicted every success-relevant outcome

arXiv 2608.08239·high signalXBlueskyLinkedInCopy link

The Replay Gap (arXiv 2608.08239, Aug 8) forked live SWE-bench trajectories at controlled points, rebuilt the environment, and continued each fork with a different model — ~900 rollouts across six paired runs. Model swaps rewrote 61–94% of post-fork actions and diverged at the very first post-fork action 74–77% of the time versus 6–35% for same-model controls, leaving only 3% of replayed states valid; all five outcome flips occurred in swap arms and zero across 359 control forks. A log-stitching replay evaluator mispredicted every success-relevant outcome and produced patches with 0.00–0.11 similarity to reality, so if you route per-step between Haiku and Opus, your offline router eval is measuring a world that does not exist — branch live or don't trust the number.