Replaying identical tokens changed answers in 83% of runs

Researchers got different answers 166 times out of 200 when they replayed identical tokens through a Qwen2.5-derived model, per arXiv paper 2607.28495.

For anyone building prompt caching or session replay, that's the gap between what you saved and what the model does next.

The researchers ran a matched 200-item experiment. One session kept its key/value cache alive, a second rebuilt it by re-feeding identical integer tokens. At 16-bit floating point (BF16), the two paths diverged on 166 of 200 suffixes.

At 32-bit floating point (FP32), the divergence disappears. Zero mismatches, with a 95% Wilson upper bound of 1.88%. The paper doesn't say whether the effect holds outside this Qwen2.5-derived setup or scales to other model families.

To prove the cache itself was steering the output, the researchers ran a bidirectional transplant. They swapped all 48 key/value layers between a live session and its rebuilt twin. Every divergent continuation followed its cache donor, 24 of 24 at the primary checkpoint and 43 of 43 in a second, outcome-blind run.

The fix in this test is precision, not prompt design. Switching cached inference to FP32 removes the divergence, though the paper doesn't say what that costs in memory or latency at scale.