One Additive Correction Term Fixes the Training-Inference Mismatch That Destabilizes LLM RL
arXiv 2609.20807 (submitted 17 Sep 2026) attributes RL instability under training-inference mismatch to drift, a persistent bias between the training and inference engines that accumulates every training step, rather than to per-step noise. The authors derive an additive score centering correction that cancels this drift, and when training models from 0.6B to 30B parameters it matches or beats importance-sampling methods under quantization, with the gap widening as the mismatch grows. Because the correction is additive it composes with importance sampling, and the combination outperforms pure importance-sampling baselines in staleness experiments. Practical because eliminating the mismatch outright costs too much rollout throughput to be worth doing.
↳ Follow the thread