One correction term fixes a drift bug destabilizing LLM RL

Drift between training and inference engines destabilizes reinforcement learning training runs for large language models, per a study posted to arXiv on September 20.

The distinction carries stakes for anyone running RL training at scale. Noise averages out over time. Drift compounds every step instead. The authors tested the fix on models ranging from 0.6B to 30B parameters.

Running inference at training-level precision would close the gap, but the arXiv paper (2609.20807) says that costs too much rollout throughput to be practical. Teams quantize inference to keep rollouts cheap, and that quantization is what introduces the drift.

Their fix is an additive score-centering correction that cancels the drift directly. Under quantization, it matches or beats importance sampling, the standard correction method, and the gap between them widens as the mismatch grows.

Because the correction is additive, it stacks with importance sampling rather than replacing it. That combination beat pure importance-sampling baselines under staleness, meaning when the rollout policy used to generate training data lags behind the policy being trained.

How the correction performs past 30B parameters, or under mismatch worse than what was tested, isn't in the study. Frontier-scale RL training is where that gets tested next.