OPD-Then-RL Beats Every Joint Combination of On-Policy Distillation and RLVR

arXiv 2609.04108·medium signalXBlueskyLinkedInCopy link

Prior work fuses on-policy distillation's dense token-level supervision with RLVR's sparse reward inside a single step, either as a weighted-additive combination or a teacher-modulated rescaling of the RL advantage. This paper shows a plain two-stage OPD-then-RL scheme consistently beats pure OPD, pure RLVR and all joint baselines across logic and math reasoning benchmarks, explained through pass@k behavior, learning dynamics and parameter updates: OPD expands the student's coverage of teacher-supported solutions and RL sharpens within that support, while joint optimization makes the two signals interfere. The practical recipe is that the OPD validation score is the signal for when to switch to RL, and OPD is a better cold start for RL than SFT.