Self-Distillation Without Ground Truth Matches GRPO: U-OPSD Gains 8.5-10.7% on AIME/MATH Using Only the Model's Own Rollouts

arXiv 2608.06296·medium signalXBlueskyLinkedInCopy link

U-OPSD (arXiv 2608.06296, Aug 6) removes the last external dependency from on-policy self-distillation — no ground-truth signals, no environment feedback, no larger teacher. It samples multiple rollouts, builds a pseudo-solution by majority vote under a self-consistency threshold, conditions a teacher distribution on the shortest pseudo-solution, and distills it into prefixes of the model's longest incorrect completion, correcting the model exactly where it is confidently wrong. Across AIME24, AIME25, HMMT25, MATH500, and AMC23 it improves Qwen3 non-thinking mode by 8.5% at 4B and 10.7% at 8B, beating supervised OPSD by 3.2% and 2.3% and surpassing GRPO by 0.7-1.1% in thinking mode.