Rethinking On-Policy Distillation: First Systematic Study of OPD Training Dynamics and Failure Modes

arXiv·medium signal

Li et al. provide the first systematic investigation of on-policy distillation dynamics in LLM post-training. They identify previously unknown failure modes in OPD training and propose a recipe that addresses them. As OPD becomes a core technique for reasoning model training (used in DeepSeek-R1, QwQ), understanding its dynamics is essential for practitioners building custom post-training pipelines.