Dense Supervision, Sparse Updates: the sparsity and geometry of on-policy distillation

arXiv·medium signal

This paper analyzes on-policy distillation (OPD) — a post-training recipe combining on-policy student trajectories with dense teacher supervision — and shows that the resulting updates are geometrically sparse. The insight has practical implications for making distillation cheaper and more stable when post-training smaller models from a stronger teacher.