Length inflation in on-policy distillation traces to teacher and student betting on different EOS tokens

arXiv / HuggingFace Daily Papers·medium signalXBlueskyLinkedInCopy link

arXiv 2609.20511 (submitted 2026-09-17) identifies termination-token mismatch as a major source of runaway response length in on-policy distillation. Across Qwen3, Llama and Gemma, base students and post-trained teachers place stopping probability on different EOS tokens even when their declared stopping sets are identical, so the student's preferred termination is suppressed without the teacher's alternative reliably transferring. Aligning the decoding stopping set alone does not fix it; treating functionally equivalent EOS tokens as one shared semantic stopping action does, across all three families. The authors also show a second, distinct length inflation appearing late in training that survives termination alignment, and release an implementation of the correction.