SimpleOPD distills across tokenizers, lifts Intern-S2 21.2 points

SimpleOPD distills language models across tokenizer boundaries by working in shared text space, lifting Intern-S2-Preview 21.2 points on ProofBench, per a new arXiv paper.

Teacher and student models rarely share a tokenizer. That gap has forced teams to pick a compatible but weaker teacher instead of whichever model scores highest.

The method fixes this by distilling in shared text space instead of matching token IDs. It adds a student reference KL loss and masks the advantages of special termination tokens, which stops the student's generations from running long.

Intern-S2-Preview reached 55.2 on ProofBench with the technique, past Gemini-2.5-Pro's score. The paper reports the same gains across Qwen3, Qwen3.5, Intern-S2, GLM-4.7 and Gemma-4, and the improvement generalizes to two other benchmarks, HLE and HiPhO.

The paper doesn't report what the extra KL loss costs in training time or compute. For a technique meant to widen teacher choice, that's the number builders adopting it will want next.

The tokenizer match that used to gate which teacher a team picked no longer applies. Benchmark score becomes the only reason left to pick one model over another for distillation.