TIDE: First Cross-Architecture Distillation Framework for Diffusion Language Models — 8B Teacher to 0.6B Student

arXiv·low signal

TIDE introduces the first framework for distilling knowledge across diffusion LLM architectures with different attention mechanisms and tokenizers, comprising TIDAL (adaptive distillation strength), CompDemo (complementary mask splitting), and Reverse CALM (cross-tokenizer objective with bounded gradients). Distilling 8B dense and 16B MoE teachers into a 0.6B student achieved 1.53 point average improvement across eight benchmarks, with HumanEval code generation reaching 48.78 vs 32.3 for autoregressive baselines. Relevant for deploying efficient small agent models that retain capabilities of larger teachers.