Multiverse Computing cuts distillation VRAM 15.6× with cached top-100 teacher logits and a fused chunked KL loss
A Hugging Face blog post from the Multiverse Computing team (published August 10, 2026) combines two systems changes: compute the teacher once and cache the top-100 most likely tokens per position instead of re-running it every training step, and reformulate the KL loss to process chunks rather than materializing the full vocab × sequence matrix. At 32K context peak VRAM drops from 85.2 GB to 5.45 GB (15.6×); at 256K tokens it is 11.6 GB versus 134.2 GB for the next-best variant. Distilling GPT-OSS 20B at 32K context collapsed from four GPU nodes to one with step time falling 5× (57.0s → 12.23s), and a 3.2B student distilled from Llama 3.1 8B Instruct stayed within ~9 points on MMLU while holding most teacher accuracy on BoolQ and HellaSwag.
↳ Follow the thread