Multiverse Computing Shrinks Distillation VRAM 15.6x

Multiverse Computing cut peak distillation VRAM from 85.2 GB to 5.45 GB at 32K context by caching teacher logits instead of rerunning every step, per its Hugging Face post.

That frees smaller teams from needing a multi-node GPU cluster just to distill a large model. Distilling GPT-OSS 20B at the same 32K context needed four GPU nodes before. With this method it took one, and each training step ran 5x faster, 57.0 seconds down to 12.23.

The caching keeps the top 100 teacher logits per position instead of storing the full vocabulary distribution. The team also reformulated the KL divergence loss to process the sequence in chunks. That avoids materializing the full vocabulary-by-sequence matrix in memory, part of the VRAM cost.

The gap widens with context length. At 256K tokens, the method holds at 11.6 GB while the next-best variant they tested needs 134.2 GB.

Quality held up in their test case: a 3.2B student distilled from Llama 3.1 8B Instruct landed within about 9 points of the teacher on MMLU.

Yes, but nine points on MMLU is still a real gap, and the post doesn't say how that student performs on tasks beyond the benchmark.

The real constraint on distillation was memory, not compute. Fix the memory curve and GPU count and step time fall with it, no bigger cluster required. Worth watching whether chunked KL distillation makes it into standard training libraries, or stays a one-off recipe.