Fetching from the wire…
01
02
03
04
05
06
07
08
Public story · 2026-08-10 · high
Caching teacher logits and chunking the KL loss let the team distill GPT-OSS 20B on one GPU node instead of four. Step time fell 5x.
Why now: Multiverse Computing published these benchmarks on Hugging Face, dated August 10, 2026.
Multiverse Computing cut peak distillation VRAM from 85.2 GB to 5.45 GB at 32K context by caching teacher logits instead of rerunning every step, per its Hugging Face post.
That frees smaller teams from needing a multi-node GPU cluster just to distill a large model. Distilling GPT-OSS 20B at the same 32K context needed four GPU nodes before. With this method it took one, and each training step ran 5x faster, 57.0 seconds down to 12.23.
The caching keeps the top 100 teacher logits per position instead of storing the full vocabulary distribution. The team also reformulated the KL divergence loss to process the sequence in chunks. That avoids materializing the full vocabulary-by-sequence matrix in memory, part of the VRAM cost.
The gap widens with context length. At 256K tokens, the method holds at 11.6 GB while the next-best variant they tested needs 134.2 GB.
Quality held up in their test case: a 3.2B student distilled from Llama 3.1 8B Instruct landed within about 9 points of the teacher on MMLU.
Yes, but nine points on MMLU is still a real gap, and the post doesn't say how that student performs on tasks beyond the benchmark.
The real constraint on distillation was memory, not compute. Fix the memory curve and GPU count and step time fall with it, no bigger cluster required. Worth watching whether chunked KL distillation makes it into standard training libraries, or stays a one-off recipe.
Each link below shares sources, entities, or timing with this story.
Meta released Llama / Shared entity: Llama / Shared topic / Earlier coverage
Linked by a graph relationship (Meta released Llama); both cover Llama; overlapping topics (compute, down).
Meta released Llama / Shared entity: Llama / Same source domain / Earlier coverage
Linked by a graph relationship (Meta released Llama); both cover Llama; reported by the same outlet (huggingface.co).
Meta released Llama / Shared entity: GPU / Earlier coverage / Tension
Linked by a graph relationship (Meta released Llama); both cover GPU; earlier GPU coverage from 2026-07-25.
Meta released Llama / Shared entity: Llama / Earlier coverage
Linked by a graph relationship (Meta released Llama); both cover Llama; earlier Llama coverage from 2026-06-15.
Linked by a graph relationship (Meta released Llama); both cover Llama; earlier Llama coverage from 2026-05-02.
Linked by a graph relationship (Meta released Llama); both cover Llama; earlier Llama coverage from 2026-05-01.
Meta released Llama / Shared entity: GPU / Earlier coverage
Linked by a graph relationship (Meta released Llama); both cover GPU; earlier GPU coverage from 2026-03-23.
Meta released Llama / Shared entity: Llama / Earlier coverage
Linked by a graph relationship (Meta released Llama); both cover Llama; earlier Llama coverage from 2026-03-12.