Fetching from the wire…
01
02
03
04
05
06
07
08
Public story · 2026-08-16 · high
The blocks train inside a frozen aligner, then recompose to match a monolithic model's perplexity, using more total compute.
Why now: The paper describing MoT lands in the August 16 briefing as a proof-of-mechanism result, with no report yet of the aligner getting reused across separate runs.
Researchers split a 1.3B-parameter model into independently trained layer blocks that recompose to match its original perplexity, per the arXiv paper describing MoT, short for Mixture of Training.
That matters for teams that schedule pretraining runs. MoT lets pretraining decompose into small, independently schedulable jobs instead of one continuous run. Its schedule cut the idealized critical path versus a monolithic run, once the shared aligner scaffold already existed.
The method partitions a Transformer into contiguous blocks of layers. Each block trains inside a frozen aligner pretrained ahead of time. The blocks then recompose with an optional short end-to-end adaptation pass. On a 1.3B Gemma-style model trained on the C4 dataset, a quality-parity schedule matched the monolithic baseline's perplexity.
Yes, but the blocks process more aggregate tokens than the monolithic run does. The shorter critical path only becomes a real compute win if the aligner gets reused across many training runs, not built once and discarded. The paper's own authors call MoT a proof of mechanism, not a replacement for standard pretraining.
Each link below shares sources, entities, or timing with this story.
DiffusionGemma benchmarked against Gemma / Shared entities / Earlier coverage
Linked by a graph relationship (DiffusionGemma benchmarked against Gemma); both cover Gemma, Mixture; earlier Gemma coverage from 2026-06-11.
DiffusionGemma benchmarked against Gemma / Shared entity: Gemma / Earlier coverage / Tension
Linked by a graph relationship (DiffusionGemma benchmarked against Gemma); both cover Gemma; earlier Gemma coverage from 2026-06-14.
Gemma built by Google / Shared entity: Gemma / Earlier coverage
Linked by a graph relationship (Gemma built by Google); both cover Gemma; earlier Gemma coverage from 2026-06-07.
Ollama supports Gemma / Shared entity: Gemma / Earlier coverage
Linked by a graph relationship (Ollama supports Gemma); both cover Gemma; earlier Gemma coverage from 2026-05-06.
Linked by a graph relationship (Ollama supports Gemma); both cover Gemma; earlier Gemma coverage from 2026-04-05.
Linked by a graph relationship (Ollama supports Gemma); both cover Gemma; earlier Gemma coverage from 2026-04-02.
LoRA benchmarked against Gemma / Shared entity: Gemma / Same source domain / Earlier coverage / Tension
Linked by a graph relationship (LoRA benchmarked against Gemma); both cover Gemma; reported by the same outlet (arxiv.org).
OpenCode uses Gemma
Linked by a graph relationship (OpenCode uses Gemma).