Mixture of Training Recomposes Independently Pretrained Layer Blocks Into a Working 1.3B Model at Parity Perplexity

arXiv 2608.13277·medium signalXBlueskyLinkedInCopy link

MoT (arXiv 2608.13277, 2026-08-13) asks whether pretraining can be decomposed into small independently schedulable jobs. It partitions a target Transformer into contiguous layer blocks, trains each inside a frozen pretrained aligner scaffold, then recomposes with an optional short end-to-end adaptation pass. On a 1.3B Gemma-style model trained on C4, a quality-parity schedule reaches the same reported perplexity as the monolithic baseline with a shorter idealized layer-equivalent critical path after aligner preparation — though it processes more aggregate tokens, so the compute advantage depends on reusing the aligner across runs. The authors are careful to present this as a proof of mechanism, not a replacement for monolithic pretraining.