SMELT Loops the Middle Half of an MoE Transformer Twice and Saves 6.8-18% of Training FLOPs

arXiv 2609.01343·high signalXBlueskyLinkedInCopy link

Most looped-Transformer results compare at fixed model size, conflating the architecture with extra compute. SMELT matches per-token FLOPs, total non-embedding parameters, and KV cache against an unlooped baseline, then loops the middle half of layers twice, scaling to four sizes up to 54B non-embedding parameters with a separate Chinchilla-style scaling law fit per architecture. Loss drops faster with compute for a 6.8 to 18.0% saving on the compute-optimal frontier, the advantage is largest on code and grows with sequence length and in-context example count, and mechanistic analysis attributes it to the second pass reducing the attention sink and redirecting mass toward content tokens. It drew 59 upvotes on HuggingFace Daily Papers.