SMELT loops a transformer twice, at matched compute

A new paper loops the middle half of a mixture-of-experts transformer through a second pass, called SMELT, at 6.8 to 18% more training compute than an unlooped model. Most published looped-transformer results compare at a fixed model size. That folds the extra compute of the second pass into the reported win, instead of accounting for it.

SMELT's authors match per-token FLOPs, non-embedding parameters and KV cache against an unlooped baseline. They then scale the comparison up to 54 billion non-embedding parameters. Each architecture gets its own Chinchilla-style scaling law, rather than sharing one curve.

The advantage is largest on code, and it grows with sequence length and the number of in-context examples. The authors trace this to the second pass cutting the attention sink, the tendency of transformers to dump weight on early tokens. It redirects that attention toward content tokens instead.

If that mechanism is real, the gap should keep widening as context windows grow, not level off. A looped MoE that wins by a flat margin regardless of context length is describing something other than sink reduction.