Attention Residuals (Moonshot AI / Kimi): Drop-In Residual Replacement Shows Consistent Scaling Improvements — Already in Production at 48B Scale

arXiv / Moonshot AI·high signalXBlueskyLinkedInCopy link

Moonshot AI's Kimi team proposes replacing standard fixed residual connections in transformers with softmax attention over preceding layer outputs — letting each layer selectively weight earlier representations rather than blindly accumulating them. A practical 'Block AttnRes' variant partitions layers into blocks for memory efficiency and validates consistent improvement across model sizes via scaling laws. Kimi Linear (48B total / 3B active, trained on 1.4T tokens) already uses this in production; the paper is the #1 trending on HuggingFace with 1,330 upvotes.