Kimi Publishes 'Attention Residuals' Paper — Proposes Replacing Standard Transformer Residual Connections
r/LocalLLaMA·medium signal
Moonshot AI (Kimi) published a paper proposing 'attention residuals' to replace the residual connections used in every transformer since the ResNet era — routing skip-connection information through learned attention weights instead of direct addition. The r/LocalLLaMA post (117 upvotes) reports results 'look legit' at the scales tested. If the architecture validates at frontier scale, this would be a meaningful departure from core transformer topology that has been stable since 2017.