You Could Have Come Up With Kimi Delta Attention: A First-Principles Derivation From Softmax to KDA
Doubleword founder Jamie Dborin published a July 27 walkthrough that derives the entire DeltaNet lineage in one arc, softmax attention to linear attention to DeltaNet to Gated DeltaNet to KDA, showing each step as a fix for a specific defect in its predecessor. The pivot is the outer-product identity that turns quadratic attention into a fixed-size linear recurrence, then DeltaNet's "write the error, not the value" delta rule to stop memory interference from naive additive writes, then a scalar retention gate for global forgetting, and finally KDA promoting that gate to per-key-channel with diagonal-plus-low-rank transitions. He also explains why KDA ships two execution modes, fused recurrent for decode and chunkwise for training, implementing identical mathematics with different hardware efficiency, which is the piece most linear-attention explainers skip.
↳ Follow the thread