Entity trail

Kimi Linear

Source-backed findings, relationship evidence, citations, and briefing history from the public MindPattern archive.

Briefing refs

Findings

Edges

Sources

Corpus findings

2026-07-29 / sources-researcherYou Could Have Come Up With Kimi Delta Attention: A First-Principles Derivation From Softmax to KDADoubleword founder Jamie Dborin published a July 27 walkthrough that derives the entire DeltaNet lineage in one arc, softmax attention to linear attention to DeltaNet to Gated DeltaNet to KDA, showing each step as a fix for a specific defect in its predecessor. The pivot is the outer-product identity that turns quadratic attention into a fixed-size linear recurrence, then DeltaNet's "write the error, not the value" delta rule to stop memory interference from naive additive writes, then a scalar retention gate for global forgetting, and finally KDA promoting that gate to per-key-channel with diagonal-plus-low-rank transitions. He also explains why KDA ships two execution modes, fused recurrent for decode and chunkwise for training, implementing identical mathematics with different hardware efficiency, which is the piece most linear-attention explainers skip.
2026-07-29 / sources-researcherSebastian Raschka's Kimi K3 Teardown: It Is Kimi Linear Scaled From 48B to 2.8T, and the First Frontier Model to Drop RoPE EntirelyRaschka's July 28 architecture notes argue K3 is far less exotic than the release framing suggests: it is a scaled-up production version of Kimi Linear, taken from 48B to 2.8T parameters, with Kimi Delta Attention as the hybrid attention layer and LatentMoE compressing large linear layers by down-projection in the same spirit as multi-head latent attention. The genuinely novel piece is that K3 replaces every RoPE layer with NoPE, the first frontier-scale implementation of no positional embeddings, alongside attention residuals that weight cross-layer residual connections by attention scores for roughly 4% added training cost and 2% added inference cost in exchange for better validation loss. He places K3 with Nemotron 3 Ultra, Nemotron 3 and DeepSeek V4 in a clear industry pivot toward inference efficiency, and contrasts attention residuals with DeepSeek V4's mHC, which widens the residual path rather than connecting across layers.
2026-07-27 / reddit-researcherEvery Kimi K3 GGUF on Hugging Face Is Currently Unloadable — llama.cpp Has No kimi_k3 Architecture YetAt least three Kimi-K3-GGUF repos (GrEarl, Kuberwastaken, AtomicChat) appeared within hours of the weight drop, and r/LocalLLaMA's 'Kimi K3 text-only for llama.cpp' thread is openly waiting for someone who can run the conversion. The llama.cpp discussion #26041 states plainly that no working K3 port exists: MODEL_ARCH.KIMIK3 is not registered in gguf-py constants, the 'situ' activation (activation_situ_beta = 4.0) is missing, AttnRes needs self_attention_res_proj / mlp_res_proj tensors at attn_res_block_size = 12, Stable LatentMoE's 3,584-dim latent expert space is unhandled, and KDA variants (mla_use_output_gate, use_full_rank_gate, gate_lower_bound) are unimplemented. Groundwork exists — ggml_gated_delta_net, LLM_ARCH_KIMI_LINEAR, and GGML_TYPE_MXFP4 are already in mainline — but builders downloading these quants today are downloading files nothing can run.
2026-05-06 / hn-researcherSubQ Launches: First Sub-Quadratic LLM With 12M-Token Context Window, $29M Seed RoundMiami-based Subquadratic emerged from stealth with $29M in seed funding and SubQ, a closed-weight LLM using Subquadratic Sparse Attention (SSA) that scales linearly with context length. The research model supports 12M tokens (~9M words), the production API exposes 1M tokens, and it runs ~52x faster than FlashAttention at 1M tokens at roughly 1/5 the cost of Claude Opus or GPT-5.5. Skepticism is warranted — the model is closed-weight with no technical report yet, and prior sub-quadratic architectures (Mamba, RWKV, Kimi Linear) have underperformed quadratic attention at frontier scale.
2026-05-01 / sources-researcherMoonshot AI Open-Sources FlashKDA: 1.7–2.2x Attention Speedup for Kimi Linear's Delta Attention ArchitectureMoonshot AI released FlashKDA, a CUTLASS-based kernel implementation of Kimi Delta Attention delivering 1.72x–2.22x prefill speedup over flash-linear-attention on NVIDIA H20 GPUs. Works as a drop-in backend — existing codebases using flash-linear-attention get the speedup without code changes. KDA is the core mechanism in Kimi Linear (48B total / 3B activated params), using a 3:1 KDA-to-MLA ratio that reduces KV cache 75% and achieves 6x higher decoding throughput at 1M context versus full attention.
2026-03-17 / sources-researcherAttention Residuals (Moonshot AI / Kimi): Drop-In Residual Replacement Shows Consistent Scaling Improvements — Already in Production at 48B ScaleMoonshot AI's Kimi team proposes replacing standard fixed residual connections in transformers with softmax attention over preceding layer outputs — letting each layer selectively weight earlier representations rather than blindly accumulating them. A practical 'Block AttnRes' variant partitions layers into blocks for memory efficiency and validates consistent improvement across model sizes via scaling laws. Kimi Linear (48B total / 3B active, trained on 1.4T tokens) already uses this in production; the paper is the #1 trending on HuggingFace with 1,330 upvotes.
2026-03-16 / hn-researcherLLM Architecture Gallery — 45+ Model Diagrams Surges to 443pts, 34cmtsSebastian Raschka's LLM Architecture Gallery hit 443 points after posting March 15-16, covering architecture diagrams and fact sheets for 45+ models from 3B to 1T parameters. Covers dense models (Llama 3, Gemma 3, OLMo 3), sparse MoE (DeepSeek V3/V3.2, Llama 4 Maverick, Mistral 3 Large), and hybrid architectures (Qwen3 Next, Kimi Linear) with parameter counts, release dates, attention type, and technical report links. Physical high-resolution poster available.

Graph relationships

RELEASED
Moonshot AI
Moonshot AI released Kimi Linear with 48B parameters.
Source finding
SUPPORTS
FlashKDA
FlashKDA is the backend for Kimi Linear kernel.
Source finding
RELEASED
Moonshot AI -> Kimi Linear
Moonshot AI released Kimi Linear with 48B parameters.
Source finding
SUPPORTS
FlashKDA -> Kimi Linear
FlashKDA is the backend for Kimi Linear kernel.
Source finding

Source trail

Sebastian Raschka's July 28 teardown Source Interconnects Doubleword Blog (Jamie Dborin)r/LocalLLaMA (primary: github.com/ggml-org/llama.cpp discussion #26041)Subquadratic / Hacker News Moonshot AI / GitHub Sebastian Raschka

Graph sources

entity graphfindings textkg edgeskg entitiesnewsletter issues

Fetching from the wire…