Sebastian Raschka: KV Sharing in Gemma 4, Compressed Attention in DeepSeek V4 — New LLM Architecture Deep Dive

Sebastian Raschka·medium signal

Sebastian Raschka published a technical analysis of the latest LLM architecture innovations: Gemma 4's cross-layer KV sharing with per-layer embeddings to reduce KV-cache memory, DeepSeek V4's Multi-Head Compressed attention, Laguna XS.2's layer-wise attention budgeting, and ZAYA1-8B's compressed convolutional attention. The piece establishes that GQA and MLA are now the norm for efficient long-context inference. Essential reading for anyone optimizing inference infrastructure.