Entity trail

Softmax

Source-backed findings, relationship evidence, citations, and briefing history from the public MindPattern archive.

Briefing refs

Findings

Edges

Sources

Corpus findings

2026-07-29 / sources-researcherYou Could Have Come Up With Kimi Delta Attention: A First-Principles Derivation From Softmax to KDADoubleword founder Jamie Dborin published a July 27 walkthrough that derives the entire DeltaNet lineage in one arc, softmax attention to linear attention to DeltaNet to Gated DeltaNet to KDA, showing each step as a fix for a specific defect in its predecessor. The pivot is the outer-product identity that turns quadratic attention into a fixed-size linear recurrence, then DeltaNet's "write the error, not the value" delta rule to stop memory interference from naive additive writes, then a scalar retention gate for global forgetting, and finally KDA promoting that gate to per-key-channel with diagonal-plus-low-rank transitions. He also explains why KDA ships two execution modes, fused recurrent for decode and chunkwise for training, implementing identical mathematics with different hardware efficiency, which is the piece most linear-attention explainers skip.
2026-07-28 / sources-researcherUpstage Ships Solar Open 2: a 250B/15B-Active Agent-First Open-Weight MoE That Runs on Two H200s QuantizedKorea's Upstage released Solar Open 2 (250B total, 15B active, 320 routed + 1 shared experts, top-8) under a commercially usable license, trained on ~12T tokens over 2M B200 GPU-hours. The architecture is deliberately serving-cheap: hybrid softmax + linear attention in a [Softmax, Linear×3]×12 pattern with NoPE and a 1M context, fitting on four H200s in BF16 or two quantized — CEO Kim Sung-hun pointedly contrasted this with models needing 16 B200s. Reported scores include 70.4 SWE-Bench Verified, 92.4 LiveCodeBench, 86.2 MMLU-Pro and 58.2 on MCP-Atlas tool calling; their Selective Weight Transfer init reached target loss in ~12B tokens versus ~22B for random init.
2026-07-27 / arxiv-researcherATLAS Automates Per-Layer FHE Approximation for Transformers, Searching a 10^225 Configuration Space for LLaMA3Deploying transformers under fully homomorphic encryption is bottlenecked by polynomial approximations of softmax, normalization, and activation, whose multiplicative depth dominates inference cost — and every prior framework applies one manually tuned configuration uniformly across all layers. ATLAS (arXiv 2607.23478) formulates per-layer configuration as a multi-objective optimization over latency and predictive accuracy, confronting a search space of roughly 10^84 configurations for 12-layer BERT/ViT and 10^225 for 32-layer LLaMA3, with 120 or 320 decision variables. The problem is hard on three axes the authors quantify: each configuration takes 70–1,000 seconds to evaluate even in cleartext, and 35–50% of candidates yield numerically invalid solutions; ATLAS uses two-stage constraint relaxation plus surrogate models to make it tractable within an hour.
2026-07-25 / sources-researcherSANA-Video 2.0 Generates 720p Video on a Single GPU in ~13 Seconds Using a 3:1 Linear-to-Softmax Attention RatioSANA-Video 2.0 (arXiv 2607.21553, 23 July, Junsong Chen plus 13 co-authors) uses gated linear attention for O(N)-dominated mixing with periodic gated-softmax anchors at a 3:1 ratio, plus Block Attention Residuals that raise effective rank about 12%. The 5B and 14B models produce 720p on a single GPU: 13.06 seconds for a 720p 5-second clip fully optimized, 13.2 seconds for 40-step sampling on an H100, with a compiled DiT forward pass 3.2× faster than the full-softmax baseline at 720p and a VBench score of 84.30 at 480p. The authors claim roughly 120× the throughput of Wan 2.2-A14B — the hybrid-attention recipe is the transferable idea here, not just the video results.
2026-06-28 / arxiv-researcherDnA: Denoising Attention for Visual TasksReplaces the de facto softmax activation in multi-head attention with a denoising attention mechanism to reduce the noisy attention patterns that softmax can produce in visual perception tasks. The drop-in change targets cleaner attention maps and improved perception accuracy. A lightweight technique for vision-model builders.
2026-05-23 / arxiv-researcherPlug-in Losses Unify Softmax and Evidential Deep Learning for Uncertainty EstimationShows that the standard softmax classifier is a special case of evidential deep learning, providing a simplified framework for uncertainty estimation. Offers drop-in loss functions that give calibrated uncertainty without requiring architectural changes — directly useful for safety-critical sensor-based systems. Evaluated on real-world audio classification tasks.
2026-05-22 / arxiv-researcherGated DeltaNet-2: Decoupling Memory Edit Operations Improves Linear Attention QualityHatamizadeh, Choi, and Kautz introduce Gated DeltaNet-2, which decouples the erase and write operations in linear attention's fixed-size recurrent state using separate gates instead of a single scalar. Existing delta-rule models scramble stored associations when editing compressed memory. The approach addresses the core quality gap between linear and softmax attention, relevant for practitioners building efficient long-context models.
2026-04-21 / arxiv-researcherGSQ: Gumbel-Softmax Sampling Closes Gap Between Simple and Vector Quantization for LLMs at 2-3 BitsGSQ introduces a Gumbel-Softmax sampling approach to scalar quantization that matches the accuracy of complex vector/trellis methods (QTIP, GPTVQ, AQLM) while retaining the simplicity and deployment compatibility of GPTQ/AWQ. Weight quantization at 2-3 bits per parameter is standard for local LLM inference — GSQ eliminates the accuracy-vs-complexity tradeoff that forced practitioners to choose between simple-but-lossy and accurate-but-exotic quantization.
2026-04-18 / arxiv-researcherAdaSplash-2: Histogram-Based Initialization Makes Differentiable Sparse Attention Competitive with Softmax SpeedAdaSplash-2 addresses the computational overhead that has prevented differentiable sparse attention (alpha-entmax) from competing with softmax in practice. Through a novel histogram-based initialization for computing the normalizer, the method makes sparse attention — which enables input-dependent sparsity patterns — fast enough for practical use in long-context training where the quadratic cost of full attention is the central bottleneck.
2026-04-07 / arxiv-researcherGPU-Accelerated TFHE Enables High-Precision Encrypted LLM InferenceChen et al. demonstrate GPU acceleration of Fully Homomorphic Encryption (TFHE) for running LLM inference on encrypted data, addressing the key bottleneck of nonlinear layers (GELU, Softmax). Current FHE methods struggle with precision requirements for LLM inference; this work achieves high-precision computation on encrypted inputs using GPU parallelism. A practical step toward privacy-preserving cloud LLM deployment.
2026-04-03 / arxiv-researcherHCCS: Clipped-Linear Softmax Surrogate Eliminates Exponentiation Bottleneck for Integer-Native Edge InferenceHead-Calibrated Clipped-Linear Softmax (HCCS) replaces the exponential softmax function with a bounded, monotone clipped-linear mapping that produces stable probability distributions without exponentiation or normalization. Targets small Transformer models under low-precision inference where softmax becomes a computational bottleneck in multi-head attention. For edge deployment of small language models, this removes a key overhead without requiring model retraining.
2026-04-02 / arxiv-researcherMultiscreen: Attention Mechanism with Absolute Query-Key Relevance Enables Explicit Key RejectionIntroduces 'screening' — an attention mechanism that defines absolute rather than relative query-key relevance, allowing irrelevant keys to be explicitly rejected rather than receiving redistributed probability mass. Standard softmax attention cannot distinguish 'weakly relevant' from 'irrelevant' keys. Multiscreen language model architecture built on screening could improve retrieval-augmented generation and long-context tasks where most keys are noise.

Graph relationships

COMPETES WITH
alpha-entmax
Differentiable sparse attention (alpha-entmax) can now compete with softmax in practical speed
Source finding
COMPETES WITH
alpha-entmax -> Softmax
Differentiable sparse attention (alpha-entmax) can now compete with softmax in practical speed
Source finding

Source trail

Upstage GSQ Doubleword Blog (Jamie Dborin)arXiv 2607.23478 arXiv / HuggingFace Daily Papers arXiv arXiv arXiv arXiv arXiv arXiv arXiv

Graph sources

entity graphfindings textkg edgeskg entitiesnewsletter issues

Fetching from the wire…