AdaSplash-2: Histogram-Based Initialization Makes Differentiable Sparse Attention Competitive with Softmax Speed

arXiv·medium signal

AdaSplash-2 addresses the computational overhead that has prevented differentiable sparse attention (alpha-entmax) from competing with softmax in practice. Through a novel histogram-based initialization for computing the normalizer, the method makes sparse attention — which enables input-dependent sparsity patterns — fast enough for practical use in long-context training where the quadratic cost of full attention is the central bottleneck.