You Only Index Once: Cross-Layer Sparse Attention with Shared Routing
arXiv·high signal
A new sparse attention method that computes top-k token routing once and shares it across layers, attacking the efficiency-quality tradeoff that limits long-context decoding in reasoning-heavy models. Structured block-sparse methods give speed but lose quality; token-sparse methods are accurate but slow due to per-layer routing — this work amortizes the routing cost. Practical for serving long chain-of-thought workloads cheaply.