CompKV Picks Sparse-Attention Tokens by Expected Compensation Error and Reaches a 6.85x Self-Attention Speedup

arXiv·low signalXBlueskyLinkedInCopy link

Sparse-attention methods usually pick tokens by attention mass and then approximate the omitted tail afterwards. CompKV shows that the residual after block-mean compensation depends on both block attention mass and within-block logit variation, and it selects blocks to minimize that residual using compact block statistics. On RULER and LongBench-Pro it scored best among the sparse baselines tested, with up to 6.85x self-attention speedup over full attention and an asynchronous implementation.