Sigmoid Attention Makes Learned KV Cache Eviction Actually Transfer From Soft Gates to Hard Deletion

arXiv 2608.23296·medium signalXBlueskyLinkedInCopy link

Learned KV eviction suffers a soft-to-hard mismatch: training uses differentiable gates that attenuate token contributions, but inference only saves memory when entries are physically removed. A controlled 2x2x2 comparison over attention type, learned gating, and positional encoding on GPT-2-scale models trained on OpenWebText found that although sigmoid attention is worse as a dense language model, sigmoid-gated models delete KV entries with negligible perplexity change against their own no-eviction reference. Under a matched live-cache protocol, learned sigmoid gates beat the authors' H2O and KeyDiff implementations, while softmax gates did not uniformly beat those post-hoc methods.