Public story · 2026-08-25 · high

Sigmoid attention fixes a KV cache eviction gap

In GPT-2-scale tests, sigmoid-gated models beat H2O and KeyDiff eviction baselines; the softmax-gated version didn't.

Why now: As of August 25, the paper tests learned gating against the established H2O and KeyDiff eviction methods under a matched live-cache protocol. Perplexity alone would have missed the gap.

Issue

2026-08-25

Confidence

high

Sources

Redaction

passed

Story

Learned KV cache eviction has a mismatch between training and inference. Training uses soft, differentiable gates that fade a token's contribution toward zero. Inference only saves memory once a token gets physically removed from the cache. A paper testing KV cache eviction checked whether gates trained the soft way survive the hard version.

The gap matters for anyone running long-context inference on a memory budget. A gate that behaves well during training can still fail once the cache drops that token for real.

The team ran a 2x2x2 comparison across attention type, learned gating, and positional encoding. Sigmoid attention scored worse than softmax as a plain dense language model on its own. Paired with learned gating, though, sigmoid-gated models deleted KV entries with almost no change in perplexity against their own no-eviction reference. Under a matched live-cache protocol, only the sigmoid-gated models beat the paper's own H2O and KeyDiff eviction baselines. Softmax-gated models, trained with the same gating setup, didn't.

It's a narrow finding. The paper doesn't test past GPT-2 scale, so it doesn't say whether the pattern holds for larger models running production caches, or whether softmax-gated models could close the gap with more tuning.

If it does hold at scale, teams building eviction into serving stacks can't treat gate training as a stand-in for delete-safety testing. The failure only shows up once you check the live-cache protocol, not the training loss.

Source trail

arXiv 2608.23296 More from arxiv.org

Entities

GPT H2O KeyDiff Learned KV Sigmoid Under

Claim evidence

Sigmoid attention fixes a KV cache eviction gap
arXiv 2608.23296

Provenance

Canonical issue: 2026-08-25
AI generated: yes
Story unit: 2026-08-25-sigmoid-attention-makes-learned-kv-eviction-actually-transfer-from-soft-gates-to-hard-deletion
Labels: source-backed, canonical briefing excerpt

Fetching from the wire…

← The Wire

Public story · 2026-08-25 · high

Sigmoid attention fixes a KV cache eviction gap

In GPT-2-scale tests, sigmoid-gated models beat H2O and KeyDiff eviction baselines; the softmax-gated version didn't.

Why now: As of August 25, the paper tests learned gating against the established H2O and KeyDiff eviction methods under a matched live-cache protocol. Perplexity alone would have missed the gap.

Issue

2026-08-25

Confidence

high

Sources

Redaction

passed

Story

The gap matters for anyone running long-context inference on a memory budget. A gate that behaves well during training can still fail once the cache drops that token for real.

Source trail

arXiv 2608.23296 More from arxiv.org

Entities

GPT H2O KeyDiff Learned KV Sigmoid Under

Claim evidence

Sigmoid attention fixes a KV cache eviction gap
arXiv 2608.23296

Provenance

Canonical issue: 2026-08-25
AI generated: yes
Story unit: 2026-08-25-sigmoid-attention-makes-learned-kv-eviction-actually-transfer-from-soft-gates-to-hard-deletion
Labels: source-backed, canonical briefing excerpt

Sigmoid attention fixes a KV cache eviction gap

Story

Related stories

PIMiner builds a transferable prompt-injection strategy library with ~10 queries per sample.

Claude changes its answers when it thinks it's talking to an AI safety researcher.

Sparse autoencoders find jailbreaks that transfer across AI labs

Long-context AI accuracy drops 30-50% before hitting the limit

GitHub Ships Cross-Model Code Review, and the Pattern Is Bigger Than It Looks

NVIDIA showed a fine-tuned 30B open model matching frontier models at attacking AI agents for 70–125x less.

LLM Web Agents Fail Dark Patterns 41–72% of the Time

GLM 5.2 and the coming AI margin collapse

Source trail

Entities

Claim evidence

Provenance

Sigmoid attention fixes a KV cache eviction gap

Story

Related stories

PIMiner builds a transferable prompt-injection strategy library with ~10 queries per sample.

Claude changes its answers when it thinks it's talking to an AI safety researcher.

Sparse autoencoders find jailbreaks that transfer across AI labs

Long-context AI accuracy drops 30-50% before hitting the limit

GitHub Ships Cross-Model Code Review, and the Pattern Is Bigger Than It Looks

NVIDIA showed a fine-tuned 30B open model matching frontier models at attacking AI agents for 70–125x less.

LLM Web Agents Fail Dark Patterns 41–72% of the Time

GLM 5.2 and the coming AI margin collapse

Source trail

Entities

Claim evidence

Provenance