On-Demand Attention: a lightweight recall head lets a model decide when long-context reads are worth paying for

arXiv·medium signalXBlueskyLinkedInCopy link

arXiv 2609.20734 (2026-09-17) targets the cost structure of agentic long-context inference, where full-attention decoding reads the entire growing history at every step whether or not that read helps the next token. The authors show a pretrained model's decoding states already carry information predictive of that benefit before the global read happens, and train only a small recall head to invoke global attention selectively while leaving pretrained weights untouched and the full historical KV cache available for later recall. They implemented GPU-side conditional execution in vLLM so the reduced global reads turn into real decoding speedups at long context. Across Qwen and Gemma models including hybrid-attention backbones, selective recall recovered most of the accuracy lost under pure local attention.