Attention Once Is All You Need: Stateful Transformers Cut Streaming Query Latency to O(|q|)
arXiv·medium signal
Paper introduces stateful transformer sessions for streaming workloads: a persistent KV cache is advanced incrementally as new data arrives, moving prefill off the critical path so query latency becomes O(|q|) independent of accumulated context size. Eliminates the O(n) prefill cost that conventional transformer engines pay on every query. Directly applicable to streaming inference pipelines, real-time monitoring, and continuous agent loops.