PRECOG Pre-Encodes Documents as SSM Hidden States, Cutting RAG Prefill From 27 Seconds to Under 6 Milliseconds on Edge Hardware

arXiv 2608.02560·high signalXBlueskyLinkedInCopy link

A new arXiv paper introduces PRECOG (Pre-Computed Context Injection), which exploits a property unique to State-Space Models: the fixed-size, position-agnostic recurrent hidden state is a complete summary of everything read, so document corpora can be pre-encoded offline and injected directly at query time. On TENNs-LLM, a 1.2B gated-SSM model with a 192 KB hidden state, PRECOG matches in-context RAG answer quality while collapsing prefill from O(L_context) to O(1) — roughly a 4,500x latency reduction. The authors note the mechanism is architecturally impossible for Transformer KV-caches, which are position-entangled and grow linearly with context.