PRECOG Cuts RAG Prefill From 27s to Under 6ms

PRECOG cuts the time a retrieval-augmented model needs to process a document from 27 seconds to under 6 milliseconds, per arXiv 2608.02560.

That's the tax every RAG system pays: re-running full documents through the model at query time. Collapsing it by roughly 4,500x changes what retrieval at scale can cost.

The trick exploits a property unique to SSMs. Their recurrent hidden state is fixed-size and doesn't care about token position, so it works as a complete summary of whatever text it's read. That means a whole corpus can get encoded once, offline, and dropped into a query later instead of reprocessed every time.

The authors tested it on TENNs-LLM, a 1.2B-parameter gated SSM with a 192 KB hidden state. It matched in-context RAG quality while dropping prefill from O(L) to O(1), turning a cost that scales with document length into a flat one.

Transformers don't get this option. Their KV-cache is entangled with token position and grows linearly with input length, and the authors call porting PRECOG's approach to Transformers architecturally impossible.

PRECOG is really an argument for switching to state-space models, not a caching trick any RAG team can bolt onto a Transformer stack.

The only evidence so far is a single 1.2B-parameter model. Whether the trade holds at frontier scale is untested.