Fetching from the wire…
01
02
03
04
05
06
07
08
Public story · 2026-08-04 · high
PRECOG pre-encodes documents into a state-space model's hidden state, skipping reprocessing at query time entirely.
Why now: The paper's arXiv identifier, 2608.02560, dates the posting to August 2026.
PRECOG cuts the time a retrieval-augmented model needs to process a document from 27 seconds to under 6 milliseconds, per arXiv 2608.02560.
That's the tax every RAG system pays: re-running full documents through the model at query time. Collapsing it by roughly 4,500x changes what retrieval at scale can cost.
The trick exploits a property unique to SSMs. Their recurrent hidden state is fixed-size and doesn't care about token position, so it works as a complete summary of whatever text it's read. That means a whole corpus can get encoded once, offline, and dropped into a query later instead of reprocessed every time.
The authors tested it on TENNs-LLM, a 1.2B-parameter gated SSM with a 192 KB hidden state. It matched in-context RAG quality while dropping prefill from O(L) to O(1), turning a cost that scales with document length into a flat one.
Transformers don't get this option. Their KV-cache is entangled with token position and grows linearly with input length, and the authors call porting PRECOG's approach to Transformers architecturally impossible.
PRECOG is really an argument for switching to state-space models, not a caching trick any RAG team can bolt onto a Transformer stack.
The only evidence so far is a single 1.2B-parameter model. Whether the trade holds at frontier scale is untested.
Each link below shares sources, entities, or timing with this story.
Simon Willison released LLM / Shared entities / Earlier coverage / Tension
Linked by a graph relationship (Simon Willison released LLM); both cover LLM, RAG; earlier LLM coverage from 2026-07-27.
Simon Willison released LLM / Shared entity: LLM / Earlier coverage / Tension
Linked by a graph relationship (Simon Willison released LLM); both cover LLM; earlier LLM coverage from 2026-06-19.
Linked by a graph relationship (Simon Willison released LLM); both cover LLM; earlier LLM coverage from 2026-06-18.
Simon Willison released LLM / Shared entity: LLM / Earlier coverage
Linked by a graph relationship (Simon Willison released LLM); both cover LLM; earlier LLM coverage from 2026-07-31.
Linked by a graph relationship (Simon Willison released LLM); both cover LLM; earlier LLM coverage from 2026-06-19.
Simon Willison released LLM / Shared entity: RAG / Earlier coverage
Linked by a graph relationship (Simon Willison released LLM); both cover RAG; earlier RAG coverage from 2026-04-24.
Simon Willison released LLM / Shared entity: LLM / Earlier coverage
Linked by a graph relationship (Simon Willison released LLM); both cover LLM; earlier LLM coverage from 2026-08-03.
Linked by a graph relationship (Simon Willison released LLM); both cover LLM; earlier LLM coverage from 2026-07-19.