Fetching from the wire…
01
02
03
04
05
06
07
08
Public story · 2026-08-03 · high
A study of 153,951 agent calls found the bottleneck survives prompt caching, and a fix already cut real TTFT up to 34% in vLLM.
Why now: As of Aug. 3, no latency dashboard in the paper's benchmarks separates tokenization from model inference, which is why TokTier ships already wired into vLLM instead of pitched as a future fix.
Tokenization eats up to 64% of time-to-first-token in agent workloads, per a profiling study of 153,951 real calls by Zhenyu Zhang and Zhichao Cao.
That's latency burned before the model generates a single token, and it happens even with prompt KV caching turned on. At high cache hit rates, re-tokenizing the full conversation transcript becomes the dominant cost in the request, not the model's forward pass.
Nobody separates that cost out. It gets folded into whatever bucket a team calls model latency, so a tokenizer bottleneck reads like a model problem.
Their fix, TokTier, does incremental repair. Instead of re-tokenizing the whole transcript, it re-tokenizes only a window around the newly appended text and uses GPU pre-tokenization. It guarantees the resulting token IDs are bit-identical to a standard Hugging Face tokenizer. Alone, it benchmarks 437 times faster than HF tokenizers and handles a 1-million-character request in 0.87 milliseconds. It sustains 1,821 requests per second against 40 for a stateless tokenizer.
Wired into vLLM, it cut median TTFT 16-34% and P99 latency 23%, per the paper.
If your agent loop replays long transcripts turn after turn, your latency dashboard probably has no line item for tokenization. You're paying that tax and calling it model slowness. The paper profiles two agent ecosystems and doesn't say whether incremental repair holds up once a transcript gets edited or trimmed, not just appended to.
Each link below shares sources, entities, or timing with this story.
Shared entities / Same source domain / Shared topic / Earlier coverage
Both cover GPU, TTFT; reported by the same outlet (arxiv.org); overlapping topics (agent, cache, ttft).
Shared entities / Same source domain / Earlier coverage
Both cover IDs, Nobody; reported by the same outlet (arxiv.org); earlier IDs coverage from 2026-07-22.
Shared entity: TTFT / Shared topic / Earlier coverage / Tension
Both cover TTFT; overlapping topics (agent, cache, caching); earlier TTFT coverage from 2026-07-25.
Shared entity: P99 / Same source domain / Shared topic / Earlier coverage
Both cover P99; reported by the same outlet (arxiv.org); overlapping topics (agent, call).
Shared entity: GPU / Same source domain / Shared topic / Earlier coverage
Both cover GPU; reported by the same outlet (arxiv.org); overlapping topics (cache, dominat).
Both cover GPU; reported by the same outlet (arxiv.org); overlapping topics (agent, cache).
Shared entity: IDs / Same source domain / Shared topic / Earlier coverage
Both cover IDs; reported by the same outlet (arxiv.org); overlapping topics (cache, caching).
Shared entities / Earlier coverage / Tension
Both cover GPU, P99; earlier GPU coverage from 2026-02-17; pushes against this story (but).