Tokenization steals up to 64% of agent response time

Tokenization eats up to 64% of time-to-first-token in agent workloads, per a profiling study of 153,951 real calls by Zhenyu Zhang and Zhichao Cao.

That's latency burned before the model generates a single token, and it happens even with prompt KV caching turned on. At high cache hit rates, re-tokenizing the full conversation transcript becomes the dominant cost in the request, not the model's forward pass.

Nobody separates that cost out. It gets folded into whatever bucket a team calls model latency, so a tokenizer bottleneck reads like a model problem.

Their fix, TokTier, does incremental repair. Instead of re-tokenizing the whole transcript, it re-tokenizes only a window around the newly appended text and uses GPU pre-tokenization. It guarantees the resulting token IDs are bit-identical to a standard Hugging Face tokenizer. Alone, it benchmarks 437 times faster than HF tokenizers and handles a 1-million-character request in 0.87 milliseconds. It sustains 1,821 requests per second against 40 for a stateless tokenizer.

Wired into vLLM, it cut median TTFT 16-34% and P99 latency 23%, per the paper.

If your agent loop replays long transcripts turn after turn, your latency dashboard probably has no line item for tokenization. You're paying that tax and calling it model slowness. The paper profiles two agent ecosystems and doesn't say whether incremental repair holds up once a transcript gets edited or trimmed, not just appended to.