SageMaker's new routing mode cuts LLM latency up to 77%

Amazon SageMaker added a routing strategy called PREFIX_AWARE that sends requests sharing a prompt prefix to the same inference instance. That keeps the instance's KV cache warm instead of getting evicted and rebuilt on every call, per AWS's prefix-aware routing post.

The gain only shows up for one shape of workload: requests that repeat a long prompt prefix, like an agent's system instructions. Short, one-off conversations barely move.

The benchmark numbers are specific. Running Llama 3.1 70B on seven p5.48xlarge instances with 8K-token shared prefixes, median time-to-first-token dropped 71-77%, and the cache hit rate climbed to 82% from around 25%. Routing itself added 1.3-1.9 ms of overhead, small next to what it saves.

Short conversations without a long shared prefix only improved 13-16%. AWS's own numbers confirm the split: this is a fix for repeated long prompts, not for latency in general.

Self-hosters got a parallel option. Mesh-LLM shipped a radix prefix cache in version 0.76.0 on September 10 that survives node restarts and routes repeated prompts to workers with verified cache hits, instead of guessing which node might still have the prefix warm.

Neither release says what happens when prefixes don't line up cleanly, like an agent that appends a few tokens of user context before the shared instruction block. If your routing layer doesn't know your prompts share a prefix, you're paying for cache misses you don't have to.