SageMaker prefix-aware routing cut P50 time-to-first-token by up to 77%

AWS Machine Learning Blog·medium signalXBlueskyLinkedInCopy link

SageMaker Inference added a PREFIX_AWARE routing strategy that sends requests sharing a prompt prefix to the same instance so the KV cache stays warm. On Llama 3.1 70B across seven p5.48xlarge instances with 8K-token shared prefixes, P50 TTFT fell 71 to 77%, throughput rose 15 to 16%, and cache hit rate went from about 25% to 82%, with 1.3 to 1.9 ms of routing overhead. Short conversations gained much less (13 to 16% P50), it needs at least two instances, and prefix caching must be on in the serving framework.