SageMaker HyperPod Inference Gateway routes on KV cache and queue depth instead of round robin

AWS Machine Learning Blog·medium signalXBlueskyLinkedInCopy link

AWS launched a Kubernetes-native, GPU-aware routing add-on for EKS on 18 September that scores pods on KV cache utilization, queue depth, whether the requested LoRA adapter is already loaded, prefix cache hit probability and active request count. AWS reports up to 97% lower time-to-first-token on mixed GPU fleets, up to 98% better P99 under bursty traffic, and 8-50% throughput gains, with no gain over round robin on uniform fleets with steady traffic. Tier 1 is per-cluster and combines an Envoy Gateway, a body-based router and an endpoint picker.