SageMaker HyperPod Inference Gateway routes on KV cache and queue depth.

The Kubernetes-native GPU-aware routing add-on for EKS scores pods on KV cache utilization, queue depth, whether the requested LoRA adapter is already loaded, prefix cache hit probability and active request count. AWS reports up to 97% lower time-to-first-token on mixed GPU fleets, up to 98% better P99 under bursty traffic, and 8-50% throughput gains, with no gain over round robin on uniform fleets with steady traffic. AWS The honest null case is stated up front, which I appreciate.