SageMaker's concurrency-sweep recipe found Nemotron-3 Nano 30B saturating at 2,823 tok/s, but only 80 concurrent requests fit a 1.5s TTFT SLA

AWS Machine Learning Blog·low signalXBlueskyLinkedInCopy link

AWS walks through CreateAIBenchmarkJob with the AIPerf engine against a vLLM endpoint on a Blackwell-backed ml.g7e.2xlarge. Throughput plateaued at 2,823 output tokens/s at 256 concurrent requests. Adding a 50s end-to-end and 1.5s time-to-first-token SLA cut safe concurrency to 80, and the SLA search converged in 8 iterations. For anyone sizing self-hosted inference, the useful number is the gap between peak concurrency and SLA-safe concurrency.