AWS cut ASR inference from 16 GPUs to 4 using CUDA MPS, a 75% drop with no model change

AWS Machine Learning Blog·medium signalXBlueskyLinkedInCopy link

Running a clinical fine-tune of NVIDIA Parakeet TDT 0.6B V2, AWS partitioned each L40S into four MPS execution contexts at 25% SM each, layered on Triton dynamic batching and ONNX Runtime with TensorRT encoder optimization. On g7e.4xlarge at concurrency 32 the setup reached 92.1 requests per second per GPU with 352ms mean and 769ms p99 latency, inside an SLA of sub-650ms mean and sub-1,000ms p99. Peak load that previously needed 16 g6e.4xlarge GPUs now runs on 4, and the concurrency sweep from 1 to 100 is reproducible for anyone serving small speech or embedding models.