SonicSampler Unifies LLM Sampling and Speculative Verification Into One Triton Kernel, Up to 16× Faster on Heterogeneous Batches

arXiv 2607.20475·medium signal

Pragaash Ponnusamy, Shivam Sahni, Jue Wang, and Tri Dao attack a real serving problem: existing sampling implementations accelerate only parts of the logit-processing/token-selection/verification pipeline, need multiple kernel launches, or assume every request in a batch samples the same way — which breaks CUDA Graph execution for dynamic workloads. Their Triton kernel suite handles grammar-constrained decoding, penalties, filtering, and speculative verification inside a single batched kernel while staying CUDA Graph compatible, with a hierarchical two-stage top-k giving up to 10× speedup and up to 16× across heterogeneous workloads. This is a drop-in serving-layer win for anyone running mixed sampling configs behind one endpoint.