Enable speculative decoding only at low batch sizes — it's net-negative when the GPU is already saturated

PremAI·medium signal

Speculative decoding (a small draft model proposes tokens the target verifies in parallel) delivers 2–5x latency wins, but only in memory-bound, low-batch regimes; at large batch sizes the GPU is already compute-bound and the extra draft and verification work makes inference slower than plain autoregressive decoding. The actionable rule: gate speculative decoding on batch size rather than leaving it always-on. SGLang's EAGLE implementation handles this adaptively — it tracks an EMA of accepted draft length and switches between predefined speculative-length tiers (e.g., [1,3,7]), each with a pre-captured CUDA graph so tier changes are cheap.