Baseten Splits LLM Inference Techniques Into Two Classes: Ones That Move You Along the Latency-Throughput Frontier and Ones That Push It Out

Baseten / Hacker News·medium signalXBlueskyLinkedInCopy link

Philip Kiely's September 1, 2026 piece argues that batch sizing, parallelism strategy (tensor, expert, and attention data parallelism) and quantization only trade latency against throughput, while kernel and runtime optimization, speculative decoding, and prefill/decode disaggregation move the whole frontier. The framing matters because frontier-pushing techniques compound multiplicatively, so two 2x improvements give you 4x of budget to spend on whichever axis you care about. The analysis assumes serving models like GLM-5.3 or Kimi K3 for agentic coding with KV cache reuse and KV-aware routing enabled.