TEMPO Shows MoE Expert-Parallel Load Balancers Optimize the Wrong Quantity Below ~160 Tokens
arXiv 2608.13057 (2026-08-13) measures expert time on two datacenter GPU generations and finds it is linear in neither token count (EPLB, LPLB, UltraEP) nor activated-expert count (METRO): below roughly 156-168 tokens HBM weight streaming dominates so cost attaches to activated replicas, and above it grouped GEMM rounds tokens to 128-tile M-tiles so splitting an expert adds padded compute. Real decode batches hold hot and cold experts in both regimes at once, and recorded batches show proxy dispatchers differ by 1.4-1.6x in modeled block time. TEMPO's makespan-aware dispatcher, integrated out-of-process with SGLang, gains 4-6% throughput and cuts p99 latency ~15.6% on Qwen3-235B — while showing only mechanism cost on communication-dominated DeepSeek-V3. The authors explicitly claim a phase diagram, not a universal win.
↳ Follow the thread