DeaMoE Cuts Per-Step Expert Weight Loading Up to 50.9% by Grouping MoE Experts Into Parameter-Sharing 'Departments'

arXiv 2608.14385·high signalXBlueskyLinkedInCopy link

Small-batch decoding — the regime coding assistants and real-time voice apps actually run in — makes MoE inference memory-bound on expert weight loading, a bottleneck the paper argues has been underserved by post-training compression and fine-grained expert designs that trade accuracy or add communication. DeaMoE groups experts into departments that share most parameters plus a small private set per expert, with a two-stage router that avoids redundant loads. It reports up to 50.9% less loaded weight per step and 1.33x end-to-end TPOT speedup on a 7B model on an A40, with microbenchmark peaks of 2.00x (A40) and 1.97x (H100) for DeepSeek-V3.