DeaMoE Cuts MoE Expert Weight Loading Up To 50.9%

DeaMoE cuts the expert weight a mixture-of-experts model loads each decoding step by up to 50.9%, per the arXiv paper describing it. Small-batch decoding is the mode coding assistants and voice apps run in. It makes MoE inference memory-bound on loading expert weights, so trimming that load speeds up the exact workload builders ship. The paper reports a 1.33x end-to-end speedup in time-per-output-token on a 7B model tested on an A40 GPU.

DeaMoE groups experts into departments that share most of their parameters, and keeps a small private set of weights for each expert. A two-stage router picks the department and the expert, avoiding the redundant loads that come from moving whole experts one at a time.

Microbenchmarks push the numbers higher: DeaMoE peaks at a 2.00x weight-loading speedup on an A40 and 1.97x on an H100, both measured against DeepSeek-V3. The paper's framing is that post-training compression and fine-grained expert designs already tried to fix this bottleneck. Both approaches trade away accuracy or add communication cost between GPUs, the paper argues. The paper doesn't say whether the 1.33x end-to-end number holds on models bigger than the 7B one it tested.

The real test is whether that 1.33x holds on models bigger than 7B, not whether the microbenchmarks climb higher on DeepSeek-V3. If it holds, expert-weight loading stops being a fixed tax on every decoding step for coding assistants and voice apps. The timing matters because more products already ship as thin wrappers around large MoE models, exactly the small-batch regime this paper targets.