Fetching from the wire…
01
02
03
04
05
06
07
08
Public story · 2026-08-17 · high
MoE inference turns memory-bound during the low-batch decoding that coding assistants and voice apps use, per a new arXiv paper.
Why now: The paper posted to arXiv on August 17.
DeaMoE cuts per-step expert weight loading up to 50.9% by grouping experts into parameter-sharing departments, per a paper posted to arXiv on August 17.
Coding assistants and real-time voice apps run in exactly the low-batch regime where that weight loading turns memory-bound. The paper says post-training compression and fine-grained expert designs haven't fixed it, trading accuracy or adding communication overhead instead.
DeaMoE groups experts into departments that share most parameters, with a small private set kept per expert. A two-stage router then picks experts while avoiding redundant weight loads.
On a 7B model on an A40 GPU, the paper reports a 1.33x speedup in time-per-output-token end to end. Microbenchmarks on DeepSeek-V3 peaked higher: 2.00x on the A40, 1.97x on an H100.
A technique that cuts loaded weight without retraining or added cross-device communication changes the cost math for anyone serving MoE models at low concurrency.
The paper doesn't report accuracy deltas from the department grouping, only speed and weight-loading numbers.
Each link below shares sources, entities, or timing with this story.
Same source domain / Semantically similar
Reported by the same outlet (arxiv.org); covers closely related ground (similarity 0.75).
Same source
Cite the same source (arXiv 2608.14385).
Same source domain / Semantically similar
Reported by the same outlet (arxiv.org); covers closely related ground (similarity 0.75).
Reported by the same outlet (arxiv.org); covers closely related ground (similarity 0.73).
Reported by the same outlet (arxiv.org); covers closely related ground (similarity 0.72).
Reported by the same outlet (arxiv.org); covers closely related ground (similarity 0.72).
Reported by the same outlet (arxiv.org); covers closely related ground (similarity 0.72).
Reported by the same outlet (arxiv.org); covers closely related ground (similarity 0.72).