Redesign Mixture-of-Experts Routers with Manifold Power Iteration

arXiv·medium signal

This paper reframes the MoE router — treating its rows as expert proxies — and applies Manifold Power Iteration to compute routing similarity more faithfully, addressing instability and poor expert utilization in standard top-k routers. The method targets better load balancing and representation quality without retraining experts from scratch. Relevant to anyone training or serving sparse MoE models where routing collapse hurts efficiency.