Predicting the Next Layer's Routing One Token Ahead Serves a 35B MoE From SSD at 20 tok/s in 3GiB

arXiv 2609.18063·medium signalXBlueskyLinkedInCopy link

arXiv 2609.18063 (16 Sep 2026) attacks the reason naive SSD offloading fails for MoE inference: layer N+1's experts must be chosen before layer N's output exists, so reads cannot start early enough to hide behind compute. Edge0 adds a per-layer prerouter that predicts the next layer's routing one token ahead and then uses that prediction as the routing, so the staged expert set equals the routed set and nothing is dropped, with an unmerged recovery LoRA paying back int4 quantization loss. On a single 24GB machine it serves a 35B-class MoE at 20 tok/s inside 3GiB of peak active memory, within a few points of its fp16 teacher across five public benchmarks, with framework, checkpoints and adapters open sourced.