slotstream adds expert lookahead for SSD-streamed MoE on Macs, and the router beats a trained predictor

GitHub / carloslfu (via r/LocalLLaMA, 27 upvotes)·medium signalXBlueskyLinkedInCopy link

carloslfu (slotstream, 364 stars, pushed September 15) reports a 10%+ throughput gain for Qwen3.8-Flash-Next running with expert offloading on low-memory Macs by running the model's own router two layers ahead on the current hidden state to prefetch the next experts. The notable negative result is that he first trained a small model to predict experts N layers ahead and it lost to simply reusing the router; N=2 was the empirical sweet spot, and a correction model layered on top adds another 3-4%. slotstream's premise is running a 125B MoE that needs 104GB at 4-bit on a Mac with a fraction of that RAM, via MLX and Swift with an Ollama-compatible API.