Simon Willison: Streaming MoE Experts — Running Larger Models on Hardware Without Enough RAM

Simon Willison·medium signal

Willison expands on Dan Woods' experiments with streaming experts, a technique for running Mixture-of-Experts models on hardware that doesn't have enough RAM to fit the entire model by streaming expert weights on and off disk as needed. The approach trades inference latency for the ability to run models that would otherwise be impossible on consumer hardware. Practically relevant for the r/LocalLLaMA community and anyone running large MoE models like Mixtral or similar architectures on constrained hardware.