slotstream runs a 104GB model on a 48GB Mac

slotstream runs Qwen3.8-Flash-Next, a 103.8GB model split across 24 files at 4-bit precision, on a Mac with 48GB of RAM. It gets 12 tokens a second. The project, written in Swift and MLX, posted August 28 and picked up 220 points on Hacker News.

The trick is what stays in memory and what doesn't. slotstream loads only the 3.8GB dense trunk into RAM, which takes about 2 seconds to start up. Everything else, the routed mixture-of-experts weights, gets read straight off SSD with pread into a fixed pool of cache slots shared across all 48 layers. Hot layers borrow slots from cold ones instead of each layer claiming its own permanent space.

That's a deliberate departure from the usual approach. Most local-inference tools memory-map the model file and let the OS page it in and out. slotstream skips mmap entirely, because MLX can't materialize part of a memory-mapped tensor, so partial reads don't work the way they need to. Managing the slots by hand is more code, but it's what makes the streaming possible at all.

The published numbers scale with RAM: 3 tokens a second at 8GB, 8 at 24GB, 12 at 48GB. Running the full setup needs about 110GB of free disk space, since the model has to sit on the SSD in its entirety even though only a fraction loads into memory at once.

For anyone trying to run models bigger than their machine's RAM, the ceiling here isn't how much memory you have. It's how fast your SSD can feed a 48-layer cache pool without stalling generation. A Mac with a slower drive would presumably see worse throughput than one with a fast NVMe setup, though the numbers above don't say what drive they were measured on.