Swiftlet runs an 80B Qwen model in 4.3GB of RAM

Swiftlet runs an 80-billion-parameter Qwen model inside 4.3 GB of RAM by streaming its mixture-of-experts weights straight off the SSD, according to its Show HN post.

That's small enough for a consumer laptop, not a server rack. On an M5 Mac, the setup manages 4.5 to 5 tokens per second, and a 35B version runs on an iPhone.

Commenters didn't let the headline number stand. Prefill, the step where the model reads the prompt before generating, is the real cost. It takes roughly half an hour for 10,000 tokens on an M5, per critics in the thread.

Generation is fast once it starts. Getting there isn't.

The trick only works because Qwen is a mixture-of-experts model that activates about 3B of its 80B parameters for any given token. Swiftlet has to pull those active experts off disk, not the full model. Dense models don't get that shortcut, since every parameter fires on every token regardless of size.

Disk-streaming inference only works because Qwen activates about 3B of its 80B parameters per token. That's why the trick scales with MoE adoption, not with local AI in general. Watch whether more labs ship MoE-first architectures, because that's what decides if this stays a demo or becomes a real way to run big models on small hardware.

The demo is sitting at 215 points on Show HN as of the Aug 4 roundup, with the prefill objection still topping the comments.