WASTE Runs the Full 2.78-Trillion-Parameter Kimi K3 on a 64 GB MacBook by Streaming Experts From NVMe at 0.45–0.62 tok/s
sqliteai/waste (769 stars, created July 28) is a dependency-free C inference engine that keeps only K3's 27.28 GB trunk resident and streams the ~4% of experts activated per token off SSD, opening the full model in 29.06 GB of RAM — the published 1.42 TB of weights convert to a 982 GB container, and this is the undistilled model. The counterintuitive result is the cache table: raising the expert cache from 17.32 GB to 23.32 GB lifts hit rate from 36.2% to 38.4% while throughput collapses eightfold (0.63 → ~0.08 tok/s) because a cache hit becomes a page fault. Storage bandwidth is the binding constraint — a cold token reads ~17 GB of experts, and a USB enclosure at 0.94 GB/s versus internal NVMe at 12.78 GB/s is the difference between working and not. Layers validate against a PyTorch reference to within 3.6e-06 on final logits.
↳ Follow the thread