Colibri Runs GLM-5.2's 744B Parameters on a 25GB Consumer Machine by Streaming Experts Off Disk

GitHub·high signal

JustVugg/colibri is a dependency-free C engine that treats VRAM, RAM, and storage as one memory hierarchy, keeping GLM-5.2's ~17B dense params resident as int4 (~9.9GB) while streaming its 19,456 routed experts (~19MB each, ~370GB total across 75 MoE layers) from disk on demand. It hit 14,728 stars since 2026-07-01 (~981/day) with 1,273 forks. Community benchmarks are honest about the cost: 1.23 tok/s on a Ryzen 9 9950X with Gen5 NVMe, 1.06 tok/s on an M5 Max, and 0.05–0.1 tok/s on a 25GB WSL2 box with slow NVMe — cold decode reads ~11GB per token. Native MTP speculative decoding gives 2.2–2.8 tokens/forward and int4 weights are pre-converted on Hugging Face.