Colibrì runs 2.8T-parameter models off consumer hardware

Colibrì runs mixture-of-experts models as large as 2.8 trillion parameters on ordinary consumer hardware by streaming experts off disk instead of loading a full model into memory first. JustVugg built it in plain C with no engine dependencies, released it under Apache 2.0, and it has picked up 27,290 stars on GitHub.

Most local inference tools treat VRAM and RAM as the whole budget and fail once a model exceeds them. Colibrì treats storage, RAM and VRAM as one hierarchy instead. It moves experts between those tiers as a MoE model needs them, with one C file per model family. That's what lets someone run a model sized for a multi-GPU cluster on a single consumer machine, without buying the cluster.

Eight model families work in the current release. Kimi K3 tops out at 2.8T parameters, and Inkling runs at 975B, both served through the same three commands, coli chat, coli serve, coli web.

The project frames itself as a research platform for inference-side performance work covering model formats, storage I/O, placement, scheduling and CPU/GPU overlap. That's a wider claim than running a big model on a small machine.

The repo doesn't say what throughput looks like once a model is mostly running off disk. Streaming experts avoids an out-of-memory crash, but disk read speed still bounds how fast tokens come out, and nothing in the project's materials puts a number on that tradeoff.