vllm.cpp Is a 1:1 C++ Port of vLLM in a 66 MiB Binary — 140x Smaller Install, 6.1x Faster Cold Start, Token-Exact Output

mudler/vllm.cpp (GitHub)·medium signalXBlueskyLinkedInCopy link

Surfaced August 6, mudler/vllm.cpp mirrors vLLM's V1 / Model Runner V2 architecture in pure C++ with no Python, PyTorch, or ggml at runtime, shipping a shared/static `libvllm` with a stable 17-symbol C ABI, an example CLI, and an OpenAI-compatible server. The install footprint is 66 MiB against vLLM's 9.1 GiB, and cold start is 36.5s versus 221.5s — a 6.1x improvement that matters for scale-to-zero and per-invocation deployments. On Qwen3.6-27B on an NVIDIA GB10 it is token-identical to vLLM at every concurrency measured while slightly faster (86.05 vs 82.32 tok/s at concurrency 1; 1,095 vs 1,076 at 32), and 1.18x faster than llama.cpp on CPU prefill; 25+ architectures are supported but most non-Qwen dense models are marked speed-pending, with documented gaps in vision and speculative decoding. 249 stars.