Entity trail

vllm.cpp

Source-backed findings, relationship evidence, citations, and briefing history from the public MindPattern archive.

Briefing refs

Findings

Edges

Sources

Corpus findings

2026-08-23 / reddit-researcherSelf-Hosting Kimi K3's 2.8T Parameters Cost $190 Per Million Output Tokens on 8 B300s, and the 1-Bit Quant Was 3.3x WorseA practitioner ran Kimi K3 on 8x B300 via Modal at $56.79/hour with vLLM, tensor parallel 8 and native MXFP4: 27-minute cold boot for a 1.56 TB load, TTFT 0.92 to 1.02s, 92 tok/s steady decode, about $36 of GPU time per clean run and $1,363/day left warm. The cheaper path was worse, not better. Unsloth's 1-bit UD-IQ1_S at 594 GB on 8x A100-80GB via llama.cpp cost $19.99/hour but delivered roughly 9 tok/s and about $620 per million tokens. Top commenters push back that per-token cost only works out when you batch many concurrent users, which is the real lesson for anyone pricing a self-hosted frontier open model.
2026-08-20 / hn-researcherDFlash 2 Adds a Path Selector and a Two-Tap Convolution to Speculative Decoding, Reaching 3.4x Throughput on Qwen3.8-27BInco AI released DFlash 2 on 2026-08-18, keeping parallel block drafting but adding a bilinear-attention path selector that scores adjacent token pairs across DFlash's top-16 candidates for 0.6% latency overhead, plus a two-tap dynamic depthwise convolution that fixes suffix decay for 3% more parameters. The result is a 21% gain in mean acceptance length over DFlash, 2.7-3.4x throughput versus autoregressive decoding on Qwen3.8-27B, and 3.1-4.6x on Meta's Muse Glimmer 30B, consistent across GSM8K, MATH-500, HumanEval, MBPP and MT-Bench. Two drafters are on Hugging Face, tested on Apple M5 Max, NVIDIA GPUs and TPUs across SGLang, vLLM, llama.cpp, Ollama and oMLX.
2026-08-09 / sources-researchervllm.cpp Is a 1:1 C++ Port of vLLM in a 66 MiB Binary — 140x Smaller Install, 6.1x Faster Cold Start, Token-Exact OutputSurfaced August 6, mudler/vllm.cpp mirrors vLLM's V1 / Model Runner V2 architecture in pure C++ with no Python, PyTorch, or ggml at runtime, shipping a shared/static `libvllm` with a stable 17-symbol C ABI, an example CLI, and an OpenAI-compatible server. The install footprint is 66 MiB against vLLM's 9.1 GiB, and cold start is 36.5s versus 221.5s — a 6.1x improvement that matters for scale-to-zero and per-invocation deployments. On Qwen3.6-27B on an NVIDIA GB10 it is token-identical to vLLM at every concurrency measured while slightly faster (86.05 vs 82.32 tok/s at concurrency 1; 1,095 vs 1,076 at 32), and 1.18x faster than llama.cpp on CPU prefill; 25+ architectures are supported but most non-Qwen dense models are marked speed-pending, with documented gaps in vision and speculative decoding. 249 stars.
2026-06-07 / projects-researcherPewDiePie Drops 'Odysseus,' a Self-Hosted AI Workspace That Hit ~54K Stars in Its First WeekOdysseus (github.com/pewdiepie-archdaemon/odysseus) is an MIT-licensed, local-first AI workspace published around May 31, 2026 that reached roughly 54,000 stars and 6,300 forks by June 5. It bundles chat over local/remote models (vLLM, llama.cpp, Ollama, OpenRouter, OpenAI), autonomous agents with bash/files/web/memory tools plus MCP servers, a document editor, image generation/inpainting, IMAP/SMTP email triage, and multi-step research runs — all with no telemetry. The viral launch, driven in part by the high-profile creator, shows how fast a polished self-hosted alternative to commercial AI suites can capture mindshare.
2026-05-11 / projects-researchervLLM Trends at 79,633 Stars as Dominant Open-Source LLM Inference Engine — Model Runner V2 Rewrite and Blackwell GPU Support Drive Continued AdoptionvLLM maintains its position as the go-to open-source inference engine with 79,633 stars, featuring PagedAttention and continuous batching for high-throughput memory-efficient LLM serving. The March 2026 Model Runner V2 ground-up rewrite delivered significant performance improvements, with the engine now supporting NVIDIA Blackwell GPUs, DeepSeek V3, and AMD ROCm acceleration. An independent May 2026 inference engine benchmark comparing vLLM ROCm, TurboQuant, and LLaMA.cpp confirms vLLM's throughput leadership on supported hardware.
2026-04-18 / agents-researcherNVIDIA Co-Optimizes Google Gemma 4 for Edge Agent Deployment: 31B Dense to 2.3B Edge Across RTX, DGX Spark, and JetsonNVIDIA announced April 2 that it co-optimized Google's Gemma 4 family for local and edge agentic AI deployment. The four-model bundle — 31B dense, 26B MoE (128 experts), E4B, and E2B — supports reasoning, code generation, agent tool use, and multimodal input. The 31B model runs on DGX Spark's 128GB unified memory while E2B (2.3B effective parameters) targets Jetson Orin Nano for robotics. Models deploy via vLLM, Ollama, llama.cpp, and NeMo Automodel with commercial-friendly licensing.
2026-04-05 / sources-researcherGoogle DeepMind Releases Gemma 4: Four Open Models with Multimodal, 256K Context, and Agentic Capabilities Under Apache 2.0Google DeepMind released Gemma 4 on April 2 with four model sizes (E2B, E4B, 26B MoE, 31B Dense) under Apache 2.0 featuring multimodal capabilities (text, vision, audio), 256K context windows, and native thinking/tool-calling optimized for agentic workflows. Day-0 ecosystem support shipped across vLLM, llama.cpp, Ollama, and Unsloth, with the 26B MoE variant hitting 162 tok/s decode on RTX 4090 at 19.5GB VRAM and 34 tok/s on Mac mini M4 — the strongest open-weight agentic model family yet for local deployment.
2026-03-17 / sources-researcherMistral Small 4: 119B MoE Apache 2.0 Model Unifies Reasoning, Vision, and Code with Per-Request reasoning_effort ControlMistral released Mistral Small 4, a 119B parameter Mixture-of-Experts model with only 6B active parameters per token, licensed Apache 2.0. The model introduces a reasoning_effort parameter (none/high) giving developers direct cost-vs-depth control per request, and delivers 40% faster completions and 3x more requests/second than its predecessor. Available day-0 on NVIDIA NIM, vLLM, llama.cpp, and HuggingFace — the first single open checkpoint unifying vision, code agents, and configurable chain-of-thought reasoning under a permissive license.

Graph relationships

BENCHMARKED AGAINST
vLLM
vllm.cpp demonstrated token-identical output with vLLM at equal or faster speeds
Source finding
BUILT BY
mudler
mudler created vllm.cpp as a pure C++ port of vLLM
Source finding
BENCHMARKED AGAINST
llama.cpp
vllm.cpp is 1.18x faster than llama.cpp on CPU prefill
Source finding

Source trail

r/LocalLLaMA (81 upvotes, 34 comments); full write-up at books.vizuara.ai/book/kimi-k3-hosting Inco AI / Hacker News mudler/vllm.cpp (GitHub)GitHub GitHub NVIDIA Developer Blog Google Blog simonwillison.net

Graph sources

entity graphfindings textkg edgeskg entities

Fetching from the wire…