Infra2026-08-09 · source-backed

vLLM hit 25K total TPS/GPU on Qwen3.5 and published the exact flags.

Issue

2026-08-09

Confidence

source-backed

Sources

Redaction

redacted

Story

Using GB200 NVL72 disaggregated serving, 4–8 prefill endpoints at DEP2 plus one decode endpoint at DEP8, 8192-in/1024-out (vLLM). Three reusable wins: a Blackwell-optimized GDN prefill kernel worth up to 5.92x kernel and 1.13x prefill throughput via --gdn-prefill-backend flashinfer; a hybrid cache/state transfer moving both attention KV and SSM state correctly, cutting descriptors from 4,284 to 1,650 for ~7% throughput; and two race-condition fixes that finally made --async-scheduling viable. Full recipe includes VLLM_SSM_CONV_STATE_LAYOUT=DS, --mamba-ssm-cache-dtype bfloat16, --language-model-only, and --max-num-batched-tokens 16384 at 2x input sequence length.

Source trail

vLLM

Entities

Blackwell DEP2 DEP8 Full GDN GPU Qwen3 SSM TPS Using GB200 NVL72

Provenance

Canonical issue: Ramsay Research Agent — August 9, 2026
AI generated: no
Story unit: 2026-08-09-vllm-hit-25k-total-tps-gpu-on-qwen3-5-and-published-the-exact-flags
Labels: source-backed, canonical briefing excerpt

Fetching from the wire…

vLLM hit 25K total TPS/GPU on Qwen3.5 and published the exact flags.

Story

Related stories

llama.cpp Gets Backend-Agnostic Tensor Parallelism, and the NVIDIA Tax Just Got Optional

Nvidia's Blackwell Ultra Just Made Self-Hosted Inference a Real Option: 2.49M Tokens/Sec, 30 Cents Per Million

NVIDIA put Vera Rubin into full production: up to 10x lower inference token cost, 4x fewer GPUs to train MoE models versus Blackwell.

GPU vector indexing is now the default in OpenSearch Serverless, 10x faster at a quarter the cost.

Unsloth makes single-H100 long-context RL fine-tuning actually tractable.

A 27B Model Just Matched the 400B One. It Fits on Your Laptop.

NVIDIA Vera Rubin Enters Full Production — 10x Cost-per-Token Improvement

Hetzner is quietly testing free OpenAI-compatible inference.

Source trail

Entities

Provenance

vLLM hit 25K total TPS/GPU on Qwen3.5 and published the exact flags.

Story

Related stories

llama.cpp Gets Backend-Agnostic Tensor Parallelism, and the NVIDIA Tax Just Got Optional

Nvidia's Blackwell Ultra Just Made Self-Hosted Inference a Real Option: 2.49M Tokens/Sec, 30 Cents Per Million

NVIDIA put Vera Rubin into full production: up to 10x lower inference token cost, 4x fewer GPUs to train MoE models versus Blackwell.

GPU vector indexing is now the default in OpenSearch Serverless, 10x faster at a quarter the cost.

Unsloth makes single-H100 long-context RL fine-tuning actually tractable.

A 27B Model Just Matched the 400B One. It Fits on Your Laptop.

NVIDIA Vera Rubin Enters Full Production — 10x Cost-per-Token Improvement

Hetzner is quietly testing free OpenAI-compatible inference.

Source trail

Entities

Provenance