Shard the KV cache by sequence position, not by head: vLLM's Decode Context Parallelism triples long-context agent throughput with two flags

vLLM Blog·high signalXBlueskyLinkedInCopy link

vLLM shipped Decode Context Parallelism (DCP), which splits KV cache across GPUs by token position rather than attention head, eliminating the duplication that caps tensor parallelism on long contexts. On 8×B200 with Kimi K2.6, baseline TP plateaued at ~1,863 tok/s/GPU at concurrency 64 while DCP reached 6,091 tok/s/GPU at concurrency 512 using only 82% of KV memory, staying stable past 200K-token sequences. Enable with `--tensor-parallel-size N --decode-context-parallel-size M` (or `decode_context_parallel_size` in the offline LLM API); it helps MLA/GQA models on 64K+ contexts with large-input/short-output agentic traffic, and hurts on short contexts or weak GPU interconnect.