Together AI Deep Dive: Serving DeepSeek-V4 at Million-Token Context Is an Inference Systems Problem, Not a Model Problem

Together AI Blog·high signal

Together AI published a detailed engineering analysis of serving DeepSeek-V4 on NVIDIA HGX B200 infrastructure, revealing that million-token context requires compressed KV layouts, prefix caching, and kernel maturity — not just a bigger context window. DeepSeek-V4-Pro (1.6T params, 49B activated) requires only 27% of single-token inference FLOPs and 10% of KV cache compared to V3.2, achieved through a hybrid Compressed Sparse Attention + Heavily Compressed Attention mechanism. This is the first major serving-side analysis of V4, framing long-context as an infrastructure challenge rather than a model capability story.