Most of Qwen3.8-Flash-Next's KV cache can live in system RAM, giving 1M context on three 3090s
An r/LocalLLaMA builder (128 upvotes) patched vLLM to offload most of the KV cache to host RAM for Qwen3.8-Flash-Next and reports 1M context on 3x RTX 3090: about 80 tok/s at short context, dropping to roughly 60 tok/s once QSA hits its 2,048-token budget and then staying flat as context grows, ~150 tok/s at four concurrent requests, and 3,701 tok/s prefill at 248K. The argument is architectural, not a trick: decode speed is bound by the per-step memory read, and in the qwen4_exp family the layers that read the whole cache every step are sparse enough that the rest can sit across PCIe. Patches and the model are on the author's Hugging Face page, and the claim is that this generalizes to any qwen4exp-based model, which is what Qwen's next local releases are built on.
Source
↳ Follow the thread