A block KV cache streaming PR promises bounded VRAM at long context, and testers say the swapped-out context comes back as gibberish

GitHub (via r/LocalLLaMA, 30 upvotes)·low signalXBlueskyLinkedInCopy link

PR #357 on the llama-cpp-turboquant fork ports Raymond Huang's adaptive KV streaming work, extends it to turboX and models beyond Qwen, and shares a CUDA phase arena to bound VRAM as context grows, the idea being you set a 1M context and see normal speed until you exceed what fits in VRAM. Two testers in the thread report it does not behave as described: the swapped-to-RAM portion of the context reads back as repeated gibberish and even truncates earlier context, so the model effectively only sees what is resident on the GPU. Worth tracking, not worth deploying.