FastDMS Achieves 6.4X KV-Cache Compression Running Faster Than vLLM BF16/FP8

r/LocalLLaMA / arXiv·medium signal

Researchers from NVIDIA, University of Warsaw, and University of Edinburgh published FastDMS, a production-ready implementation of Dynamic Memory Sparsification that achieves 6.4x KV-cache compression while running faster than vLLM's BF16 and FP8 baselines. The technique dynamically prunes attention heads at inference time based on contribution scores, enabling dramatically longer context windows on existing hardware without accuracy loss. Posted to r/LocalLLaMA (91 upvotes, 17 comments).