DepthKV: Layer-Dependent KV Cache Pruning Cuts Memory Bottleneck for Long-Context LLM Inference

arXiv·high signal

Dehghanighobadi and Fischer propose DepthKV, a KV cache pruning method that assigns different pruning budgets per Transformer layer rather than the uniform approach used by existing methods. The key insight is that different layers exhibit different attention patterns — some need full context retention while others can be heavily pruned with minimal quality loss. This directly addresses the primary memory bottleneck for long-context inference where KV cache grows linearly with sequence length.