Tangram: Non-Uniform KV Cache for Efficient Multi-turn LLM Serving

arXiv·medium signal

Multi-turn serving suffers because the KV cache grows linearly with conversation length, pressuring GPU memory and bandwidth. Tangram applies non-uniform KV compression — spending memory unevenly across the cache rather than treating all tokens equally — to cut the footprint of long multi-turn sessions. Practical for anyone self-hosting models behind a chat or agent loop. Published 2026-06-04 (cs.LG).