413-Config KV Cache Quantization Benchmark: 6-Bit KVarN Beats q8_0 on Quality While Saving 488 MiB

anbeeld.com via r/LocalLLaMA (84 upvotes / 33 comments)·medium signalXBlueskyLinkedInCopy link

A practitioner benchmark tested 413 cache configurations across Qwen 3.6 27B (238) and Gemma 4 31B (175) using KLD against BF16 baselines on BeeLlama.cpp v0.4.0. KVarN — which rotates and normalizes each 128-token group before quantizing — beat standard widths by 2.6–12.6% on Qwen at matched suffixes: kvarn6 hit 0.000889 median KLD versus q8_0's 0.000901 while using 488 MiB less memory, and kvarn4 hit 0.001111 versus q4_0's 0.001206. The 'precision tail 1024' finding — keeping the most recent 1024 tokens in exact precision while older context stays quantized — is where the KLD curves flatten on Wikitext, and is the single most actionable knob here for anyone running long-context local inference.