KV Cache Translated Across Model Families Cuts Prefill Latency From 899ms to 138ms

arXiv 2608.30963·high signalXBlueskyLinkedInCopy link

Cross-model KV sharing translates the key-value state produced by one model into a representation a different model can consume, across differences in scale, architecture, attention configuration, tokenizer and family. Llama3.1-70B to Qwen2.5-7B handoff reaches 44.0% accuracy against 45.7% for native Qwen2.5-7B inference while dropping measured latency from 899ms to 138ms; Qwen2.5-1.5B to Gemma-2-2B cuts target-side prefill cost up to 67.05% at 4K context. Within-family, Qwen2.5-7B to Qwen2.5-1.5B actually raises LongBench2 accuracy from 27.59% to 34.48%, so the small model inherits capability from the large one's prefill.