CARE: Post-Hoc Conversion of Pretrained GQA Attention to MLA via Covariance-Aware Rank-Enhanced Decomposition

arXiv 2603.17946·medium signal

CARE enables upgrading pretrained models from grouped-query attention (GQA) to the more expressive and KV-cache-efficient multi-head latent attention (MLA) without full retraining. By preserving covariance structure during low-rank decomposition—rather than naively projecting—CARE retains model quality while achieving MLA's memory efficiency gains. Practitioners holding GQA-based models (LLaMA, Qwen lineage) can now retrofit MLA-style KV compression as a post-training operation.