Elastic weight consolidation preserves general benchmarks and still loses formality and gender control.

Fine-tuning Llama 3.1 8B Instruct on Arabic-English and Spanish-English data, EWC held the general-benchmark drop to 1.7 points against 11.0 for standard fine-tuning. Formality and grammatical-gender instruction following degraded about as much as with no mitigation at all (arXiv). Mixing in control-task examples was the only thing that preserved them, and it didn't transfer to unseen prompts. Retention on MMLU-style benchmarks is a bad proxy for retention of the instruction-following you care about.