IAR Bakes a Document Corpus Into Model Weights for Retrieval-Free QA and Keeps 12.1 Points More General Ability Than SFT

arXiv 2608.20281·medium signalXBlueskyLinkedInCopy link

arXiv 2608.20281 treats a fixed corpus as something to internalize rather than retrieve, running three separate post-training stages: Inject converts source documents into continuation, rewrite, and instruction-conditioned reconstruction objectives; Align adapts with answer-only QA supervision; Recover merges the domain-adapted model back with the base instruction model. Across Common Corpus and CCI with Llama, Phi, Qwen, and SmolLM, IAR beats vanilla SFT on all four reported metrics in 7 of 8 dataset-model settings, averaging +3.6 points of domain QA accuracy and +12.1 points of mean general performance on IFEval, MMLU, and MSBench. That second number is the point: the usual cost of fine-tuning knowledge in is wrecking everything else, and the merge step recovers most of it.