Test your RAG under corpus mutation, not against a static snapshot — metamorphic oracles score F1 0.927–1.000 where the best RAGAS metric manages 0.570

arXiv 2607.26843·medium signalXBlueskyLinkedInCopy link

RAGAS-style evaluation checks correctness against a frozen snapshot, which means routine document updates, factual corrections, and noise can silently break a production RAG system without moving any dashboard. This ASE 2026 paper (July 29) defines a fault taxonomy plus 11 mutation operators that perturb at both the pre-chunk retrieval-index level and the post-chunk retrieved-context level, then measures consistency. Across five datasets and 28,000+ mutants, metamorphic violation rates ran 4.9–10.2%, and the metamorphic oracles detected them at F1 0.927–1.000 versus 0.570 for the best RAGAS metric — a large enough gap that snapshot-only evaluation should be treated as insufficient for any evolving corpus.