Research
A Foundation-Model Oracle Reading Git Diffs Found 13 Real Bugs in Rope's Python Refactorings — 12 of 13 Reports Accepted by Maintainers
Posted 2026-08-10, this work uses a foundation-model oracle over git-style diffs to detect behavioral changes introduced by supposedly behavior-preserving Python refactorings. Reusing 1,152 refactoring attempts from a prior study and analyzing 217 resulting transformation pairs, the oracle surfaced 13 distinct bugs across seven refactoring types. All were filed with the Rope maintainers, and issue-tracker evidence shows 12 of the 13 reports accepted. This is a rare case of LLM-based analysis producing externally validated defect reports rather than self-scored benchmark numbers.
↳ Follow the thread