RepoProbe Uses GitHub Discussions Instead of Bug Reports to Catch Models That Edit Code Before Understanding It

arXiv 2608.04783·medium signalXBlueskyLinkedInCopy link

Repo-level benchmarks built on GitHub Issues let models pattern-match on error logs and skip genuine comprehension, which under-measures what the authors call Edit Bias — proposing code modifications before understanding the existing architecture. RepoProbe instead draws open-ended architectural Q&A from GitHub Discussions and replaces high-variance LLM-as-a-Judge scalar scoring with a Checklist-Based Verification Protocol that decomposes each answer into atomic, verifiable facts. Evaluating SOTA LLMs reveals a persistent gap between answer clarity and evidence-grounded technical correctness, quantitatively confirms edit bias, and shows the checklist protocol is more reliable than scalar rating.