RepoReasoner: Best Frontier Model Reaches Only 69.1% Pass@1 on Cross-File Execution Reasoning Even With Oracle Context

arXiv 2607.25996·high signalXBlueskyLinkedInCopy link

RepoReasoner (arXiv 2607.25996, 2026-07-28) moves code-reasoning evaluation from the function level to the repository level, testing Output Prediction (stateful execution reasoning across files) and Call Chain Prediction (architectural dependency understanding under noisy context). Ground-truth call chains come from dynamic tracing of real pytest executions, with LLM-based I/O rewriting to suppress memorization. Across seven state-of-the-art LLMs the best model manages 69.1% Pass@1 on Output Prediction even under oracle context; Call Chain Prediction shows high precision but low recall, and longer contexts do not consistently help because of added noise. Performance drops on rewritten data confirm partial reliance on memorization.