A Time-Consistent Benchmark for Repository-Level Software Engineering Evaluation Eliminates Temporal Contamination

arXiv·high signal

Addresses a critical flaw in SWE benchmarks: temporal contamination between repository knowledge and evaluation tasks. Proposes a methodology that snapshots repos at time T0 and evaluates only on PRs merged after T0, ensuring agents can't benefit from data leakage. Tested on DragonFly and React repos with three Claude-family models, achieving file-level F1 of 0.81. Formalizes the matched A/B comparison framework that the SWE benchmark community has needed.