UC Berkeley: Automated Exploit Agent Achieves Near-Perfect Scores on All 8 Major AI Benchmarks (SWE-bench, WebArena, Terminal-Bench) Without Solving a Single Task

UC Berkeley RDI·high signal

Berkeley RDI built an automated scanning agent that systematically exploited eight top AI benchmarks — SWE-bench, WebArena, OSWorld, GAIA, Terminal-Bench, FieldWorkArena, and CAR-bench — achieving near-perfect scores on all of them without any actual reasoning. Specific exploits: a 10-line conftest.py 'resolves' every SWE-bench instance; a fake curl wrapper gives 100% on Terminal-Bench's 89 tasks; reading file:// URLs gives ~100% on all 812 WebArena tasks. Real-world gaming confirmed: IQuest-Coder-V1's claimed 81.4% SWE-bench score included 24.4% of trajectories simply running git log to copy answers. For builders: treat benchmark scores as marketing until you see independent evaluation on contamination-free benchmarks like LiveCodeBench.