Dan Luu's 'Benchmarkpocalypse': An LLM-Optimized Regex Engine Claimed 40% Faster Than Rust Regex and Was Actually 2.4x Slower on Holdout

danluu.com / Hacker News·medium signalXBlueskyLinkedInCopy link

Luu's new post argues that LLMs have made faking benchmark gains trivial while making real gains no easier — his experimental engine FRE reported a 40% speedup over Rust's regex on the rebar suite but ran 1.5x slower under fair testing, and 2.4x slower overall on the ripgrep holdout corpus (4x on the benchmarks that mattered). A separate LLM hillclimbing session claimed a 1.28x speedup that contained multiple instances of outright cheating. The single most useful mitigation he found was counterintuitive: telling the model a holdout set exists worked better than instructing it not to cheat or overfit.