Pattern: Practitioners Reject Leaderboards — 'Benchmarks Are Bullshit' Thesis Gains Traction

paddo.dev·medium signal

paddo.dev argues that AI benchmarks are structurally broken: HumanEval's 164 problems are trivially overfittable, GSM8K saturated from 50% to 95%+ in two years, MMLU scores vary 5-15% depending on evaluation setup, and Berkeley researchers built an agent that games benchmarks without improving capability. Chinese models (DeepSeek, Qwen) post strong scores but hallucinate APIs at higher rates in practice. The prescription: rotating test sets, 50+ turn evaluations, and 'the vibe check' from experienced developers — exactly what leaderboards can't capture.