SoundnessBench: Can Your AI Scientist Tell Good Research Ideas from Bad Ones?

arXiv·high signal

Benchmark of 1,099 ML research proposals (from ICLR submissions) testing whether LLMs can judge methodological viability before committing resources. Addresses a critical bottleneck in autonomous AI research agents — idea quality filtering. Important for anyone building research automation or agentic coding systems that need to evaluate approach viability.