MIT Technology Review: AI Benchmarks Are Broken — Models Tested in Isolation While Deployed in Teams

MIT Technology Review·medium signal

MIT Technology Review publishes a deep analysis arguing that the fundamental methodology behind AI benchmarking is flawed: models are tested against individual humans on isolated problems with clear answers, but deployed within human teams and organizational workflows where performance dynamics are entirely different. Even newer dynamic evaluation methods still evaluate AI outside the human contexts where it must actually perform. The piece calls for organizational-context benchmarks, not just model-vs-human comparisons.