SRE-Bench: 5,000 expert hours to build the first contamination-free reverse-engineering benchmark, and frontier agents solve only 31.5% of it

arXiv·high signalXBlueskyLinkedInCopy link

Researchers built 19 private, real-world-scale programs from scratch (averaging 16.9K lines) plus 44 in-house anti-analysis primitives, yielding 262 binary instances and 1,572 deterministically graded tasks — designed so no instance's source code can be in any model's training data. Across GPT-5.6-sol, Claude Opus 5, GPT-5.5, Grok-4.5 and GLM-5.2, the strongest model scored 61.4% per instance and fully solved just 31.5% of instances. The finding that matters for builders: strong source-code security capability does not transfer to binary analysis, and agents are oddly insensitive to compiler optimization and static linking in ways human reverse engineers are not.