Best AI Agent Fully Solves Just 31.5% of SRE-Bench

SRE-Bench, a new binary-analysis benchmark, stumped every frontier model tested, with the best agent fully solving just 31.5% of its instances, per the paper. The real finding sits underneath that score: strong source-code security skill doesn't carry over to reading compiled binaries. That gap matters for anyone betting on AI to handle reverse engineering.

Researchers wrote 19 real-world-scale programs from scratch, averaging 16.9K lines each. They added 44 anti-analysis primitives to keep the material out of any training set. That work produced 262 binary instances and 1,572 tasks, each graded by a deterministic pass-fail check, not a human judge.

Five models took the test: GPT-5.6-sol, Claude Opus 5, GPT-5.5, Grok-4.5, and GLM-5.2. The top model scored 61.4% on a partial-credit, per-instance measure but fully solved only 31.5% of instances outright, according to the paper.

Where the blindness shows up

Agents were also insensitive to compiler optimization and static linking, changes that make binaries far harder for human reverse engineers to read, the paper found.

A model that can't tell when a binary got harder to crack is a worse problem than a 31.5% score. Anyone testing an agent for reverse-engineering work must run it against optimized, statically linked builds, not just the easy case. The paper is part of the August 16 briefing.