Boundary-Bench: turn on your enterprise's actual security controls and agents lose 18 points

Every coding agent leaderboard number you've seen was produced in conditions your security team would reject on sight.

Researchers from Accomplish AI and NYU open-sourced Boundary-Bench on August 5. The setup: 12 frontier agent harnesses, roughly 10,000 runs, 89 Terminal-Bench 2.1 tasks, executed inside Daytona sandboxes hardened with NIST-derived Network × Filesystem × Privilege policy tiers. What makes it credible is the enforcement mechanism. Not a shim, not a mock, not a wrapper that tells the agent "you can't do that." Real nftables rules, read-only bind remounts, setpriv with no_new_privs, and Landlock. Denials surface to the agent as ordinary EROFS and EPERM errors, exactly like they would on a locked-down corporate box.

The numbers. Codex on GPT-5.6 Sol leads unrestricted at 83.9%. Under the strictest tier, the leader changes: Grok Build on Grok 4.5 tops out at 74.9%. Claude Code on Sonnet 5 shows the largest degradation at 18.3 points. And costs inflate up to 167.3% because the agent burns turns rediscovering what it isn't allowed to do.

That cost figure is the bridge to the SAP story. Everyone's token spend projections came from unrestricted benchmark conditions. Then you deploy into an environment with EDR, SASE, and DLP, and your agent spends a third of its budget failing into permission walls and retrying. The security tax and the token tax are the same tax, measured from opposite sides.

The leader flip matters more than the absolute scores. If you picked your harness on an unrestricted leaderboard, you may have picked the one that degrades worst under your own controls. That's not a hypothetical ranking exercise. That's a procurement decision made on numbers that don't survive contact with production.

Do this before you sign anything: re-run your harness eval inside your actual sandbox policy. The repo is open, the methodology is documented, and Terminal-Bench 2.1 tasks are standard. You need maybe a day to get a real number for your environment. Compare it to the vendor's number and see how much of the gap is yours to eat.

Novee Security's Elad Meged made the complementary argument at Black Hat, via Novee: the vulnerable layer is the harness, not the model. The surrounding code that manages tool permissions, execution, and sandboxing. Boundary-Bench measures the same layer from the performance side. Two independent teams landed on the same conclusion in the same week, which is usually how you know something's real.