Research
Thinkingbox Finds the Best Agent Hits 65.36% Pass@1 on Business Workflows but Only 25.25% When Reliability Is Required
Thinkingbox is an MCP-compatible sandbox for tool-agent-user interaction with isolated sessions, complete execution traces, and outcome evaluation against terminal backend state. Its benchmark contains 507 policy-conditioned workflows across retail, hospitality, auto insurance, neobank internal IT, and consulting IT/HR support, each graded by executable checks that accept valid trajectories while rejecting wrong, missing, or extra effects. Across proprietary and open-weight models the strongest reached 65.36% pass@1 but only 25.25% under the repeated-attempt reliability measure, which is the number that matters for anything running unattended.
↳ Follow the thread