Benchmarked against workflows real users already pay for, the best agent finishes only ~30% end to end

arXiv 2608.17800·medium signalXBlueskyLinkedInCopy link

StartupBench inverts the usual benchmark construction: instead of researcher-invented tasks, the authors studied AI startup products with demonstrated market adoption, their workflows and their users, then translated those into complete deliverable-oriented tasks scored with fine-grained rubrics. Under a unified agent harness, even the strongest evaluated model completes only approximately 30% of the benchmark despite making substantial partial progress on many tasks. The named failure sources are the actionable part for anyone shipping agent products: complex instruction following and domain-specific expertise - meaning rubric-shaped task decomposition and injected domain context are where the remaining 70% lives, not raw model capability.