Fetching from the wire…
01
02
03
04
05
06
07
08
Agents2026-08-19 · source-backed
StartupBench (arXiv 2608.17800) inverts benchmark construction. Instead of researcher-invented tasks, the authors studied AI startup products with demonstrated market adoption, their workflows, and their users, then translated those into complete deliverable-oriented tasks with fine-grained rubrics. Under a unified harness, the strongest model completes roughly 30% end to end, while making substantial partial progress on many tasks. The named failure sources: complex instruction following and domain-specific expertise. Neither is a raw capability problem. Both are things you fix with rubric-shaped decomposition and injected domain context, which means the missing 70% is addressable by product engineering rather than by waiting for the next model.
Each link below shares sources, entities, or timing with this story.
Shared entity: Under / Same source domain / Shared topic / Earlier coverage
Both cover Under; reported by the same outlet (arxiv.org); overlapping topics (adoption, agent, author, model, task).
Shared entity: Under / Same source domain / Shared topic / Earlier coverage / Tension
Both cover Under; reported by the same outlet (arxiv.org); overlapping topics (agent, capability, model).
Both cover Under; reported by the same outlet (arxiv.org); overlapping topics (actually, agent, author).
Shared entity: Under / Same source domain / Shared topic / Earlier coverage
Both cover Under; reported by the same outlet (arxiv.org); overlapping topics (actually, agent, model).
Both cover Under; reported by the same outlet (arxiv.org); overlapping topics (actually, agent, benchmark).
Shared entity: Instead / Same source domain / Shared topic / Earlier coverage
Both cover Instead; reported by the same outlet (arxiv.org); overlapping topics (agent, task, workflow).
Same source domain / Shared topic / Tension
Reported by the same outlet (arxiv.org); overlapping topics (agent, benchmark, best, capability, their); pushes against this story (but).
Shared entity: Under / Same source domain / Shared topic / Earlier coverage
Both cover Under; reported by the same outlet (arxiv.org); overlapping topics (benchmark, model).