ProgramDistill makes agents infer features by interacting with a working app instead of reading an issue, and success falls from 100% to 32% as restoration depth goes 1 to 8
Most coding benchmarks hand the agent a written issue. ProgramDistill instead factors 26 fully functional web applications into features, mines 1,975 replay-verified behaviors, and auto-constructs 4,063 tasks with no human labeling, so the agent must discover the target behavior by driving the reference app. Across nine frontier agents, GPT-6 Astra and Claude Opus 5 reach 49.2% and 28.8% on cumulative workflows in full-application reconstruction; in partial reconstruction, success drops from 100% to 64.0% and from 96% to 32% as restoration depth rises from 1 to 8. The depth curve is the useful number for builders: agent reliability collapses with the number of interdependent missing pieces, which argues for slicing agent work into shallow independently verifiable units rather than handing over a deep restoration.
↳ Follow the thread