Tip: Give Long Agent Runs a Machine-Checkable Oracle, Not a Prose Definition of Done
arXiv / Daring Fireball·high signal
The common thread between Cherny's two-week Swift port (screenshot diff against the running Electron app) and the HANDBOOK.md results (36.2% compliance with prose policy) is that agents drift against text and hold against executable checks. Concretely: replace "match the existing UI" with a screenshot-comparison step the agent runs itself, and replace binding prose rules with a hook or tool-level assertion that fails the turn. HANDBOOK.md's third failure mode — details lost over extended sequences — is exactly the regime multi-hour autonomous runs live in.