'Agentic auto-research is fuzz testing': the generate-and-rank paradigm is the wrong control loop

arXiv·medium signalXBlueskyLinkedInCopy link

He et al. argue the field's response to research agents outrunning validation — scale the proposer, rank more samples with a learned judge — misses that the real problem is sparse feedback. A research agent runs the same loop as a greybox fuzzer (propose, execute, observe, choose next), and fuzzers make progress because coverage gives a cheap dense signal on every execution that then directs mutation, not just post-hoc ranking. The prescription for anyone building research or exploration agents: instrument each experiment to emit a cheap progress signal before final validation, use that signal to pick the next intervention rather than to sort finished runs, and keep validation evidence protected from adaptive reuse.