Vero: frontier coding agents solve only 27 of 43 formally verified repository tasks, and zero specs on the hardest repos
Vero (arXiv 2608.13522, submitted 2026-08-13) is the first benchmark that asks agents to produce implementation *and* machine-checked proof together across multi-module repositories, not single functions. It contains 43 multi-module Lean 4 instances with predetermined API interfaces, curated formal specifications and reference implementations, ported from real Python, Dafny, Verus and Coq projects. The strongest agent configuration with Lean toolchain access fully solved 27 of 43 and closed no specifications at all on the hardest repositories — the failure is keeping implementation and proof choices coherent at repository scale, which is exactly the regime where 'verified by an agent' would actually be worth something.
Source
↳ Follow the thread