EvalCEGAR Evolves a 55-Line Python Metric That Beats 15 Hand-Written Operators and Charges No Model Calls

arXiv 2608.18744·medium signalXBlueskyLinkedInCopy link

Asking a model to write evaluation operators directly does not work: 183 candidates in arXiv 2608.18744 (2026-08-19) realized only 96 distinct behaviours from one narrow region of the space. EvalCEGAR instead imports counterexample-guided abstraction refinement from program verification, treating the operator pool as an abstraction and searching for collisions, two answers scored identically where one is correct and one is not, and using that pair as the authoring request. On MBPP+ and HumanEval+ the loop writes a 55-line operator that closes 15.4% of the gap to a perfect filter on 428 unseen tasks (+0.0065, p=0.0010) at a quarter of the flag rate of the best hand-written operator, while the authors' 15 hand-written operators applied together lose accuracy; an LLM judge ties the delta but charges a model call per candidate forever.