ICAE-Bench evaluates coding agents as interactive project builders rather than instruction-followers

arXiv·medium signal

Submitted July 23 by a team spanning Meituan and SMU (including David Lo), ICAE-Bench targets the gap between completing fully-specified code and transforming incomplete product intent into working software. It grounds tasks in real open-source repositories with executable behavior, uses User Agent Data to make interaction simulation reproducible, and scores with standardized black-box tests plus multi-dimensional diagnostics across functional correctness, semantic similarity, structural fidelity, design quality, and interaction quality. The reproducible-user-simulation piece is the transferable idea — most agent evals still can't replay ambiguous requirement negotiation deterministically.