Can Coding Agents Reproduce Findings in Computational Materials Science? First Systematic Evaluation
arXiv·medium signal
Researchers evaluated autonomous coding agents on reproducing computational materials science results (arXiv 2605.00803), finding that while agents achieve strong SWE-bench scores, scientific reproducibility requires different capabilities — understanding domain constraints, interpreting physical results, and making judgment calls about numerical precision. This benchmarks coding agents on real scientific workflows rather than software engineering tasks alone.