TraceBench Finds Agents Do Worse at Root-Cause Attribution When Asked to Write a Script Than When Answering Directly

arXiv 2608.27182·medium signalXBlueskyLinkedInCopy link

arXiv 2608.27182 generates controlled root-cause attribution tasks by simulating physical dynamical systems, where an agent receives time-series observations and must decide whether a system parameter was altered and which one. Evaluating four LLM agents on tasks from three interpretable mechanical systems, agents benefit substantially from domain context and explore data mainly through numerical console output rather than visualizations. Notably they perform worse when required to emit a Python script mapping each sample to a predicted label than when submitting predictions directly, which is a concrete argument against forcing programmatic output formats on observability agents. Datasets, trajectories and a leaderboard are at tracebench.github.io.