Encoding the Expected Answer as Executable Code Keeps Data-Science Agent Evals Valid on Live Data

arXiv 2609.16487·medium signalXBlueskyLinkedInCopy link

Skill-based Agentic Evaluation for Real-time Data Science Tasks (arXiv 2609.16487, submitted 15 Sep 2026) targets queries like 'what were last week's audience sizes' where the correct answer changes as the underlying data changes, so static references go stale and LLM-as-a-judge has nothing fixed to verify against. Their ground-truth-as-code approach encodes each expected answer as an executable reference function that recomputes from live data at evaluation time. A factoid-level, format-agnostic judge then decomposes both the agent response and the computed ground truth into atomic claims and scores precision, recall, and accuracy regardless of whether the agent answered in prose, a list, a table, or HTML.