Fetching from the wire…
01
02
03
04
05
06
07
08
Public story · 2026-08-25 · high
The judges credit steps for looking related to the outcome, not for causing it, across 499 file-localization episodes.
Why now: The paper posted this August, aimed squarely at the full-trajectory LLM judging setup most agent benchmarks use to score coding-agent runs.
Full-trace LLM judges of coding agents measure relevance instead of causal contribution, per a new framework for attributing agent steps. That matters for teams that use judge scores to decide which agent version ships. Across 499 file-localization episodes in 12 repositories, the paper found next-step choices tracked execution history more than code structure.
The framework separates three things process evaluations usually blur together: action prediction, task uncertainty and step-level causal attribution. It scores each step with a replay-based estimator built over a structural causal model of agent execution. Task uncertainty, the paper found, concentrates at the level of the whole task rather than individual steps.
The sharper result concerns the judges themselves. When a judge reads a full trace, including steps that come after the one it's grading, it develops what the paper calls collider bias. It credits steps that look related to the eventual outcome, whether or not they caused it.
For teams building or buying agent evals, the fix isn't a better prompt for the judge. Swapping full-trajectory grading for step-level attribution changes which runs score best.
Each link below shares sources, entities, or timing with this story.
Simon Willison released LLM / Shared entity: Full / Same source domain / Earlier coverage / Tension
Linked by a graph relationship (Simon Willison released LLM); both cover Full; reported by the same outlet (arxiv.org).
Simon Willison released LLM / Shared entity: LLM / Shared topic / Earlier coverage
Linked by a graph relationship (Simon Willison released LLM); both cover LLM; overlapping topics (agent, task).
Simon Willison released LLM / Shared entity: Full / Same source domain / Earlier coverage
Linked by a graph relationship (Simon Willison released LLM); both cover Full; reported by the same outlet (arxiv.org).
Simon Willison released LLM / Shared entity: LLM / Earlier coverage / Tension
Linked by a graph relationship (Simon Willison released LLM); both cover LLM; earlier LLM coverage from 2026-08-16.
Linked by a graph relationship (Simon Willison released LLM); both cover LLM; earlier LLM coverage from 2026-07-27.
Linked by a graph relationship (Simon Willison released LLM); both cover LLM; earlier LLM coverage from 2026-06-19.
Linked by a graph relationship (Simon Willison released LLM); both cover LLM; earlier LLM coverage from 2026-06-18.
Simon Willison released LLM / Shared entity: Full / Earlier coverage / Tension
Linked by a graph relationship (Simon Willison released LLM); both cover Full; earlier Full coverage from 2026-05-09.