Fetching from the wire…
01
02
03
04
05
06
07
08
Public story · 2026-06-30 · high
Agents can land on the right answer through an unsafe path, and scoring only the output misses it.
Why now: This lands as one specific fix for a known blind spot: scoring answers without ever checking how an agent got there.
Agent-as-judge evaluation checks an AI system's intermediate steps and tool calls, not just its final output, per a guide from Confident AI.
That catches the failure output-only scoring misses: an agent that reaches the right answer through a wrong or unsafe path. In production, that's the run that blindsides a team. It looks clean. It wasn't.
The method runs a cheap agent-as-judge on 100% of traffic: every trace, every tool call, every intermediate step, scored automatically. Teams then pull a small human-reviewed sample, 1-2% of traffic, and check it against the judge's calls. If the two agree, the automated grading holds up.
The 1-2% human sample is the part most teams will cut first. When teams skip it, the judge's blind spots go unchecked: agents start passing on the exact steps a human reviewer would have flagged. Watch for teams that report a judge's pass rate but never publish the human-agreement number. That's the tell they skipped the check.
This lands as one specific fix for a known blind spot: scoring answers without ever checking how an agent got there.
Each link below shares sources, entities, or timing with this story.
output uses Claude Code / Shared entity: Score / Shared topic / What happened next / Tension
Linked by a graph relationship (output uses Claude Code); both cover Score; overlapping topics (agent, answer, failure).
output uses Claude Code / Shared entity: Output / Earlier coverage / Tension
Linked by a graph relationship (output uses Claude Code); both cover Output; earlier Output coverage from 2026-04-02.
output uses Claude Code / Shared topic / Tension
Linked by a graph relationship (output uses Claude Code); overlapping topics (actually, agent, call); pushes against this story (but).
output uses Claude Code / Shared entity: Output / What happened next
Linked by a graph relationship (output uses Claude Code); both cover Output; picks up the Output thread on 2026-08-17.
Linked by a graph relationship (output uses Claude Code); both cover Output; picks up the Output thread on 2026-08-15.
output uses Claude Code / Shared topic
Linked by a graph relationship (output uses Claude Code); overlapping topics (actually, agent, answer, call).
output uses Claude Code / Shared entity: Output / Earlier coverage
Linked by a graph relationship (output uses Claude Code); both cover Output; earlier Output coverage from 2026-06-12.
output uses Claude Code / Shared topic / Tension
Linked by a graph relationship (output uses Claude Code); overlapping topics (actually, agent); pushes against this story (against).