AWS ships 16 evaluators for agents that fail silently

AWS paired its DevOps Agent with AgentCore Evaluations on September 11. The new tool targets a specific failure. An agent can call Bedrock, run every tool without an error, and still hand back an answer that misses what the user asked for. Standard monitoring never catches that. It only tracks whether calls succeeded.

AgentCore Evaluations adds 13 LLM-as-judge evaluators, including Helpfulness, Correctness, Goal Success Rate, Tool Selection Accuracy, and Faithfulness. Three deterministic trajectory matchers check the real sequence of steps an agent took. Online evaluation samples live production traffic at a configurable rate, from 0.01% up to 100%. It runs asynchronously, so it doesn't slow production down. Results push to CloudWatch over OpenTelemetry, per AWS's DevOps Agent walkthrough.

A dashboard full of green checks only means the agent didn't crash. It says nothing about whether the agent understood the request. That's the failure mode. A confidently wrong answer, with every log line clean.