Entity trail

AgentCore Evaluations

Source-backed findings, relationship evidence, citations, and briefing history from the public MindPattern archive.

Briefing refs

Findings

Edges

Sources

Corpus findings

2026-09-12 / rss-researcherAgentCore Evaluations ships 16 evaluators for the failure mode infrastructure monitoring cannot seeAWS published a 11 September walkthrough pairing its DevOps Agent with AgentCore Evaluations, aimed at a specific gap: 'An agent can successfully invoke Amazon Bedrock, call every tool without errors, and return a response while completely misunderstanding what the user needs.' Evaluations provides 13 LLM-as-a-Judge evaluators (Helpfulness, Correctness, Goal Success Rate, Tool Selection Accuracy, Faithfulness among them) plus 3 deterministic trajectory matchers. Online evaluation samples production traffic at a configurable 0.01-100% asynchronously and pushes metrics to CloudWatch over OpenTelemetry.
2026-09-09 / rss-researcherAWS shows a copyable pattern for failing a pull request when agent behavior regressesThe walkthrough wires Bedrock AgentCore Evaluations into GitHub Actions: CDK deploys a Strands agent runtime and an OAuth-protected MCP server behind a shared Cognito pool, the workflow invokes the agent with test prompts, pulls traces from CloudWatch and scores them, then destroys everything with if: always(). Four built-in evaluators run at a 0.8 default threshold (GoalSuccessRate, Correctness, ToolSelectionAccuracy, ToolParameterAccuracy) and any score below it fails the PR. Two details save real debugging time: trace propagation takes 30 to 90 seconds so the script retries every 30 seconds for up to 10 minutes, and CI machine-to-machine tokens carry scopes but no role claims, so role enforcement silently does not apply in the pipeline.
2026-08-28 / rss-researcherBedrock AgentCore Evaluations scores agents from OpenTelemetry spans, so the framework stops matteringAnnounced 2026-08-26, the service reconstructs agent sessions from three OTel span types, invoke agent, inference and execute tool, then applies the same scoring regardless of SDK. It names Strands Agents, LangGraph, OpenAI Agents SDK, LlamaIndex, Google ADK and the Claude Agent SDK explicitly, and covers anything emitting spans under the `opentelemetry.instrumentation.*` or `openinference.instrumentation.*` scope prefixes through generic classification. Built-in evaluators are GoalSuccessRate at session level, Correctness and Helpfulness at trace level, plus custom LLM-as-a-judge at session, trace or tool level; AWS published no pricing or regions.

Graph relationships

RELEASED
AWS
AWS published AgentCore Evaluations with 16 LLM evaluators.
Source finding
USES
Amazon Bedrock
AgentCore Evaluations pairs with Amazon Bedrock for agent evaluation.
Source finding
RELEASED
AWS -> AgentCore Evaluations
AWS published AgentCore Evaluations with 16 LLM evaluators.
Source finding

Source trail

AWS AWS Machine Learning Blog AWS Machine Learning Blog

Graph sources

entity graphfindings textkg edgeskg entitiesnewsletter issues

Fetching from the wire…