Build a trace-to-eval flywheel: convert failed production traces into eval cases and run cheap distilled scorers online

Confident AI·medium signal

The 2026 eval stack splits into cheap distilled evaluators running continuously on production traffic and a full agent-as-judge run selectively for deep verification — because a model call per judgment is too costly to run on every turn. Production traces that fail an online scorer are automatically promoted into the offline eval suite, so the suite grows from real user behavior and future regressions are caught without hand-authoring cases. This closes the loop between observability and evaluation instead of maintaining them as separate artifacts.