Skills
Balanced Eval Problem Sets: Always Test Both Trigger and Non-Trigger Cases to Prevent Behavioral Drift
Anthropic's eval design guide mandates pairing every positive-case test (when agent should act) with a negative case (when it should not), using Claude.ai web search as the example: test 'what's the weather?' (should search) alongside 'who founded Anthropic?' (should answer from knowledge). Without negative cases, optimization maximizes activation, turning a well-tuned web-search agent into one that searches for everything. This two-sided pattern prevents over-triggering and under-triggering bugs that appear late in production.
↳ Follow the thread