LangChain shipped an eval-engineering skill that interviews you instead of one-shotting evals — and it writes Harbor-format tasks

LangChain Blog·medium signal

A July 22 post describes an installable skill that reads your repo to map the agent surface (prompts, models, tools, skills, hooks), pulls production traces via langsmith-cli, then interviews you to converge on evals you approve one at a time. Output is a containerized Harbor task per eval under `evals/<task-id>/` with `instruction.md`, an `environment/` Dockerfile, and `tests/` verifier logic. The reported lesson is that single-pass generation produced weak evals — the good ones came from inspecting both the agent's and the verifier's trajectories to catch reward-hacked shortcuts.