Skills
Plugin evals mock MCP servers from one markdown file per tool, and an `expect:` block that aborts the run with score 0
A plugin eval run never starts your real MCP servers. You write `evals/mocks/<server>/<tool>.md` whose body is the tool's return value, interpolate `{{input.<field>}}`, and put an `expect:` block in frontmatter that asserts the shape of what the agent sent. A violating call aborts the run at score 0 and records why, so a case can assert what your plugin asked the service to do rather than only what it replied; `type: agent` mocks get recorded under `mock-recordings/` and copied into `mocks/.replay/` so CI stops paying for them.
↳ Follow the thread