smevals: Willison and Jesse Vincent's Prime Radiant Ship a Small Eval Harness That Grades Models, Prompts and Harnesses as Three Separate Variables
Announced July 31, smevals is a compact evaluation framework built with Jesse Vincent's Prime Radiant applied AI research lab, structured around a four-layer model: an eval contains tasks, each task runs against one or more configs (a model plus parameters like system prompt), each execution produces a run, and graders then apply checks — which can be simple string matches, custom scripts, or other models acting as judges. It answers questions like 'how good is that model at generating SVGs?' and emits both an interactive dashboard and a shareable static HTML report. The design point worth stealing is separating the harness from the model and the prompt as independent axes, since in agentic systems the harness is frequently the actual bottleneck.
↳ Follow the thread