Use an LLM judge to filter your fine-tuning dataset, not just to score outputs

Confident AI·medium signal

Beyond evaluation, an LLM judge can curate your fine-tuning set: run candidate training answers through a rubric-scoring judge and reject low-quality ones before SFT, so the model learns from reliably vetted examples. In 2026 judges agree with human reviewers about 85% of the time — higher than two humans agree with each other — making this filtering economical at scale. For domain work, fine-tune a small open judge (Prometheus-style, with chain-of-thought feedback) instead of paying frontier rates per evaluation.