Run a tiered eval stack: cheap distilled judges on 100% of traffic, full LLM judge on 10–20% sample, 100% in CI
Vinayak Talikot·high signal
Replace single LLM-as-judge with a multi-tier architecture: millisecond distilled evaluators (e.g. hallucination/grounding checks) run continuously on all production traffic, a frontier LLM judge does nuanced scoring on a 10–20% sample, and 100% of CI regression cases get the full judge before any prompt or model change ships. This controls eval cost while keeping coverage where regressions actually hurt.