LangChain measured judge variance and the number is 913x

The reason your LLM-judged eval suite can't gate a deploy isn't that the judge is wrong. It's that the judge is inconsistent, and LangChain published the measurement on September 20.

They evaluated a weather agent across five scenarios, comparing a small decision model against three LLM judges on accuracy against a human oracle, run-to-run consistency, latency and cost. The decision model's mean per-case variance was 0.0000149. Claude as judge: 92x higher. Luna: 433x. Terra: 913x.

Accuracy across 500 repeated binary pass/fail evaluations: the decision model matched the human oracle on all 500. Terra got 99.8%, Luna 96.4%, Claude 80.0%. Cost for the run was $0.34 against $28.17 for the Claude judge, at 0.44 seconds and $0.00035 per call.

I've built eval suites with LLM judges and watched them flake. You change nothing, rerun, and three cases flip. Then you spend an afternoon deciding whether your change broke something or the judge had a mood. The usual diagnosis is "the judge needs a better rubric," so you rewrite the rubric, and it flakes differently.

That diagnosis is wrong, and the variance numbers explain why. A judge that's 80% accurate but deterministic is a usable regression gate, because you can characterize its 20% and route around it. A judge that's 99.8% accurate and non-deterministic is not, because you can't tell a real regression from a resample. Variance, not accuracy, is what makes an eval gate.

Openlayer's jevals arrived the same weekend with the cost side of the same argument. Their published comparison puts Ragas at 6 to 11 requests per sample, $2.60 per sample, 22 to 35 seconds for 20 samples, against one request at $0.03 per sample and 0.8 seconds on a decision model, with p50 latency of 244ms. Local backends drop the cost to zero. It's a week-old alpha and the numbers are vendor-run, so discount accordingly, but the architecture claim holds independent of the benchmark: one request per trace instead of six to eleven.

Arize's September 18 analysis gives the honest tradeoff. Jev at 68% accuracy for $0.0004 and 0.4 seconds per case, against Opus 5 at 73% for $0.18 and 38 seconds. You lose five accuracy points and you lose the explanation. For a CI gate that runs on every commit, I'll take determinism and no explanation. For debugging why a gate failed, I want the frontier model and its reasoning.

So run both. Decision model gates the build, frontier model explains the failure when the gate goes red. That's cheap enough to run on every commit and expensive enough to be useful exactly when you need it.

The pattern generalizes past evals. Three unrelated projects hit Hacker News in 48 hours attacking the same assumption, that a frontier model must make the yes/no call: jevals for eval gating, Kev as open weights, and a CoreML port running a multilingual decision model fully offline on an M4 Mac at a 560MB physical footprint. Any product whose cost line is frontier-model classification should be repricing this week.