LangChain measured LLM judges against Jev in LangSmith: 92x to 913x more score variance, 80x the cost

LangChain·high signalXBlueskyLinkedInCopy link

LangChain published a head-to-head on 2026-09-20 evaluating a weather agent across five scenarios, comparing Jev against three LLM judges on accuracy versus a human oracle, run-to-run consistency, latency and cost. Jev's mean per-case variance was 0.0000149, against 92x higher for Claude, 433x for Luna and 913x for Terra; it matched the human oracle on all 500 repeated binary pass/fail evaluations while Terra hit 99.8%, Luna 96.4% and Claude 80.0%. Jev averaged 0.44s and $0.00035 per call, $0.34 total against $28.17 for the Claude judge. The practical takeaway for anyone running an eval suite is that judge non-determinism, not judge accuracy, is what makes LLM-judged tests unusable as regression gates.