Calibrate LLM-as-judge before trusting it, and budget evals at 30–50% of inference cost

DeepEval·medium signal

A calibrated LLM judge now agrees with human reviewers ~85% of the time — higher than two humans agree with each other — but uncalibrated judges are unreliable, so anchor them to a small human-labeled set first. Plan the spend realistically: judge evaluation runs roughly 30–50% of your inference cost (a $4K/month inference budget implies $1.2K–$2K/month in eval). The payoff is referenceless, continuous scoring (relevancy, faithfulness, task completion) over live traffic.