Skills
Run a tiny fine-tuned judge model on live production traffic instead of a frontier judge
Teams are moving LLM-as-judge out of offline eval and onto real-time samples of production traffic, using small fine-tuned judges (e.g. Galileo's Luna at ~440M params) that run in milliseconds at a fraction of a frontier model's per-call cost while still flagging hallucinations and factuality issues. You set quality thresholds and alert when a live metric drops, catching regressions the moment they ship rather than in a weekly eval. The builder move: fine-tune or adopt a small dedicated judge for one or two high-value metrics and wire it to a random-sample monitor, reserving expensive frontier judges for offline deep-dives.
Source
↳ Follow the thread