Your LLM judge works in the test harness. Here's why it fails in production.

The evaluation pipeline looked solid. The judge model correlated with human raters on 84% of your held-out test set. You shipped it. Three months later it's been scoring production traffic every day, and nobody has checked whether those scores still mean anything.

That gap, between the test harness and six months of production traffic, is the part almost nobody writes about. Tutorials cover setup. Academic papers cover bias taxonomy. Vendor blogs explain how their platform handles position bias and verbosity preference. What they skip is the operational layer: what breaks after you go live, and how you catch it before it costs you.

The calibration you forget about

Setting up an LLM judge involves an implicit calibration decision. You pick examples that represent good and bad outputs. You tune the prompt until the scores feel right. You check correlation against your human raters. Then you ship.

That calibration is baked into a moment in time. Your production traffic is not.

In January, your LLM product serves formal queries with consistent phrasing. By April, users have discovered that casual language works just as well. Context lengths grow. Ambiguity patterns shift. The judge calibrated against January traffic is now scoring April traffic on a scale that no longer lines up with what you care about.

The judge doesn't crash. It doesn't return errors. It keeps scoring. But "6.5 / 10" now means something different than it did when you shipped, and nothing in your dashboard is telling you that.

Decision boundaries are where judges fail silently

Most teams set a threshold. Above 7: ship it. Below 7: flag for review. That threshold was calibrated against a distribution that no longer exists. But even if it did, continuous scores from LLM judges have enough variance in the middle range that the threshold is doing very little useful work.

Consider what the judge is doing. It reads your rubric, reads the response, and produces a number sampled from the model's output distribution. Run it twice on the same input with temperature > 0 and you'll often get scores differing by 1–2 points. At a threshold of 7.0, a 6.5 and a 7.5 make opposite routing decisions based on noise.

The fix isn't a better threshold. It's a different output structure. A judge that produces three tiers — accept / review / reject — is more reliable than one that outputs 0–10. You can calibrate tier boundaries with far less data than a continuous scale requires, and the uncertainty zone (the middle tier) becomes explicit rather than invisible.

If you need a continuous score for trend analytics, keep it. But route production decisions through tiers, not through a point estimate with a hard cutoff.

Cascade failures in multi-step pipelines

Single-step evals give you a misleading picture of end-to-end quality. If each step in a 5-step pipeline has 95% accuracy, the end-to-end accuracy is 0.95⁵ ≈ 77%. A 10-step pipeline: roughly 60%. This is standard probability, but it gets systematically ignored when deploying LLM judges — teams evaluate the final output, the final output looks fine, because the failure was subtle and happened two steps back.

An LLM judge reading a finished response has no visibility into the intermediate decisions that produced it. It can't see that step 3 retrieved the wrong context, or that step 5 hallucinated a date that the final step dressed up convincingly. The judge scores the surface.

What works: instrument at step boundaries, not just at the output. A targeted deterministic check for each intermediate step is often sufficient and far cheaper than a full LLM judge at each one. Reserve the judge for final output quality, but don't treat it as your only signal. If a step-3 failure shows up in 3% of traces, a final-output judge running at 95% overall accuracy will miss nearly all of them.

The meta-evaluation problem

To know if your LLM judge is reliable, you need to evaluate it. To evaluate it, you need ground truth. Your ground truth is your own judgment, which is what the judge is supposed to approximate. The loop is unavoidable.

The sustainable way to manage it: a quarterly calibration panel. 100 examples sampled from recent production traffic (not recycled from the original test set), covering the full distribution including edge cases from your review queue. Three human raters scoring each independently. Calculate inter-rater agreement. Compare to what your judge said.

If judge-to-human agreement has dropped since last quarter, your judge has drifted. Track that number over time. For a small team this is 2–3 person-hours per quarter — the actual minimum cost of knowing whether your eval system is working or just running.

Three scenarios where LLM-as-judge holds up

None of this means LLM-as-judge is wrong for production. It means it has a specific reliability profile, and using it outside that profile is where the failures come from.

Binary pass/fail with a tight rubric. If correct and wrong are unambiguous — the response includes a required citation or it doesn't; it stays in the specified language or it doesn't. An LLM judge can enforce this reliably. The decision boundary is clear, scores aren't involved, and the judge is being asked to classify rather than to rank.

Pairwise comparison. Asking a judge "which of these two responses is better?" is substantially more reliable than "how good is this response on a 0–10 scale?". Pairwise anchors the judge to a relative decision, which reduces both position bias and verbosity preference. For A/B testing prompts or models, pairwise ranking is the right evaluation shape.

Format and structure validation. When the thing you're checking is machine-readable (valid JSON, a template-following response, a structured extraction with required fields), use a deterministic validator. But if you need to verify that a response feels complete and structured, an LLM judge does this reasonably well as a first pass before human review.

What to wire around your judge

Treating an LLM judge as a point-in-time artifact rather than an ongoing system is the root of most production failures. The operational scaffold:

Distribution monitoring. Track your score distribution daily. Alert when it compresses below your baseline standard deviation. This catches calibration drift before it affects routing decisions.

Cost circuit-breaker. If scoring costs spike, whether from longer inputs or higher retry rates, something in your production distribution has changed. A cost ceiling that triggers an alert is a cheap proxy for input drift.

Uncertain-zone review. Define an explicit uncertainty band and route a random sample of outputs in that band to human review each week. This gives you a continuous calibration signal without a full formal panel every month.

Quarterly calibration panel. 100 examples, three raters, compare to judge. Refresh the examples from recent production traffic. Track judge-to-human agreement over time as your primary system health metric.

Deterministic first pass. For anything where a rule works, use the rule. JSON schema validation, regex for required elements, programmatic presence checks — these are faster, cheaper, and more reliable than an LLM judge for the problems they can solve. Reserve the judge for what's left.

The most common mistake isn't picking the wrong row. It's using continuous LLM scoring where deterministic checks or pairwise comparison would be more reliable and cheaper. The judge gets deployed because it feels comprehensive; it survives because nobody has measured whether it's still honest.

An LLM judge calibrated against a stale distribution, routing on a noise-dominated threshold, and scoring only final outputs isn't measuring quality. It's measuring how convincingly the output resembles what you trained the judge on six months ago. Knowing that distinction is the difference between a monitoring system and a false sense of one.