Applied AI / Forward Deployed Engineer · Evaluation & Quality · Lesson 4 of 4
LLM-as-judge, offline vs online & guardrails against overfitting
For subjective or open-ended output (summaries, chat helpfulness), use LLM-as-judge: a strong model scores outputs against a rubric. Scalable and surprisingly good — with caveats.
- Give the judge a clear rubric and ask for a score + justification; consider pairwise comparison (A vs B) over absolute scores.
- Prefer binary pass/fail checks per failure mode over 1–10 scores — humans and models both judge binaries far more consistently; decompose 'quality' into independent checks.
- Beware judge biases: position bias (favoring first/second), length bias (favoring longer), self-preference. Randomize order; validate the judge against human labels on a sample.
- Validate the judge like a classifier: label ~100 examples, hold out a test split, and report TPR and TNR (>90% target) — raw '% agreement' hides failure on the rare class.
Offline vs online evaluation
- Offline: run your eval set in CI before shipping — catches regressions pre-deploy.
- Online: measure in production — user feedback (👍/👎), A/B tests, task-success signals, human review of samples.
- Guardrail metrics: track cost, latency, refusal rate, and safety alongside quality — a 'better' answer that's 3× slower/costlier may not ship.
Don't overfit the evalIf you tune endlessly against the same 50 cases, you overfit to them. Keep a held-out set, refresh cases from real traffic, and watch online metrics — the eval set is a proxy, not the goal.
◆ Lock it in
- LLM-as-judge scales subjective eval — use rubrics, pairwise comparisons, and watch position/length bias.
- Run offline evals in CI and online metrics (feedback, A/B, success) in prod; track cost/latency/safety guardrails.
- Keep a held-out set and refresh from real traffic to avoid overfitting the eval.
Feynman drill — say it out loudWhat biases might an LLM judge have, and how would you defend against them?
Step 1 rate your confidence · Step 2 pick your answer
You use an LLM to judge which of two summaries is better. A key bias to control for is:
Step 1 — how sure are you?