LLM Evaluation (Evals) Interview Questions
"How do you know your AI system actually works?" is the differentiating interview question. Evals turn vibes into measurement.
Key concepts to know
- You can't eyeball non-deterministic output — evals make quality measurable.
- Evals catch regressions and enable fast, safe iteration.
- 'How do you know it's good?' → lead with your eval strategy.
- Evals are discovered, not invented: real traces → open coding (first upstream failure) → counted failure taxonomy → binary judges.
- Validate each judge with TPR/TNR on held-out human labels; prefer binary checks per failure mode.
- Wire the loop: trace → dataset → CI gate → online monitoring. Guardrails block inline; evals measure async.
- Build an eval set from real + edge-case inputs with expected outputs/rubrics; grow it from every bug.
- Prefer deterministic checks; use classification metrics, reference similarity, or human eval as fit.
- For RAG, measure retrieval and generation separately (relevance + faithfulness).
- LLM-as-judge scales subjective eval — use rubrics, pairwise comparisons, and watch position/length bias.
- Run offline evals in CI and online metrics (feedback, A/B, success) in prod; track cost/latency/safety guardrails.
- Keep a held-out set and refresh from real traffic to avoid overfitting the eval.
Interview questions & answers
Why are evals especially critical for LLM systems (vs traditional deterministic software)?
Output is non-deterministic and open-ended, so systematic measurement is the only reliable way to detect quality changes and regressions.
Why: Non-determinism + open-ended outputs mean manual spot-checks miss regressions. Evals give you a reliable, repeatable quality signal to iterate against.
Your team wants to 'add evals' to a new LLM feature. Per 2026 best practice, what's the first step?
Read ~100 real traces, note the first upstream failure in each, and cluster them into a counted failure taxonomy.
Why: Error analysis comes first: real traces → open coding → taxonomy. Judges are then built per observed failure mode and validated against human labels. Imagined-failure evaluators and self-ratings measure the wrong things; public benchmarks don't reflect your task.
For a JSON-extraction task, the most reliable primary eval metric is:
Deterministic checks: JSON validity plus field-level exact/normalized match against expected values.
Why: Structured extraction has a ground truth — use deterministic, field-level checks (valid JSON + exact/normalized match). They're cheap, reliable, and unambiguous.
You use an LLM to judge which of two summaries is better. A key bias to control for is:
Position/length bias — judges often favor the first-shown or the longer answer; randomize order and control length.
Why: LLM judges exhibit position and length biases. Randomizing presentation order, controlling for length, and validating against human labels keep the judge honest.