Applied AI / Forward Deployed Engineer · Evaluation & Quality · Lesson 2 of 4
Error analysis: where evals come from
The 2026 eval bar moved past 'have a golden set.' The method interviewers now expect (codified by Hamel Husain & Shreya Shankar, and echoed in frontier-lab guidance on agent evals) starts with error analysis: you discover what to measure by reading real failures — you don't invent metrics in a vacuum.
- Collect real traces (production logs, or realistic pilot traffic) — ~100 to start.
- Open coding: read each trace and write a free-form note on the first upstream failure only (don't log downstream symptoms of the same root cause).
- Axial coding: cluster the notes into a small failure taxonomy ('retrieved wrong policy,' 'ignored refund limit,' 'wrong tone') and count them — now you know what actually breaks, and how often.
- Build one binary pass/fail judge per failure mode, validated against human labels. Fix the biggest bucket, re-run, repeat.
- Trace-to-dataset: every production failure becomes a permanent regression test. A great eval suite is harvested from traffic, not authored up front.
- Guardrails ≠ evals: guardrails are synchronous, deterministic, inline blocking (PII filters, schema checks); evals are async quality measurement feeding the improvement loop. Naming this distinction reads senior.
- As capabilities stabilize, migrate saturated capability evals into the always-run regression suite that gates deploys.
The eval-harness design round2026 design rounds increasingly make the measurement system the deliverable — 'how would you evaluate this agent?' can be the entire hour. The winning shape, said as a loop: error analysis → failure taxonomy → binary judges validated against human labels → CI regression gate → online monitoring → production traces feed the dataset.
Don't write evals for imagined failuresPre-authoring dozens of evaluators before you've read traces is a named anti-pattern — you'll measure what you imagined instead of what breaks. Start with a small smoke set, then grow the real suite from observed failures. Teams that ship well spend the majority of their applied-AI time in this loop.
◆ Lock it in
- Evals are discovered, not invented: real traces → open coding (first upstream failure) → counted failure taxonomy → binary judges.
- Validate each judge with TPR/TNR on held-out human labels; prefer binary checks per failure mode.
- Wire the loop: trace → dataset → CI gate → online monitoring. Guardrails block inline; evals measure async.
Feynman drill — say it out loudWalk through running error analysis on a support bot's last 100 conversations — what do you write down per trace, and what artifact do you end up with?
Step 1 rate your confidence · Step 2 pick your answer
Your team wants to 'add evals' to a new LLM feature. Per 2026 best practice, what's the first step?
Step 1 — how sure are you?