Product Manager Interview (+ AI PM) · AI Product Management · Lesson 2 of 4
Evals, hallucination & AI UX
AI features are probabilistic: the same input can give different, sometimes wrong, outputs. So the PM owns two new things — how you measure quality (evals) and how the UX handles being wrong.
Evals as a PM
- Build a golden dataset of real inputs with expected/acceptable outputs — this is the AI PM's equivalent of a test suite and your source of truth.
- Define quality criteria: accuracy/groundedness, relevance, format, safety, tone. Measure them, don't eyeball a demo.
- Use LLM-as-judge to grade open-ended outputs at scale, but validate the judge against human ratings.
- Track quality over time and per release — treat prompt/model changes like code changes behind evals.
Hallucination & UX
- Ground the model with RAG and cite sources so users can verify.
- Design for wrongness: show confidence, make outputs editable/reviewable, keep a human in the loop for high-stakes actions.
- Set expectations in the UI ('AI can make mistakes'); prefer assistive (draft/suggest) over fully autonomous where errors are costly.
- Add feedback loops (thumbs up/down, edits) that feed back into your eval set.
Own the eval loopThe clearest AI-PM signal: 'I'd build a golden eval set from real usage, define quality metrics, gate model/prompt changes behind it, and close the loop with in-product feedback.' That separates AI PMs from PMs who just prompt a demo.
◆ Lock it in
- Because outputs are probabilistic, the PM owns evals (a golden dataset + quality metrics) and error-tolerant UX.
- Reduce hallucination with grounding (RAG) + citations; design for wrongness with review, editing, and confidence cues.
- Close the loop: in-product feedback feeds the eval set; gate prompt/model changes behind evals.
Feynman drill — say it out loudExplain how you'd measure and improve the quality of an AI summarization feature as its PM.
Step 1 rate your confidence · Step 2 pick your answer
What's the most reliable way for a PM to measure an AI feature's quality?
Step 1 — how sure are you?