Product Manager Interview (+ AI PM) · Metrics & Analytics · Lesson 3 of 3
Guardrails, trade-offs & A/B tests
Optimizing one metric can quietly wreck another. Guardrail metrics are the health checks you refuse to break while chasing a primary goal — they keep you from winning the battle and losing the product.
- Primary metric: the thing you're trying to move (e.g. conversion, engagement).
- Guardrails: metrics that must not degrade — e.g. latency, unsubscribe/complaint rate, refund rate, retention, error rate.
- Classic tension: pushing more notifications raises short-term engagement but can spike uninstalls — watch both.
- Balance engagement vs retention vs monetization; a metric that trades long-term health for a short-term spike is a trap.
Validating a change
- A/B test: split users, change one thing, compare the primary metric with the guardrails held in view.
- Watch for statistical significance and enough sample/time; don't ship on a noisy 2-day blip.
- Beware novelty effects (a bump that fades) and segment differences (a win for new users can hurt power users).
The lever that breaks the productAny metric can be gamed if you ignore guardrails — max 'time in app' by making things hard to find, or max clicks with dark patterns. Always pair a primary metric with the guardrails that protect user trust.
◆ Lock it in
- Guardrail metrics are health checks you won't break while chasing a primary goal.
- Name the tension: engagement vs retention vs monetization, short vs long term.
- Validate with A/B tests — check significance, sample/time, and novelty effects.
Feynman drill — say it out loudExplain why you'd pair 'increase notifications sent' with a guardrail metric, and which guardrail you'd choose.
Step 1 rate your confidence · Step 2 pick your answer
What is the purpose of a guardrail metric?
Step 1 — how sure are you?