Data Analyst & Analytics Interview · The Analytics Case Interview · Lesson 2 of 2
A worked case: measuring a feature launch
Case: 'A messaging app just launched voice notes. How would you measure whether it's a success?' Here is the framework in action.
- Clarify: success for whom and by when? Assume goal = increase overall engagement and retention within 90 days without harming message volume.
- Adoption: % of active users who send at least one voice note; trend over time (is it growing or a one-week novelty?).
- Engagement: voice notes per active user, and whether voice adds to or cannibalizes text messages (guardrail).
- Retention: do voice-note adopters retain better than a comparable non-adopter cohort? Beware selection bias — power users adopt everything.
- Quality / guardrails: latency, failed sends, report/block rate, storage cost.
- Causal read: ideally this shipped as an A/B test so lift is causal; if 100%-rolled-out, use a pre/post or matched-cohort comparison and caveat it.
Adoption is not successHigh adoption of a shiny feature can still be a failure if it cannibalizes a more valuable behavior or doesn't move retention. Always pair the adoption metric with the north-star and a guardrail.
Correlation vs causation'Voice-note users retain 20% better' is likely selection bias — engaged users adopt new features anyway. The clean read is an A/B test; absent that, a matched-cohort or difference-in-differences design and an explicit caveat.
◆ Lock it in
- Measure a launch across adoption, engagement, retention, and guardrails — not one number.
- Check for cannibalization: a new feature can win locally but hurt the north star.
- 'Adopters retain better' is usually selection bias; prefer an A/B test for a causal claim.
Feynman drill — say it out loudFor a launched feature, explain why 'adopters have higher retention' is not proof the feature works, and what design would give a causal answer.
Step 1 rate your confidence · Step 2 pick your answer
Data shows users who adopt the new feature retain 20% better. What's the most rigorous conclusion?
Step 1 — how sure are you?