Data Analyst & Analytics Interview · Product & Business Metrics · Lesson 2 of 2
Diagnosing a metric change: 'why did X drop?'
The most common analytics interview question is open-ended: 'Metric X dropped 15% last week — why?' There is no single answer; they are testing your structured diagnostic process.
- Clarify & quantify: which metric exactly, how much, over what window, is it a step change or gradual? Confirm the definition.
- Rule out artifacts first: is it a data/logging bug, a tracking change, a broken pipeline, or a duplicate? Most 'drops' are instrumentation.
- Internal vs external: did we change something (release, pricing, UI, experiment) or did the world (seasonality, holiday, competitor, outage, macro)?
- Segment / slice: break the metric down by platform, geo, new-vs-returning, device, channel, cohort. A drop concentrated in one slice localizes the cause.
- Numerator or denominator: for a rate, decide which side moved (fewer conversions, or more traffic diluting the rate?).
- Form a hypothesis and validate with data; state confidence and next steps.
Segment, segment, segmentThe move that impresses is slicing. 'Overall conversion fell 15%' becomes tractable when you find it's entirely iOS in Germany after Tuesday's release. A drop that's uniform across all slices points to data/logging or something global.
Don't jump to a causeWeak candidates immediately guess ('probably the new feature'). Strong ones first confirm the number is real (not a logging bug), then systematically narrow with segments before naming a cause.
◆ Lock it in
- Structure beats guessing: clarify → rule out data bugs → internal vs external → segment → validate.
- Most sudden 'drops' are instrumentation/logging — check that first.
- Slicing by platform/geo/cohort localizes the cause; decompose rates into numerator vs denominator.
Feynman drill — say it out loudWalk through your first three steps when told 'daily active users fell 20% overnight,' and why you check data quality before blaming a feature.
Step 1 rate your confidence · Step 2 pick your answer
Signups suddenly dropped 30% overnight. What is the best FIRST move?
Step 1 — how sure are you?