Data Analyst & Analytics Interview · Statistics & Experimentation · Lesson 3 of 3
A/B testing: design & pitfalls
An A/B test is a randomized controlled experiment: randomly split users into control and treatment, change one thing, and compare a pre-declared metric. Randomization is what lets you claim causation, not just correlation.
Designing it
- Pick one primary metric and a hypothesis before you start.
- Do a power / sample-size calculation: the minimum detectable effect, alpha (0.05), and power (0.8) determine how many users and how long you must run.
- Randomize at the right unit (usually user, not session) and check the split is balanced (an A/A test).
- Run for full business cycles (whole weeks) and decide the stop date in advance.
Where it goes wrong
- Peeking / early stopping: checking daily and stopping at the first 'significant' result inflates false positives dramatically.
- Multiple comparisons: test 20 metrics at alpha 0.05 and ~1 will look significant by chance — correct for it (Bonferroni / FDR).
- Novelty & primacy effects: users react to change itself; short tests overstate impact.
- Sample ratio mismatch (SRM): a 50/50 split that arrives 53/47 signals a broken assignment — invalidate and investigate.
- Network / spillover effects: treatment leaks to control (marketplaces, social) — use cluster randomization.
Peeking is the #1 killerStopping a test the moment it crosses p < 0.05 inflates the 5% false-positive rate several-fold — easily 20%+ in simulations of daily peeking. Fix the sample size up front, or use a sequential testing method designed for continuous monitoring.
The senior answerWhen asked 'is this result real?', a strong candidate checks: was the sample size pre-computed, was there peeking, is the split balanced (SRM), were multiple metrics corrected, and is the effect big enough to ship — not just statistically significant.
◆ Lock it in
- Randomization at the user level is what licenses a causal claim.
- Compute sample size / power up front; run full weekly cycles; fix the stop date.
- Guard against peeking, multiple comparisons, novelty effects, and SRM.
Feynman drill — say it out loudExplain why 'peeking' at an A/B test daily and stopping when p < 0.05 inflates false positives.
Step 1 rate your confidence · Step 2 pick your answer
Your experiment was meant to be a 50/50 split but assignment logs show 55/45. What should you do first?
Step 1 — how sure are you?