ML & Data Science Interviews · Statistics & Probability · Lesson 3 of 3
Bayes' theorem & A/B test design
Bayes' theorem updates a prior belief with evidence: P(A|B) = P(B|A) * P(A) / P(B). It's the backbone of the classic 'rare-disease test' question and of reasoning about base rates.
The base-rate trapA 99%-accurate test for a disease affecting 1 in 10,000: a positive result is still probably a false positive, because true positives are far rarer than false positives. Base rates dominate — always ask 'how common is it to begin with?'
A/B testing is hypothesis testing applied to product changes: randomly split users into control and treatment, compare a metric, and check significance.
- Design first: define the metric and hypothesis, compute the sample size / power for the effect you care about, and fix the duration up front.
- Randomize properly and check the groups are balanced (A/A test / sanity checks).
- Pitfalls: peeking (stopping early when it looks significant inflates false positives), multiple comparisons (test many metrics → some 'win' by chance; correct for it), novelty effects, and network/interference between groups.
- Report the confidence interval and effect size, not just 'significant'.
Name the pitfallsThe fastest way to sound experienced in an A/B question: proactively raise peeking, multiple testing, and adequate power/duration, and insist on fixing sample size before launch.
◆ Lock it in
- Bayes updates priors with evidence; base rates dominate rare-event tests.
- A/B testing = randomized hypothesis test; power/sample size before launch.
- Guard against peeking, multiple comparisons, novelty, and interference.
Feynman drill — say it out loudDesign an A/B test for a new checkout button and list three ways it could give a false result.
Step 1 rate your confidence · Step 2 pick your answer
An analyst checks an A/B test daily and stops the moment p < 0.05. Why is this a problem?
Step 1 — how sure are you?