Data Analyst & Analytics Interview · Statistics & Experimentation · Lesson 2 of 3
Hypothesis testing, p-values & CIs
Hypothesis testing asks: could the difference I see be explained by random chance alone? You assume a null hypothesis (no effect), then measure how surprising your data would be if the null were true.
- Null (H0): no difference. Alternative (H1): there is a difference.
- p-value: probability of seeing a result this extreme if the null were true. Small p = the data is surprising under H0.
- Significance level (alpha): threshold (commonly 0.05). p < alpha → reject the null.
- Confidence interval: a range of plausible values for the true effect; a 95% CI excluding 0 mirrors p < 0.05.
- Type I error: false positive (cry wolf). Type II: false negative (miss a real effect). Power = 1 - P(Type II).
What a p-value is NOTA p-value is not the probability the null is true, and not the probability your result happened by chance. It is P(data this extreme | null true). And statistical significance is not business significance — a 0.01% lift can be 'significant' yet worthless.
Prefer the intervalA confidence interval beats a bare p-value: it shows both direction and magnitude. '+2.1% conversion, 95% CI [+0.5%, +3.7%]' tells a stakeholder the effect is positive and roughly how big — a p-value alone hides the size.
◆ Lock it in
- p-value = P(data this extreme given the null is true) — not P(null is true).
- Reject H0 when p < alpha; a 95% CI excluding 0 says the same thing plus magnitude.
- Type I = false positive, Type II = false negative; power is the ability to detect real effects.
Feynman drill — say it out loudState in one sentence what a p-value of 0.03 means, and why that is different from 'a 3% chance the result is a fluke.'
Step 1 rate your confidence · Step 2 pick your answer
An A/B test reports p = 0.02. Which statement is correct?
Step 1 — how sure are you?