ML & Data Science Interviews · Statistics & Probability · Lesson 2 of 3
Hypothesis testing, p-values & confidence intervals
Hypothesis testing asks: could the effect I see be just noise? You assume a null hypothesis (no effect), compute how surprising your data would be under it, and decide.
- p-value: the probability of observing data at least as extreme as yours if the null were true. Small p → the null looks implausible.
- Significance level (alpha): your threshold (commonly 0.05). p < alpha → reject the null. Alpha is your false-positive rate.
- Type I error = false positive (reject a true null); Type II error = false negative (miss a real effect). Power = 1 - P(Type II).
- Confidence interval: a range that, under repeated sampling, contains the true parameter X% of the time. Wider = less certain; if a 95% CI for a difference excludes 0, it's significant at 0.05.
What a p-value is NOTA p-value is not the probability the null is true, and not the probability your result happened by chance. It is P(data this extreme | null true). Also: statistical significance is not practical significance — a tiny, useless effect can be significant with enough data.
◆ Lock it in
- p-value = P(data this extreme | null true); compare to alpha to decide.
- Type I = false positive, Type II = false negative, power = 1 - P(Type II).
- A CI gives the effect's plausible range; significance is not the same as importance.
Feynman drill — say it out loudDefine a p-value precisely, then say what it is NOT (two common misinterpretations).
Step 1 rate your confidence · Step 2 pick your answer
Which is the correct interpretation of a p-value of 0.03?
Step 1 — how sure are you?