ML & Data Science Interviews · ML Fundamentals · Lesson 3 of 3
Train/val/test splits & cross-validation
You need an honest estimate of how the model does on unseen data. That means splitting data into a training set (fit parameters), a validation set (tune hyperparameters / choose models), and a test set (touched once, for a final unbiased estimate).
- k-fold cross-validation: split into k folds, train on k-1 and validate on the held-out fold, rotate, average. Uses data efficiently and gives a variance estimate of the score.
- Stratified k-fold: preserves class ratios in each fold — essential for imbalanced classification.
- Time-series split: never train on the future. Use expanding/rolling windows so validation always comes after training in time.
- Fit all preprocessing (scalers, encoders, imputers) on the training fold only, then apply to validation — otherwise you leak.
The test set is not a tuning setIf you keep peeking at the test set to pick models, you overfit to it and your reported number is a lie. Tune on validation / CV; unlock the test set exactly once, at the end.
◆ Lock it in
- Train fits params, validation tunes choices, test is touched once.
- k-fold CV gives an efficient, lower-variance score estimate.
- Stratify for imbalance; respect time order for temporal data; fit preprocessing on train only.
Feynman drill — say it out loudDescribe 5-fold cross-validation to someone who has only ever done a single train/test split.
Step 1 rate your confidence · Step 2 pick your answer
Why use k-fold cross-validation instead of a single train/validation split?
Step 1 — how sure are you?