ML & Data Science Interviews · Core Algorithms · Lesson 2 of 3
Trees, random forests & gradient boosting
A decision tree splits the feature space with a series of if/else questions chosen to reduce impurity (Gini / entropy for classification, variance for regression). Single trees are interpretable but overfit — so we ensemble them two different ways.
- Random forest (bagging): train many deep trees on bootstrap samples with random feature subsets, then average/vote. Reduces variance; robust, hard to overfit, minimal tuning.
- Gradient boosting (XGBoost / LightGBM / CatBoost): build shallow trees sequentially, each correcting the previous ensemble's errors (fitting the gradient of the loss). Reduces bias; usually the top performer on tabular data.
- Bagging trees are independent (parallel); boosting trees are dependent (sequential) and more sensitive to hyperparameters / overfitting if unregularized.
- Trees need no feature scaling and handle non-linear interactions natively.
Bagging vs boosting in one lineBagging averages many low-bias/high-variance models to cut variance. Boosting adds many high-bias/low-variance weak learners to cut bias. Same base learner, opposite attack on the error.
Default for tabularFor structured/tabular problems, saying 'I'd start with a gradient-boosted trees baseline (XGBoost/LightGBM), with a random forest as a robust sanity check' is a strong, current answer.
◆ Lock it in
- Single trees overfit; ensembles fix it two ways.
- Random forest = bagging (parallel, cuts variance); boosting = sequential (cuts bias, top tabular performer).
- Trees need no scaling and capture non-linear interactions.
Feynman drill — say it out loudContrast how a random forest and a gradient-boosted ensemble each build and combine their trees.
Step 1 rate your confidence · Step 2 pick your answer
What is the key difference between a random forest and gradient boosting?
Step 1 — how sure are you?