ML & Data Science Interviews · ML Fundamentals · Lesson 2 of 3
Regularization: L1, L2 & how overfitting is tamed
Regularization adds a penalty on model complexity to the loss so the optimizer prefers simpler weights — trading a little bias for a lot less variance. The two classics penalize the size of the coefficients.
- L2 (Ridge): penalizes the sum of squared weights. Shrinks coefficients smoothly toward zero but rarely to exactly zero. Great when many features each contribute a little.
- L1 (Lasso): penalizes the sum of absolute weights. Drives some coefficients to exactly zero → automatic feature selection / sparsity.
- Elastic Net: a mix of L1 and L2 — sparsity plus stability with correlated features.
- Beyond linear models: dropout, early stopping, tree depth limits, and more data all regularize.
# The knob is lambda (alpha): strength of the penalty. loss = data_loss + lambda * penalty # L2: penalty = sum(w_i ** 2) -> small, spread-out weights # L1: penalty = sum(abs(w_i)) -> sparse weights (some exactly 0) # lambda too high -> underfit (bias up) # lambda too low -> overfit (variance up) -> tune via cross-validation
Why L1 zeroes weightsL1's diamond-shaped constraint region has corners on the axes, so the optimum often lands exactly on an axis (a weight = 0). L2's circular region has no corners, so it shrinks but seldom zeroes. That geometry is the whole answer.
◆ Lock it in
- Regularization penalizes complexity: trade a bit of bias for much less variance.
- L1 = sparse (feature selection); L2 = smooth shrinkage of all weights.
- The penalty strength (lambda) is a hyperparameter — tune it by cross-validation.
Feynman drill — say it out loudExplain why Lasso (L1) can zero out a feature while Ridge (L2) only shrinks it.
Step 1 rate your confidence · Step 2 pick your answer
You have 500 features and suspect most are useless. Which regularizer helps you select features automatically?
Step 1 — how sure are you?