ML & Data Science Interviews · Feature Engineering & Data · Lesson 3 of 3
Feature selection & the curse of dimensionality
More features is not more better. Irrelevant and redundant features add variance, cost, and overfitting risk — the curse of dimensionality: as dimensions grow, data becomes sparse and distances lose meaning. Feature selection keeps only what pays its way.
- Filter methods: rank features by a statistic (correlation, mutual information, chi-squared) independent of the model. Fast, but ignore interactions.
- Wrapper methods: search subsets by training the model (recursive feature elimination). Accurate but expensive.
- Embedded methods: selection happens during training — L1/Lasso zeroing weights, tree feature importances. Good default.
- Drop highly collinear features — they destabilize linear-model coefficients and importances.
Fewer, better featuresInterviewers reward 'I'd cut features that don't earn their keep' with a concrete method (Lasso, RFE, importances) plus the why (less overfitting, lower latency, easier to maintain and explain).
◆ Lock it in
- High dimensionality → sparsity, overfitting, and meaningless distances.
- Selection families: filter (stats), wrapper (search), embedded (L1/importances).
- Remove collinear/redundant features for stable, interpretable models.
Feynman drill — say it out loudExplain the curse of dimensionality and name one method from each feature-selection family.
Step 1 rate your confidence · Step 2 pick your answer
Which is an EMBEDDED feature-selection method?
Step 1 — how sure are you?