ML & Data Science Interviews · Deep Learning Fundamentals · Lesson 1 of 2
Gradient descent & how neural nets learn
Every modern model — logistic regression to LLM — trains the same way: define a loss, compute its gradient with respect to the parameters, and step downhill. Interviewers use this as a breadth question because it separates people who used a library from people who know what the library did.
# One step of gradient descent w = w - lr * gradient(loss, w) # lr too high -> overshoot: loss bounces or explodes # lr too low -> crawls, may stall in a poor region # mini-batch SGD: estimate the gradient on a small batch # -> cheap, noisy steps; the noise even helps escape bad spots
- Batch vs mini-batch vs stochastic: full-data gradients are exact but slow; mini-batches (the default) trade a little noise for far cheaper steps.
- Momentum / Adam: momentum smooths the step direction across batches; Adam adds a per-parameter learning rate — the robust default optimizer.
- The learning rate is the single most important hyperparameter — say that, then describe the two failure modes above.
A neural network stacks linear layers with non-linear activations (like ReLU, max(0, x)) between them. Without the non-linearity, stacked linear layers collapse into one linear model — the depth buys nothing. Backpropagation is just the chain rule: one backward pass computes every layer's gradient from the output back to the input, so gradient descent can update all weights at once.
- Regularizing networks: dropout (randomly zero activations in training), early stopping (halt when validation loss turns up), weight decay (the L2 idea again).
- Same diagnosis skills transfer: train/validation gap for overfitting; a loss that will not fall points to learning rate, data, or architecture.
Backprop is blame assignmentThe network makes an error; backprop walks backward asking 'how much did this weight contribute to the miss?' and nudges each one in proportion. The chain rule is the bookkeeping that makes the blame exact.
No math requiredBreadth rounds want the mechanism, not the derivation: loss → gradient → step, learning-rate failure modes, why non-linearities exist, and backprop as the chain rule. Two crisp minutes beats a whiteboard of calculus.
◆ Lock it in
- Training = loss → gradient → step; the learning rate is the first hyperparameter to name and to blame.
- Without non-linear activations, a deep network collapses into a single linear model.
- Backprop = the chain rule: one backward pass assigns each weight its share of the error.
Feynman drill — say it out loudExplain gradient descent and backpropagation to someone who has only ever fit linear regression with a formula.
Step 1 rate your confidence · Step 2 pick your answer
Why does a deep network need non-linear activation functions between its layers?
Step 1 — how sure are you?