ML & Data Science Interviews · ML System Design · Lesson 2 of 3
Training vs serving, pipelines & labels
A production ML system is two connected pipelines: an offline training path and an online serving path. Most real-world bugs live in the seam between them.
- Training (offline): batch over historical data, heavier compute, produces a versioned model + metrics.
- Serving (online): low-latency predictions, often from a feature store, under a strict latency/cost budget. Batch (precompute) vs real-time (on request).
- Training/serving skew: features computed differently in the two paths → the model sees inconsistent inputs. Share feature code / a feature store to prevent it.
- Labels: explicit (ratings, fraud confirmations) vs implicit (clicks, dwell); watch label delay (churn/fraud labels arrive weeks later) and feedback loops (the model influences the data it later trains on).
Training/serving skewIf a feature is computed one way in the training notebook and another way in the serving service, offline metrics look great and production quietly degrades. A shared feature store / feature pipeline is the standard fix — call it out.
◆ Lock it in
- Two paths: offline training vs online serving (batch or real-time).
- Training/serving skew is a top failure mode — share feature computation.
- Mind label delay and feedback loops in how labels are generated.
Feynman drill — say it out loudExplain training/serving skew and one concrete way to prevent it.
Step 1 rate your confidence · Step 2 pick your answer
A model performs great offline but degrades in production. Features are computed in a notebook for training and in a separate service for serving. What's the likely cause?
Step 1 — how sure are you?