ML & Data Science Interviews · ML System Design · Lesson 3 of 3
Online vs offline metrics & drift
A model that wins offline can lose online — and a model that ships well still decays over time. Senior answers connect the two metric worlds and plan for monitoring from day one.
- Offline metrics (AUC, precision@k, NDCG, RMSE) measure the model on historical data — necessary but not sufficient.
- Online metrics (CTR, conversion, revenue, retention) measure real user impact via A/B tests. Offline gains must be validated online.
- Data drift: input distribution shifts (new users, seasonality). Concept drift: the input→output relationship itself changes (fraud tactics evolve).
- Mitigate with monitoring (input + prediction + performance), alerting, and scheduled or triggered retraining.
Offline up, online flat?It's common for an offline metric to improve while the online metric doesn't move — proxy misalignment, feedback loops, or skew. That's exactly why you A/B test every model change rather than shipping on offline numbers alone.
◆ Lock it in
- Validate offline gains with online A/B tests — proxies can mislead.
- Data drift (inputs shift) vs concept drift (relationship shifts) both degrade models.
- Ship with monitoring, alerting, and a retraining plan.
Feynman drill — say it out loudDistinguish data drift from concept drift, and explain why you A/B test even a model that looks better offline.
Step 1 rate your confidence · Step 2 pick your answer
Fraud patterns change and your once-accurate model steadily loses precision even though input volume looks normal. What is this?
Step 1 — how sure are you?