DevOps, SRE & Cloud Interview · Observability & SRE · Lesson 3 of 3
Alerting, incidents & postmortems
When something breaks, SRE practice is about fast, humane response and learning without blame.
Alerting
- Alert on symptoms, not causes: page on user-facing pain (error rate, latency, SLO burn), not on 'CPU is 80 percent' which may be fine.
- Page only on actionable, urgent things: every page should need a human now. Noisy alerts cause alert fatigue and missed real incidents.
- Burn-rate alerts: page faster when you are spending the error budget quickly.
Incident response
- Declare an incident, assign an Incident Commander (coordinates, does not fix), keep a comms channel and timeline.
- Prioritize mitigation over root cause during the incident — roll back or fail over first, diagnose after.
- Track MTTR (mean time to recovery); fast, safe rollback is often the biggest reliability lever.
Blameless postmortemsAfter an incident, write a blameless postmortem: what happened, timeline, impact, and systemic action items — assuming everyone acted reasonably with the information they had. The goal is to fix the system, not punish a person. Saying 'blameless' unprompted signals real SRE maturity.
◆ Lock it in
- Alert on symptoms (user pain / SLO burn), keep pages actionable to avoid alert fatigue.
- In an incident: an Incident Commander coordinates; mitigate first (rollback/failover), diagnose later; track MTTR.
- Run blameless postmortems that fix the system, not blame a person.
Feynman drill — say it out loudExplain why you alert on symptoms rather than causes, and what makes a postmortem 'blameless'.
Step 1 rate your confidence · Step 2 pick your answer
What should page an on-call engineer?
Step 1 — how sure are you?