DevOps, SRE & Cloud Interview · Live Troubleshooting · Lesson 2 of 2
Kubernetes triage: CrashLoopBackOff & friends
The other guaranteed scenario: 'a pod is in CrashLoopBackOff — go.' The workflow is always the same loop: kubectl get → describe (read the events) → logs (including --previous) → exec or a debug container. Then match the symptom to a known failure state.
kubectl get pods # what state, how many restarts kubectl describe pod <pod> # EVENTS: the diagnosis is usually here kubectl logs <pod> --previous # why the LAST container died kubectl get endpoints <svc> # is anything behind the Service? kubectl rollout undo deploy/<name> # mitigate first if a deploy caused it
- CrashLoopBackOff: the container keeps exiting — read
logs --previous. Exit code 137 = killed (usually OOMKilled — confirm in describe); a failing liveness probe shows as probe-failure events. - ImagePullBackOff: bad tag, deleted image, or missing registry credentials (imagePullSecret).
- Pending: unschedulable — no node fits the CPU/memory requests, taints without tolerations, or an unbound PVC.
describesays which. - Service gets no traffic: readiness probe failing or label selector mismatch — an empty
kubectl get endpointsis the tell. - Flaky in-cluster DNS: test from a debug pod (
nslookup svc.namespace), check CoreDNS pods/logs and their resource limits; know thatndots:5amplifies external lookups.
Mitigate, then diagnoseIf the crash loop started right after a deploy, say it: 'I'd roll back first (
kubectl rollout undo) to restore service, then diagnose the bad revision.' Restoring users before finding root cause is the SRE-grade instinct interviewers listen for.137 vs livenessBoth look like 'pod keeps restarting'. OOMKilled shows
Reason: OOMKilled / exit 137 in describe (fix: the limit or the leak). A liveness kill shows probe-failure events (fix: probe timing or a startup probe). Read the events — do not guess.◆ Lock it in
- The loop:
get → describe (events) → logs --previous → exec; mitigate withrollout undowhen a deploy is the trigger. - Map symptom → cause: CrashLoopBackOff (app/OOM/liveness), ImagePullBackOff (tag/creds), Pending (resources/taints/PVC).
- Empty
endpoints= readiness or selector mismatch; DNS issues start at CoreDNS with a debug pod.
Feynman drill — say it out loudA pod restarts every 40 seconds. Walk the exact commands you'd run, and how you'd tell an OOM-kill from a failing liveness probe.
Step 1 rate your confidence · Step 2 pick your answer
A pod is in CrashLoopBackOff; describe shows Last State: Terminated, Reason: OOMKilled, exit code 137. What happened?
Step 1 — how sure are you?