AI System Design · Reliability, Safety & Observability · Lesson 1 of 3
Retries, fallbacks & graceful degradation
LLM systems depend on external model providers that rate-limit, time out, and have outages. Non-determinism means the same input can fail intermittently. A production design must answer: what happens when the model call fails?
The reliability toolkit
- Retries with backoff + jitter on transient errors (429/5xx/timeouts); cap attempts so you don't amplify an outage.
- Timeouts on every model/tool call — never hang a user request on a stuck upstream.
- Provider fallback: a secondary model or provider when the primary is down (e.g. failover across providers/regions). Abstract the model behind an interface so you can swap.
- Graceful degradation: fall back to a smaller/cached answer, a non-AI path, or an honest "can't do that right now" instead of a hard error.
- Idempotency on any action-taking call so a retry can't duplicate a side effect.
- Circuit breakers + load shedding: stop hammering a failing dependency; shed low-priority traffic under overload.
Retries can make an outage worseNaive immediate retries during a provider incident create a thundering herd that deepens the outage. Always use exponential backoff with jitter and a hard attempt cap, plus a circuit breaker.
Design like the model is a flaky network callTreat every LLM call as an unreliable remote dependency — because it is. The same patterns that protect microservices (timeouts, retries, breakers, fallbacks) apply directly.
◆ Lock it in
- Model calls are flaky remote dependencies — design for failure.
- Retries+backoff, timeouts, provider fallback, graceful degradation, idempotency, circuit breakers.
- Naive retries cause thundering herds — always backoff + jitter + caps.
Feynman drill — say it out loudYour model provider starts returning 429s and timeouts. Describe how a well-designed system responds without falling over.
Step 1 rate your confidence · Step 2 pick your answer
During a partial provider outage, which retry strategy is safest?
Step 1 — how sure are you?