Applied AI / Forward Deployed Engineer · Production: Latency, Cost, Reliability & Safety · Lesson 2 of 3
Reliability: timeouts, retries, fallbacks & idempotency
Model APIs fail, rate-limit, and occasionally return garbage. Production systems assume this and degrade gracefully.
- Timeouts + retries with backoff: handle transient errors and rate limits (429s) with exponential backoff + jitter.
- Fallbacks: a secondary model/provider or a cached/canned response when the primary is down. Avoid single-provider outages taking you down.
- Validate every output: schema-check and have a repair/retry path; never trust the first response blindly.
- Idempotency: if a tool call has side effects (charge a card, send an email), guard against duplicate execution on retry.
- Graceful degradation: if the AI can't produce a confident answer, fall back to 'I'm not sure — here's a human/handoff' rather than a confident wrong answer.
Non-determinism is a reliability propertyThe same input can yield different outputs. Your system must be correct despite variability: validate, constrain, and build the safety net rather than assuming the model behaves.
◆ Lock it in
- Assume failure: timeouts, retries w/ backoff, fallbacks, output validation.
- Make side-effecting tool calls idempotent; guard against duplicate retries.
- Degrade gracefully to a human/handoff instead of a confident wrong answer.
Feynman drill — say it out loudYour agent sends emails via a tool. What must you add so a retry doesn't send the email twice?
Step 1 rate your confidence · Step 2 pick your answer
A tool call charges a customer's card. To stay safe under retries you should:
Step 1 — how sure are you?