AI System Design · Scaling, Cost & Latency · Lesson 1 of 2
Model routing, cascades, caching & batching
The fastest way to fail an AI design round is to answer every scale/cost question with "use a bigger model." The senior toolkit is routing, caching, and batching — spend expensive compute only where it's needed.
Model routing & cascades
- Routing: send each request to the cheapest model that can handle it — small/fast model for easy queries, frontier model for hard ones. A classifier or heuristic picks.
- Cascade: try the cheap model first; if a confidence/verifier check fails, escalate to the stronger model. Most traffic never reaches the expensive tier.
- This alone can cut cost several-fold while keeping p95 quality — a great thing to volunteer.
Prompt & prefix caching
- Prompt/prefix caching: providers cache the processed KV state of a stable prompt prefix (system prompt, long instructions, retrieved context reused across turns). Cache hits are much cheaper and lower-latency.
- Design prompts so the stable part comes first (system + tools + context) and the variable user turn comes last — that maximizes cache hits.
- Semantic caching: return a stored answer when a new query is near-duplicate of a past one — deflects whole requests.
Batching
For non-interactive work (bulk classification, embedding a corpus, evals), use batch APIs (often ~50% cheaper) or server-side continuous batching. Distinguish interactive traffic (stream, low latency) from batch traffic (throughput-optimized, latency-tolerant) up front.
The lever orderWhen cost is the problem, reach in this order: cache → route/cascade to smaller models → shorten prompts/context → batch → fine-tune a small model to replace a big one. 'Bigger model' is rarely the answer to a cost question.
◆ Lock it in
- Route/cascade: cheap model first, escalate hard cases — big cost cuts.
- Prefix-cache stable prompt heads; put variable content last; add semantic caching.
- Split interactive vs. batch; batch APIs are cheaper for tolerant workloads.
Feynman drill — say it out loudCosts are too high on a chat product. Walk through the levers you'd pull, cheapest-effort first.
Step 1 rate your confidence · Step 2 pick your answer
To cut LLM serving cost without tanking quality, the highest-leverage first moves are:
Step 1 — how sure are you?