Applied AI / Forward Deployed Engineer · Production: Latency, Cost, Reliability & Safety · Lesson 1 of 3
Cost & latency: the two dials you always tune
LLM apps live and die on cost per request and latency. You're billed per token (input + output), and generation is sequential (slow). Levers:
- Right-size the model: use a small/cheap model for easy steps (classification, routing) and the big model only where needed. Massive savings.
- Model cascade / routing: try the cheap model first; escalate to the expensive one only on hard cases.
- Prompt caching: cache stable prefixes (long system prompts, docs) so you don't pay to reprocess them each call — big cost + latency win.
- Shorten context: retrieve less but better; trim history; summarize old turns.
- Cap output tokens and stream: streaming tokens to the UI cuts perceived latency dramatically even if total time is unchanged.
- Batch / parallelize independent calls; use async.
Do the napkin mathInterviewers love 'estimate the cost.' Cost ≈ (input tokens × input price + output tokens × output price) × requests. Know that output tokens are usually 3–5× pricier than input, and that caching + right-sizing are your biggest levers.
Worked example — say the numbers out loud"1M requests, 2,000 input + 500 output tokens each. At a mid-tier price of ~$3 / M input and ~$15 / M output: input = 1M × 2,000 = 2B tokens × $3/M = $6,000; output = 1M × 500 = 0.5B × $15/M = $7,500. Total ≈ $13.5k before optimization. Then prompt-cache the shared prefix and route easy requests to a cheaper model — realistically halving it." Producing this arithmetic live is a strong signal.
◆ Lock it in
- Tune cost/request and latency constantly; you pay per token and generation is sequential.
- Biggest levers: right-size the model, route/cascade, prompt caching, shorten context, stream.
- Streaming slashes perceived latency; output tokens usually cost more than input.
Feynman drill — say it out loudA feature costs too much per request. List four levers you'd pull, in order of impact.
Step 1 rate your confidence · Step 2 pick your answer
Your chatbot re-sends the same 3,000-token system prompt + policy docs on every call, and it's expensive. Best fix?
Step 1 — how sure are you?