Production concerns, the feedback loop & naming tradeoffs

A working prototype is not a system. The last third of a strong answer covers production concerns, the feedback loop, and — crucially — naming the tradeoffs of what you built.

Production concerns to volunteer

• Latency: streaming, caching, model routing (covered later).
• Cost: tokens per request x volume; the number that kills projects.
• Reliability: provider outages, timeouts, retries, fallbacks.
• Safety & privacy: prompt injection, PII, access control on retrieved data.
• Observability: tracing, evals in CI, online quality monitoring.

The feedback loop

Production LLM systems get better by closing the loop: capture user signals (thumbs, edits, escalations, task success) → log full traces → mine failures → expand the eval set and improve prompts/retrieval/models → redeploy. Explicitly drawing this loop shows you think about systems that improve over time, not one-shot demos.

Name the tradeoffs

End by stating what your design gives up. There is no perfect answer — the interviewer is grading whether you see the tradeoffs.

• Quality vs. latency vs. cost — the eternal triangle (bigger model = better + slower + pricier).
• Freshness vs. cost (re-index frequency), recall vs. precision (retrieve more vs. less).
• Autonomy vs. control (agent flexibility vs. predictability), build vs. buy.