AI System Design Interview Questions
"Design an AI assistant for X" is a staple round. A repeatable framework — clarify, evals, architecture, production, tradeoffs — wins it.
Interview questions & answers
How do you approach an open-ended 'design an AI system for X' prompt?
Clarify first: scale/QPS, task type, data volume/freshness/permissions, latency budget, quality bar, cost/compliance constraints — state explicit assumptions. Define success/evals before architecture: primary metric + guardrail metrics + how you'll measure. Sketch high-level architecture (request path + data path + feedback loop). Choose the AI core on the ladder: prompt → RAG → agent → fine-tune, justified. Cover production (latency/cost/reliability/safety/observability) and close by naming tradeoffs.
RAG vs fine-tuning vs long context vs an agent — how do you pick?
RAG: answers must be grounded in a large/private/fresh corpus with citations — the default for 'chat with our knowledge'. Fine-tune: you need consistent style/format/behavior or to compress a narrow task into a cheaper model — not for fresh facts. Long context: the whole relevant material fits and is small/stable — simplest, but costly and lossy at scale. Agent: the task needs actions or dynamic multi-step tool use, not just retrieval. Climb the ladder cheapest-first; combine when needed (e.g. fine-tune + RAG).
Design a RAG system to answer questions over a company's internal knowledge base.
Offline: structure-aware chunking + overlap + metadata (source, ACL, date), embed with a fixed model, index. Online: hybrid (dense+sparse) retrieval → cross-encoder rerank → grounded generation with citations. Permission-filter at retrieval time via ACL metadata so users only see authorized docs. Evals: retrieval recall/precision@k + generation groundedness/citation correctness on a golden set. Production: cache, route, stream, trace; incremental re-indexing for freshness.
Your RAG system gives a wrong answer. How do you debug it?
Split the failure: did we retrieve the right chunk? If no → chunking/embedding/retrieval problem (chunk boundaries, wrong embedder, need hybrid/rerank, missing metadata filter). If yes → generation/prompt problem (model ignored context, prompt doesn't enforce grounding, context too long). Use traces to inspect retrieved chunks + prompt; add the case to the eval set.
How do you scale a vector index to tens of millions of documents?
ANN indexes (HNSW/IVF) for sub-linear search — never brute force. Shard by tenant/topic for size + parallelism; replicate for QPS/availability. Metadata pre-filtering to shrink the search space (tenant, recency, doc type). Quantization (int8/binary) and dimensionality choices to cut memory 4-32x. Incremental indexing for updates/deletes; treat embedder upgrades as a migration.
Design an AI customer-support assistant that can answer and take actions.
Core: RAG for policy answers + an agent for actions (get_order, issue_refund, escalate) with schema'd tools. ReAct loop with step/cost caps; validate each tool result; answer from context with citations. Human-in-the-loop for refunds over a threshold; least-privilege tools; idempotent actions. Treat ticket + retrieved text as untrusted (injection); guardrails on outbound. Evals + tracing: golden tickets, task-success + tool-error rate + groundedness; observe every step.
When should you NOT build an agent?
When steps are fixed/known — a workflow is cheaper, faster, more testable. When you need predictable latency/cost — agent loops are variable. When the task is high-stakes with low error tolerance and needs no autonomy. When you can't evaluate the trajectory yet. Rule: agent only if the path is genuinely dynamic AND measurable; otherwise constrain it.
The interviewer says your design is too expensive. Walk through cutting cost.
Do the math: tokens/request x price x volume — find the dominant term. Cache: prefix-cache stable prompts, semantic-cache repeated queries. Route/cascade: small model for easy traffic, escalate only hard cases. Shorten prompts/retrieved context; cap output length. Batch tolerant workloads; consider fine-tuning a small model to replace a big one. 'Bigger model' is rarely the answer.
How do you hit a sub-2-second interactive latency target?
Stream tokens so perceived latency = TTFT, not full completion. Cut TTFT: shorter prompts, tight retrieved-context budget, prefix caching. Cut generation time: cap output tokens, prefer concise formats, faster/smaller model where quality allows. Parallelize retrieval + prep; precompute/cache where possible. Reason with TTFT + output_tokens x TPOT.
How do you make an LLM system reliable given provider outages and non-determinism?
Treat model calls as flaky remote deps: timeouts on every call. Retries with backoff + jitter + hard caps; circuit breakers to avoid thundering herds. Provider/model fallback behind an abstraction; multi-region where needed. Graceful degradation: cached/smaller answer or honest failure over a hard error. Idempotency on action-taking calls so retries can't duplicate side effects.
How do you secure an AI agent that reads untrusted content and can act externally?
Frame with the lethal trifecta: private data + untrusted content + external comms — cut a leg. Treat retrieved/tool content as data, not instructions (delimit; don't blindly concatenate). Least-privilege tools; human approval before outbound actions; allow-list recipients/actions. Output filtering; keep secrets the model doesn't need out of context. Log/trace tool calls to detect abuse.
How do you know your production LLM system is actually working?
Trace every request: prompt, retrieved context, tool calls, tokens, latency, cost, model version (LangSmith/Langfuse/Braintrust). Offline evals: golden set in CI gating every prompt/model change — catch regressions before ship. Online evals: user signals (thumbs/edits/escalations), LLM-as-judge sampling, A/B tests. Feed failures back into the eval set and prompts/retrieval — close the loop. Offline gates deploys; online watches prod — you need both.