AI System Design · The Framework · Lesson 2 of 3
Define success, sketch the architecture, choose the AI core
Once scoped, move in three beats: define success (evals) → high-level architecture → pick the AI core. Doing evals before architecture is a subtle senior move — you can't design a system you can't measure.
1. Define success & evals
- Name a primary metric tied to the task: answer accuracy / groundedness, task-success rate, deflection rate, extraction F1.
- Name guardrail metrics: latency p95, cost per request, hallucination/refusal rate, safety violations.
- Decide how you'll measure: a golden eval set, LLM-as-judge, human review, or online A/B.
- Be ready for the round to stay here: 2026 design interviews increasingly make the eval harness itself the deliverable — failure taxonomy → binary judges validated against human labels → CI regression gate → online monitoring.
2. High-level architecture
Draw the request path first, then the data path: client → API/gateway → orchestration → AI core (model + context) → post-processing → response, with offline ingestion/indexing and a feedback/eval loop on the side. Keep it boxes-and-arrows; you'll zoom into the hard part next.
3. Choose the AI core
The heart of the interview: what actually powers the intelligence? Reason through the ladder, cheapest first:
- Prompt + in-context: task fits in context, no private knowledge needed. Cheapest, fastest to ship.
- RAG: answers must be grounded in a private/large/fresh corpus. Default for 'chat with our knowledge'.
- Agent (tools): the task requires actions or dynamic multi-step tool use, not just retrieval.
- Fine-tune: you need a consistent style/format/behavior or to compress a narrow task into a cheaper/smaller model — not to inject fresh facts.
RAG vs fine-tune, the clean lineRAG changes what the model knows (facts, freshness, citations); fine-tuning changes how the model behaves (style, format, task shape). Most 'answer over our data' problems are RAG, not fine-tuning. You can also combine them.
Don't reach for the fanciest coreJumping straight to a multi-agent swarm or fine-tuning when a prompt or RAG would do is a classic over-engineering flag. Justify each rung you climb.
◆ Lock it in
- Order: evals → architecture → AI core. Measure before you build.
- AI core ladder: prompt → RAG → agent → fine-tune; climb only as needed.
- RAG = knowledge; fine-tune = behavior. They solve different problems.
Feynman drill — say it out loudExplain when you'd choose RAG over fine-tuning, and give one problem where fine-tuning is the right call.
Step 1 rate your confidence · Step 2 pick your answer
A team wants their model to answer using the company's constantly-updated internal wiki, with citations. Best AI core?
Step 1 — how sure are you?