AI System Design · Designing RAG Systems · Lesson 3 of 3
Evaluating RAG & scaling to millions of docs
"How do you know your RAG system is good?" is where most candidates go vague. Strong answer: evaluate retrieval and generation separately — a bad answer is either 'we didn't retrieve the right chunk' or 'we had it and the model ignored it', and you must tell them apart.
Evaluating RAG
- Retrieval metrics: recall@k, precision@k, MRR/NDCG against a labeled set of (question → relevant chunk) pairs.
- Generation metrics: groundedness/faithfulness (is every claim supported by retrieved context?), answer relevance, and citation correctness — often scored by LLM-as-judge.
- The RAG triad: context relevance, groundedness, answer relevance. Frameworks: Ragas, TruLens, LangSmith, Braintrust.
- Watch the two failure classes: retrieval miss (right chunk absent) vs. generation miss (chunk present, ignored or misread).
Scaling to millions (or tens of millions) of docs
- ANN, not brute force: HNSW/IVF indexes make search sub-linear; brute-force cosine over 50M vectors is a non-starter.
- Shard & replicate: partition the index (by tenant/topic) for size and parallelism; replicas for QPS and availability.
- Metadata pre-filtering to shrink the search space before ANN (by tenant, recency, doc type).
- Quantization (e.g. int8/binary) and dimensionality choices to cut memory 4-32x at large scale.
- Incremental indexing: stream updates/deletes; don't rebuild the world. Handle re-embedding on model upgrades as a migration.
Two-stage caching pays offCache embeddings (don't re-embed unchanged docs) and cache frequent query results. At scale, a semantic cache on common questions can deflect a large fraction of retrieval+generation cost entirely.
Separate the two failuresWhen asked to debug a wrong RAG answer, always split it: "Did we retrieve the right chunk? If yes, it's a generation/prompt problem; if no, it's a chunking/embedding/retrieval problem." That diagnostic framing is a senior tell.
◆ Lock it in
- Evaluate retrieval (recall/precision@k) and generation (groundedness, relevance) separately.
- Scale with ANN + sharding + replication + metadata pre-filtering + quantization.
- Use incremental indexing and caching; re-embedding on model change is a migration.
Feynman drill — say it out loudYour RAG system gave a wrong answer. Describe the two-step diagnosis you'd run to locate the failure.
Step 1 rate your confidence · Step 2 pick your answer
Scaling a RAG index to 50M documents, which is the essential move?
Step 1 — how sure are you?