AI System Design · Designing RAG Systems · Lesson 1 of 3
Ingestion, chunking & embeddings
RAG is two pipelines: an offline indexing pipeline and an online retrieval pipeline. Interviewers probe the offline side because that's where quality is silently won or lost. The flow: load → parse → chunk → embed → index.
Chunking — the highest-leverage knob
- Chunk too big → you dilute the embedding and blow context; too small → you lose the surrounding meaning.
- Prefer structure-aware / semantic chunking (by heading, section, paragraph) over blind fixed-size splits.
- Use overlap (e.g. 10-20%) so facts spanning a boundary aren't cut in half.
- A typical starting point: ~200-500 tokens with overlap, then tune against evals — never guess and ship.
- Attach metadata to each chunk (source, title, section, date, permissions) — it powers filtering and citations later.
Embeddings
An embedding model maps text to a vector so semantically similar text lands nearby. Choices that matter: dimensionality (storage/speed vs. quality), domain fit (general vs. code/legal/multilingual), and consistency — you must embed queries and documents with the same model, and re-embed everything if you switch.
Chunking is like citationsA chunk is the unit you retrieve and cite. If a human couldn't answer from that snippet alone, the model can't either. Chunk so each piece is a self-contained, quotable fact.
The silent version-skew bugEmbed documents with model A and queries with model B (or upgrade the embedder and forget to re-index) → retrieval quietly degrades with no error. Pin the embedding model and re-index on change.
◆ Lock it in
- RAG = offline indexing + online retrieval; quality is won offline.
- Chunking is the top knob: structure-aware, overlap, ~200-500 tokens, tuned to evals.
- Embed queries and docs with the same model; attach metadata to every chunk.
Feynman drill — say it out loudExplain why chunk size and overlap matter, and what goes wrong if you upgrade your embedding model but don't re-index.
Step 1 rate your confidence · Step 2 pick your answer
Why is chunking often the highest-leverage decision in a RAG system?
Step 1 — how sure are you?