Applied AI / Forward Deployed Engineer · Retrieval-Augmented Generation · Lesson 2 of 4
The RAG pipeline end to end
Every RAG system has two phases. Know them cold — this is a whiteboard staple.
Indexing (offline, once / on update)
- Load source documents (PDFs, HTML, DB rows, tickets).
- Chunk them into passages (e.g. 200–800 tokens) with some overlap.
- Embed each chunk with an embedding model → vectors.
- Store vectors (+ metadata + original text) in a vector database.
Retrieval + generation (online, per query)
- Embed the query with the same embedding model.
- Search the vector store for the top-k nearest chunks (often + keyword search = hybrid).
- (Optional) Rerank the candidates with a cross-encoder for precision.
- Assemble the prompt: instructions + retrieved chunks + user question.
- Generate the answer, instructing the model to use only the context and cite chunks.
The whole game is retrieval qualityIf the right chunk isn't retrieved, the model can't answer — 'garbage in, garbage out.' Most RAG failures are retrieval failures, not generation failures. Spend your effort on chunking, hybrid search, and reranking.
◆ Lock it in
- Indexing: load → chunk → embed → store. Retrieval: embed query → search → (rerank) → assemble → generate.
- Use the same embedding model for chunks and queries.
- Most RAG failures are retrieval failures — invest there first.
Feynman drill — say it out loudDraw the RAG pipeline from memory: name every step in both the indexing and query phases.
Step 1 rate your confidence · Step 2 pick your answer
Your RAG bot gives wrong answers. It turns out the correct passage was never in the retrieved set. This is primarily a failure of:
Step 1 — how sure are you?