Applied AI / Forward Deployed Engineer · Retrieval-Augmented Generation · Lesson 3 of 4
Chunking & embeddings that don't sabotage you
Chunking — how you split documents — quietly determines retrieval quality.
- Too big: a chunk covers many topics; embeddings get muddy; you waste context and money.
- Too small: chunks lose the surrounding meaning; answers get fragmented.
- Overlap (e.g. 10–20%) prevents cutting a sentence/idea across a boundary.
- Structure-aware chunking (by heading, section, or semantic boundary) beats naive fixed-size splits.
- Attach metadata (source, title, section, date, permissions) — you'll need it for filtering and citations.
Practical defaultsStart ~400–600 tokens per chunk with ~15% overlap, split on semantic/heading boundaries, store rich metadata. Then measure and tune — don't guess forever.
Embedding mismatchYou must embed queries and chunks with the same model. Changing embedding models means re-indexing everything. Choose deliberately and version it.
◆ Lock it in
- Chunk size is a tradeoff: too big = muddy, too small = fragmented; add overlap.
- Prefer structure/semantic-aware chunking and attach rich metadata.
- Query and chunk embeddings must use the same model; switching = full re-index.
Feynman drill — say it out loudExplain why 4,000-token chunks might hurt retrieval accuracy even though they contain more info.
Step 1 rate your confidence · Step 2 pick your answer
Retrieval keeps pulling loosely-related chunks. A reasonable first tuning move is:
Step 1 — how sure are you?