RAG Interview Questions & Answers
Retrieval-Augmented Generation (RAG) is the default architecture for grounding LLMs in real data. Interviews probe the pipeline, failure modes, and evaluation.
Key concepts to know
- RAG injects relevant data at query time so the model answers from your sources.
- It fixes freshness, private knowledge, and hallucination/attribution together.
- RAG adds knowledge; fine-tuning changes behavior — different tools, often combined.
- 2026: retrieval is a tool in the agent loop; classic RAG for big/fresh/permissioned corpora, agentic search for navigable ones.
- Indexing: load → chunk → embed → store. Retrieval: embed query → search → (rerank) → assemble → generate.
- Use the same embedding model for chunks and queries.
- Most RAG failures are retrieval failures — invest there first.
- Chunk size is a tradeoff: too big = muddy, too small = fragmented; add overlap.
- Prefer structure/semantic-aware chunking and attach rich metadata.
- Query and chunk embeddings must use the same model; switching = full re-index.
- Layer hybrid search + reranking for precision; add query rewriting and metadata/permission filters.
- Instruct the model to answer only from context and admit ignorance.
- Know the failure modes: missing chunk, ignored context, stale index, conflicts, doc-borne injection.
Interview questions & answers
You need answers grounded in a constantly-changing internal knowledge base, with citations. Best approach?
RAG: retrieve the relevant chunks at query time and have the model answer from them with source citations.
Why: Changing facts + citations = RAG. Nightly fine-tuning is costly, slow to update, and gives no attribution. Pasting everything is expensive, slow, and hits lost-in-the-middle.
Your RAG bot gives wrong answers. It turns out the correct passage was never in the retrieved set. This is primarily a failure of:
Retrieval — chunking/embedding/search didn't surface the right passage.
Why: If the right context isn't retrieved, generation can't recover. This is a retrieval-quality problem: revisit chunking, hybrid/keyword search, and reranking.
Retrieval keeps pulling loosely-related chunks. A reasonable first tuning move is:
Reduce chunk size and split on semantic/heading boundaries with some overlap, then re-measure.
Why: Muddy retrieval often means chunks are too large or split mid-idea. Smaller, structure-aware chunks with overlap sharpen embeddings. Metadata helps (don't remove it); generation model/temperature aren't the issue.
Users search by exact SKU codes and semantic vector search keeps missing them. Best fix?
Add keyword/BM25 search (hybrid) so exact tokens like SKUs are matched, alongside semantic search.
Why: Exact identifiers are a classic weakness of pure vector search. Hybrid (semantic + keyword/BM25) recovers exact-term matches like SKUs and IDs.