Applied AI / Forward Deployed Engineer · LLM Foundations · Lesson 3 of 5
Context windows & why they rule your design
The context window is the maximum number of tokens the model can attend to at once — its working memory. Everything must fit: system prompt + conversation history + retrieved documents + the user's message + room for the response. Modern models range from ~8K to 200K–1M+ tokens.
- The model has no memory between calls. Each API call is stateless — you resend the whole context every time. "Memory" in chat apps is you re-sending history.
- Bigger context ≠ free. More tokens = more cost and higher latency, and quality can degrade.
- Lost in the middle: models attend best to the start and end of a long context; facts buried in the middle get missed. Put critical instructions/data at the edges.
Long context ≠ RAG replacement"Just stuff all the docs in the 1M window" fails on cost, latency, and the lost-in-the-middle effect at scale. Retrieval that sends only the relevant chunks is usually cheaper, faster, and more accurate. Know when each wins.
Name the discipline: context engineeringManaging what's in the window is now a named craft — context engineering — and prompt engineering is framed as a subset of it. The four verbs: Write, Select, Compress, Isolate. Vocabulary interviewers recognize: attention budget (every token spends it), context rot (quality degrades as the window fills — measured across models), compaction (summarize/prune stale turns), just-in-time retrieval (fetch when needed, don't preload), progressive disclosure (load tool defs/docs on demand). Using this vocabulary instantly dates your knowledge to 2026.
◆ Lock it in
- Context window = working memory; everything (system + history + docs + query + answer) shares it.
- APIs are stateless — you resend history each call; that's what "memory" is.
- Watch lost-in-the-middle: put key instructions/data at the start or end.
Feynman drill — say it out loudA colleague says 'we have a 1M-token model, so we don't need retrieval anymore.' Give two reasons they're wrong.
Step 1 rate your confidence · Step 2 pick your answer
Where should you place the single most important instruction in a very long prompt?
Step 1 — how sure are you?