Applied AI / Forward Deployed Engineer · LLM Foundations · Lesson 5 of 5
How models are made: pretraining, SFT, RLHF
You won't train frontier models, but you must understand the three stages to reason about behavior, refusals, and fine-tuning.
- Pretraining — predict the next token over a huge web-scale corpus. Produces a base model: knowledgeable but not helpful/aligned; it just autocompletes.
- Supervised fine-tuning (SFT / instruction tuning) — train on curated (prompt → ideal response) pairs so the model follows instructions and answers rather than autocompletes.
- RLHF / preference tuning (RLHF, DPO, RLAIF) — humans (or AI) rank responses; the model is optimized toward preferred behavior. This produces the helpful, harmless, honest chat model you actually call.
Why this matters for youRefusals, tone, and 'safety' behavior come from the alignment stage — you steer them with prompting, not retraining. Knowledge comes from pretraining (frozen at a cutoff date) — you add fresh/private knowledge with RAG or tools, not fine-tuning.
Knowledge cutoffThe model's world knowledge stops at its training cutoff. For anything recent or private, it will confidently hallucinate. Ground it with retrieval or live tool calls.
◆ Lock it in
- Three stages: pretraining (knowledge) → SFT (instruction-following) → RLHF/DPO (alignment).
- The base model just autocompletes; alignment makes it a helpful assistant.
- Add fresh/private knowledge via RAG/tools, not fine-tuning; steer behavior via prompting.
Feynman drill — say it out loudExplain the difference between what pretraining gives a model and what RLHF gives it.
Step 1 rate your confidence · Step 2 pick your answer
A model confidently makes up facts about your company's internal policy. The best fix is usually:
Step 1 — how sure are you?