Applied AI / Forward Deployed Engineer · LLM Foundations · Lesson 4 of 5
Temperature, sampling & determinism
The model outputs a probability distribution; sampling parameters decide how you pick from it.
- Temperature (range is provider-dependent — OpenAI 0–2, Anthropic 0–1): scales randomness. Low (~0–0.3) = focused, deterministic-ish, best for extraction/classification/code. High (~0.8+) = creative, varied, best for brainstorming/writing.
- Top-p (nucleus): sample only from the smallest set of tokens whose cumulative probability ≥ p. Another way to control diversity.
- Max tokens: caps the response length (and cost).
- Stop sequences: strings that force generation to halt.
Determinism reality checkEven at temperature 0, LLMs are not perfectly deterministic in production (floating-point, batching, model updates). Never write code that assumes byte-identical outputs. Design for validation and retries, not exact reproducibility.
Interview answer"For a data-extraction task I'd use temperature 0 to minimize variance, ask for structured JSON, validate it against a schema, and retry on failure." That sentence signals production maturity.
◆ Lock it in
- Temperature low for precision tasks (extraction, code); high for creative tasks.
- Top-p, max tokens, and stop sequences are your other sampling controls.
- Temperature 0 is not a guarantee of identical output — design for validation + retries.
Feynman drill — say it out loudYou're extracting invoice fields into JSON. What sampling settings do you choose and why?
Step 1 rate your confidence · Step 2 pick your answer
You need consistent JSON extraction from documents. Best setting?
Step 1 — how sure are you?