LLM Interview Questions & Answers
LLM fundamentals are table stakes for any AI-engineering interview. Know how models actually work — tokens, embeddings, context, sampling, and training.
Key concepts to know
- LLMs predict the next token from a probability distribution; capability is emergent from scale.
- Models operate on tokens (~4 chars each), not words or letters — this drives cost, limits, and quirks.
- The engine is the Transformer; self-attention lets every token weigh every other; cost scales ~quadratically with length.
- An embedding = a vector encoding meaning; similar meaning → nearby vectors.
- Cosine similarity measures closeness; it powers semantic search and RAG.
- Embedding models are separate and cheaper than chat models.
- Context window = working memory; everything (system + history + docs + query + answer) shares it.
- APIs are stateless — you resend history each call; that's what "memory" is.
- Watch lost-in-the-middle: put key instructions/data at the start or end.
- Temperature low for precision tasks (extraction, code); high for creative tasks.
- Top-p, max tokens, and stop sequences are your other sampling controls.
- Temperature 0 is not a guarantee of identical output — design for validation + retries.
- Three stages: pretraining (knowledge) → SFT (instruction-following) → RLHF/DPO (alignment).
- The base model just autocompletes; alignment makes it a helpful assistant.
- Add fresh/private knowledge via RAG/tools, not fine-tuning; steer behavior via prompting.
Interview questions & answers
Fundamentally, what is a language model computing at each step?
A probability distribution over the next token given the preceding tokens.
Why: At each step it outputs a distribution over the next token. Sampling from that distribution (repeatedly) produces text. It's not a lookup or a guaranteed-correct oracle — hence hallucinations.
Two sentences have an embedding cosine similarity of 0.92. What does that imply?
They are semantically very similar in meaning.
Why: High cosine similarity ≈ close in meaning-space, i.e. semantically similar — even if the wording differs. It says nothing about identical grammar/tokens or fine-tuning.
Where should you place the single most important instruction in a very long prompt?
At the beginning or the end, because models attend best to the edges (lost-in-the-middle).
Why: The lost-in-the-middle effect means edge positions get the most reliable attention. Critical instructions belong at the start or end of a long context.
You need consistent JSON extraction from documents. Best setting?
Temperature ~0, request structured output, validate against a schema, retry on failure.
Why: Extraction wants low variance: temperature ~0, structured output, and schema validation with retries — because even temp 0 isn't perfectly deterministic.
A model confidently makes up facts about your company's internal policy. The best fix is usually:
Ground it with retrieval (RAG) or a tool that fetches the real policy, and instruct it to only answer from provided context.
Why: Private/fresh knowledge isn't in pretraining. Retrieval or tools supply the facts; instructing 'answer only from context' curbs hallucination. Retraining/fine-tuning on web data won't add your private policy.