Applied AI / Forward Deployed Engineer · Prompt Engineering · Lesson 4 of 4
Structured output & making prompts robust
Production systems need machine-parseable output. Free-form text is a liability. Techniques, best to okay:
- Structured outputs (constrained decoding): every major provider now guarantees schema-valid output at the API level — the model can't emit invalid JSON. This is the default; retry-on-parse-failure loops are legacy.
- Ask for JSON + validate + retry: the fallback when structured outputs aren't available — instruct the schema, parse, and on failure send the error back and retry once.
- Prefill / response prefix: legacy technique for models without structured outputs — start the assistant's turn with { to force JSON.
Schema compliance is solved; semantic correctness is notThat's the 2026 framing to say out loud. Constrained decoding killed malformed-JSON bugs, but the model can still fill a perfectly valid schema with wrong values, insane numbers, or hallucinated IDs. Validation now guards meaning (Pydantic/zod + business-rule checks), not syntax.
Iterating on prompts like an engineer
- Keep prompts in version control, not hardcoded strings scattered in code.
- Change one thing at a time and measure against an eval set — never 'it feels better'.
- Build a small set of failure cases and re-run them on every prompt change (regression testing).
◆ Lock it in
- Prefer structured output / schema-constrained generation; else JSON + validate + retry.
- Treat model output as untrusted: parse, validate, fallback.
- Engineer prompts: version control, change one variable, test against an eval + regression set.
Feynman drill — say it out loudDescribe how you'd guarantee a downstream service always receives valid JSON from an LLM.
Step 1 rate your confidence · Step 2 pick your answer
The most robust way to ensure valid structured output is to:
Step 1 — how sure are you?