Product Manager Interview (+ AI PM) · AI Product Management · Lesson 1 of 4
Scoping an LLM feature (build vs buy vs API)
AI PM starts the same as any PM work — user problem first — but adds AI-specific judgment. The first question is not 'which model' but 'is this a real user problem, and is probabilistic AI the right tool?' Many 'AI features' are better solved by deterministic software.
Build vs buy vs API
- Use a hosted API (OpenAI, Anthropic, etc.): fastest to ship, no infra, pay per token. Default for most features — buy before you build.
- Fine-tune / open-weight self-host: only when you need domain adaptation, data control/privacy, lower per-unit cost at scale, or latency guarantees a hosted API can't meet.
- Train from scratch: almost never for a PM — enormous cost/time, rarely justified.
- Decide on differentiation: build where AI is your moat; buy the commodity parts.
Prompt → RAG → fine-tune → trainClimb the cost ladder only as needed: try prompting first, add retrieval (RAG) to ground it in your data, fine-tune for style/format at scale, and train essentially never. Most 'we need a custom model' asks are solved by better prompts + RAG.
AI as a solution looking for a problemDon't bolt an LLM onto a problem that a form, a rule, or search solves better. Probabilistic AI adds cost, latency, and a new failure mode (wrong answers). Use it where language/understanding/generation is the actual need.
◆ Lock it in
- Lead with the user problem; ask whether probabilistic AI is even the right tool.
- Buy/API before build; self-host or fine-tune only for data control, scale cost, or latency needs.
- Climb the ladder: prompt → RAG → fine-tune → train, only as far as required.
Feynman drill — say it out loudA stakeholder wants to 'add AI' to the product. Walk through how you'd decide whether and how to build it.
Step 1 rate your confidence · Step 2 pick your answer
For most new LLM features, what's the right default build approach?
Step 1 — how sure are you?