Worked example: a customer-support AI assistant

Let's design a support assistant that answers customer questions from a company's help docs + can take actions (check order status, issue refunds). This blends RAG + agents + production concerns.

Clarify (assume answers)

• 10k help articles + order DB; ~5 QPS peak; <3s first-token latency; must cite sources; refunds need a guardrail; multilingual later.

Architecture

1. Indexing: chunk help articles (structure-aware, ~500 tokens, metadata: product, section, permissions) → embed → vector store. Re-index on doc updates.
2. Retrieval: hybrid search (semantic + keyword for error codes/SKUs) → rerank top-k → assemble context.
3. Orchestration: router classifies intent → simple FAQ = RAG answer; action requests (order status/refund) = tool calls with schemas.
4. Generation: answer only from retrieved context, cite chunks, 'I don't know → escalate to human' fallback.
5. Guardrails: refunds above a threshold require human approval (human-in-the-loop); treat retrieved/user text as untrusted (injection defense); PII handling.
6. Production: stream tokens; cache stable system prompt; small model for routing, big model for hard answers; retries/fallback provider; log everything.
7. Evals + feedback: eval set of real tickets (retrieval relevance + answer faithfulness + action correctness); thumbs up/down and resolution rate online; failures feed back into the eval set and retrieval tuning.