Streaming, token-budget math & multi-tenancy

Interviewers love a candidate who can do the arithmetic. LLM latency and cost are governed by tokens, and you should be able to reason about them out loud.

Latency & streaming

• Two numbers dominate: TTFT (time to first token — dominated by prompt length + retrieval) and TPOT (time per output token, i.e. generation speed).
• Total latency ≈ TTFT + (output tokens x TPOT). Long outputs are slow — cap generation and prefer concise formats.
• Stream tokens to the user so perceived latency = TTFT, not full completion. Non-negotiable for interactive UX.
• Shrink prompts and retrieved context to cut TTFT; long stuffed contexts hurt both latency and quality.

Token-budget & capacity math

A worked example you can reproduce on the whiteboard: 1,000 in + 500 out tokens/request. At, say, $3 per 1M input and $15 per 1M output, that's ~$0.003 + ~$0.0075 ≈ $0.01/request. At 1M requests/day that's ~$10k/day — now caching and routing aren't optional. For capacity, size against peak QPS x tokens/request vs. provider TPM/RPM limits, and plan for rate-limit backpressure.

Multi-tenancy

• Isolation: partition data and vector indexes per tenant; never let one tenant's retrieval reach another's docs.
• Fairness: per-tenant rate limits and token quotas so one heavy user can't starve others (noisy-neighbor).
• Cost attribution: meter tokens per tenant for billing and abuse detection.
• Caching must be tenant-scoped so a cache hit can't cross a tenant boundary.