DevOps, SRE & Cloud Interview · Networking, Security & LLMOps · Lesson 3 of 3
LLMOps & AI infrastructure
Serving models in production (LLMOps / MLOps) is DevOps with new constraints: models are large, GPU-bound, and non-deterministic. The pipeline and reliability instincts transfer; the resource profile does not.
- Serving: expose a model behind an API with a specialized inference server (vLLM, SGLang, TGI, Triton) that batches requests and manages the KV cache to raise GPU throughput.
- GPUs are the scarce, expensive resource: memory (VRAM) usually bounds you before compute. Watch utilization, batch to fill the GPU, and autoscale on queue depth / GPU metrics, not just CPU.
- Latency vs throughput: batching raises throughput but adds latency; measure time-to-first-token and tokens/sec, not just request rate.
- Model deployment: version models and prompts like artifacts; roll out with canary + evals, keep the ability to roll back to the prior model.
- Cost control: right-size/quantize models, cache responses, and route easy requests to smaller/cheaper models.
Same discipline, new bottleneckCI/CD, observability, canary rollouts and IaC all apply — but the bottleneck moves to the GPU. A model service that ignores VRAM limits, batching, and GPU utilization will be slow and ruinously expensive no matter how clean the pipeline is.
◆ Lock it in
- LLMOps = DevOps discipline (CI/CD, canary, observability, IaC) with a GPU-bound, non-deterministic workload.
- Use an inference server that batches + manages the KV cache; the scarce resource is usually VRAM.
- Measure time-to-first-token and tokens/sec; version + canary models; control cost with quantization, caching, and model routing.
Feynman drill — say it out loudExplain how serving an LLM differs from serving a normal web service, and why GPU utilization drives the design.
Step 1 rate your confidence · Step 2 pick your answer
When scaling an LLM inference service, which resource is usually the binding constraint?
Step 1 — how sure are you?