Design an LLM inference serving platform at scale

Expected question

"Design an LLM inference serving platform that can handle production traffic at scale. How do you achieve low latency, high throughput, and cost efficiency? Walk through KV cache, batching, and scaling."

Variant forms

Interviewers often ask the same design with different framing — recognize the archetype:

• "Design the inference stack for ChatGPT-scale token generation — how do you serve millions of concurrent users?"
• "Our API latency P99 spiked to 8 seconds under load. How would you architect LLM serving to fix it?"
• "Design a model-serving layer that routes between a fast cheap model and a slow capable model behind one API."
• "How would you deploy Llama-3 70B on a fixed GPU budget for 10,000 requests/minute at peak?"
• "Design continuous batching and PagedAttention-style KV management for a multi-tenant inference API."
• "You need 99.9% availability for /v1/chat/completions across three regions — walk me through serving architecture."
• "How do you scale from a single vLLM pod to a fleet that handles Black Friday traffic for an AI product?"
• "Design inference infrastructure where prefill is compute-bound and decode is memory-bandwidth-bound — what changes?"

Where this actually gets asked

GPU scheduling, batching, and autoscaling for model serving is one of the most consistently reported AI-infra system design topics across OpenAI, Anthropic, and Google DeepMind interview loops — but treat it as a well-documented archetype, not a single verbatim question. The best-attributed source: a Blind post on DeepMind's Applied AI Engineer loop names "efficiency (quantization/distillation)" and "system architecture/scale" as explicit focus areas of its ML-system-design round. Prep aggregators (designgurus.io, systemdesignhandbook.com) repeatedly describe "serve a model like GPT-4/Claude to millions of users" as the framing at OpenAI and Anthropic specifically, but I could not confirm a single verbatim quote from a primary source (Glassdoor's own page for this was inaccessible to fetch directly). Write your answer assuming the interviewer wants to see you reason from first principles about GPU memory and scheduling, not recite a specific company's stack.

Executive summary

30-second thesis

I'd start with continuous batching and a real KV-cache budget — not tensor-parallel diagrams. Admit what fits, shed what doesn't, and keep a warm previous revision so a bad canary isn't a weekend outage.

2-minute answer

First I'd pin the SLOs: interactive vs batch, model size, and the GPU cost ceiling. Then I'd serve with continuous batching and paged KV, and I'd keep TTFT and TPOT as separate budgets — mixing them is how teams lie to themselves about P99.

Under load I'd reject or queue by priority instead of letting the backlog grow forever. Quantization only after a quality gate says the slice still passes. GPU death is a product mode: drain, restart, replay or fail clean — and roll back to the previous revision you kept warm.

What I'd refuse: opening with exotic TP/PP topology before you've measured a hotspot. Custom kernels come after the scheduler and KV math, not instead of them.

Quantitative trade-offs

Numbers and thresholds in interview delivery should be labeled H unless the candidate can defend a measured baseline. Open repositories are O; researched public patterns are R. Do not upgrade O/R into employer P adoption.

Migration and rollout

This is a model-revision / serving-pool cutover, not an index migration speech.

1. Stand up the new revision in a separate replica pool. Pin traffic by model id / alias so clients don't silently jump.
2. Shadow or dual-run on a sample: compare TTFT, TPOT, tokens/s, and the eval slice that gates quality — especially if you quantized.
3. Canary interactive traffic first on a small tenant cohort with a holdback. Batch jobs can wait; they're cheaper to replay than angry chat users.
4. Drain in-flight sequences on the old pool before you reclaim GPUs. Hard cut mid-stream is how you ship truncated answers.
5. Keep the previous revision warm until soak passes. Rollback = flip the alias back. Kill switch = stop admitting new interactive work when KV or queue depth trips.

Org ownership and operating model

• Inference / ML platform owns scheduler policy, KV allocator, replica pools, and model aliases.
• Model owners own weights, quantization choices, and the quality gate that blocks a bad revision.
• Product owns interactive vs batch SLO budgets and which tier gets shed first under load.
• SRE owns GPU failure runbooks, capacity pages, and multi-AZ / multi-region drain drills.
• FinOps owns GPU-hour budgets and the halt when a tenant or model burns the ceiling.

If model owners can ship a quantized revision without the quality veto, you will learn about it from customers, not from CI.

Requirements

Functional

• Send a prompt, get a streamed completion token-by-token.
• Multiple model sizes/variants behind one API (fast/cheap vs slower/better).
• Interactive (low-latency chat) and batch (throughput-optimized) workloads on the same fleet without letting batch starve chat.

Non-functional

• Tight TTFT for interactive traffic (illustrative H: p50 under ~200ms — confirm their budget).
• High aggregate tokens/sec without silently degrading quality.
• GPU memory is the scarce resource — at long context and high concurrency, KV often beats weights.
• Under load: reject or queue with backpressure. Don't pretend every request still gets the same latency.

Core entities

• Request: prompt, model, sampling params, arrival time, priority/tier.
• Sequence: generation state — tokens so far, KV blocks it owns.
• KV cache block: fixed-size GPU memory chunk for attention K/V; allocated on demand.
• Model replica: one loaded weight set on one or more GPUs (tensor-parallel shards).

API / interface

Auth: Authorization: Bearer <tenant_token>. Interactive vs batch is a routing signal on the wire.

POST /v1/chat/completions
Authorization: Bearer <token>
Idempotency-Key: <uuid>
{
  "model": "llama-3.1-70b",
  "messages": [{"role":"user","content":"..."}],
  "max_tokens": 1024,
  "stream": true,
  "priority": "interactive",
  "metadata": {"request_id":"...","tenant_id":"..."}
}
→ 200 text/event-stream (token deltas) | 429 capacity_exhausted | 503 model_warming

GET /v1/models
→ {"data":[{"id":"llama-3.1-70b","status":"ready","p50_ttft_ms":120,"inflight":42}]}

GET /v1/metrics/serving?model=llama-3.1-70b
→ {"ttft_p99_ms":...,"tpot_p50_ms":...,"gpu_util":0.81,"kv_cache_used_pct":0.64}

Staff+ callout: expose queue depth / TTFT breach as client-visible signals so callers can shed load themselves.

Data Flow

Happy path for interactive completion, then where NFRs show up (admission, KV pressure).

Rendering architecture diagram…

Admission is where you say no. If everything reaches the GPU, you've already lost the TTFT fight.

High-level design

Rendering architecture diagram…

Router classifies by model and priority. Scheduler decides which sequences enter the next forward pass. The needle-mover is continuous batching: add/remove sequences every decode step as memory allows, instead of waiting for a static batch to finish. That's the difference between GPUs sitting at ~20–30% and actually earning their rent.

Deep dive 1: KV cache memory management (the actual bottleneck)

Hard constraint: at long context and high concurrency, KV — not weights — runs you out of memory first. If you don't budget blocks, everything else is theater.

Naive approach: pre-allocate each sequence's max KV contiguously. You pay for fragmentation and for sequences that finish early.

Default: paged KV (block table per sequence, allocate on demand). When memory runs out, preempt — swap or recompute a victim — instead of taking the replica down. I've worked through that allocator/scheduler math in an open lab (O); the interview signal is deriving kv_bytes_per_token × layers × concurrent sequences, not name-dropping PagedAttention.

Deep dive 2: scheduling and fairness under load

Hard constraint: interactive TTFT stays bounded while long generations are in flight.

Continuous batching alone isn't fairness. Pure FCFS lets one long decode starve a queue of short chat requests. Separate interactive and batch queues. Inside interactive, cap how long one sequence can hog decode steps before yielding.

Common mid/senior miss: "we'll just sort by priority." Fine until every request is high priority. You still need admission control — reject or backpressure when KV or queue depth crosses a threshold.

Deep dive 3: cost and capacity planning

Hard constraint: GPU-hours dominate the bill. If cost isn't a design input on day one, FinOps will make it one later — loudly.

Meter real tokens per model/tenant into both a dashboard and a budget gate. Guessed token costs and seed-data "cost" dashboards are how you ship optimism. Same scar as any fake FinOps layer: the invoice is the only honest metric until metering is real.

Staff+/Principal signal rubric

• Mid-level: queue + replica pool; may need prompting for KV pressure.
• Senior: continuous batching unprompted; KV as binding constraint at high concurrency.
• Staff+: real KV budget math, explicit preemption/swap, interactive vs batch fairness beyond "priorities."
• Principal: cost model as a day-one input; quantization / model-size trade-offs with gates, not vibes; migration that drains in-flight work.

Follow-up questions to expect

• "GPU dies mid-generation?" KV on that GPU is gone. Detect fast, fail clean, or replay from a checkpoint if you actually built that. Streams don't continue by magic.
• "1M+ context without KV eating everything?" Sparse/sliding window, aggressive prefix cache, eviction — each trades quality; say which quality you'll lose.
• "New model version without downtime?" Blue/green pools, drain in-flight on the old revision, then cut. Hard cutover pages you.
• "Out of scope in 45 minutes?" Stop after scheduler + KV + admission; defer full multi-region/tenant to 09.

What I'd ask them

• Interactive vs batch mix and the real TTFT/TPOT budgets?
• Fixed GPU count or elastic — and what's the kill switch when KV fills?
• Which quality gate blocks quantization or a model swap?

Related

• vllm-architecture-lab — open PagedAttention / continuous-batching lab (O, not hyperscale prod ownership)
• agent-finops — real vs guessed cost metering
• scalability-governance-tradeoffs/01: Cost vs. latency vs. safety