Design a multi-tenant AI platform

Expected question

"Design a multi-tenant AI platform where many customers share inference, agents, and data planes. How do you isolate tenants, quota usage, and customize per customer?"

Variant forms

Interviewers often ask the same design with different framing — recognize the archetype:

• "Design Azure OpenAI-style shared vs dedicated capacity for enterprise tenants."
• "How do you prevent noisy-neighbor token spikes from starving other customers?"
• "Design per-tenant API keys, budgets, custom system prompts, and fine-tuned adapters."
• "Scale from 10 to 10,000 tenants on one control plane — what do you shard?"
• "Design a hub-and-spoke agent architecture with centralized governance and isolated runtimes."
• "How do you bill accurately when one tenant runs 200-agent background jobs?"
• "Design data residency: EU tenants' prompts and logs never leave EU regions."
• "Architect tenant-scoped tool credentials so Customer A's Stripe key is never visible to B."

Where this actually gets asked

Best-sourced at Microsoft: the official Azure Architecture Center has a real, published document explicitly discussing per-tenant vs. shared-instance trade-offs for Azure OpenAI, noisy-neighbor risk, and TPM/RPM (tokens/requests-per-minute) quota mechanics — with Provisioned Throughput Units (PTUs, dedicated reserved capacity) documented as the real mitigation for noisy-neighbor contention. Google's Vertex AI Dedicated Endpoints (cloud.google.com) confirm dedicated-endpoint isolation as a real, documented pattern. OpenAI publishes its own tier/TPM/RPM rate-limit structure but no internal architecture document. Two claims caught and rejected during research: a specific, widely-repeated "a tenant's recursive workflow took down services for 47 other customers for 3 hours" Vertex AI incident traced to an unsourced personal blog post with no citation trail — not used here. A claim that "system design for a multi-tenant Claude deployment" is a specific, confirmed Anthropic interview stage traced only to a blog author's own generic characterization, not any leaked candidate account — also not presented as verified. Treat multi-tenancy/isolation as a classic distributed- systems archetype asked broadly across big tech; the GPU-batching/KV-cache/token-quota angle below is the genuinely AI-specific wrinkle that makes this harder than a typical multi-tenant web service question.

Executive summary

30-second thesis

I'd isolate identity, data, cache, and secrets per tenant first — share only control-plane machinery that can't become a cross-tenant channel. Density is a cost optimization after isolation proofs.

2-minute answer

Tenant comes from verified auth, never a body field. Indexes, caches, logs, and tool credentials are scoped by tenant policy. Meter and budget per tenant so one runaway job doesn't eat the fleet. Noisy-neighbor protection and dedicated/stability channels for enterprises that paid for them. Before chasing denser packing, I'd document blast radius and incident fan-out — "shared GPU pool" without quota ledgers is how you page every customer at once.

What I'd ask them: Shared vs dedicated for the top tenants? Soft vs hard quota at 100%? Prompts/logs stay in-region?

Principal verbal board (8 minutes)

HM screens often want this before the 40-minute platform design. Same default. Threat model first.

I'd start with the threat model, not a tenancy column. For agents the leak is usually RAG bleed — Tenant A's query pulling Tenant B's chunks — or a tool call as the wrong tenant. Authorization happens before ranking, and every write carries tenant_id through a gateway. If we can't fail a negative test in CI, we don't have isolation. We have a slide.

What I'd refuse: a shared default corpus in prod. Fail-open gateway on irreversible tools. Cache keyed only on prompt text. An "admin bypass" with no dual control.

Request
  → Identity: JWT with tenant_id + principal + roles/clearance
  → AuthZ before retrieval: filter corpus by tenant + clearance BEFORE hybrid rank
  → Data plane: per-tenant collections / namespaces / RLS; no shared default corpus in prod
  → Control plane: per-tenant policy bundles, tool allowlists, HITL rules
  → Runtime: tenant-keyed cache + budgets; no cross-tenant keys
  → Audit: signed logs with tenant_id on every tool decision

Any write (CRM, publish, notify, deploy, payment): tool args must include tenant_id matching the JWT; gateway rejects mismatch; high-risk → HITL with tenant visible; fail-closed in production if the gateway is unreachable.

Happy-path RAG quality is not an isolation test. Week one on their stack is identity — no anonymous retrieve. Week two is filter-before-rank plus NT-1. Week three is gateway on writes. Week four is cache keys and the NT suite in CI. I wouldn't start with a multi-region rewrite.

Demo vs Strict: live demos may use a labeled Demo principal. Production is JWT, durable policy, AuthZ-before-ranking, fail-closed side effects. I wouldn't sell an in-memory registry as multi-tenant SaaS.

Quantitative trade-offs

Numbers and thresholds in interview delivery should be labeled H unless the candidate can defend a measured baseline. Open repositories are O; researched public patterns are R. Do not upgrade O/R into employer P adoption.

Migration and rollout

This is a tenancy cutover. Wrong order creates silent cross-tenant channels.

1. Prove identity: tenant id only from verified auth — never from request body — in a dual-run that logs mismatches.
2. Migrate data planes first: indexes, caches, object prefixes, tool credentials scoped by tenant. Shared semantic cache stays off until keyed by tenant+policy+model.
3. Turn on per-tenant meters, quotas, and noisy-neighbor limits in observe-only mode; page on would-be breaches.
4. Move low-risk tenants to the new isolation path; keep regulated / dedicated customers on silo until isolation tests (cache/log/index) pass.
5. Rollback = pin tenant back to previous pool/silo. Kill switch = freeze new tenant onboarding + deny cross-tenant cache.

Org ownership and operating model

• AI platform owns shared control plane, quota ledgers, and isolation invariants.
• Security / identity owns tenant derivation from IdP and cross-tenant access reviews.
• SRE owns noisy-neighbor pages and dedicated-pool capacity for enterprise tiers.
• Customer success / enterprise owns which tenants paid for silo vs pool — platform doesn't silently densify them.
• FinOps owns per-tenant budgets and halt-on-breach.

Requirements

Functional

• Multiple customers (tenants) share the same underlying model-serving infrastructure, each with their own API key, usage accounting, and rate limits.
• Support per-tenant customization (fine-tuned model variants, custom system prompts, tool configurations) without each tenant requiring a fully separate deployment.
• Provide a path to fully dedicated capacity for tenants who need guaranteed throughput or stronger isolation, without re-architecting the platform for them specifically.

Non-functional

• No tenant's traffic pattern should degrade another tenant's latency or throughput — the "noisy neighbor" problem, which is more severe here than in typical multi-tenant web services because GPU batching means tenants can share the same physical compute at the request level, not just the same rack.
• Per-tenant quota enforcement (TPM/RPM) needs to be accurate under concurrent load, not eventually-consistent — a tenant hitting their quota should be rejected promptly, not after they've already burned through GPU capacity meant for others.

Core entities

• Tenant: an isolated customer/account with its own API key, quota (TPM/RPM), and billing.
• Model variant: the base model plus tenant-specific customization (fine-tune, system prompt, tool access) — may or may not require dedicated GPU capacity depending on tier.
• Capacity tier: shared (multiplexed across tenants), or dedicated (Provisioned Throughput Unit-style reserved capacity for one tenant).
• Quota ledger: real-time tracking of each tenant's token/request consumption against their configured limit.

API / interface

Auth: tenant-scoped API keys or org JWT with tenant_id claim verified server-side.

POST /v1/tenants
{"name":"acme","plan":"enterprise","isolation":"namespace"} → 201 {"tenant_id":"ten_..."}

PUT /v1/tenants/{tenant_id}/quotas
{"rpm":600,"tpm":2e6,"max_concurrent":50,"budget_usd_day":500}
→ 200 {"quotas":{...},"effective_at":"..."}

POST /v1/tenants/{tenant_id}/keys
{"scopes":["inference:chat"],"expires_at":"..."} → 201 {"key_id":"key_...","secret":"..."}  # secret once

GET /v1/tenants/{tenant_id}/usage?window=24h
→ {"rpm_used":120,"tpm_used":4.2e5,"cost_usd":88.4,"throttled_requests":12}

POST /v1/chat/completions
Authorization: Bearer <tenant_key>
{"model":"...","messages":[...]}
→ 200 ... | 429 {"error":"quota_exceeded","quota":"tpm","reset_at":"..."}

Staff+ callout: quotas and keys are control-plane APIs; data-plane errors must name which quota fired.

Data Flow

Control plane sets quotas/keys; data plane enforces on every inference call and meters usage.

Rendering architecture diagram…

High-level design

Maps to functional requirements from step 1 — the component architecture that makes the API and data flow real.

Rendering architecture diagram…

The critical design decision: quota enforcement happens at the gate, before scheduling — not as a post-hoc billing reconciliation. A design that only tracks usage after the GPU has already done the work has already let the noisy-neighbor problem happen; it can bill for it accurately, but it hasn't prevented it.

Deep dives below target non-functional requirements (latency, scale, failure, cost, security).

Deep dive 1: shared vs. dedicated capacity, and the real mitigation for noisy neighbors

Hard constraint: one tenant's token spike must not starve everyone else — quotas and reserved capacity are the product, not a nice dashboard.

Common mistake at the mid/senior level: proposing only per-tenant rate limits (TPM/RPM caps) as the complete isolation story. Rate limits bound how much of the shared pool one tenant can consume, but if the shared pool's batching itself isn't tenant-aware, a large batch dominated by one tenant's long-context requests can still add latency variance for other tenants within their quota — quota caps the blast radius, it doesn't eliminate cross-tenant interference within the allowed range.

Deep dive 2: customization without fragmenting the platform

Hard constraint: per-tenant prompts/adapters/tools can't fork the control plane into N snowflake deployments.

A common weak design gives each tenant a fully separate deployment the moment they need any customization (a fine-tune, a custom system prompt), which doesn't scale operationally past a handful of tenants. The better pattern, consistent with how Azure OpenAI and Vertex AI structure this: keep the base model and serving infrastructure shared, and express tenant customization as a layer applied at request time — a fine-tuned adapter loaded alongside the shared base weights, or a system prompt injected before the tenant's user prompt — rather than a fully forked deployment per tenant. Dedicated capacity (the top row of the table above) is reserved for tenants who need isolation guarantees, not merely customization.

Deep dive 3: the AI-specific isolation problem generic multi-tenancy design doesn't cover

Hard constraint: prompt caches, embeddings, and tool credentials leak across tenants if keys aren't tenant+policy scoped — classic SaaS isolation isn't enough.

A generic multi-tenant system design answer (connection pooling, per-tenant database schemas, row-level security) doesn't address the specific mechanism unique to LLM serving: continuous batching multiplexes multiple tenants' requests into the same GPU forward pass (see system-design/01's deep dive on this same scheduler). This means tenant isolation isn't just an API-gateway concern — it extends into the scheduler itself, which needs to be quota-aware and, for dedicated tiers, needs to guarantee a tenant's requests never share a batch with another tenant's at all. This is the genuinely AI-specific wrinkle: isolation has to be reasoned about one layer deeper than the request/response boundary most multi-tenant system designs stop at.

Deep dive 4: quota-ledger failure mode (must say out loud)

Hard constraint: if the quota ledger is wrong or down, fail closed on spend for shared pools — inventing capacity is how you take the neighborhood down.

If the quota ledger is unavailable, fail closed (503 / reject) — never open unlimited inference. Dedicated pools isolate noisy neighbors for batch; shared pools need scheduler fairness. Name a concrete interference budget (e.g., neighbor burst adds ≤50ms to interactive P99 TTFT vs dedicated). Skip billing/invoicing design in 45 minutes; cover gate-before-schedule + tier table.

Staff+/Principal signal rubric

• Mid-level: proposes per-tenant API keys and rate limits without addressing GPU-batching- level interference between tenants sharing the pool.
• Senior: identifies TPM/RPM quota enforcement at the gate (before scheduling, not after) as the mechanism bounding noisy-neighbor impact.
• Staff+: designs the shared-vs-dedicated capacity tiering explicitly, and connects tenant customization (fine-tunes, prompts) to a layering mechanism rather than per-tenant forked deployments. Unprompted: AuthZ-before-rank and tenant on tool args.
• Principal: scheduler-level isolation (continuous batching multiplexes tenants at the GPU); negative evals NT-1..NT-5 in CI; cache/trace/prompt-injection as leak channels; week-1–4 ship on their identity stack; Demo vs Strict honesty. Org blast radius: who inherits the isolation standard. In 8 minutes they get the threat model; in 40 they get quotas and dedicated tiers.

Follow-up questions to expect

• "What do you skip in 45 minutes?" Multi-region SaaS rewrite. Lock threat model, AuthZ-before-rank, NT suite, quota-before-schedule.

• "A tenant on the shared tier complains about inconsistent latency. How do you diagnose it?" (Answer: check whether their requests are being batched alongside high-token-count requests from other tenants sharing the same pool — this is a scheduler-visibility problem, not just an API-gateway metric.)

• "How would you migrate a growing tenant from shared to dedicated capacity without downtime?" (Answer: provision the dedicated pool, dual-route a small percentage of that tenant's traffic to verify it behaves identically, then cut over fully — the same canary pattern used for model version rollouts, applied to a capacity-tier migration instead.)

• "A noisy neighbor burns everyone's TPM." Per-tenant token budgets at the gateway first, fair scheduling second, dedicated capacity for tenants who pay for isolation. Don't shard the control plane on day one.

• "EU residency — prompts never leave EU." Pin stores and cache keys to the region. Scope tool credentials so an EU tenant's key never leaves. Honest limit if your public demo isn't multi-region.

• "Per-tenant LoRA / custom prompts." Adapter registry keyed by tenant. Prompts are data. Same tenant boundary as chunks.

• "How do you bill this?" Meter at the gateway, not in the model client. Budgets and deny-on-overrun are policy.

• "We already have one shared vector index." First ship is a mandatory pre-rank filter plus NT-1 — not a six-month re-index. Per-tenant collections when the contract or leak class demands it.

• "Isn't this just Postgres RLS?" RLS is a data-plane control. Agents still leak through caches, traces, prompt injection, and tool args that don't carry tenant.

• "Prompt injection that says ignore tenant." AuthZ bug, not a prompt contest. Scope comes from the token, not the completion.

What I'd ask them

• Shared pool vs dedicated for the top enterprise tenants?
• Soft vs hard quota on tokens — what happens at 100%?
• Data residency: prompts/logs stay in-region or only weights?

Related

• system-design/01: LLM inference serving at scale — the shared scheduler this entry's isolation problem builds on
• 02 RAG — access before ranking
• 24 Agent IAM / MCP
• 33 Context engineering — long-context is not an ACL
• 11 Timebox / 8-minute altitude
• cloud-architecture/01: GPU capacity planning and procurement
• agent-finops — real per-tenant/per-call cost metering, the accounting half of this problem