Design an enterprise LLM gateway (unified multi-provider platform)

Expected question

"40 teams call LLMs directly — different providers, no cost visibility, PII flowing to third parties, no fallback when a provider degrades. Design a unified gateway that every internal LLM call flows through."

Variant forms

Interviewers often ask the same design with different framing — recognize the archetype:

• "Design one internal API surface over OpenAI, Anthropic, Bedrock, and a self-hosted model."
• "How do you fail over an in-flight streamed LLM call when a provider degrades mid-response?"
• "Design token-aware rate limiting for LLM traffic — why doesn't plain RPS work?"
• "Architect per-team cost attribution and hard budget enforcement for LLM spend."
• "Where does PII redaction belong when a call must leave your boundary for a third-party model?"
• "Design a migration path to move 40 teams off direct provider calls without a big-bang cutover."

Where this actually gets asked

Common 2025–26 infra pattern as companies centralize LLM spend and risk behind one platform team. Modeled on Uber's public GenAI Gateway, which serves 60+ internal use cases behind one API. Distinct from 01 LLM inference serving: that entry assumes you own the model-serving stack; here you don't own any model — you own the platform between every internal caller and every external or self-hosted provider. Also closely related to ../cloud-architecture/07 LLM gateway + semantic cache, which is grounded in this org's shipped gateway/cache planes (ADR-028/029/033) and goes deep on gateway-vs-sidecar topology and cache-safety. This entry stays at the platform-product layer — routing/fallback economics, token-aware admission control, and a 40-team migration story — cite 07 for the deployment-topology and cache-correctness depth.

Executive summary

30-second thesis

I'd put a stateless gateway in the hot path for every internal LLM call — auth, PII boundary, token-aware budgets, and health-aware routing with fallback chains — and decouple teams from providers behind model aliases so the platform can remap "fast/smart/cheap" without client changes.

2-minute answer

One OpenAI-compatible API surface for fastest adoption, with documented escape hatches for provider-specific capabilities so we don't flatten everything to the lowest common denominator. PII is detected and redacted before the provider and rehydrated in the response; some routes can't tolerate redaction's quality hit, so I'd classify those to self-hosted models instead of trying to redact them well enough.

Rate limiting has to be token-aware, not RPS-based — one request can be 100 tokens or 100K. Admission estimates tokens from the prompt and caps max_tokens, then reconciles with actuals. Fallback chains (provider A → B → self-hosted) need an explicit answer for mid-stream failure: either requests are idempotent and restart from scratch on failover, or streamed partials are lost — I'd say which, out loud, and why. Circuit breakers key on provider+region+model and trip on latency degradation, not just error rate, because a slow provider rarely trips a naive error-rate breaker.

Cost governance is enforced at admission — reject or downgrade tier on budget exhaustion — not just surfaced on a dashboard after the fact.

What I'd ask them: Is PII redaction acceptable for the highest-value use cases, or does that route need self-hosting? What's the added-latency budget for the gateway itself? Hard budget enforcement or advisory only?

Requirements

Functional

• One API surface (OpenAI-compatible) with provider routing, streaming passthrough, and tool-call normalization across providers.
• Prompt/completion logging with a retention policy.

Non-functional

• Added gateway latency under ~50ms p99 (H) — the gateway must be nearly invisible.
• Survive a single provider outage without failing user requests.
• Per-team cost attribution and enforced budgets.
• PII never leaves the trust boundary unredacted; full auditability for compliance.

Core entities

• Team / caller: identity, budget, quota, allowed model tiers.
• Model alias: logical name ("fast", "smart", "cheap") mapped to a ranked list of provider candidates.
• Route decision: the resolved provider + model + region for one request, with the reason recorded.
• Usage record: tokens in/out, cost, latency, team/use-case/environment tags — the unit cost attribution is built from.

API / interface

POST /v1/chat/completions
Authorization: Bearer <team-key>
{ "model": "smart", "messages": [...], "stream": true }
→ 200 (SSE stream) | 429 {"error":"budget_exhausted","downgrade_available":"cheap"}

GET /v1/usage?team=checkout&window=24h
→ 200 { "tokens_in": 1200000, "tokens_out": 340000, "cost_usd": 41.20 }

Staff+ callout: the client never names a provider or model directly — only an alias. Remapping an alias is a platform-side config change, not a fleet-wide client migration.

Data Flow

Rendering architecture diagram…

High-level design

Rendering architecture diagram…

Compose with ../cloud-architecture/07 for gateway-vs-sidecar deployment topology and cache-correctness depth; this entry owns multi-provider routing economics and the 40-team migration story.

Deep dive 1: routing and fallback

Model aliases decouple teams from providers. Fallback on mid-stream failure needs an explicit policy: either requests are idempotent and restart from scratch on failover, or streamed partials are lost — state which and why, don't leave it implicit. Circuit breakers key on provider+region+model and trip on error-rate and latency-percentile degradation; a provider that goes 2x slower with no errors will sail past a naive error-rate-only breaker.

Deep dive 2: token-aware rate limiting

RPS limits are the wrong unit for LLM traffic — one request can be 100 tokens or 100K. Admission estimates tokens from the prompt and caps max_tokens, then reconciles against actuals post-response. Two levels: per-team budgets (business) and per-provider quotas (protect upstream contracts) — a bursty team inside its own quota still needs weighted fair queuing against other teams sharing the same provider quota.

Deep dive 3: the PII boundary

Detect and redact before the provider (NER + regex + allowlist per data classification), rehydrate placeholders in the response. The trade-off: redaction can degrade output quality when the model has to reason over <PERSON_1> instead of a real name. The fix isn't a smarter redactor — it's classifying which use cases can tolerate redaction and routing the rest to self-hosted models instead. Saying that trade-off out loud, per-route rather than globally, is the Principal-level signal.

Deep dive 4: caching and cost governance

Exact-match cache on normalized (prompt, params) for deterministic temp-0 calls. Semantic cache (embedding similarity) only for idempotent, user-invisible workloads, opt-in per route — a stale-or-wrong-hit on a personalized or tool-using answer is a correctness bug, not a cost win. Cost governance means every request is tagged team/use-case/environment and budgets are enforced at admission — reject or downgrade tier — not just visible on a dashboard after the spend already happened.

What's expected at each level

• Mid-level: single API surface, basic routing to one or two providers.
• Senior: fallback chains, exact-match caching, per-team usage dashboards.
• Staff+: token-aware admission control, latency-percentile circuit breakers, explicit mid-stream-failure policy.
• Principal: per-route redact-vs-self-host policy instead of a global redaction rule, opt-in semantic caching with a stated correctness boundary, and a shadow-mode migration plan for cutting 40 teams over without a big-bang.

Follow-up questions to expect

• "Provider A degrades slowly — latency 2x, no errors — how does your breaker catch it?" Latency-percentile-based tripping, not just error-rate.
• "Two teams share a provider quota; one is bursty." Weighted fair queuing within the shared quota; priority tiers for user-facing vs. batch traffic.
• "How do you migrate 40 teams without a big bang?" SDK shim that mirrors traffic in shadow mode before cutover, per-team rollout with diff-based response comparison.

Related

• 01 LLM inference serving at scale
• 07 LLM evaluation & observability
• 17 LLM application security / prompt injection
• 19 Model release, canary & rollback
• ../cloud-architecture/07 LLM gateway + semantic cache — gateway-vs-sidecar topology and cache-correctness depth
• ../scalability-governance-tradeoffs/02 Build vs. buy for shared services