Design an enterprise LLM gateway with semantic cache — gateway vs sidecar; cache-in-process vs cache-as-service
Expected question
"Design an enterprise LLM gateway with semantic cache and model routing. How do you cut cost/latency while enforcing policy, keys, and observability?"
Variant forms
Interviewers often ask the same design with different framing — recognize the archetype:
- "Design a gateway that caches semantically similar prompts — when is it safe?"
- "How do you route prompts to GPT-4 vs Haiku vs self-hosted Llama based on complexity?"
- "Design centralized API keys, rate limits, and PII redaction before upstream LLM calls."
- "Our LLM bill doubled — architect caching, routing, and budget caps per team."
- "Gateway vs sidecar vs SDK — where does policy enforcement belong?"
- "Design cache invalidation when the underlying model version changes."
- "How do you log prompts/responses for compliance without storing secrets?"
Where this actually gets asked
Rising 2026 infra question as companies centralize LLM spend: "Design an LLM gateway," "API gateway for OpenAI/Anthropic," "semantic cache for prompts," "gateway vs sidecar." Sits between multi-tenant platforms and FinOps — Staff+ cares about deployment topology, cache safety, and blast radius — not a reverse proxy demo. Distinct from ../ai-system-design/28 Enterprise LLM gateway, which stays at the platform-product layer (multi-provider routing economics, token-aware admission, a 40-team migration story) — cite this entry for deployment topology and cache correctness.
Org grounding: aegis-llm-gateway +
aegis-semantic-cache (ADR-028)
— federated model plane; tool governance stays in AegisAI.
Shipped enforce (ADR-029): apps select; gateway denies confidential→cloud and
verifier.provider == generator.provider, records RoutingDecisionV2.
Thin geo (ADR-033): optional X-Allowed-Regions — not a residency envelope.
Executive summary
30-second thesis
I'd put a tenant-aware LLM gateway in front of providers for auth, routing, budgets, and optional semantic cache — and keep it separate from the tool-policy gateway that owns irreversible side effects.
2-minute answer
Scar I've seen: one "AI gateway" that both burns tokens and writes to production systems. When either half needs to change cadence, you get either rushed policy or frozen product.
So: authenticate, meter, and route by capability / latency / cost class. Semantic cache only for safe, FAQ-like prompts with tenant + policy + model in the key — never personalized or tool-using answers across tenants. Fail closed on budget and auth in strict mode. Provider adapters stay swappable. FinOps tags leave here (10).
What I'd refuse: caching "similar" prompts without a correctness story, or conflating model ingress with side-effect authorization. I'd also refuse claiming a geo/residency envelope when only optional X-Allowed-Regions exists (ADR-033) — sensitivity (confidential→private) is not sovereignty.
Quantitative trade-offs
| Decision | Trade-off and reversal evidence | Evidence class |
|---|---|---|
| Shared gateway vs sidecar | Shared gateway centralizes FinOps/policy; sidecar wins at edge/single-app constraints. Reverse when blast radius or latency mandates local. | H |
| Semantic cache hit rate vs correctness | Aggressive cache saves money; reverse/disable for user-specific or tool-using prompts. | H |
| Multi-provider routing vs complexity | Routing improves resilience/cost; reverse if contract/token portability cannot support failover safely. | H |
Numbers and thresholds in interview delivery should be labeled H unless the candidate can defend a measured baseline. Open repositories are O; researched public patterns are R. Do not upgrade O/R into employer P adoption.
Migration and rollout
Gateway + cache + router is a traffic-plane change. Wrong cache keys leak tenants.
- Put gateway in front in passthrough mode; meter tokens and routes.
- Enable semantic cache with tenant+policy+model keys in shadow; measure hit rate and mismatch rate.
- Turn on router policies for cheap-vs-capable models with quality holdback.
- Canary tenants with explicit cost and quality budgets.
- Rollback = passthrough to previous default model. Kill switch = bypass cache + pin model.
Org ownership and operating model
- AI platform owns gateway, cache keying, and router policy engine.
- Model owners own which models are eligible for a route.
- Security owns tenant isolation in cache and prompt logging redaction.
- FinOps owns savings targets and spend alerts.
- Product owns UX when router picks a cheaper model.
Requirements
Functional
- Single ingress for chat/completions across multiple model providers (or stub → BYOK).
- Route by policy: tenant, latency class, cost class, capability (tools, vision).
- Optional semantic cache for idempotent / FAQ-like prompts.
- Central auth, rate limits, usage metering, and kill switches.
- Explicit choice: shared gateway plane vs per-app sidecar; cache-as-service vs in-process.
Non-functional
- P99 overhead of the gateway itself is small vs model latency.
- Cache must not serve personalized or safety-sensitive answers across tenants.
- Provider / FinOps outages: fail-closed vs fail-open is a documented posture, not silent.
- FinOps: per-tenant budgets enforced before tokens burn.
- No fake 99.9% SLO claims on free-tier demos.
Core entities
- Route policy: match rules → model/provider, timeout, retry, fallback chain.
- Cache key:
tenant_id+ embedding(prompt) + model_id + policy_version (+ optional tags). - Usage event: tenant, model, tokens_in/out, cached_hit, cost_usd.
- Provider adapter: normalized request/response + error taxonomy.
- Control posture:
strict|demo— budget/cache/auth failure behavior.
API / interface
POST /v1/chat/completions
Authorization: Bearer <gateway_key>
X-Tenant-Id: vap|acf|…
{ "messages":[...], "model":"auto"| "stub-small"| "gpt-…", "stream": false }
→ 200 json + headers: X-Model-Used, X-Cache: HIT|MISS
→ 402 budget breached (strict) | 503 FinOps down (strict)
POST /v1/cache/lookup
{ "tenant_id":"…", "model":"…", "messages":[...] }
→ { "hit": true|false, "response": … }
POST /v1/cache/store
{ "tenant_id":"…", "model":"…", "messages":[...], "response":{…} }
→ 204
GET /v1/posture
→ { "control_plane_mode":"demo"|"strict", "fail_open":… }
GET /v1/ops/metrics
→ completions, cache_hits/misses, finops_* (gateway) | hits, misses, hit_rate (cache)
Staff+ callout: default deny semantic cache for user-specific / tool-using prompts; opt-in for safe classes. Tenant is a first-class header, not inferred from model name.
Data Flow
Client → auth → tenant bind → budget pre-check → cache lookup (if allowed) → provider/stub → meter → (optional) cache store → response. Failover walks the provider chain; posture decides whether FinOps/cache outages block or degrade.
Rendering architecture diagram…
High-level design
Maps to functional requirements — shared plane vs sidecar is a first-class topology choice.
Rendering architecture diagram…
Overlaps ../ai-system-design/09 and ../scalability-governance-tradeoffs/01. Tool side effects stay on AegisAI — this entry is the model HTTP plane only.
Deep dives below target non-functional requirements (topology, tenancy, safety, failure, cost).
Deep dive 1: gateway vs sidecar
Shared gateway (org default): one OpenAI-shaped service; apps set LLM_GATEWAY_URL. Wins for
central FinOps, one policy surface, Control Room metrics, and cross-app cache hit rates.
Sidecar / in-process: library or container next to a single app. Wins at the edge (low RTT), air-gapped single product, or when the shared plane is down and you need a documented local fallback (OmniForge “self-contained OR plane-connected”).
Staff+ trap: absorbing the LLM proxy into the tool-governance monolith. Separating planes keeps blast radius and ownership clear (ADR-028).
Deep dive 2: cache-in-process vs cache-as-service
In-process / sidecar cache: lowest latency; hard to share hits across apps; memory pressure on every replica; invalidation is local.
Cache-as-service (org default): aegis-semantic-cache scales independently; gateway stays thin;
tenant isolation and ops metrics (hit_rate) are one API. Cost: extra hop + availability coupling
— mitigate with fail-open/closed posture and short timeouts.
Deep dive 3: tenancy keying and cache poisoning
Keys must include tenant_id (logical multi-tenant). Cross-tenant lookup must miss — test it.
Never cache: tool-using agent traces, PII-heavy prompts, regulated advice without review.
False-hit story: cosine threshold + optional exact-match secondary check; prefer miss over wrong answer.
Deep dive 4: TTL vs tag invalidation; fail-closed vs fail-open; FinOps placement
TTL alone is blunt; prefer tag invalidation when RAG/KB versions change (kb_v42).
Fail-closed (strict): budget breached → 402; FinOps down → 503. Fail-open (demo): degrade
to stub/direct with honest /v1/posture — never silent.
FinOps placement: pre-flight budget in the gateway (before provider call); meter after success;
cache hits still record saved_usd for exec dashboards. Align with agent-finops.
Streaming: cache complete responses only (or skip cache for SSE) — say how X-Cache: HIT works.
Staff+/Principal signal rubric
- Mid-level: reverse proxy to OpenAI.
- Senior: multi-provider + rate limits + basic cache.
- Staff+: gateway vs sidecar trade-off; cache-as-service vs in-process; tenant-scoped keys; fail-closed posture; FinOps pre-flight.
- Principal: org-wide spend control, Control Room ops, incident playbooks for provider/FinOps outages, clear non-goals (no fake SLO).
Follow-up questions to expect
- "How do you prevent cache poisoning across tenants?" Hard tenant partition plus authz on the cache key. Shared embeddings without tenant in the key is how you leak answers.
- "When does sidecar beat a shared gateway?" Edge RTT, single-app blast radius, or offline fallback — not because someone likes sidecars.
- "Where do you meter — app, gateway, or provider webhook?" Gateway pre-flight and post-success; webhooks only reconcile the invoice.
- "What is your false-hit story?" Similarity threshold plus optional exact-match check — and disable semantic cache for personalized or tool-using prompts.
Related
- ../ai-system-design/28 Enterprise LLM gateway — platform-product layer: routing economics, admission control, migration
- ../ai-system-design/09 Multi-tenant AI platform
- 05 Security and compliance
- ../scalability-governance-tradeoffs/01 Cost vs latency vs safety
- Shipped planes: aegis-llm-gateway · aegis-semantic-cache
- ADR: ADR-028 federated AI control plane