Playbook / AI system design / Design a retrieval-augmented generation (RAG) platform at scale

Design a retrieval-augmented generation (RAG) platform at scale

Expected question

"Design a retrieval-augmented generation (RAG) platform for enterprise knowledge. How do you ingest, index, retrieve, and generate grounded answers at scale with access control?"

Variant forms

Interviewers often ask the same design with different framing — recognize the archetype:

  • "Design an internal search + Q&A system over 50 million documents with per-user ACLs."
  • "How would you build a copilot over a company's SharePoint, Confluence, and tickets?"
  • "Our RAG answers hallucinate citations — architect a system that enforces grounding and decline paths."
  • "Design hybrid retrieval (keyword + vector) for a legal/compliance corpus where wrong answers are costly."
  • "Scale RAG from 10K to 10B chunks — what breaks first in indexing, retrieval, and generation?"
  • "Design agentic RAG where the model decides when to retrieve, rerank, and call tools."
  • "How do you keep RAG fresh when documents change hourly across thousands of sources?"
  • "Design a multi-tenant RAG SaaS where Tenant A must never see Tenant B's embeddings or chunks."

Where this actually gets asked

RAG is the best-attributed AI-infra system design topic in this whole set. A Blind post on Google DeepMind's Applied AI Engineer loop names RAG (factuality/grounding) explicitly as a focus area. Prep aggregators covering Microsoft AI-adjacent roles cite "RAG ranking and retrieval pipelines" as a recurring theme. Treat the exact phrasing as company-specific, but the underlying archetype — "design a system that answers questions grounded in a large, access-controlled document corpus, at low latency, for millions of documents and thousands of concurrent users" — as close to universal across every company in this list.

Executive summary

30-second thesis

I'd filter by verified access before I rank — hybrid retrieve, cite what you used, decline when grounding fails. Optimizing recall by sneaking unauthorized neighbors into context is how demos look smart and prod leaks.

2-minute answer

I'd start on the write path: hard ingest contracts, chunks with lineage, indexes stamped with tenant/ACL predicates. On the read path: authenticate a real principal, access-filter before scoring, hybrid retrieve, rerank a bounded set, generate only if grounding passes — otherwise decline out loud.

Under overload I'd shed quality (smaller top_k, lexical-only) and never skip the access filter to "go faster." Deletion and ACL changes get their own invalidation path; treating them as "next reindex" is a silent leak.

What I'd refuse: a post-filter that redacts citations after the model already saw the chunk. That's theater.

Quantitative trade-offs

DecisionTrade-off and reversal evidenceEvidence class
Access-before-rank vs post-filterFiltering after vector search can still fetch unauthorized neighbors into memory/traces; reverse only if the store cannot push predicates and you can prove no residual leakage path.O/H
Hybrid+RRF vs dense-onlyHybrid improves sparse/entity queries; reverse if latency budget cannot afford BM25+dense and eval shows dense alone meets slices.H
Freshness vs rebuild costIncremental upsert keeps minutes-level freshness; reverse to scheduled rebuild only when consistency windows are unacceptable and cost is funded.H

Numbers and thresholds in interview delivery should be labeled H unless the candidate can defend a measured baseline. Open repositories are O; researched public patterns are R. Do not upgrade O/R into employer P adoption.

ML fundamentals

Evaluate grounding/faithfulness and citation precision by slice. Retrieval recall alone is not product success — a high-recall leak is still a leak.

Migration and rollout

This is an index cutover, not a generic canary speech.

  1. Dual-write ingest into the new chunk/index schema while reads stay on the old path. Prove lineage (owner, source URI, content hash) lands on every chunk.
  2. Shadow answers on a holdout query set: same principal, compare citations + faithfulness + decline rate — not just latency. Kill if grounding regresses even when p99 looks prettier.
  3. ACL invalidation drill before traffic: delete a doc, revoke a group, confirm both stop appearing in retrieve and in citations within the agreed window. If that drill fails, you don't canary.
  4. Flip read traffic by tenant or corpus slice with a holdback. Keep the old index warm until soak on faithfulness + leak tests pass.
  5. Rollback is "point reads back at the previous index alias," not "redeploy hope."

Org ownership and operating model

  • Knowledge / search platform owns ingest contracts, index schemas, access-filter invariants, and the invalidation path.
  • Product copilots own prompts, UX, and domain eval slices — they don't get to weaken ACL predicates for recall.
  • Identity / security owns how Principal is derived from the IdP token and how fast group revocation must land.
  • Corpus owners (legal, HR, eng wiki, etc.) own classification and source-of-truth ACLs on documents.
  • Eval owners can veto a release when citation faithfulness or leak tests fail — even if the model "sounds better."

If those veto rights aren't written, someone will eventually ship "just for the demo" and you'll discover it in an access review.

Requirements

Functional

  • Ingest messy real sources — uploads, crawlers, internal wikis — without inventing owners.
  • Answer natural-language questions with grounded, cited responses.
  • Never show a user content or a citation they aren't allowed to see.

Non-functional

  • Scale toward very large corpora (billions of chunks is the stress case interviewers imply).
  • Retrieval budget that leaves room for generation (illustrative H: retrieval p99 under ~300ms).
  • Freshness in minutes for new docs; faster invalidation for revocations than for routine upserts.
  • A confident wrong answer is worse than a slow decline — design for that product preference.

Core entities

  • Document: raw source, owner, classification/ACL metadata, version.
  • Chunk: retrievable unit — embedding + text + lineage back to the document.
  • Principal: verified querying identity — user, tenant, groups, clearance. Not a client-asserted blob.
  • Citation: chunk used in an answer, with a pointer back to source.

API / interface

Auth: verified JWT → server-derived Principal. Never trust client-asserted ACLs alone.

POST /v1/documents
Authorization: Bearer <token>
{
  "source_uri": "s3://corp-wiki/page-42",
  "owner": "team-platform",
  "classification": "internal",
  "allowed_groups": ["eng","support"],
  "content_hash": "sha256:..."
}
→ 202 {"document_id":"doc_...","ingestion_job_id":"job_...","status":"queued"}
→ 422 {"error":"ingestion_contract_failed","violations":["missing_owner"]}

GET /v1/documents/{document_id} → {"status":"indexed","chunk_count":128}
DELETE /v1/documents/{document_id} → 202 {"invalidation_job_id":"..."}

POST /v1/answer
Authorization: Bearer <token>
{"query":"...","top_k":8,"require_citations":true}
→ 200 {"answer":"...","citations":[...],"grounded":true,"declined":false,"trace_id":"tr_..."}
→ 200 {"answer":null,"declined":true,"reason":"insufficient_grounding","trace_id":"tr_..."}

Staff+ callout: ingest, answer, and invalidation are separate contracts. ACL changes after index time need an explicit path — not "we'll catch it on the next crawl."

Data Flow

Two flows: ingest (write) and answer (read). On the answer path, access filter runs before ranking.

Rendering architecture diagram…

Write path fails closed on missing owner/lineage. Read path fails closed on weak grounding — decline beats a confident guess.

High-level design

Rendering architecture diagram…

I'd keep retrieval two-stage: cheap hybrid recall first, expensive rerank on a small candidate set. Running a cross-encoder over millions of chunks is how you burn the latency budget before the model says a word.

Deep dives below are the NFRs that actually separate a Senior tour from a Staff+/Principal answer.

Deep dive 1: access control ordering (the detail that separates Staff+ from Senior)

Hard constraint: an unauthorized chunk must never enter the model context. Redacting the citation afterward is already too late.

The instinctive design retrieves top-k by score, generates, then checks ACL on citations. That ordering is wrong. If access control runs after ranking, the unauthorized chunk still entered the context window — the model read it. A leading follow-up can pull it back out.

Default I'd defend: filter by verified Principal before scoring. Unauthorized chunks never get scored, reranked, or assembled into context. I've implemented that pattern in open proof (O) — and the scar that came with it: filter-before-rank is worthless if Principal is client-asserted from the request body. You need both halves: ordering and a verified identity.

Demos hide this bug because everyone in the room has the same access.

Deep dive 2: ingestion data contracts and lineage

Hard constraint: no owner, no source, no content hash → not citable. "Garbage in, cited out" is a trust failure, not a data-quality nicety.

Naive ingest accepts anything posted to the endpoint. The failure mode I've hit in practice (O): quality checks were computed, logged, and then ignored — so orphan docs still got indexed and cited as if they were trustworthy.

Fix: reject hard violations with 422. Stamp every chunk with a content hash and ingest timestamp that survive rerank and optional persistent stores. Callers should always be able to answer "when was this indexed, and has the bytes changed since?"

Ingestion validationWhat it catchesCost
No validationNothing — garbage in, cited outCheap now, expensive in an audit
Compute checks, log onlyVisibility without teethThe anti-pattern
Reject on hard violationsNo-owner / no-lineage / near-empty never enterCallers must handle real 422s

Deep dive 3: freshness vs. index rebuild cost

Hard constraint: full re-embed-and-rebuild at billions of chunks is unaffordable on every write.

Default: incremental upsert for routine freshness, plus a separate fast invalidation path for deletes and ACL revocations. Upsert lag of a few minutes is usually fine. Citing a revoked policy doc for a week because "the next crawl will get it" is not.

Name the inconsistency window across replicas out loud. Most RAG products can live with brief replica lag. Safety-critical revocation lists cannot.

Deep dive 4: overload and stale-index degradation (45-min critical)

Under retrieval/rerank overload, shed quality before inventing answers: lexical-only or smaller top_k, return degraded=true + trace_id. Never skip the access filter to go faster.

Brief replica lag after ACL or ingest updates is acceptable if you name the staleness window. Silent cross-tenant leakage is not.

In 45 minutes, don't spend the round on chunking bake-offs or embedding beauty contests unless they steer there.

Staff+/Principal signal rubric

  • Mid-level: retrieval → generation with citations; may not raise access control until prompted.
  • Senior: hybrid + rerank; some access filtering.
  • Staff+: access-control ordering unprompted (before ranking) and why post-hoc redaction fails.
  • Principal: trust chain behind the filter (verified identity, not asserted), ingest contracts as first-class design, and a migration story that drills invalidation — not just latency canaries.

Follow-up questions to expect

  • "Doc deleted or ACL changed after index — what happens?" Separate invalidation path, or you're citing revoked docs until the next crawl. I'd rather over-invalidate than under-invalidate.
  • "How do you know it's grounded, not just plausible?" Citation precision / faithfulness suites as a release gate — see 07. Vibe checks die on the first model swap.
  • "Multi-tenant with per-tenant embedding models?" Tenant-scoped namespaces or separate indices. Don't assume one shared embedding space for the reranker.
  • "What do you skip in 45 minutes?" Chunking/embedding bake-off. Cover access-before-rank, hybrid retrieve, grounding/decline, one overload path.

What I'd ask them

  • Who owns ACL truth — IdP groups, doc ACLs, or both — and how fast must revocation land?
  • Is a confident wrong answer worse than a decline? (Usually yes — that choice drives UX and gates.)
  • What's the staleness window you'll accept after ingest vs after ACL change?
← Design an LLM inference serving platform at scaleDesign an agent orchestration platform with real tool use →