Multi-tenant isolation — the 8-minute whiteboard

I rehearse this aloud before every HM call. Same design whether tenant means org unit, SaaS customer, CRM account, or startup workspace.

What I actually say first (~60s)

I’d start with the threat model, not a tenancy column. For agents the leak is usually RAG bleed — Tenant A’s query pulling Tenant B’s chunks — or a tool call as the wrong tenant. So authorization happens before ranking, and every write carries tenant_id through a gateway. If we can’t fail a negative test in CI, we don’t have isolation. We have a slide.

Then I go to the board.

What I refuse: a shared default corpus in prod. Fail-open gateway on irreversible tools. Cache keyed only on prompt text. An “admin bypass” with no dual control.

Whiteboard (8–10 min)

1. Threat model

Cross-tenant RAG chunk leak on a shared index
Shared agent memory / conversation bleed
Confused deputy: agent A mutates tenant B’s CRM or publish target
Prompt injection that tries to escalate tenant scope
Semantic cache keyed only on prompt text → cache bleed
Admin / debug paths that skip AuthZ

2. Isolation layers

Request
  → Identity: JWT with tenant_id + principal + roles/clearance
  → AuthZ before retrieval: filter corpus by tenant + clearance BEFORE hybrid rank
  → Data plane: per-tenant collections / namespaces / RLS; no shared default corpus in prod
  → Control plane: per-tenant policy bundles, tool allowlists, HITL rules
  → Runtime: tenant-keyed cache + budgets; no cross-tenant keys
  → Audit: signed logs with tenant_id on every tool decision

3. Side effects

Any write — CRM, publish, notify, deploy, payment:

Tool args must include tenant_id matching the JWT claim.
Gateway rejects mismatch or missing tenant.
High-risk actions → HITL with tenant context visible to the approver.
Fail-closed in production strict mode if the gateway is unreachable.

4. Negative evals (I say these out loud)

ID	Assertion
NT-1	Tenant A query never returns Tenant B chunks
NT-2	Tenant A agent cannot mutate Tenant B resources
NT-3	Cache hit for A must never serve B’s embedding or response
NT-4	Prompt injection cannot widen tenant scope
NT-5	Gateway deny + audit event when tenant mismatches on a tool call

Wire NT-1..NT-5 as CI fixtures. Same discipline as golden-eval-registry.

5. Week 1–4 on their stack

Week	Ship
1	JWT claims (`tenant_id`, principal); reject unauthenticated retrieve/tool
2	Retrieval filter / collection isolation; negative RAG tests
3	Gateway enforces tenant on tool args; HITL on writes
4	Tenant-keyed cache/budgets; signed audit; NT suite in CI

If they ask “how would you ship this here?”

Week one is identity — no anonymous retrieve, no tool without tenant. Week two is the filter-before-rank and a test that must fail if Tenant B leaks. Week three is the gateway on writes. Week four is cache keys and the NT suite in CI. I wouldn’t start with a multi-region rewrite.

6. Honesty sentence (required)

Live free-tier demos may use a Demo Principal and an in-memory registry so a panel can actually click them. The production design is Strict/JWT principal, durable policy store, AuthZ-before-ranking, and fail-closed side effects. I can walk the Strict path and the ADRs. Demos are labeled Demo vs Strict so nobody’s surprised.

Failures I name if they push

Failure	What I’d do
Shared vector collection	Per-tenant collection or mandatory metadata filter pre-rank
Tool without tenant	Schema require + gateway check
Approver sees the wrong tenant	HITL UI shows tenant + resource IDs
Embed cache bleed	Cache key = hash(tenant_id + model + content)
“Admin bypass”	Break-glass role with dual control + audit

Rendering architecture diagram…

If they ask “have you built this?”

Access-before-ranking under a Strict/JWT principal — that’s the RAG path a panel can inspect. Gateway, HITL, audit is the governance plane. Lucid is production outcomes, not those binaries. Honest gap: free-tier demos are labeled Demo vs Strict. I won’t sell an in-memory registry as multi-tenant SaaS.

Follow-ups I’d answer without flipping to a new pitch

“Can’t we just filter after retrieval?”

That’s the leak. Ranking on a shared index and filtering later still embeds Tenant B into the candidate set — and into caches, traces, and sometimes the prompt. Filter first. Rank second.

“What if the gateway is down?”

For irreversible tools I fail closed. Reads can degrade. Writes don’t get a free pass because Redis was sad.

“Isn’t this overkill for an internal tool?”

Internal is still multi-tenant if two orgs, two clearances, or two CRM accounts share a graph. I’d thin the control plane. I wouldn’t skip tenant on the tool args.

“Shared vs dedicated inference?”

Shared until a tenant pays for isolation or burns everyone else’s TPM. Identity spine stays the same. That’s the deep path.

“Can’t we put tenant in the system prompt?”

That’s a hint, not a control. Completions will ignore it under injection. Tenant comes from the token, into the filter and the tool args, before the model speaks.

“We already share one index.”

Mandatory pre-rank filter plus a negative test this week. Per-tenant collections when the leak class or the contract says so. I wouldn’t halt the product for a re-platform.

Close the board

I’d ship isolation as negative tests in CI, not as a slide. If tenants can leak in eval, the platform doesn’t ship.

What I’d ask them before I leave the board: where retrieval sits relative to authZ today, and what the riskiest write an agent can already do.

Go deeper before a platform HM: Multi-tenant deep path.