Enterprise deploy — VPC, SSO, hosted vs self-hosted AI

Expected question

"A Fortune 500 wants to deploy our AI platform in their AWS VPC with Okta SSO and Snowflake as the data source. Walk the deployment architecture — including hosted API vs self-hosted / air-gapped trade-offs."

Variant forms

• "Design a private, VPC-deployed RAG for healthcare with HIPAA and 50M documents."
• "Compare hosted (OpenAI/Anthropic API) vs self-hosted open-weights on customer infra."
• "Customer's data spans SAP, Salesforce, and Postgres. Unify for an agent without a 2-year MDM program."
• "Customer demands sub-100ms LLM search; naive RAG is 1.5s. How do you get closer?"
• "Design ingestion for 12 fragmented retail sources with no clean schema."
• "How do you stage value in week one, not month six?"
• "Air-gapped / GPU-constrained customer — what changes?"
• "How would you handle rate limits and latency integrating LLMs into high-traffic production?"
• "Design observability for an agent in the customer's environment — logs, alerts, dashboards."

Where this actually gets asked

OpenAI / Databricks / Cohere FDE system-design and deploy rounds. Bar: trust boundaries (identity, data, model, audit) with an honest hosted vs self-hosted rule — not “put it in a VPC.”

The question, as it might actually be asked

"Draw the trust boundaries. Where do secrets, data, and model calls live?"

The framework

30-second thesis

I'd put the app and retrieval inside the customer's trust boundary under Okta SSO, filter Snowflake/docs by ACL before ranking, and send model calls only via private endpoint, allowlisted egress, or on-prem GPU — with audit to their SIEM. I self-host for isolation needs, not vanity. Week one is SSO + one source + read-only assist, not GPU theater.

2-minute method

I'd draw four planes and talk through them:

Users → Okta SSO → App in customer VPC
                 → Private AI gateway (policy/HITL/metering)
                 → Retrieval under Snowflake/ACL filters
                 → Model: private endpoint OR allowlisted egress OR on-prem GPU
Audit → customer SIEM

Hosted vs self-hosted rule: isolate the data plane first. Hosted private endpoint wins time-to-wedge for most commercial SaaS AI. Self-host / air-gap when defense, strict residency, or customer GPU policy demands it — Cohere-style on-prem included. Ops burden flips to the customer (and you) the moment you self-host.

Same control brand either way: gateway + HITL, access-aware RAG, evals, audit. Not a different architecture story per logo.

Hosted API vs self-hosted

Requirements

Functional

• SSO-authenticated principal on every request.
• ACL-aware retrieval; no cross-tenant/doc leakage.
• Gateway policy on tools; HITL on irreversible writes.
• Audit events to customer-controlled sink.

Non-functional

• Private networking / egress control for model path.
• Latency budget with explicit cache/routing plan.
• Week-one staging value (SSO + one source + read-only assist).
• Observability: quality, cost, latency, safety alerts.

Core entities

• Trust boundary: customer VPC / account.
• Identity principal: Okta groups → ACL predicates.
• Gateway: policy, HITL, quotas, audit.
• Data plane: Snowflake / docs / SAP slice projections.
• Model plane: hosted private endpoint vs on-prem weights.
• Secrets: customer KMS / vault — never FDE laptop.

Data flow

Rendering architecture diagram…

High-level design

1. Identity: OIDC/SAML via Okta; map groups to retrieval predicates.
2. App + gateway: in customer VPC; no dual policy systems.
3. Retrieval: Snowflake secure views / row filters; access-before-rank.
4. Model: prefer provider private link for speed-to-wedge; self-host when air-gap/GPU policy demands.
5. Writes: idempotent tools to SoR via gateway + HITL.
6. Obs: traces without raw secrets/PHI; dashboards for latency, $/task, HITL reject, groundedness samples.

Deep dive 1: unify messy systems without MDM theater

I'd pick one workflow’s entities, read APIs/CDC for that slice, and build a canonical projection for agent tools only. Writes return to SoR with idempotency + HITL. I would refuse a two-year MDM program as wave one.

Deep dive 2: latency path (1.5s → tighter)

Cache embeddings/retrieval; skip rerank on easy queries; smaller/faster model for FAQ; speculative retrieval; stream tokens; async tools — then exotic infra. Sub-100ms “LLM search” often means retrieval+cache UX, not full generation in 100ms. Say that aloud (H).

Deep dive 3: air-gap / HIPAA deltas

Minimize PHI in prompts/logs; BAAs where hosted; fail-closed egress; on-prem weights if required. GPU capacity and model updates become joint customer+FDE ownership. Week-one value still: SSO + one source + read-only assist in staging — not standing up a GPU cluster first.

Quantitative trade-offs

Migration and rollout

1. Week one: SSO + Snowflake (or one source) + read-only assist in staging.
2. Shadow beside humans; no irreversible writes.
3. Private model path proven (latency + egress tests).
4. Canary; HITL writes later; handoff runbooks + SIEM alert ownership.

Org ownership

• Customer cloud/security owns VPC, KMS, egress exceptions (expiry).
• IdP owners own Okta app + group mappings.
• FDE owns gateway wiring and first connectors.
• Customer SRE owns SIEM alerts after handoff.
• Platform product absorbs connector patterns.

Situation

Enterprise deploy interviews mirror the same constraints you design for in portfolio cloud docs (O / R): private connectivity, landing zones, Enterprise RAG Strict path. Lucid (P) supplies instinct for real change windows and irreversible ops — framed as internal-customer discipline, not “we deployed OpenAI in Lucid’s VPC.”

Task

Draw trust boundaries for Okta + VPC + Snowflake (or analogous) and make an honest hosted vs self-hosted call with a week-one value path.

Action

1. Place app, gateway, and retrieval inside customer trust boundary under SSO.
2. Enforce ACL filters before ranking; audit to SIEM.
3. Choose model plane by residency/ops — not brand preference.
4. Avoid MDM theater: one workflow projection first.
5. Attack latency with cache/routing before exotic infra.
6. Stage week-one read-only assist; HITL before writes; hand off obs ownership.

Result

A deploy design that survives Staff+/Principal: identity + data + model + audit, with a clear isolation decision rule and staged value.

The follow-up question you should expect

"Where do secrets live, and who can read prompts?"
Customer KMS/vault; gateway service role least-privilege; prompt/log redaction for secrets/PHI; break-glass time-boxed and audited. FDE does not take production credentials home.

What I'd ask them

1. What's the residency / egress policy in writing — DPA, private link, or air-gap?
2. Who owns Okta group → ACL mapping, and how stale is it?
3. What's the irreversible write in scope for wave one?
4. What's the week-one success demo they'll accept without prod writes?

Candidate-owned evidence prompts

1. Which portfolio doc will you cite for private connectivity / landing zone (O)?
2. What latency number will you label H vs measured?
3. What is your one-sentence hosted vs self-host rule?
4. What week-one demo will you refuse (irreversible writes)?

Author reference (do not memorize)

Trust-boundary diagrams are the artifact — practice drawing them blank. HIPAA/residency details are R unless you have a dated customer engagement.

Staff+/Principal signal rubric

• Mid-level: "Put the app in a VPC and call the API."
• Senior: SSO, private networking, one data source, basic audit.
• Staff+: Trust boundaries, ACL-aware retrieval, staged wedge, latency/cost plan, hosted vs self-hosted decision rule.
• Principal: Landing-zone fit, exception expiry, productization of connectors, customer ops handoff.

Related

• 03-customer-discovery-to-mvp-wedge.md
• 05-production-ai-evals-cost-guardrails-takehome.md
• ../cloud-architecture/09-enterprise-ai-landing-zone.md
• ../cloud-architecture/11-private-connectivity-and-egress-control-for-ai.md
• ../ai-system-design/02-rag-platform-at-scale.md