Design private connectivity and egress control for AI

Expected question

"Design private networking and egress control so enterprise data can use cloud LLMs/RAG without exposing corpora to the public internet — PrivateLink, private endpoints, and controlled egress."

Variant forms

"How do you call OpenAI/Azure OpenAI / Bedrock from a locked-down VPC?"
"Design on-prem RAG that uses cloud LLMs without shipping raw documents publicly."
"Prevent prompt/data exfiltration via employee shadow ChatGPT usage — network + platform controls."
"Private endpoints for vector DBs and model endpoints across accounts."
"Zero-trust access to an internal AI assistant."
"What breaks when you force all AI traffic through an egress proxy?"
"Design DNS, TLS inspection trade-offs, and allowlists for foundation-model APIs."
"Hybrid: factory OT network needs an assistant — where does inference run?"

Where this actually gets asked

Frequent Customer/Cloud AI Architect follow-up after landing zone and security rounds. Complements 05 and 09 with a network data-path focus.

Executive summary

30-second thesis

Default deny public AI egress from data subnets. Approve paths only through private endpoints or a governed egress proxy tied to the AI gateway identity — never raw provider keys on laptops talking to production data.

2-minute answer

Here's where demos lie: "we put it in a VPC" while every app still NATs to a public model URL with a shared key.

I'd start by putting model and vector endpoints on private connectivity (PrivateLink / private service connect). Apps reach a shared AI gateway privately. The gateway holds provider credentials and calls providers via private offerings or an allowlisted egress proxy. DNS and security groups enforce the allowlist; mis-opened SGs to a vector DB are the scar I watch for.

On-prem: private circuit into the hub; prefer sending queries or embeddings over shipping whole corpora. Audit connection metadata. Be careful with TLS break-and-inspect on regulated prompts — legal often wins that fight. Pair network controls with DLP/CASB for humans pasting into consumer ChatGPT; VPC design alone doesn't stop that.

Fail closed for regulated tiers when the private path is down. Optional local/small model only if the policy already approved it.

Quantitative trade-offs

Decision	Trade-off and reversal evidence	Evidence class
Fully private provider offerings vs public API + proxy	Private reduces exposure; reverse when region/feature lag blocks product and legal accepts controlled egress.	H
TLS inspection vs end-to-end TLS	Inspection aids DLP; reverse when legal/crypto policy forbids it.	H
Embeddings offsite vs local models	Cloud LLM quality; reverse for highest residency tiers.	H

ML fundamentals

Data minimization: retrieve/generate with least data movement. Residency class drives topology more than model brand.

Migration and rollout

Inventory public AI egress and laptop keys.
Deploy private gateway path; dual-run.
SCP/firewall deny direct provider domains from data subnets.
Migrate high-sensitivity apps first.
Keep break-glass monitored path with expiry.

Org ownership and operating model

Network owns PrivateLink, DNS, firewall.
AI Platform owns gateway egress identity.
Security owns allowlists, DLP, inspection policy.
App teams own not embedding secrets in clients.

Requirements

Functional

Private paths app → gateway → model/vector.
Egress allowlists for approved AI domains/services.
Audit of connection metadata and gateway authz.
Hybrid on-prem connectivity pattern.

Non-functional

Added latency budget for private path (measure; illustrative H: tens of ms).
No single mis-opened SG exposing vector DB.
Fail closed when private path down (degrade or error — policy by tier).

Core entities

PrivateEndpoint, EgressProxy, Allowlist, TrustZone, GatewayIdentity, ResidencyClass.

API / interface

Private gateway endpoints only (no public model URLs from app subnets); DNS allowlists for approved AI domains; break-glass monitored egress path with expiry and dual control.

Data Flow

App subnet → private link → AI gateway → (private provider or allowlisted egress) → response; mirrors to audit.

Rendering architecture diagram…

High-level design

Network is part of the AI control plane — not an afterthought once the demo works on a laptop.

Deep dive 1: provider private offerings

Prefer Bedrock / Azure OpenAI private endpoints / VPC endpoints when residency requires it. Document feature gaps vs the public API so product doesn't discover them in week twelve.

Deep dive 2: what still leaks

Humans paste data into consumer chat apps. You need endpoint management, training, and DLP — not only VPC design. I'd say that aloud before claiming "we're locked down."

Deep dive 3: on-prem corpora

Keep documents on-prem; cloud gets chunk embeddings or query text under contract. Cross-border embedding storage is still a residency decision.

Deep dive 4: failure modes

PrivateLink outage: fail closed for regulated. Optional cached/local small model for status pages only if already approved — don't invent that under pressure.

What I'd ask them

Can any subnet still reach api.openai.com today, and who owns the deny list?
Where do provider credentials live — gateway role or app env vars?
What's the break-glass egress expiry, and who co-approves?

Staff+/Principal signal rubric

Mid-level: NAT gateway to OpenAI.
Senior: security groups and a proxy.
Staff+: private endpoints, deny-direct, gateway identity, hybrid pattern, human shadow-AI controls.
Principal: residency-tier topologies, legal constraints on inspection, org migration off laptop keys.

Follow-up questions to expect

"Does PrivateLink make RAG GDPR-safe?" No — you still have processing location, subprocessors, and retention. Private path is necessary, not sufficient.
"Multi-cloud private AI?" Only with real residency or contract drivers. Vanity multi-cloud doubles the deny-path surface.
"What if product needs a model feature only on the public API?" Time-bound exception through the gateway allowlist, dual control, expiry — or wait for the private offering. I wouldn't open the data subnet.