Design an enterprise AI landing zone

Expected question

"Design a cloud landing zone for an enterprise that will run GenAI, RAG, and agent workloads across many teams — accounts, identity, networking, logging, and AI service guardrails."

Variant forms

"Walk me through a landing zone for a 500-person org with three regulated business units that want to use LLMs."
"How do you structure AWS Organizations / Azure Management Groups / GCP folders for AI platforms?"
"Design account vending so product teams can experiment without bypassing security."
"Where do shared LLM gateways, vector stores, and eval services live vs app accounts?"
"How do you prevent a team from attaching a public LLM endpoint to production data?"
"Design centralized logging, GuardDuty/Security Hub equivalents, and AI audit trails."
"Multi-account AI platform: shared services vs per-team stacks."
"Landing zone for hybrid: on-prem data, cloud inference."

Where this actually gets asked

Top-frequency Cloud / Customer AI Architect whiteboard in 2025–2026 hiring guides: landing zones, multi-account governance, and AI workload patterns together. Distinct from generic K8s cost (06) — this entry owns org/account topology and AI guardrails.

Executive summary

30-second thesis

I'd separate identity, logging, networking, and a shared AI platform account from vended app accounts. Every LLM/RAG path exits through a governed gateway — SCPs deny public model endpoints from data accounts. Easy approved path beats prohibition theater.

2-minute answer

What I'd refuse first: fifty accounts each with their own provider API key and a NAT to the public internet. That's how you get shadow AI and an unownable deny path.

Org hierarchy: security/log archive, networking hub, shared AI platform (gateway, eval, registry), and short-lived app accounts per product. SCPs/policies deny public foundation-model APIs from regulated data accounts and force PrivateLink/egress controls. Central SSO with break-glass. App teams consume AI via approved APIs — not raw cloud AI keys in each account. Chargeback tags on every AI resource so FinOps can attribute spend (10).

Canary new AI services in a sandbox OU before org-wide allow. For hybrid, corpora stay home; cloud gets queries/embeddings over private connectivity (11). Residency beats multi-cloud for vanity.

Quantitative trade-offs

Decision	Trade-off and reversal evidence	Evidence class
Shared AI platform vs per-team stacks	Shared reduces sprawl and key leakage; reverse when a BU needs hard isolation and will fund duplicate platform cost.	H
Strict SCPs vs developer speed	Strict prevents data exfil to public LLMs; reverse exceptions with expiry and dual control.	H
Hub networking vs flat VPC mesh	Hub simplifies inspection; reverse for tiny orgs where hub latency/cost dominates.	H

ML fundamentals

Not model math — platform fundamentals: blast radius, least privilege, and stopping ungoverned training/inference from becoming a shadow AI estate.

Migration and rollout

Inventory existing AI keys, vector DBs, and GPU projects.
Stand up security/log/network OUs; freeze new public AI endpoints.
Deploy shared gateway + registry in AI platform account.
Vend first app accounts with private connectivity only.
Migrate teams wave-by-wave; revoke standalone keys after dual-run.

Org ownership and operating model

Cloud Platform / Landing Zone owns org policy, networking, account vending.
AI Platform owns shared gateway, eval, registry interfaces.
Security owns SCP/deny lists, key management, audit retention.
FinOps owns tags, budgets, commitment strategy.
App teams own product UX inside approved contracts.

Requirements

Functional

Multi-account/subscription hierarchy with AI-aware OUs.
Central identity, logging, and security findings aggregation.
Shared AI platform services consumable by app accounts.
Policy that blocks unapproved public LLM egress from sensitive accounts.
Account vending with baseline network, tags, and budgets.

Non-functional

New account ready in hours, not weeks (illustrative H).
Audit completeness for AI API calls and admin actions.
Fail closed on missing tags or missing private path for regulated data.
Documented RTO for control-plane vs data-plane incidents.

Core entities

OU/Management Group, Account, SCP/Policy, Hub Network, AI Platform Account, App Account, Egress Gateway, Budget/Tag Policy.

API / interface

Account vending API/IaC module; AI platform endpoints (/v1/chat, /v1/retrieve) reachable only via PrivateLink; break-glass role with time-bound elevation.

Data Flow

User/app → app account → private link → AI platform gateway → model/RAG → audit to log archive; security findings → SOC.

Rendering architecture diagram…

High-level design

Security + network + AI platform + vended apps. AI is a shared platform product, not a DIY service in every account.

Deep dive 1: where AI services live

Put gateway, prompt/registry, eval, and shared vector infra in the AI platform account. App accounts hold product data and call in. Do not scatter long-lived provider API keys across fifty accounts — that's the scar.

Deep dive 2: deny lists that matter

Deny: public internet to foundation-model APIs from regulated accounts; open SGs to model endpoints; untaxed GPU projects without budget. Allow: sandbox OU for experimentation with synthetic data only. If the approved path is painful, teams will invent a worse one.

Deep dive 3: hybrid / customer edge

On-prem corpora stay home; cloud receives embeddings or queries over private connectivity. Residency beats "multi-cloud for vanity."

Deep dive 4: adoption without shadow AI

Chargeback + easy approved path beats prohibition. Measure shadow-key detections and time-to-vend — those are the health metrics I'd watch.

What I'd ask them

How many provider keys exist outside the platform account today?
Who owns the SCP that denies public LLM domains from data OUs?
What's the exception expiry and dual-control path when a BU needs isolation?

Staff+/Principal signal rubric

Mid-level: one VPC and a model endpoint.
Senior: multi-account with SSO and logging.
Staff+: AI-aware OUs, shared gateway, SCP deny for public LLM from data accounts, vending + chargeback.
Principal: org migration off shadow AI, exception expiry, regulated-unit isolation patterns.

Follow-up questions to expect

"When is multi-cloud real vs vanity?" Residency, M&A, hard negotiation leverage — not default HA. Each cloud is another deny path to maintain.
"How do startups differ?" Collapse OUs; keep gateway + audit + key centralization. Topology shrinks; invariants don't.
"A regulated BU wants its own gateway." Fine if they fund it and mirror the same audit contract — federation with a shared policy schema, not a free-for-all.