Design a model and experiment registry with promotion gates

Expected question

"Design the ML/LLMOps registry that tracks experiments, model versions, datasets, and prompts — and that only promotes artifacts to production through enforced gates."

Variant forms

Interviewers often ask the same design with different framing — recognize the archetype:

• "Design MLflow/SageMaker-style model registry for classical models and LLMs."
• "How do you prevent a notebook model from becoming production without eval evidence?"
• "Design lineage from dataset → experiment → checkpoint → canary release."
• "Our teams promote models by copying S3 paths — fix the platform."
• "Design stage transitions: Staging → Production → Archived with approval and rollback."
• "How do prompts, adapters, and safety classifiers register beside model weights?"
• "Design multi-tenant registries so teams cannot overwrite each other's champions."
• "Walk through promoting a LoRA adapter that depends on a base model digest."

Where this actually gets asked

Core MLOps/LLMOps platform loop at enterprise AI platform and applied ML infra interviews. Distinct from progressive delivery (ai-system-design/19) — this entry owns identity, lineage, and promotion authority for artifacts before traffic routing.

Executive summary

30-second thesis

Treat every shippable AI artifact as an immutable, lineage-linked registry object. Promotion is a gated state machine — never an S3 path copy someone ran from a notebook.

2-minute answer

Scar: "the model in prod" is actually three different S3 prefixes and nobody can say which digest served last Tuesday's incident.

I'd register datasets, experiments, builds, prompts, adapters, and policies with digests and parent links. Stages (Dev → Staging → Production → Archived) need offline eval reports, ownership, and human approval by risk class. Production pointers are aliases to digests; rollback retargets the alias. Multi-tenant namespaces stop cross-team overwrite. Couple the registry to canary so traffic can only pin registered digests.

What I'd refuse: unsigned path-based deploys after the platform is live. Research sandboxes can be looser — they just don't get a production alias path.

Quantitative trade-offs

Numbers and thresholds in interview delivery should be labeled H unless the candidate can defend a measured baseline. Open repositories are O; researched public patterns are R. Do not upgrade O/R into employer P adoption.

ML fundamentals

ML fundamentals: experiment tracking, champion/challenger identity, leakage-safe dataset versioning, and promotion only with pre-registered metrics — not notebook folklore.

Migration and rollout

1. Inventory shadow-copied S3/model paths; import as digests without promoting.
2. Dual-write new training jobs to the registry while old paths still serve.
3. Block new production aliases that lack eval_report_id and lineage.
4. Canary traffic routers to digest pins (illustrative H: 1% → 10% → default).
5. Retire unsigned path-based deploys after two clean release cycles.

Org ownership and operating model

• ML platform owns registry, stages, and promote API.
• Model owners own eval packs and stage-move requests.
• SRE owns prod alias health after promote.
• Security owns who can attach credentials to a digest.
• Audit owns immutable promote history.

Requirements

Functional

• Register datasets, experiments, model builds, prompts/adapters, and policies with content digests.
• Record lineage: parents, code commit, feature/dataset versions, hyperparameters.
• Enforce stage transitions with required evidence (eval report, approver, risk class).
• Expose production aliases that resolve only to registered digests.
• Support search, compare, and audit ("what is in prod for tenant X?").

Non-functional

• Metadata write P99 under an illustrative 200 ms in-region (H); strong consistency for alias updates.
• Immutable artifacts; no silent overwrite of digests.
• Multi-tenant isolation; cross-namespace promote requires explicit grant.
• Audit every promote/rollback with actor, evidence IDs, and timestamp.
• Fail closed: traffic routers reject unregistered digests.

Core entities

• Artifact: type, digest, URI, created_by, tenant, lineage_parents.
• ExperimentRun: metrics, params, dataset_versions, code_sha.
• ModelVersion: base_model_digest, adapter_digest?, stage, eval_report_ids.
• Alias: name (e.g. fraud-ranker@prod), current_digest, previous_digest.
• PromotionRequest: from_stage, to_stage, evidence, approvers, status.
• PolicyBundle: safety/prompt/tool versions linked into a release candidate.

API / interface

POST /v1/artifacts
{ "type":"model","digest":"sha256:...","lineage":{...},"tenant":"payments" }
→ 201 { "artifact_id":"a_..." }

POST /v1/experiments/{id}/runs
{ "metrics":{"auc":0.91},"dataset_version":"d_...","code_sha":"..." }
→ 201 { "run_id":"r_..." }

POST /v1/promotions
{ "artifact_id":"a_...","to_stage":"Production","eval_report_id":"e_..." }
→ 201 { "promotion_id":"p_...","status":"pending_gates" }

PUT /v1/aliases/fraud-ranker@prod
{ "digest":"sha256:...","promotion_id":"p_..." }
→ 200 { "previous":"sha256:old","current":"sha256:..." }

Staff+ callout: aliases are the production contract — humans never paste raw URIs into deploy manifests.

Data Flow

Train/eval job registers digests → offline gates attach reports → promotion request → policy check → alias retarget → release/canary plane consumes alias → rollback retargets previous digest.

Rendering architecture diagram…

High-level design

Rendering architecture diagram…

Deep dives below target non-functional requirements (consistency, tenancy, failure, audit).

Deep dive 1: digest identity beats file paths

Path-based promotion (s3://…/model.pt) silently replaces bytes. Content digests make lineage and rollback real. Store blob once; metadata is the product. For LLMs, register base weights, adapter, tokenizer, prompt bundle, and safety policy as separate artifacts composed into a release candidate (19).

Deep dive 2: stage machine and risk class

Low-risk prompt tweaks may auto-promote Staging→Production after golden evals. High-risk models (fraud, medical, child safety) require dual control and longer soak evidence. Exceptions expire. Never allow Dev→Production skip except break-glass with page + postmortem.

Deep dive 3: multi-tenant namespaces

Each team gets a namespace quota and IAM. Shared foundation models are read-only base artifacts; fine-tunes fork into the team namespace with parent digest links. Cross-tenant promote is a platform admin action with audit — not a convenience API.

Deep dive 4: coupling to canary without duplicating it

Registry answers "what may be served"; release plane answers "how traffic moves." Routers resolve aliases only. If canary fails, retarget alias to previous digest and freeze the bad promotion. Do not rebuild training in a 45-minute interview — show identity + gates + alias rollback.

Staff+/Principal signal rubric

• Mid-level: store models in object storage with naming conventions.
• Senior: registry UI with stages and basic metrics attached.
• Staff+: immutable digests, lineage, enforced gates, aliases, tenant isolation, fail-closed consumers.
• Principal: risk-class promotion policy, dual control, org-wide migration off path-based deploys, and clear ownership of metadata DR.

Follow-up questions to expect

• "Who can force-promote?" Break-glass role, dual control, expiry, postmortem — not a quiet admin flag.
• "How do you delete a poisoned dataset?" Tombstone lineage; block descendants from promotion; honor legal hold.
• "Notebooks?" Fine in Dev. Production aliases reject unsigned notebook artifacts.

Related

• ai-system-design/19 Model release, canary & rollback
• ai-system-design/07 LLM evaluation & observability
• ai-system-design/04 Feature store / fine-tuning data pipeline
• 02 Training–serving skew platform