Design ML CI/CD and continuous training pipelines
Expected question
"Design CI/CD for machine learning — how do code, data, and models move from commit to production with tests, registries, and safe continuous training triggers?"
Variant forms
- "How is CD different for ML than for microservices?"
- "Design a pipeline that retrains when drift fires — without thrashing."
- "What do you test in CI for ML (data, schema, model, latency)?"
- "Design GitHub Actions / Vertex / SageMaker Pipelines for weekly retrains."
- "How do you prevent a bad dataset from promoting a model?"
- "Continuous training vs continuous deployment — when to auto-ship."
- "LLMOps: CI for prompts, eval suites, and index rebuilds."
- "Walk through rollback when a retrain passes offline but fails canary."
Where this actually gets asked
Among the highest-frequency Staff MLOps/LLMOps questions across interview guides (CI/CD for ML, retraining triggers, data+model tests). Complements registry (01), drift (03), and canary (ai-system-design/19) by owning the pipeline orchestration and gates.
Executive summary
30-second thesis
Data, code, model, and metrics are versioned artifacts. CI validates all four; CD promotes via registry aliases. Continuous training is gated automation — not infinite auto-deploy.
2-minute answer
On code, schedule, or drift trigger: validate dataset schema and freshness → train with pinned feature package → offline eval vs champion → register digest → staging/shadow → canary → alias promote. Auto-retrain can stop at a registry candidate; auto-prod only for low-risk models with burn-rate limits.
LLMOps: prompts and index rebuilds share the same gate types with different artifacts. What I'd refuse: "retrain again and hope" as rollback. Roll back by alias retarget. Cooldown on drift triggers so you don't thrash the bill.
Quantitative trade-offs
| Decision | Trade-off and reversal evidence | Evidence class |
|---|---|---|
| Auto-promote to prod vs human gate | Speed; reverse for high-risk models always. | H |
| Retrain on every drift alert vs cooldown | Freshness vs thrashing/cost; require sustained breach + min sample. | H |
| Full retrain vs incremental/fine-tune | Quality vs cost; reverse when incremental drifts from champion lineage. | H |
ML fundamentals
Data tests, leakage checks, holdout hygiene, offline vs online metrics mismatch, and champion/challenger discipline.
Migration and rollout
- Wrap current train scripts as versioned pipelines.
- Add data validation + offline eval gates before any auto-registry write.
- Connect drift alerts to candidate training jobs (not prod alias).
- Enable auto-canary for one low-risk model.
- Expand risk tiers; keep dual control for money/safety models.
Org ownership and operating model
- ML platform owns CT pipeline templates and gate libraries.
- Model owners own approve-on-prod and retraining triggers.
- Data engineering owns upstream data contracts that break CT.
- SRE owns pipeline reliability and secret injection.
- FinOps owns training job cost caps.
Requirements
Functional
- Pipelines triggered by code, schedule, or approved monitoring events.
- Data/schema/validity tests; model eval gates; latency smoke tests.
- Register artifacts with lineage; promote via aliases.
- Continuous training with cooldown and budget caps.
- LLM artifact paths: prompts, tools, indexes.
Non-functional
- Pipeline reproducibility from digests.
- Training budget caps per team.
- Idempotent runs; no duplicate prod promotes.
- Audit who/what triggered each retrain.
Core entities
- PipelineRun, DatasetVersion, TrainJob, EvalReport, TriggerPolicy, CooldownCooldown, ArtifactAlias.
API / interface
POST /v1/pipelines/fraud-weekly/run { "trigger":"schedule" }
POST /v1/triggers/drift { "alias":"fraud@prod", "report_id":"..." }
→ 202 { "run_id":"...", "mode":"train_to_candidate_only" }
Data Flow
Trigger → validate data → train → offline eval → registry candidate → shadow/canary → alias → monitor → (optional) next trigger.
Rendering architecture diagram…
High-level design
Orchestrator + feature/train/eval steps + registry + delivery plane. Monitoring closes the loop with policy, not infinite recursion.
Deep dive 1: what CI must test
Schema, null budgets, segment coverage, training job dry-run, offline metrics vs floor, prediction latency on sample, prompt contract tests for LLMOps.
Deep dive 2: continuous training without thrash
Require sustained drift, minimum new labels, cost budget, and cooldown. Prefer train-to-candidate; separate promote decision.
Deep dive 3: LLMOps specifics
Index rebuild CI (embedding version pin), prompt eval suites, tool-schema compatibility — same registry/canary spine (ai-system-design/19).
Deep dive 4: failure and rollback
Failed canary: freeze trigger, retarget alias, file incident with dataset+code digests. Do not auto-retrain in a loop during incidents.
Staff+/Principal signal rubric
- Mid-level: cron train and manual upload.
- Senior: CI that runs tests then deploys.
- Staff+: data+model gates, registry aliases, CT policy with cooldown, LLM artifact parity.
- Principal: risk-tiered automation, org pipeline templates, FinOps caps on CT, clear ownership of thrash incidents.
Follow-up questions to expect
- "Offline pass, online fail — next step?" Shadow/canary evidence first; check skew. Do not keep auto-promoting while you "investigate."
- "Who can disable gates?" Break-glass only — audited, expiring, dual control. Permanent disable is how bad models ship forever.