Design data/concept drift monitoring and shadow scoring

Expected question

"Design the LLMOps/MLOps monitoring plane that detects data and concept drift and uses shadow scoring to validate challengers safely before user-visible canaries."

Variant forms

"Production accuracy fell but we lack labels for weeks — how do you monitor?"
"Design shadow traffic for a new ranker without changing user results."
"How do you separate feature drift from label/concept drift?"
"Design alert policy that does not page on every distribution wiggle."
"LLM traffic mix shifted to a new language — what monitors fire?"
"Design delayed-label evaluation joined to shadow predictions."
"Walk through freeze/roll back when shadow disagreement explodes."
"How do online evals and drift monitors divide ownership with the eval platform?"

Where this actually gets asked

ML platform and applied science loops; pairs with canary design (19) and eval platforms (07). This entry owns continuous distribution health + shadow prediction comparison, not full progressive delivery.

Executive summary

30-second thesis

I'd watch inputs and proxy outputs continuously, shadow-score challengers for disagreement, and only then spend canary budget on user-visible risk.

2-minute answer

Labels arrive late — that's the scar. So I don't wait for perfect labels to know something's wrong.

Track feature/prompt embedding distributions, null rates, segment mix, and score histograms against a baseline. When labels show up, join for concept-drift slices. Run challengers in shadow on sampled traffic; compare disagreement, latency, and cost with zero user exposure. Page on sustained, risk-weighted breaches with owners and runbooks — not every distribution wiggle.

Feed signals into promotion/canary gates. What I'd refuse: inventing a second release system next to the one you already have.

Quantitative trade-offs

Decision	Trade-off and reversal evidence	Evidence class
Sensitive drift alerts vs noise	Tight thresholds catch early decay; reverse when false pages exceed on-call budget.	H
Heavy shadow sampling vs cost	Higher sample improves challenger confidence; reverse when GPU/CPU cost exceeds expected lift.	H
Proxy metrics vs waiting for labels	Proxies enable same-day action; reverse reliance when proxy-label correlation collapses.	H

ML fundamentals

ML fundamentals: covariate vs concept drift, delayed labels, calibration of scores, slice-aware monitoring, and selection bias in labeled subsets.

Migration and rollout

Define baseline windows per model alias and critical slices.
Emit prediction and feature stats from online scorers.
Enable shadow for one challenger with cost caps.
Wire gates: drift breach blocks widen; shadow fail blocks promote.
Expand to LLM prompt/router aliases after classical models soak.

Org ownership and operating model

ML platform owns monitor pipeline, shadow runner, and alert routing.
Model owners own baselines, slice definitions, and response to pages.
Eval platform owns golden suites; drift plane owns live distribution health.
SRE owns page urgency and multi-model incident aggregation.
FinOps owns shadow cost budgets.

Requirements

Functional

Baselines and live stats for features, segments, and prediction scores.
Shadow execution of challenger digests on sampled traffic.
Delayed-label join jobs for concept-drift evaluation.
Alert → ticket/page with model alias, slice, and recommended action.
APIs for gate checks used by registry/release.

Non-functional

Monitor freshness SLO (illustrative H: dashboards < 5 min lag).
Shadow overhead capped (illustrative H: ≤5% of serving cost).
Privacy: no raw user content in general drift stores without policy.
Multi-tenant isolation of stats and alerts.

Core entities

Baseline: alias, window, feature/score histograms, slice.
DriftMetric: psi/kl/ks/population mix delta, severity.
ShadowRun: challenger_digest, sample_rate, disagreement_rate, cost.
LabelJoin: prediction_id, label, label_ts, delay.
AlertPolicy: thresholds, for-duration, owner, runbook.

API / interface

GET /v1/models/{alias}/drift
→ { "overall":"warn", "slices":[{"segment":"mobile_es","psi_max":0.35}] }

POST /v1/shadow
{ "champion":"ranker@prod", "challenger_digest":"sha256:...", "sample_rate":0.05 }
→ 201 { "shadow_id":"s_..." }

GET /v1/shadow/{id}
→ { "disagreement_rate":0.18, "latency_p99_delta_ms":12, "gate":"fail" }

Data Flow

Online predictions → stats stream → baseline compare → alerts; optional shadow branch scores same request IDs → disagreement store → gate; delayed labels → concept drift reports.

Rendering architecture diagram…

High-level design

Separate health monitors (always on) from shadow experiments (budgeted). Both feed the same gate interface used by registry and canary controllers.

Deep dive 1: what to monitor without labels

Null spikes, categorical mix shifts, embedding cluster mix, output length/toxicity proxies for LLMs, score calibration vs recent labeled slice. Prefer a small owned metric set over twenty noisy charts.

Deep dive 2: shadow semantics

Identical request context, no user-visible side effects, deterministic logging of both outputs. Disagreement thresholds depend on task: ranking top-1 disagreement differs from generative semantic distance. Cap fan-out to one challenger by default.

Deep dive 3: alert design

Require for-duration, slice-aware thresholds, and inhibit during known migrations. A platform-wide distribution shift should create one incident, not fifty model pages.

Deep dive 4: handoff to canary

Shadow pass ≠ user-safe. It clears infra/disagreement risk; UX/safety still need bounded canaries (19).

Staff+/Principal signal rubric

Mid-level: plot accuracy when labels exist.
Senior: PSI dashboards + ad hoc shadow scripts.
Staff+: baseline contracts, shadow service, delayed-label joins, gate APIs, cost caps.
Principal: org alert taxonomy, proxy governance, multi-model incident aggregation, FinOps for shadow.

Follow-up questions to expect

"All features drifted after a logging change — model bad?" Probably instrumentation. Compare to skew monitors before you retrain the world.
"Shadow agrees, canary fails CSAT — why?" UX, latency, or position effects shadow never sees. Don't trust shadow as a full product eval.