Design a training–serving skew detection and prevention platform

Expected question

"Design the platform that prevents and detects training–serving skew — features, preprocessing, and prompts that differ between offline training and online inference."

Variant forms

• "Offline AUC is great but production collapses — how do you debug train/serve skew?"
• "Design shared feature definitions for batch training and online scoring."
• "Our LLM prompt template in training eval differs from production — architect parity."
• "Design continuous skew monitors on feature distributions and null rates."
• "How do you version transformers so training and serving execute the same code?"
• "Point-in-time correct training vs real-time features — where does skew still hide?"
• "Design shadow scoring that compares logged production features to offline recomputation."
• "Walk through a skew incident: which metrics page first, and who owns the fix?"

Where this actually gets asked

Classic ML platform / MLOps deep dive; increasingly asked for LLMOps when prompt/tool schemas drift from eval harnesses. Complements feature stores (04) by owning parity contracts and detection, not just storage.

Executive summary

30-second thesis

Skew is a contract failure. Same feature or prompt identity has to run the same transform code on train and serve — I'd detect divergence continuously, not debug it after every launch.

2-minute answer

I'd start by publishing versioned transformation packages used by both offline pipelines and online scorers. Log production feature vectors (or hashes) with model/prompt versions. Continuously recompute offline from the same raw events and compare distributions, null rates, and exact mismatches. Gate promotion on skew SLOs. For LLMs, pin prompt/tool schema digests in eval and production routers.

What I'd refuse: blaming the model first when null rates diverged two weeks ago. Page when skew exceeds budget before the AUC theater starts.

Quantitative trade-offs

ML fundamentals

ML fundamentals: train-serve skew, leakage vs skew distinction, covariate shift vs pipeline bugs, and feature store point-in-time joins as necessary but not sufficient.

Migration and rollout

1. Inventory features/prompts with dual implementations; mark parity unknown.
2. Wrap online path to emit feature logs with schema_version.
3. Stand up offline recompute jobs for top revenue/risk models first.
4. Add skew dashboards; then enforce promotion gates (illustrative H: critical features mismatch rate < 0.1%).
5. Delete duplicate transform code paths after parity holds for a soak window.

Org ownership and operating model

• ML platform owns skew detection and shared feature compute contracts.
• Model owners own which skews are fatal vs warning for their model.
• Data engineering owns batch pipelines that feed training.
• Serving owners owns online feature path latency.
• Eval owns holdout sets used for skew gates.

Requirements

Functional

• Single versioned transform/prompt identity consumed by train and serve.
• Production logging of feature values or approved aggregates with versions.
• Offline recompute and comparison jobs with alerting.
• Promotion API checks latest skew report for the artifact.

Non-functional

• Detection lag SLO (illustrative H: page within 15 minutes of sustained breach).
• Sampling strategy that still bounds miss rate for rare categorical skew.
• Privacy-safe logging; no raw PII in general skew stores.
• Low online overhead (illustrative H: <2% added latency budget).

Core entities

• TransformPackage: version, code_digest, feature_schema.
• FeatureLog: request_id, feature_vector_or_hash, model_alias, ts.
• SkewReport: feature, metric (psi/ks/null_rate/exact_mismatch), severity.
• ParityContract: required shared packages per model.
• GateResult: pass/fail for promotion.

API / interface

POST /v1/transforms { "name":"user_velocity_v3", "code_digest":"sha256:..." }
GET /v1/models/{alias}/skew?window=1h
→ { "features":[{"name":"amount_z","psi":0.28,"status":"breach"}], "gate":"fail" }
POST /v1/promotions/{id}/checks/skew → 200 pass | 422 fail

Data Flow

Online scorer applies TransformPackage vN → emits FeatureLog → stream to warehouse → recompute job applies same package to raw events → SkewReport → alert + promotion gate.

Rendering architecture diagram…

High-level design

Shared transform SDK + feature log bus + continuous comparison + gates. Prefer one code path over "document the differences."

Deep dive 1: skew vs drift vs leakage

Skew = pipeline parity bug (train and serve disagree on the same event). Drift = world changed. Leakage = train saw future labels/features. Staff+ separates these in the first minute; treating skew as "the model went bad" wastes weeks.

Deep dive 2: LLM prompt/tool skew

Eval harnesses often use slightly different system prompts, tool JSON, or truncation. Pin prompt digests in registry aliases and fail CI when production router digest ≠ eval digest (01).

Deep dive 3: what to log

Log schema_version, null flags, numeric summaries, and categorical top-k. Full vectors for a sampled fraction. Join keys must survive privacy redaction policies.

Deep dive 4: ownership and incident response

When skew pages: freeze promotions, compare last good package, roll back alias if needed, then fix transform. Platform owns detector reliability; model team owns semantic fix.

Staff+/Principal signal rubric

• Mid-level: "keep preprocessing the same" without mechanism.
• Senior: shared library + some dashboards.
• Staff+: versioned packages, logs, continuous compare, promotion gates, LLM prompt digest parity.
• Principal: org migration off dual implementations, privacy-safe logging standards, risk-tiered enforcement.

Follow-up questions to expect

• "PSI spiked but exact match is fine — ship?" Investigate first. Distribution shift may be drift, not skew — don't rubber-stamp.
• "Can feature stores alone solve this?" Necessary for point-in-time joins; not enough without shared execution packages and continuous monitors.

Related

• ai-system-design/04 Feature store
• 01 Model/experiment registry
• 03 Drift monitoring & shadow scoring