Design an AI data flywheel and human-feedback platform

Expected question

"Design a data flywheel for a production LLM product. How do you sample production traffic, label or preference-rank examples, control quality, and promote datasets into training and eval without poisoning the model?"

Variant forms

Interviewers often ask the same design with different framing — recognize the archetype:

"Design the human feedback loop that turns ChatGPT thumbs-down into better SFT/RLHF data."
"How do you sample 0.1% of production traces for labeling without biasing the dataset?"
"Design annotation workflows for preference pairs with inter-annotator agreement gates."
"Our flywheel started rewarding sycophancy — architect detection and dataset surgery."
"Design PII redaction and consent before any production log enters training."
"How do you version datasets, promote to gold, and roll back a bad labeling week?"
"Design active learning: which failures should humans see first?"
"Architect a red-team / adversarial data pipeline separate from organic feedback."

Where this actually gets asked

High-frequency at OpenAI/Anthropic/Google/Meta-style ML platform / applied science Staff+ rounds — the "how does the product get better every week" question. Distinct from the training pipeline itself (08) and from offline eval (07): this entry owns sampling → labeling → dataset contracts → promotion. Prep aggregators over-attribute company names; the archetype is real.

Executive summary

30-second thesis

I'd treat the flywheel as sampling → redact → label → gated promote — never "log thumbs and fine-tune." Uniform 0.1% undersamples the failures that matter, and a bad labeling week can poison the next model if train pins latest.

2-minute answer

I'd start with consent and redaction before any human or training store sees production text. Sample with reason codes: oversample tool failures, escalations, grounding declines; cap power users. Store why each row was sampled so we can reweight later. Never train only on thumbs-down — that teaches "be conservative and useless."

Annotation needs rubrics and IAA gates; spot-check safety queues with experts. Watch sycophancy and length bias with holdout probes. Keep red-team sets out of the organic mix so adversarial coverage doesn't get diluted.

Promotion is a hard gate: PII residual, agreement, schema, train/eval leakage. Training jobs pin dataset_id, not "gold latest." If online metrics regress, roll back dataset pin and model together. Limitation I'll own: label latency and vendor quality will lag product incidents — hot queues and emergency datasets exist for a reason, but they still go through gates.

What I'd ask them: Consent model for train_ok? Who owns safety vs style queues? Eval holdout leakage checks today? Vendor IAA thresholds?

What I'd refuse: training on 'gold latest' with no dataset pin, or organic and adversarial labels in one unlabeled bucket. Clock: lock consent + holdout + rollback of dataset and model.

Migration and rollout

The thing you migrate is the dataset pin, not a labeling vendor.

Consent + redaction before any new row enters a human or train store.
Split organic vs adversarial queues; IAA gates on both. No unlabeled mixed bucket.
Train jobs pin dataset_id. Shadow a new pin against the previous; leakage check on holdouts.
Online regress → roll back dataset pin and model together.
Kill switch = freeze publishers that write latest. Emergency hot-queues still go through the same gates.

Org ownership and operating model

Data flywheel owns sampling, pins, and the promote gate.
Safety owns adversarial/red-team queues.
Product owns style/quality queues and what "good" means.
Annotation ops / vendor own IAA and rubric drift.
Training owns refusing a job on an unpinned corpus.

What I'd ask them

Consent model for train_ok?
Who owns safety vs style queues?
Vendor IAA thresholds and leakage checks on holdouts?

Requirements

Functional

Capture production signals (thumbs, edits, escalations, tool failures) with consent/policy.
Sample and route items to human raters or specialist queues.
Produce versioned datasets: SFT rows, preference pairs, eval gold, adversarial sets.
Promote datasets through quality gates; block training on failed contracts.

Non-functional

Strict PII / secrets redaction before annotator eyes or training storage.
Sampling must be auditable and reweightable (rare failures oversampled intentionally).
Label latency SLOs by queue (safety same-day; style preference weekly).
Poisoning / reward-hacking resistance: adversarial and organic streams stay separable.

Core entities

Production event: request_id, model_id, prompt_hash, outcome signals, consent_flags.
Sampled item: event_ref, sampling_reason, queue, priority.
Annotation task: schema (rating / preference / critique), rater_id, agreement_score.
Dataset version: id, lineage, row_count, quality_report, promotion_status.
Promotion gate: checks (PII, agreement, schema, toxicity, leakage of eval into train).

API / interface

POST /v1/flywheel/events
{ "request_id":"...", "signals":{"thumb":"down","user_edit":true}, "consent":"train_ok" }
→ 202 { "accepted":true }

POST /v1/flywheel/sample
{ "strategy":"active_learning","quota":1000,"filters":{"failure_class":"grounding"} }
→ 200 { "batch_id":"b_...","items":[...] }

POST /v1/annotations/{task_id}/submit
{ "preference":"a_wins","critique":"...", "time_spent_s":42 }
→ 200 { "agreement":0.81 }

POST /v1/datasets
{ "name":"sft_support_v12","sources":["batch_..."], "split":"train" }
→ 201 { "dataset_id":"ds_...","status":"pending_gates" }

POST /v1/datasets/{id}/promote
→ 200 { "status":"gold" } | 422 { "violations":["pii_residual","iaa_below_threshold"] }

Staff+ callout: promote is a hard gate — training jobs must pin dataset_id, not "latest."

Data Flow

Signals → policy/consent filter → sample → redact → annotate → quality → dataset version → consume by train/eval (never silent auto-promote).

Rendering architecture diagram…

High-level design

Rendering architecture diagram…

Deep dives below target non-functional requirements (latency, scale, failure, cost, security).

Deep dive 1: sampling without silent bias

Uniform 0.1% undersamples rare safety failures and oversamples easy thumbs-up. Staff+ designs stratified + active sampling: oversample tool failures, escalations, grounding declines, and low-confidence routes; cap any single power-user's contribution. Store sampling_reason on every row so you can reweight later. Never train only on thumbs-down — that encodes "be conservative and useless."

Deep dive 2: annotation quality and reward hacking

Preference data needs clear rubrics and inter-annotator agreement (IAA) thresholds (e.g., Cohen's κ or pairwise agreement ≥0.7 for gold). Spot-check with experts on safety queues. Watch for sycophancy and length bias in raters — measure with holdout probes. Keep red-team datasets out of the organic mix so adversarial coverage does not get diluted.

Production → train is a privacy incident waiting to happen. Redact before humans; dual-control for export; honor consent=train_ok. Hash prompts for dedup. Eval leakage: promotion gates must block train rows that near-duplicate gold eval items (07). Pin dataset versions in training configs for reproducibility and rollback.

Deep dive 4: promotion SLOs and bad-week rollback

A labeling vendor outage or rubric change can poison a week of data. Keep pending vs gold states; train only on gold. If online metrics regress after a train job, roll back dataset pin and model together (19). In 45 minutes, cover sampling bias, redaction, IAA gate, and version pins — not GPU kernel details.

What's expected at each level

Mid-level: "log thumbs and fine-tune on them."
Senior: human annotation queue + basic PII stripping.
Staff+: stratified/active sampling with reason codes; IAA gates; dataset versioning; train/eval leakage checks; consent.
Principal: reward-hacking detection, vendor quality contracts, dataset rollback, organic vs adversarial streams. Who inherits: data + eval owners. Fail-closed: don't train on unlabeled production dumps. Clock: lock labeling contract + promotion gate; skip inventing a new RL algorithm.

Follow-up questions to expect

"What do you skip in 45 minutes?" A new RL algorithm. Cover labeling contract, pins, incident rollback.
"How do you stop the model from learning to please raters?" — Clear rubrics, sycophancy probes, and outcome metrics mixed with preference data. Raters drift; assume it.
"Who sees raw user data?" — Need-to-know queues only, redact first, audit access. If we can't say who saw what, the flywheel is a privacy incident.
"How fast can a new failure class enter training?" — Hot queue → emergency dataset → still gated promote. Fast is not "skip the gates."