Design a real-time fraud and risk decisioning system

Expected question

"Design a real-time payment or account-fraud detection system. How do you make a decision under a strict latency SLA when labels are delayed, fraud is rare, and attackers adapt to the model?"

Variant forms

• "Design Stripe-style card fraud detection with a P99 decision latency below 100ms."
• "How do you detect account takeover using device, identity, velocity, and graph signals?"
• "Design a rules + ML + human-review cascade for high-risk transactions."
• "Fraud losses fell but legitimate-customer declines doubled — how do you retune the system?"
• "How do you train when chargeback labels arrive weeks later and manual review changes the data?"
• "Design degraded behavior when the online feature store or model service is unavailable."
• "How do you deploy a new fraud model against adaptive attackers without widening losses?"
• "Design explainable risk decisions, appeals, and audit evidence for regulated payments."

Where this actually gets asked

This is a longstanding, high-frequency ML system-design archetype rather than a speculative GenAI trend. Exponent's candidate-report-based ML system-design guide explicitly includes Stripe fraud detection among reported prompts. Exact company wording and frequency are not public, so treat the attribution as candidate-reported, not company-confirmed. The architecture itself is grounded in Stripe's primary engineering accounts of Radar, its ML feature platform, and its fraud ML flywheel.

Executive summary

30-second thesis

I'd optimize expected dollars of harm under a hard latency budget — not F1 on a rare-fraud dataset. Labels arrive late, attackers adapt, and "approve everything" or "decline everyone" when features die are both wrong.

2-minute answer

I'd cascade: cheap rules and cached features first, low-latency model next, expensive graph/identity or human review only in the ambiguous band. Calibrate scores, then set thresholds by amount, reversibility, and customer harm — one global F1 hides the business decision.

Point-in-time features or you leak the future into training. Chargebacks and review outcomes are delayed and biased; I'd measure on mature-label cohorts and keep some adjudication samples where it's legal and economic. Shadow/canary challengers with a loss budget; emergency rules need owners and expiry or they become permanent contradictory policy.

Degraded mode: required vs optional features, freshness, conservative fallback for low-value established accounts, step-up/hold for high-value irreversible events — degraded=true with an ops loss budget. Fraud ops owns thresholds with model owners; otherwise nobody owns the FP vs loss trade.

What I'd ask them: Decision P99 budget? Payment vs login/ATO scope? Review capacity? Who owns threshold changes today?

Quantitative trade-offs

Numbers and thresholds in interview delivery should be labeled H unless the candidate can defend a measured baseline. Open repositories are O; researched public patterns are R. Do not upgrade O/R into employer P adoption.

ML fundamentals

ML fundamentals: class imbalance, delayed labels, calibration, point-in-time features, leakage, slice debugging, and cost-sensitive thresholds.

Migration and rollout

Challenger models here burn real money and real customers. Treat canary as a loss budget.

1. Freeze feature point-in-time contracts; dual-compute challenger scores in shadow with zero user effect.
2. Compare on mature-label cohorts (chargebacks/reviews that have landed) — not same-day F1 theater.
3. Canary with an explicit dollar loss budget and FP budget; fraud ops on the pager for threshold moves.
4. Emergency rules require owner + expiry before they can override the model.
5. Rollback = previous champion alias + rule set. Kill switch = force step-up / hold for high-value irreversible events.

Org ownership and operating model

• Fraud / risk ops owns thresholds, review queues, and the FP vs loss trade — not "the model team alone."
• Model owners own training leakage controls, calibration, and challenger shadow metrics.
• Feature platform owns point-in-time correctness and freshness SLOs.
• Payments / auth product owns what "decline" and "step-up" mean in the user journey.
• Exec risk sponsor owns the loss budget that makes canary decisions non-political.

Requirements

Functional

• Score payment, login, account-change, or payout events before committing the action.
• Combine deterministic rules, real-time ML, optional expensive checks, and manual review.
• Return approve / challenge / review / decline with calibrated risk and reason codes.
• Ingest confirmed fraud, chargebacks, appeals, and review outcomes as delayed labels.
• Support analyst-authored emergency rules and champion/challenger model releases.

Non-functional

• Keep the synchronous decision inside the product's latency budget (for example, P99 <100ms).
• Optimize expected economic and customer harm, not raw accuracy on an imbalanced dataset.
• Preserve point-in-time correctness between training and online features.
• Fail safely when features or models are unavailable without declining all legitimate traffic.
• Audit every decision with feature, rule, model, threshold, and policy versions.
• Detect drift and attacks quickly enough to contain loss before normal retraining catches up.

Core entities

• Risk event: actor, account, device, instrument, merchant/resource, amount, and event time.
• Feature snapshot: point-in-time values plus freshness and provenance for each feature.
• Rule result: matched rule, severity, action, owner, version, and expiration.
• Risk score: calibrated probability/expected loss with model version and reason codes.
• Decision: approve / challenge / review / decline plus threshold and policy version.
• Label: fraud, legitimate, chargeback, appeal, or unknown with observation time and confidence.
• Case: evidence packet and reviewer outcome for ambiguous or consequential events.

API / interface

POST /v1/risk/decisions
{"event_id":"evt_...","type":"payment","account_id":"acct_...",
 "amount_cents":42000,"device_id":"dev_...","occurred_at":"..."}
→ 200 {"decision":"challenge","risk_score":0.81,
 "reason_codes":["new_device","velocity_1h"],"decision_id":"dec_...",
 "model_version":"fraud-v42","policy_version":"p17","degraded":false}

POST /v1/risk/labels
{"event_id":"evt_...","label":"confirmed_fraud","source":"chargeback",
 "observed_at":"...","confidence":1.0}
→ 202 {"accepted":true}

POST /v1/risk/rules
{"predicate":{...},"action":"challenge","expires_at":"...",
 "reason":"active_card_testing_campaign"}
→ 201 {"rule_id":"rule_...","version":3}

POST /v1/risk/cases/{case_id}/decision
{"outcome":"legitimate","reviewer_id":"analyst_...","reason_codes":[...]}
→ 200 {"label_id":"lbl_...","policy_action":"release"}

Clients receive stable reason codes, not raw model internals. Reusing event_id is idempotent and returns the original decision unless an explicit re-evaluation contract is invoked.

Data Flow

Rendering architecture diagram…

Delayed labels flow asynchronously into a point-in-time training set, monitoring, and the next champion/challenger cycle; they never block the hot path.

High-level design

Rendering architecture diagram…

The system has two clocks: a millisecond decision path and a days-to-weeks label path. A strong design connects them without pretending ground truth is immediately available.

Deep dive 1: optimize expected harm, not accuracy

Fraud is rare, so a model that predicts "legitimate" for every event can have excellent accuracy and zero utility. The decision threshold should minimize expected cost:

expected cost =
  P(fraud | x) × loss_if_approved
  + P(legitimate | x) × customer_harm_if_blocked
  + challenge/review cost

Calibrate scores, then set thresholds by amount, product, geography, account maturity, and action reversibility. Evaluate precision/recall, PR-AUC, calibration, dollars prevented, false-positive customer harm, challenge completion, and review capacity. One global F1 threshold hides the business decision the model exists to support.

Deep dive 2: a latency-bounded cascade

Run cheap deterministic blocks and cached features first, then the low-latency model. Reserve expensive graph expansion, external identity checks, or manual review for the ambiguous band.

Budget each dependency and use deadlines, not unbounded retries. If a feature arrives after the decision deadline, record it for analysis but do not let it stall the transaction.

Deep dive 3: point-in-time features and delayed labels

Training features must be reconstructed as known at event time. Joining today's account status or chargeback count onto last month's event leaks the future and inflates offline performance. Use event-time windows, immutable feature definitions, and the same transformations for batch and online serving. Labels carry both event time and observation time so backtests reproduce what was knowable at each release.

Chargebacks are delayed and incomplete; manual review creates selection bias because only some events are inspected. Maintain randomized exploration or adjudication samples where legally and economically safe, track label-source confidence, and measure performance on mature-label cohorts rather than declaring yesterday's low chargeback rate a win.

Deep dive 4: adaptive attackers and release safety

Monitor feature distributions, score distributions, rule-hit rates, attack-cluster velocity, approval/decline mix, and eventually matured loss by slice. Emergency rules stop an active attack within minutes but must have owners and expiration dates or the rule set becomes permanent, contradictory policy.

Deploy model changes in shadow first, compare counterfactual decisions, then canary by bounded traffic and loss budget. Keep champion/challenger and instant rollback. Attackers react to visible decision boundaries, so avoid exposing scores or overly specific decline reasons to untrusted clients while retaining full internal audit evidence.

Deep dive 5: degraded modes

Do not make "feature store unavailable" equivalent to "approve everything" or "decline everyone." Classify features as required or optional, attach freshness, and train/test explicit missing-value paths. Under degradation:

• hard compliance/account blocks remain local and available;
• low-value, established-account events may use a conservative fallback model/rules;
• high-value or irreversible events step up, hold, or review;
• stale/missing features set degraded=true and trigger an operational loss budget;
• a circuit breaker can narrow accepted traffic while dependencies recover.

Staff+/Principal signal rubric

• Mid-level: rules plus a classifier and fraud/legitimate labels.
• Senior: online features, class imbalance, calibrated thresholds, and a latency budget.
• Staff+: point-in-time correctness, delayed labels, cascade economics, review/challenge design, champion/challenger rollout, and explicit degraded modes.
• Principal: frames the system as economic and customer-risk control against adaptive adversaries; quantifies label bias, feedback loops, attack containment, auditability, appeals, and cross-functional threshold ownership.

Follow-up questions to expect

• "Why not maximize recall?" — Near-100% recall rejects too many good customers. Optimize expected harm under review capacity and regulation — say that trade out loud.
• "How do you know a decline was correct?" — Often you don't see the counterfactual. Mature labels, appeals, review samples, careful exploration — and causal humility.
• "Rules or ML?" — Both. Rules contain known attacks fast; ML generalizes weak signals. Version and evaluate them together.
• "What happens at 10× traffic?" — Precompute aggregates, bound fan-out, cache stable features, keep per-dependency deadlines. Don't let one slow graph hop blow the SLA.
• "What do you cover in 45 minutes?" — Cost objective, online cascade, point-in-time features, delayed labels, adaptive rollout, degraded mode.

Related

• 05 Content moderation and safety — similar cascade, different harm and label economics
• 07 LLM evaluation and observability — versioned eval and release gates
• 18 AI data flywheel and human feedback — feedback collection and label quality
• 19 Model release, canary, and rollback — progressive model delivery
• Stripe Radar architecture
• Stripe's ML feature platform