Design a human-in-the-loop operating model at org scale

Expected question

"Design human-in-the-loop for enterprise agents at org scale — not a checkbox on one graph node. Who reviews what, under what SLA, how do you sample vs 100% review, and when do you promote autonomy?"

Variant forms

Interviewers often ask the same design with different framing — recognize the archetype:

"HITL doesn't scale — we have 10 reviewers and 40 teams shipping agents."
"Design the approval queue: paging, SLAs, dual control, and what happens at 2am."
"When do you take HITL off a tool path without creating silent writes?"
"Rubber-stamp humans aren't oversight. How do you measure reviewer quality?"
"Design sampled review vs blocking review vs post-hoc audit."
"The VP wants autonomy next quarter. What's the promotion gate?"
"How do HITL, the gateway, and evals fit — aren't they the same?"
"Design exception expiry so standing bypasses don't become the architecture."

Where this actually gets asked

Principal loops that already believe in "put a human on the write." The Staff answer is interrupt_before on a node (13). The Principal question is an operating model: queues, staffing, sampling, promotion, and dual control — the same class of problem as change-management and payments authorization, applied to agents. Public patterns: high-risk AI oversight language in the EU AI Act (R); incident postmortems where autonomy outran review (R anecdotes, treat as archetypes). No claim that a named company asks this exact prompt.

Executive summary

30-second thesis

I'd risk-tier tools, not teams. Irreversible writes block on a human who can refuse; medium risk is sampled with a kill switch; low risk is audited. Autonomy is a promotion through evals and incident budget — not a launch default. A reviewer who cannot stop the write is decoration.

2-minute answer

Three layers people mash together: policy (is this action allowed at all), HITL (does a human have to say yes on this instance), evals (does the system still meet quality/safety bars). Policy is the gateway. HITL is a queue with an SLA. Evals are CI and online gates. I'd never use evals as the only brake on a refund or a CRM overwrite.

Risk tiers: T0 read/draft; T1 reversible writes with sampled review; T2 irreversible or regulated — blocking HITL, dual control above a threshold; T3 prohibited — deny, no exception without legal. Reviewers see tenant, user, tool args, and the model's justification. They need a refuse button that actually fails closed.

Capacity: if you require 100% blocking review on a high-QPS path, you designed a call center. I'd default T2 to blocking with a staffing model, and use sampling on T1 with paging when the reject rate or eval slice moves. Promotion to more autonomy: holdout evals, dual-run vs human, incident rate, and a dated expiry. Reverse on the first silent-wrong-write class.

What I'd ask them: Which tools are irreversible today? Reviewer org — central vs federated? Night/weekend coverage? What "autonomy" means in their ladder?

What I'd refuse: silent allow at 2am because 'we don't have reviewers.' T2 doesn't run, or you staff it, or you narrow the tool.

Quantitative trade-offs

Decision	Trade-off and reversal evidence	Evidence class
Blocking HITL vs sampled audit	Blocking prevents the write; sampling scales. Reverse to blocking after a silent-wrong-write or regulator finding.	H
Central review team vs federated domain approvers	Central is consistent; federated knows the business. Reverse to central when domains rubber-stamp; reverse to federated when the queue SLA is fiction.	H
Dual control vs single approver	Dual control cuts fraud/confused-deputy; it doubles latency. Use above $ / blast-radius thresholds.	H
Promote autonomy vs freeze	Promotion reduces cost; freeze after an incident until the eval that would have caught it is in CI.	H

Migration and rollout

You're moving from hero approvers in Slack to a queue.

Inventory tools. Tag T0–T3. No unlabeled irreversible write in prod.
Put T2 on gateway require_approval with a real queue UI — not a chat ping that can be missed.
Measure: time-to-decision, refuse rate, override rate, reviewer disagreement, pages missed.
Add sampling on T1; page if refuse rate or eval slices move.
Autonomy promotion is a written gate with expiry. Rollback = re-block the tool class. Kill switch = deny all T1+ writes.

Org ownership and operating model

AI platform owns gateway states (allow / deny / require_approval), queue plumbing, audit.
Domain owners own T2 decisions for their tools (payments, CRM, publish).
Risk / legal own T3 and dual-control thresholds.
Ops own SLA, on-call, and staffing math.
Eval owners own the promotion packet — not the VP's demo.

HITL that rubber-stamps 400 cases an hour is not oversight (31).

Requirements

Functional

Classify actions into risk tiers; route T2 to blocking review; T1 to sample; T3 to deny.
Approver sees tenant, principal, tool, args digest, and can refuse.
Dual control above threshold; exception records expire.
Resume durable runs after approval without double side effects.

Non-functional

Named SLA (e.g. p50 decide in minutes for interactive; hours for batch — H until they give numbers).
Fail closed if the queue is down for T2.
Audit: who approved what, with the payload digest bound to the execution token (24).

Core entities

Risk tier — T0–T3 on (tool, arg shape, tenant, amount)
Approval request — bound payload digest, SLA, assignee
Reviewer role — domain vs central vs dual
Promotion record — eval packet, incident budget, expiry
Exception — owner, expiry, kill switch

API / interface

Gateway: authorize_tool → allow | deny | require_approval
POST /approvals/{id}/decide — accept/refuse + reason
GET /approvals?sla_breach=true — ops
Eval: promotion packet id required to move T2 → T1

Data Flow

Tool call → gateway policy → if T2, persist approval + release the worker (13) → reviewer decides → mint execution token bound to digest → tool runs once → audit. Timeout → fail closed or safe degrade per tier (T2 does not silently allow).

High-level design

Rendering architecture diagram…

Deep dive 1: sampling is not a substitute for a brake

Hard constraint: if a wrong refund cannot be undone, sampling only tells you how often you already failed.

Use blocking HITL (or dual control) on that class. Sampling belongs on T1 and on reviewer quality (second-reviewer spot checks).

Deep dive 2: reviewer fatigue and rubber stamps

Hard constraint: a green button with no refuse budget becomes autonomy with extra latency.

Measure refuse rate. If it's ~0% on a noisy tool, either the tool is mis-tiered or reviewers are theater. Rotate, cap queue depth, and let refuse be the celebrated path. Show the payload diff, not a wall of JSON.

Deep dive 3: promoting autonomy

Hard constraint: "the demo worked" is not a gate.

Packet: offline eval slices including the failure class you fear, dual-run vs human on a holdout, incident budget, dated expiry, owner. First silent-wrong-write → reverse the tier. Same instinct as canary/rollback for models (19).

Staff+/Principal signal rubric

Mid-level: "We'll add an approve button."
Senior: interrupt_before on one graph; may leave Slack as the queue.
Staff+: gateway states, fail closed, digest-bound execute, durable pause.
Principal: org-scale operating model — tiers, staffing/SLA, sampling vs blocking, promotion/expiry, dual control, reviewer quality, who inherits the standard. Fail-closed: T2 does not silently allow when the queue is down. Clock: lock tiers + promotion gate; skip building a ticketing product.

Follow-up questions to expect

"We don't have reviewers at 2am." Then T2 doesn't run at 2am, or you staff it, or you narrow the tool. Silent allow is not a coverage strategy.
"Can't evals replace HITL?" Evals catch classes. HITL catches this instance. Use both. Evals promote; they don't refund.
"Every team wants a bypass for launch." Time-bound exception, owner, expiry, or it becomes the architecture (10).
"What do you skip in 45 minutes?" Building a full ticketing product. Cover tiers, fail closed, promotion gate.

What I'd ask them

Which write is irreversible in their domain — money, identity, customer message, deploy?
Central security review vs domain approvers?
What autonomy have they already reversed?