Design an automotive supply-chain AI platform (multi-model agents + multimodal RAG)

Expected question

"Design an AI platform for automotive supply chain: exceptions, scanned ASNs and invoices, packing photos, and a human who can ask 'why did we hold this shipment?' without a hallucination."

Variant forms

• "Walk the multi-agent graph — who does what, which model, where does RAG sit?"
• "Draw the router. What does it see, what does it mutate, what does it never touch?"
• "How does context flow from an inbound 856 to a planner clicking 'why' three days later?"
• "How do you ingest ASN PDFs and packing photos without leaking the next supplier's packet?"
• "Multi-model routing: why isn't one frontier model the whole case?"
• "What stays a system of record vs what goes in the vector index?"
• "Scale this: 20k suppliers, partner retries, HITL queue, plant cutover weekend."

Where this actually gets asked

This is my unfair question. Employer supply-chain / commerce is the scar (P). Public orchestration, access-aware RAG, and gateway/HITL are inspectable patterns (O) — one cite if they want a click, not a catalog tour. Workday asked a hybrid assist + provenance problem firsthand (Aug 2026) — same muscle, different domain.

Executive summary

30-second thesis

I'd put a router over specialists, not one agent on the whole case. Context is a typed object that flows; models don't own it. Scans go through multimodal RAG. The ledger and the hold rule stay structured. Writes go through a gateway. "Why is this held?" is an evidence pack, not a paragraph.

2-minute answer

I'd draw four planes before I name a model.

1. Orchestration — router + specialists + critic. LangGraph-shaped, durable case state, interrupt before writes.
2. Knowledge — multimodal RAG over paperwork only: scanned invoices, ASN PDFs, packing photos. ACL before rank. Page-aware chunks. Decline if the pack is thin.
3. Record — case, EDI ids, exception type, hold rule, partner ack. Not in the vector index.
4. Control — identity, policy, HITL, signed audit, idempotency. The graph proposes; the gateway decides.

Router. Sees CaseContext (who, which exception, what's already in the pack, budgets left). Routes to ingest, retrieve, draft, criticize, or stop. It does not call send_edi. It does not embed PDFs.

Context flow. Inbound exception opens a case. A scan lands and mutates scan_ids + chunks. Retriever mutates evidence_pack. Drafter mutates disposition_draft using only the pack. Critic stamps critic_verdict. Human + gateway mutate committed_write. "Why" later replays the pack, it does not re-invent.

Models. Cheap/fast for route. Vision/doc for scans. Stronger text for grounded draft. Critic is rules first.

What I'd refuse: one completer, one frontier model on every node, and "just embed the PDF." I reversed the autonomy bet once.

How I'd spend the 45

If they pull you into models first, draw the router anyway. The model table is a consequence.

Four planes (say this while you draw)

                         ┌─────────────────────────────────────────────────────────┐
                         │ IDENTITY / CONTROL                                      │
                         │  IdP → Principal (plant / buyer / supplier-scope)       │
                         │  Gateway: policy · HITL · signed audit · FinOps · kill  │
                         │  Graph may only PROPOSE: send_edi · release_hold · pay  │
                         └──────────────────────────┬──────────────────────────────┘
                                                    │ allow / HITL / deny
                                                    │ + short-lived execution token
┌───────────────────────────────────────────────────▼───────────────────────────────────────────────────┐
│ ORCHESTRATION                                                                                         │
│                                                                                                       │
│   inbound signals ──► ROUTER (small/fast + decision TABLE — table wins ties)                          │
│                         │                                                                             │
│                         ▼                                                                             │
│                    CaseContext bus   (partial updates only; checkpointer; hop budgets)                │
│                         │                                                                             │
│          ┌──────────────┼──────────────┬──────────────┐                                               │
│          ▼              ▼              ▼              ▼                                               │
│       Ingest         Retriever       Drafter        Critic                                            │
│       vision/doc     ACL→hybrid      stronger       rules first                                       │
│                      →rerank         text                                                             │
│                                                                                                       │
│   interrupt_before write node · resume = same CaseContext, not a new chat                             │
└────┬────────────────────────────┬─────────────────────────────┬──────────────────────┬────────────────┘
     │                            │                             │                      │
     ▼                            ▼                             ▼                      ▼
┌─────────────┐          ┌─────────────────┐          ┌─────────────────┐     ┌─────────────────────┐
│ KNOWLEDGE   │          │ RECORD          │          │ WRITE/PARTNER   │     │ PLANNER UI          │
│ Scan index  │          │ Case · EDI ids  │          │ EDI VAN / TMS   │     │ same evidence_pack  │
│ page+bbox   │          │ hold rule · ack │          │ pay / ticket    │     │ "why" = replay      │
│ ACL→hybrid  │          │ NOT in ANN      │          │ idempotency key │     │ no second generate  │
│ cite|decline│          │                 │          │                 │     │                     │
└─────────────┘          └─────────────────┘          └─────────────────┘     └─────────────────────┘

Public shape to steal vocabulary from, not to claim as the employer binary: VAP-style router + specialists + critic (O orchestration), Enterprise RAG access-before-rank + page cites + decline (O knowledge), Aegis-style gateway/HITL/audit (O control). One cite if they ask. Sit down after that.

Core nouns on the board

CaseContext — the thing that flows

This is the contract. Specialists return partial updates. The router is the only thing that decides the next hop. Models do not get a bag of raw tool credentials.

CaseContext
  principal          verified JWT → plant / buyer / supplier-scope   # never from OCR
  case_id
  exception_type     mismatch | late_asn | invoice_fail | hold | unknown
  edi                { txn_id, set, partner, ack_status }            # RECORD
  rule_hits          [{ rule_id, fired_at, severity }]               # RECORD
  scan_ids           [scan_id...]                                    # KNOWLEDGE handles
  evidence_pack      { chunks[], rule_hits[], citations[], thin: bool }
  disposition_draft  { action, why_sentences[], cite_uris[] } | null
  critic_verdict     pass | decline | rereceive | null
  write_proposal     { tool, args, idempotency_key } | null          # not executed
  committed_write    { audit_id, executed_at } | null                # gateway only
  budgets            { hops, tokens, vision_calls, deadline }
  trace_id

Who may write which field

If a node needs a field it isn't allowed to write, the design is already wrong. That's the same instinct as typed LangGraph state with partial updates — specialists don't own the bus.

Router — decisions, not a chat brain

The router is a small/fast model plus a deterministic table. The table wins ties. That's how I keep high-volume exceptions from becoming a 7B essay on every ASN.

What the router sees (and only this)

RouterView
  principal.scopes          # already verified
  exception_type            # or unknown
  edi.ack_status
  scan_ids dirty?           # bool
  evidence_pack.thin?       # bool | missing
  disposition_draft?        # present / absent
  critic_verdict            # pass | decline | rereceive | missing
  write_proposal?           # present / absent
  budgets remaining
  last_hop + hop_count      # anti-loop

It does not see raw PDF bytes, embedding scores, VAN credentials, or the other supplier's chunks.

Decision table (table wins)

Router internals (draw this second)

Rendering architecture diagram…

What the router never does: pick embeddings, talk to the VAN, trust a supplier id the VLM read off a packing slip, or hold a write execution token.

Context flows (five paths, one bus)

Flow A — inbound exception (system of record → case)

EDI / TMS / plant hold
        │
        ▼
  ingest adapter (deterministic)
        │  edi.txn_id, partner, set, ack
        ▼
  Router.open_case ──► RECORD: Case row
        │
        ├─ scans already on the ASN? ──► Flow B (ingest) then C
        └─ no scan yet ──► park case, wait for paperwork

No LLM required to open a case. If your "AI platform" starts with a prompt here, you've already lost the ledger.

Flow B — scan ingest (multimodal write path)

ASN PDF / invoice scan / packing photo
        │
        ▼
  malware + checksum + mime
        │
        ▼
  page classify (vision/doc model)
        │  invoice | asn | packing_photo | unknown
        ▼
  extract ── OCR / VLM ──► fields (qty, part, damage) as HYPOTHESES
        │
        ▼
  chunk by page (and photo region)
        │
        ▼
  embed text | image
        │
        ▼
  INDEX  (ACL from Principal / case.supplier_scope — NOT from OCR)
        │
        ▼
  Router  ←  scan_ids dirty  →  Flow C

Extracted fields are hypotheses attached to the pack, not ACL, not the case's supplier id. The next supplier's packet must be unsearchable even if the ANN neighborhood is hot.

Flow C — retrieve → draft → critic (assist path)

Router
  │
  ▼
Retriever
  1. filter ACL (principal ∩ case scope)
  2. hybrid BM25 + dense (+ image kNN if the query is a photo)
  3. rerank bounded set
  4. attach RECORD facts: rule_hits, edi.ack, case.state
  5. stamp evidence_pack.thin
  │
  ▼
Drafter (stronger text)
  may write why_sentences ONLY with cite_uris from this pack
  │
  ▼
Critic (rules first)
  every sentence has a cite? schema ok? ACL stamp intact?
  pass | decline | rereceive
  │
  ├─ pass + "why" only ──► return pack to planner UI
  └─ pass + write needed ──► Flow D

Flow D — write (graph proposes, gateway commits)

Router.propose_write
  tool + args + idempotency_key = hash(case_id, tool, edi.txn_id, action)
        │
        ▼
Gateway
  identity ── policy ── risk tier
        │
        ├─ deny  ──► stop, signed audit
        ├─ HITL  ──► interrupt; human sees THE SAME pack the critic saw
        └─ allow (low risk only, if you ever have one)
                │
                ▼
         execute once ── partner / TMS / pay
                │
                ▼
         RECORD.committed_write + provenance snapshot

HITL UI shows the pack, not a chat transcript. If the approver can't see the page box, the queue is theater. Same instinct as notify/publish sitting behind a gateway — the graph does not get a second door.

Flow E — "why did we hold this?" three days later

Do not re-run the drafter as the source of truth. Load provenance_record + evidence_pack from the commit (or the hold). The LLM, if used at all, verbalizes that snapshot. Workday's grill is this flow: interactive provenance, not a second opinionated generate.

Context token — sequence (walk this with a finger)

One inbound 856, one ASN PDF, one hold, one "why" on Thursday. Each arrow mutates named fields. If you can't point at the field, you're waving.

Rendering architecture diagram…

Partner retry of the same EDI hits the idempotency key and does not start a new graph. That's a Record lookup, not a second draft.

End-to-end architecture (router in the middle)

Draw this top to bottom. Then put your finger on CaseContext and refuse to leave it.

Rendering architecture diagram…

Dotted lines to the model plane are role bindings, not a second control path. Apps pick a role; the tool gateway is a different door. I will not put a frontier model on classify-and-send.

Durable graph (LangGraph-shaped)

Long deterministic waits — "partner hasn't sent the ASN photo yet," "HITL overnight" — are a workflow. Bounded ambiguity — "does this packing photo support the qty mismatch?" — is an agent hop.

Rendering architecture diagram…

I'd persist the graph with a checkpointer (interrupt_before the write node). Resume is approve + the same CaseContext, not a new chat. Graph version on in-flight cases: old definition finishes; new cases get the new graph. If I can't replay "who saw which page when they clicked" in five minutes, the design isn't done.

Eval gates on the assist path, not a slide: citation span hit, ACL leak test (supplier B query must return zero of A's chunks), schema on disposition_draft, decline rate on thin packs. Fail the build on regression. Public eval-registry shape is O if they want the pattern; employer volume is P and I won't invent it.

Multi-model routing (which weights, on purpose)

Same instinct as a model plane: the router picks a role, not "the smartest model I have."

I will not put a frontier model on classify-and-send. Volume is exceptions. If they ask whether I have receipts that "small/fast is a real class," that's a model-plane cite (O) — not this whiteboard's job.

Deep dive 1: multimodal RAG pipeline

Corpus I will defend here: scans only — invoices, ASN PDFs, packing photos. Contracts and email are a second index if they exist; I will not invent them onto this board.

scan → mime/checksum → page classify → extract (OCR / VLM)
     → chunk by page (photo: region + caption hypothesis)
     → embed text | image
     → index { chunk_id, page, bbox, scan_id, case_acl, content_hash }
query → Principal ACL ∩ case scope → hybrid → rerank → pack → cite | decline

Rendering architecture diagram…

Failure modes I'd name first:

1. ACL after rank. Neighbor leak. Filter first. Same instinct as the public RAG lab (O, one cite).
2. Embed-the-PDF. 40-page ASN → one vector. "Why" needs a page and a box.
3. OCR as identity. Supplier id from the slip is a hypothesis, not the ACL.
4. Ledger in ANN. Shipment state and partner acks go stale in vectors. They stay on the case.

Deletion / hold-release invalidation is its own path: drop chunks by scan_id / case_acl before the next retrieve. "We'll reindex tonight" is a leak.

Deep dive 2: irreversible writes + dual-run

A bad 856 or a wrong payment instruction is not a retryable chat error. Gateway + policy + HITL on send_edi, release_hold, pay. Idempotency key is hash(case_id, tool, edi.txn_id, action) so partner retries don't double-send.

Authorize ≠ execute. The graph holds a proposal. The gateway mints a short-lived token after policy (and HITL). If those are the same call, you can't insert a human without rewriting every specialist.

Dual-run the incumbent path until exception rate and partner rejects move. Spoken $10M licensing / $7M net live on the EDI STAR — two numbers, not mashed into this whiteboard unless they ask for the business case.

Kill switch is gateway deny-all for a tool, not "redeploy the graph."

Deep dive 3: provenance that survives "why?"

Interactive "why" is: three scan cites (page + box), the rule id, the last human, the policy version the gateway used. Generate a narrative only from that snapshot. No citation, no sentence.

Same muscle as the wait-time + provenance grill: the LLM does not get to re-rank reality.

provenance_record
  case_id
  evidence_pack_hash
  citations[]          page + bbox + scan_id
  rule_hits[]
  critic_verdict
  policy_version
  approver_id | null
  write_proposal       if any
  committed_write      if any
  graph_version
  model_roles          { router, ingest, draft, critic }  # roles, not "the smartest"

Thursday's "why" loads this row. It does not call retrieve again unless the human is opening a new question on new paperwork.

Scale, dual-run, cutover

What I'd say when they hit the 20k-supplier variant:

• Open case and partner retry stay off the LLM. That's how volume stays a workflow problem.
• Ingest is the expensive hop (vision). Batch pages; skip VLM on text-layer PDFs; budget vision_calls on the case.
• Retrieve is per-case, ACL-scoped — not a fleet-wide ANN scan. Neighbor leak is a tenant bug, not a recall win.
• HITL queue is the thing that drowns. Risk-tier writes; dry-run the queue with real approvers on a quiet cohort before you flip execute tokens.
• Cutover weekend: dual-run incumbent vs propose-only. Shadow allow/deny/HITL rates. Flip execute on low-risk notify-class tools first, then release_hold, then send_edi / pay. Rollback is gateway deny-all, not graph revert theater.
• Plant isolation: Principal carries plant scope. A case from plant A must not retrieve plant B's packing photos even when the part number is the same.

p95 / index size / tok$ stay H unless I have a measured baseline. I will not invent employer QPS onto this board.

What breaks first (name these)

Pattern references (one cite each — then sit down)

Steal vocabulary. Do not tour the org. Do not claim public binaries ran inside that employer.

What I'd ask them

• Which write is irreversible — ASN, payment, shipment?
• Who owns the HITL queue after the architect leaves?
• Dual-run: what has to match before cutover?

Quantitative trade-offs

p95 / index size / tok$ stay H unless I have a measured baseline.

Org ownership

• Orchestration platform — router, CaseContext schema, checkpointer, hop budgets
• Knowledge platform — scan ingest, ACL predicates, invalidation, decline contract
• Domain / supply-chain ops — exception taxonomy, HITL rota, what "done" means
• Integration / EDI — VAN, idempotency, partner retry semantics
• Control / security — gateway, Principal derivation, audit retention
• Eval owners — leak tests and citation faithfulness can veto a release

If nobody owns the HITL queue after I leave, the embed failed.

What I'd refuse

• One agent that classifies, retrieves, drafts, and sends
• One frontier model for every node
• Vectorizing the EDI ledger
• Trusting supplier id extracted from the scan as the ACL
• Post-filter redaction after the model already saw the other supplier's page
• Autonomy-first so the headcount slide looks good before the failure mode shows up
• Regenerating "why" instead of replaying the pack
• Letting the router hold VAN credentials or an execution token

Evidence classes (say them)

Follow-up questions to expect

• "What do you skip in 45 minutes?" A catalog of specialists. Lock four planes, context bus, one HITL write.

Staff+/Principal signal rubric

• Senior: agents + RAG + a queue.
• Staff+: router, typed context, multimodal ingest, ACL before rank, HITL on writes.
• Principal: four planes, context bus with write permissions, dual-run + who owns the queue after you leave, provenance replay instead of regenerate. Who inherits: domain + platform. Fail-closed: HITL on irreversible EDI/SAP writes. Clock: lock planes + one write path; skip a catalog of 34 specialists.

Related

• ai-system-design/02 RAG at scale
• ai-system-design/03 agent / tool orchestration
• ai-system-design/06 multimodal retrieve
• ai-system-design/13 durable long-running agents
• ai-system-design/22 PDF cites + grounding
• ai-system-design/23 hybrid retrieve + ACL
• ai-system-design/27 wait-time + provenance
• ai-system-design/28 LLM gateway platform
• model-plane/01 buy vs RAG vs PEFT
• behavioral/01 10→2
• behavioral/04 payments and EDI
• behavioral/07 failed autonomy bet
• fde-deployment/03 wedge