Design an LLM-powered customer support assistant

Expected question

"Design an LLM-powered customer support assistant. How do you combine RAG, ticket systems, escalation to humans, and measurable resolution quality?"

Variant forms

Interviewers often ask the same design with different framing — recognize the archetype:

"Design a support bot that deflects 40% of tickets with grounded answers from help center + past tickets."
"How do you escalate to a human with full context when confidence is low?"
"Design integration with Zendesk/Salesforce — create, update, and close tickets via tools."
"Our bot gave wrong refund policy — architect policy-aware RAG and decline paths."
"Design multilingual support with one knowledge base and locale-specific tone."
"How do you measure containment rate vs customer satisfaction vs handle time?"
"Design voice + chat support with shared memory and different latency budgets."

Where this actually gets asked

High-frequency product GenAI design at Amazon/Microsoft/AI startups: "Design a customer support bot," "Design an AI helpdesk." Differs from ChatGPT clone by grounded knowledge, tool actions (refund, cancel order), and human escalation with clear ownership handoff.

Requirements

Functional

Answer FAQs from a knowledge base with citations.
Take allowed actions via tools (order lookup, cancel, refund) under policy.
Escalate to human agents with full conversation + reason codes.
Multi-turn memory of the case; optional multilingual.

Non-functional

High containment (resolve without human) without sacrificing CSAT.
Hallucination of policy/order state is unacceptable — ground or escalate.
Action tools are side effects: gateway + HITL above risk thresholds.
Audit trail for regulated industries (payments, healthcare).

Core entities

Case / Ticket: customer_id, channel, status, assignee (bot|human), priority.
Turn: message, intent, confidence, citations[], tool_calls[].
Knowledge article: versioned content, locale, effective_dates.
Tool policy: tool_name, risk_tier, approval_required, rate limits.
Escalation: reason_code, transcript_ref, SLA clock start.

API / interface

POST /v1/cases
{ "customer_id":"...", "channel":"chat", "subject":"..." }
→ 201 { "case_id":"k_..." }

POST /v1/cases/{id}/messages
{ "content":"Where is my order?", "stream":true }
→ SSE tokens + final { "citations":[...], "suggested_actions":[...] }

POST /v1/cases/{id}/actions
{ "tool":"cancel_order", "args":{"order_id":"..."} }
→ 200 { "status":"done" } | 202 { "status":"approval_required" } | 403

POST /v1/cases/{id}/escalate
{ "reason":"low_confidence", "queue":"billing" }
→ 200 { "assignee":"agent_...", "sla_due":"..." }

GET /v1/cases/{id}
→ full transcript + tool audit

Staff+ callout: never let the LLM invent order state — tools are the source of truth.

Data Flow

Message → intent route (FAQ vs agent vs escalate) → retrieve KB → generate grounded reply → optional tool via gateway → escalate if confidence/policy requires.

Rendering architecture diagram…

High-level design

Maps to functional requirements from step 1 — the component architecture that makes the API and data flow real.

Rendering architecture diagram…

Reuse RAG (02) and gateway (03) patterns; the product-specific hard parts are escalation quality and action policy.

Deep dives below target non-functional requirements (latency, scale, failure, cost, security).

Deep dive 1: containment vs CSAT

Optimizing only for "deflect from humans" creates hostile bots. Track resolution rate and CSAT / reopen rate. Low-confidence or high-emotion intents should escalate early. Tone adaptation matters (empathy vs efficiency) but is secondary to correct state.

Deep dive 2: tools and money movement

Refunds/cancels are irreversible. Classify tools read-only vs mutating; mutating requires gateway authorize + optional HITL above $ thresholds (03). Idempotency keys on every mutating tool call.

Deep dive 3: knowledge freshness

Policy articles need effective dating; stale KB is a hallucination source. Invalidate retrieval on publish; cite article version in the answer. Eval suite of golden tickets as a merge gate (07).

Deep dive 4: containment SLOs and human-queue saturation

Escalate when confidence is low (e.g., <0.7) or after repeated failed tool attempts — do not optimize only for deflection. Target containment in a realistic band (often ~40–60%) with a CSAT/reopen guardrail, not 90% at any cost. Under load, stop aggressive deflection and route to humans with an explicit queue_depth_high reason; name a pickup SLA (e.g., minutes for chat). Reuse RAG (02) and gateway (03); do not rebuild them.

Deep dive 5: false-containment economics and exactly-once actions (Staff+)

False containment (bot "resolves," customer returns angrier) is more expensive than an early escalate. Track reopen-within-72h and downstream contact rate as first-class guards next to containment %. Example: if human handle cost is $8 and a false containment creates 1.5 extra contacts, a bot that boosts containment +10pp while raising reopen +4pp can lose money — show the arithmetic. For refunds/cancels: idempotency keys + reconciliation jobs against the system of record (order state wins over transcript claims). Policy articles need effective dating; the bot must cite the version active at decision time, not "latest draft." Queue math: if chat arrival is λ and human capacity is μ, when λ_escalate > μ, shed bot ambition (fewer tool retries) before SLA breach; publish expected wait to the customer.

What's expected at each level

Mid-level: FAQ chatbot over docs.
Senior: retrieval + escalation path.
Staff+: tool governance, confidence routing, citation grounding, case state machine, containment vs CSAT/reopen economics, idempotent mutating tools.
Principal: queue capacity/SLA math under surge, regulated audit trails, policy versioning, and explicit false-containment cost models.

Follow-up questions to expect

"How do you prevent the bot from promising a refund the policy forbids?" (Tool+policy engine, not prompt-only.)
"What does the human see on escalate?" (Transcript, intent, failed attempts, recommended macros.)
"How do you know containment is healthy?" (Reopen rate + CSAT + cost per resolution, not deflection alone.)