Production AI deep-dive — evals, cost, guardrails, take-home defense

Expected question

"You built a customer agent / RAG system. How do you evaluate it, control cost and latency, prevent hallucinations, and defend those choices in a take-home deep-dive?"

Variant forms

"How would you evaluate this agent? Golden dataset, regression suite, drift?"
"When do you fine-tune vs RAG vs prompt-engineer for this customer?"
"How do you keep RAG from hallucinating? What guardrails, how do you test them?"
"This agent will run 100k times/day. Token cost, latency budget, where do you cache?"
"Customer says the agent 'sometimes gives wrong answers.' Make that measurable and fixable."
"Diagnose high latency in an LLM inference pipeline end-to-end."
"Design an evaluation harness for an agent with a business KPI target (e.g. delivery rate)."
"Hosted API vs self-hosted / air-gapped open-weights for a regulated enterprise — trade-offs?"
"How do you version, A/B-test, and roll back prompts in production?"
"Walk me through your take-home: chunking, tools, evals, what you'd change at 100× scale."
"How do you design guardrails for a production LLM application?"
"What's your chunking strategy, and how would you justify it to a skeptical customer?"
"Estimate tokens/day for a Fortune-500 support org — then convert to dollars."
"Customer reports the model is 'getting worse.' Investigate drift."

Where this actually gets asked

OpenAI / Anthropic Applied AI take-home deep-dives and FDE production rounds. Offer-decider: can you turn “sometimes wrong” into measured slices + gates, not demo vibes.

The question, as it might actually be asked

"Don't tell me the demo looked good. Tell me how you'd know it's safe and economical in their production."

The framework

30-second thesis

I'd treat evals as a release gate, not a slide. Measure task success and grounding by slice, budget cost per successful task, fail closed on unauthorized or unsupported answers, and run versioned prompt/index bundles with a warm rollback. If I can't answer “who called which tool as which tenant” from logs, the design isn't done.

2-minute method

Measure → constrain → guard → operate. Spoken:

First I'd pin the customer KPI and build a stratified golden set — hard cases, rare intents, adversarial — versioned and never used for training. Offline gates cover task success, groundedness/citations, tool reliability, and safety. Online: sampled human review, reopen rates, slice drift before promote.

Cost and latency: decompose tokenize → retrieve → rerank → generate → tools. Budget tokens per successful task, not vanity totals. Cache embeddings/retrieval/safe responses; route cheap models for easy intents; hard quotas with graceful degrade.

Guardrails: ACL before ranking, allowlisted tools, HITL on irreversible side effects, decline or escalate when unsupported. Red-team in the same CI gate as quality.

Take-home defense in six beats: problem + non-goals, walking skeleton, tomorrow's eval, failure modes + rollback, what changes at 10×/100×, what's customer-specific vs productize. Same brand throughout — access-aware RAG, gateway HITL, eval CI (O).

1) Evaluation

Layer	What to define
Task success	Binary or graded rubric tied to the customer KPI
Grounding / citation	Retrieval hit quality; refuse when unsupported
Tool reliability	Success, timeout, wrong-tool rates
Safety	Injection, PII leak, unauthorized action attempts
Online	Sampled human review; thumbs; reopen/escalation
Drift	Slice monitors + shadow disagreement before broad promote

Golden set rules: stratified; versioned; regression gate in CI; never train on the eval set.

2) Fine-tune vs RAG vs prompt

Choose	When
Prompt / tools	Task is narrow; behavior changes weekly; citations matter
RAG	Knowledge is large, changing, or ACL-bound
Fine-tune	Stable style/schema/behavior; eval proves prompt+RAG insufficient
Never	Fine-tune to paper over missing access control or bad retrieval

3) Cost & latency at volume

Decompose: tokenize → retrieve → rerank → generate → tools → post-process. Budget tokens per successful task. Cache embeddings, retrieval, and safe semantic responses. Batch where async. Route cheap models for easy intents. Hard quotas with graceful degrade.

4) Guardrails

Input: injection filters, allowlisted tools, tenant ACLs before ranking. Action: HITL on irreversible side effects. Output: decline / escalate on low confidence; citation required for regulated answers. Test: red-team suites in the same CI gate as quality evals.

5) Take-home defense script

Customer problem in one sentence + non-goals.
Walking skeleton you shipped.
Eval you would run tomorrow (not “eyeball it”).
Failure modes + rollback.
What changes at 10× / 100× data or QPS.
What you’d leave as customer-specific vs productize.

Requirements

Functional

Stratified golden set tied to business KPI.
Decline path when unsupported; citation policy for regulated answers.
Prompt/index/model version pins with rollback.
HITL boundary for irreversible tools.

Non-functional

Cost per successful task and P95 latency budget.
CI regression gate blocking promote on quality/safety slices.
Online drift monitors by slice (tenant, intent, corpus version).

Core entities

Golden item: input, expected behavior, slice tags, severity.
Eval run: versioned artifact + pass/fail vs thresholds.
Bundle: prompt + retrieval config + model + tool schema pins.
Guardrail: input / action / output control with test coverage.
Unit economics: $/successful task, not vanity token totals.

Data flow — production quality loop

Rendering architecture diagram…

High-level design

Traffic → AuthZ / ACL filter → Retrieve (+cache) → Rerank (optional)
       → Generate (routed model) → Output guards → Tools via gateway/HITL
       → Telemetry (quality, cost, latency, safety) → Eval store
Promote only if offline + shadow + canary gates pass

Deep dive 1: “sometimes wrong” → measurable

I'd ask: which slice — intent, doc freshness, ACL edge, adversarial? Add the failing cases to the golden set. Fix retrieval/prompt/tool — not “raise temperature.” Re-run CI; only then widen canary.

Deep dive 2: 100k runs/day economics (H)

Intent-route FAQ to cheap model + cache; reserve expensive reasoning for the low-confidence band; cap tools per turn; budget tokens for the final answer; hard quota with degrade-to-retrieve-only. Numbers here are H until measured. Eval gate before removing HITL.

Deep dive 3: chunking defense for skeptics

I'd prefer structure-aware chunks (headings, sections) with doc_id / page / ACL lineage over naive token windows. Prove it with citation precision and groundedness slices — not chunk-size dogma. At 100× corpus: incremental index, access-before-rank, and invalidation matter more than chunk tokens alone (O Enterprise RAG instincts).

Quantitative trade-offs

Decision	Trade-off and reversal evidence	Evidence class
Strict decline vs always-answer	Decline hurts “helpfulness”; reverse only if harm of wrong answer is low and eval shows users recover	O/H
Cache aggressive vs freshness	Cache cuts cost/latency; reverse when corpus churn breaks citation trust on measured slices	H
Remove HITL to save cost	Saves reviewer hours; reverse until tool-success + reject-rate gates hold for agreed window	O/P

Migration and rollout

Register thresholds before canary (quality, safety, cost, latency).
Shadow prior bundle; promote canary with holdback.
Keep previous bundle warm; atomic rollback on gate breach.
Expand autonomy only after HITL reject-rate stability.

Org ownership

Eval / quality can veto releases that fail gates.
FDE / account owns customer golden-set seeding early; hands off ownership.
Platform owns CI harness, metering, gateway.
Finance / FinOps partners on $/successful task reviews.

Situation

Take-homes and production deep-dives ask you to defend a customer agent/RAG as if it will run under real load and real harm. Inspectable method proof is the open stack (O): Enterprise RAG (access-before-ranking, decline), AegisAI (HITL/policy), VAP, golden eval CI — not a claim Lucid runs those repos. Lucid (P) supplies muscle for measurable exception/HITL rates when you have baselines.

Task

Show you can operate production AI: evaluate, control cost/latency, prevent hallucinations, and defend take-home choices at 10×/100× — not that the demo looked good.

Action

Tie eval layers to the customer KPI and stratified golden set.
Decompose latency/cost; budget per successful task; cache and route deliberately.
Enforce ACL-before-rank, HITL on irreversible tools, decline on low support.
Version bundles; shadow → canary → rollback.
In take-home defense: problem, skeleton, tomorrow’s eval, failure modes, scale deltas, productize vs custom.
Investigate “getting worse” by slice and recent deploys before retraining.

Result

A deep-dive that sounds Staff+/Principal: measured slices, unit economics, fail-closed guards, operable rollback. Residual practice: timed take-home narration with a real artifact.

The follow-up question you should expect

"What would you change at 100× QPS or documents?"
Not “bigger model.” Caching/routing, incremental index + invalidation, stronger access predicates, async tools, tighter online sampling. Self-host only if residency/ops demand it — not for vanity.

What I'd ask them

What's the business KPI this agent is judged on — not “answer quality”?
What's the irreversible tool, and who's on the HITL queue today?
What's the cost ceiling per successful task you're willing to defend to FinOps?
Which slice fails most often in the last two weeks of shadow?

Candidate-owned evidence prompts

Which O repo/demo will you open for access-before-rank or HITL?
What $/successful-task sketch will you use (with H labels)?
What three golden-set slices will you name first?
What is your rollback sentence under 15 seconds?

Author reference (do not memorize)

Numbers without a dated baseline stay H. Do not upgrade O repos into employer P adoption.

Staff+/Principal signal rubric

Mid-level: Names RAG + vector DB; eval is “humans look at answers.”
Senior: Has a golden set and basic latency/cost awareness.
Staff+: Stratified evals, CI gates, cost per successful task, HITL boundary, take-home scale story.
Principal: Turns deep-dive findings into reusable product eval packs and account operating metrics.