Design an LLM evaluation and observability platform

Expected question

"Design an LLM evaluation and observability platform. How do you run offline evals, monitor production quality, trace agent runs, and gate releases?"

Variant forms

Interviewers often ask the same design with different framing — recognize the archetype:

"Design how you'd know a prompt change regressed quality before shipping to 100% traffic."
"How do you evaluate RAG faithfulness and citation accuracy at scale?"
"Design tracing for multi-step agent runs — cost, latency, and tool outcomes per span."
"Our eval pass rate is 95% but users complain — architect online vs offline metrics that align."
"Design a golden-set CI that blocks deploys when regression exceeds a threshold."
"How do you attribute LLM cost to team, product, and tenant for FinOps?"
"Design human-in-the-loop eval labeling without becoming the bottleneck."

Where this actually gets asked

No company-specific attributed question was confirmed for this exact topic across the six companies in scope, but prep material and industry commentary increasingly describe "evaluation methodology" as displacing generic system design as the differentiating interview topic for AI-infra roles — the reasoning being that anyone can wire up a model API, but knowing whether your system is actually working, and catching regressions before they ship, is the harder and more senior-differentiating skill. Treat this as an emerging archetype worth preparing regardless of exact attribution.

Executive summary

30-second thesis

I'd make eval a control plane with teeth — offline gates that can block a ship, online monitors that page, and traces I can replay. Spreadsheets after the fact don't count.

2-minute answer

I'd register datasets, rubrics, and slices as versioned artifacts — if the golden set can drift silently, it's worse than no eval. Releases die on golden/safety/tool-contract regressions. In prod I'd watch quality, latency, cost, and drift with traces that include prompts and tool spans (async, not on the hot path). Product analytics and safety incidents are different queues. What I'd refuse: a 10k vanity suite nobody owns. Prefer a small suite you trust. What I'd refuse: shipping on vibe checks because the suite was "too strict."

Quantitative trade-offs

Decision	Trade-off and reversal evidence	Evidence class
Offline gate strictness vs ship speed	Strict gates prevent silent regressions; reverse temporary waivers only with expiry, owner, and compensating monitor.	H
Human labels vs LLM judges	Humans calibrate trust; LLM judges scale. Reverse full automation if disagreement with gold exceeds the calibration budget.	H
Broad coverage vs slice depth	Deep slices catch hidden failures; reverse endless slice proliferation when ownership and actionability collapse.	H

Numbers and thresholds in interview delivery should be labeled H unless the candidate can defend a measured baseline. Open repositories are O; researched public patterns are R. Do not upgrade O/R into employer P adoption.

ML fundamentals

ML fundamentals: experiment design, slice regressions, calibration of judges/scores, and drift monitors tied to release bundles.

Migration and rollout

You're standing up an eval control plane, not canarying a chat model.

Inventory today's "evals": spreadsheets, ad-hoc notebooks, CI jobs nobody owns. Pick one golden suite per critical product path and version it.
Wire the suite as a blocking gate on a shadow release train first — report only — until false-fail rate is tolerable.
Turn the gate hard: prompt/model/tool-contract changes can't ship if golden/safety slices fail. Waivers need owner + expiry.
Turn on online monitors + trace sampling after offline gates are trusted; don't page on noise from an unowned vanity suite.
Rollback is "unpin the candidate bundle / keep previous prompt+model alias." Kill switch is waive-with-expiry, not delete the suite.

Org ownership and operating model

Eval platform owns suite registry, judges, trace schemas, and the release API that other trains call.
Product / model owners own slice definitions and accept/reject thresholds for their domain — platform doesn't invent "good enough" for legal vs support.
Safety owns the harm/abuse slices and can hard-block a ship.
SRE owns monitor pages and trace pipeline SLOs (async — not on the hot path).
Exec sponsor owns the rule that waivers expire; otherwise every regression becomes a permanent exception.

Requirements

Functional

Every production LLM call (or agent run) should produce a trace: inputs, outputs, intermediate steps, and enough context to debug a bad output after the fact.
A versioned suite of "golden" test cases should be runnable against any candidate model or prompt change, producing a pass/fail regression signal, not just a vibe check.
Evaluation results need to be queryable across dimensions: by suite, by model version, by time, by which specific case regressed.

Non-functional

Trace collection must not meaningfully add latency to the production request path — async, fire-and-forget instrumentation, not a blocking call.
Eval suites need to be fast enough to run on every merge/deploy (a CI gate), not just a quarterly manual audit.
Golden test data needs to be versioned and immutable — an eval that silently changed its own pass bar is worse than no eval at all.

Core entities

Trace: a full record of one production request/run — spans for each step (retrieval, generation, tool call), inputs/outputs, and metadata (model version, latency, cost).
Eval suite: a versioned, locked set of test cases with expected outcomes.
Eval case: one input + an expect block (grounded: true, specific risk flags expected, a quality-score floor) — not just "the output should look reasonable."
Eval run: one execution of a suite against one target (a model version, a deployed service), producing pass/fail per case.

API / interface

Auth: CI service account for gates; humans for overrides.

POST /v1/eval-suites
{"name":"rag-grounding-v4","suite_uri":"s3://golden/rag_v4.jsonl","thresholds":{"grounding":0.92}}
→ 201 {"suite_id":"suite_...","version":4}

POST /v1/eval-runs
{"suite_id":"suite_...","target":{"service":"rag-answer","git_sha":"abc123"},"mode":"ci_gate"}
→ 202 {"run_id":"run_..."}

GET /v1/eval-runs/{run_id}
→ {"status":"failed","scores":{"grounding":0.88},"failed_cases":["case_17"],"trace_links":["https://..."]}

POST /v1/traces
{"trace_id":"tr_...","spans":[...],"eval_case_id":"case_17"} → 202 {"accepted":true}

POST /v1/gates/{run_id}/override
{"reason":"known flaky fixture","approver_id":"u_..."} → 200 {"gate":"waived","expires_at":"..."}

Staff+ callout: eval runs must deep-link to traces; overrides are audited and time-bounded.

Data Flow

CI triggers an eval run against golden fixtures; failures deep-link to production-like traces.

Rendering architecture diagram…

High-level design

Maps to functional requirements from step 1 — the component architecture that makes the API and data flow real.

Rendering architecture diagram…

The critical design split: tracing (what actually happened in production, for debugging) and evaluation (does the system still behave correctly against known cases, for regression prevention) are related but distinct systems. A common weak answer conflates them into one "logging" system that does neither job well — traces need to capture arbitrary production diversity; eval suites need to be small, curated, and stable enough to gate a merge on.

Deep dives below target non-functional requirements (latency, scale, failure, cost, security).

Deep dive 1: fixtures that validate vs. fixtures that actually gate

Hard constraint: an eval that never blocks a bad release is theater — gates need owners, thresholds, and a veto path.

The naive version of an eval system stores golden fixtures and checks that they're well-formed — schema validation, no duplicate IDs. That's necessary but insufficient: it proves the fixtures exist, not that the system passes them.

Real, disclosed finding: golden-eval-registry had exactly this gap — versioned suites with real validation, but nothing actually executed a suite against a live system and gated a build on the result (ADR-0001's own "future work" note said as much). The fix (ADR-0002/ADR-014): a real, dependency-light, provider-agnostic scorer — each consumer service reaches itself however it can (an HTTP call, a direct function import) and hands the real output back for comparison against the suite's expect block. Running the very first suite for real immediately found a genuine bug in the fixture itself — a case whose expected answer had never actually been checked against real system behavior, contradicted by what the real pipeline produced once actually executed. This is the single strongest argument for why "the fixtures exist and validate" and "the fixtures are correct and gate something" are different claims — only real execution proves the second one.

Eval maturity level	What it catches
Fixtures exist, schema-validated	Malformed test data
Fixtures gate a real CI build	Regressions in tested behavior — but only for kinds you've wired up
Fixtures run against live production traffic samples (shadow eval)	Regressions in behavior you didn't think to write a fixture for

Deep dive 2: trace-linked evals — connecting production behavior back to eval scores

Hard constraint: offline green + online complaints means your slices don't match production failure modes — link traces to cases or you're flying blind.

An eval suite that only ever runs in CI, disconnected from what's actually happening in production, drifts from reality — the traffic patterns and edge cases users actually hit are richer than any hand-written suite. Real systems export production traces with eval-relevant scores attached (grounded: bool, citation_count, human_approval_required) directly into the same observability backend (Langfuse, in this org's reference stack) used for latency/cost dashboards — so a quality regression shows up next to a latency regression, not in a separate tool nobody checks as often.

Deep dive 3: cost as an evaluation dimension, not a separate concern

Hard constraint: a quality win that 3× token cost can still be a failed release — cost belongs on the same scorecard.

A model change that improves quality but doubles cost per request is not an unambiguous win — it's a trade-off that needs to be visible in the same evaluation report, not discovered later in a finance review. This is the same "real numbers, not guessed" discipline as agent-finops: real per-call token costs aggregated alongside quality scores, so a candidate model change is evaluated on both axes together.

Deep dive 4: trace privacy and volume (interview-critical)

Hard constraint: full prompt/response logging at chat scale is a privacy and storage bomb — sample, redact, and retain with purpose.

Production traces are async + sampled under load — they must never block the user path. Redact PII at ingest; scope trace namespaces by tenant. Eval gates block promote/merge; traces are for debug. In 45 minutes, cover the trace/eval split + one real CI gate — do not design a full Langfuse clone.

Deep dive 5: multi-agent collaboration is a vector, not trace theater

Hard constraint: has_final and has_trace is not quality — score CSS / TUE, hard gates, and multi-trial.

Single-output goldens miss specialist contradiction, invented tool results, and escalation bypass. Treat collaboration as a portable suite kind (collaboration_scorecard) with live trajectories from the AgentOps runtime — full answer in 26 — multi-agent collaboration evaluation scorecard (ADR-031).

Staff+/Principal signal rubric

Mid-level: proposes logging + a manual QA review process.
Senior: proposes a versioned eval suite and some automated scoring.
Staff+: distinguishes fixture validation from real CI-gating execution explicitly, and designs the scorer to be provider-agnostic (dependency-light, not embedding a specific service's client code).
Principal: additionally connects production tracing and offline eval into one observability story, and treats cost as a first-class evaluation dimension alongside quality, not a separate report.

Follow-up questions to expect

"How do you keep golden fixtures from becoming stale as the underlying system evolves?" (Answer: version the suite explicitly — a fixture that needs correcting gets a disclosed, versioned update, not a silent edit, exactly as happened with the real bug found above.)
"How would you catch a regression that no existing fixture covers?" (Answer: shadow-eval a sample of real production traffic against quality heuristics or a judge model, and promote interesting failures into new golden fixtures over time.)
"What's the failure mode of over-relying on an LLM-as-judge for evaluation?" (Answer: judge models have their own biases and blind spots — treat judge-based scores as one signal, not a ground truth, and keep a core of human-verified fixtures the judge is itself periodically checked against.)

What I'd ask them

Who has veto on a failed golden suite — and can they waive with expiry?
Which slices have bitten you in production that offline missed?
Trace retention vs PII: what's the legal retention window?

golden-eval-registry — real scorer + CI gate, found a real fixture bug on first execution
ADR-014: golden-eval-registry becomes a real CI gate
26 — Multi-agent collaboration evaluation scorecard — CSS / TUE / hard gates / multi-trial (ADR-031)
agent-finops — real cost metering as an evaluation dimension