Playbook / AI system design / Design a multi-agent collaboration evaluation scorecard

Design a multi-agent collaboration evaluation scorecard

Expected question

"How do you evaluate multi-agent systems beyond single-output golden tests? What is your scorecard when specialists can contradict each other, tools may not fire, and a green CI pass from 'trace exists' is theater?"

Variant forms

  • "Design evals for a multi-agent fleet — orchestrator plus specialists with tool use."
  • "How do you measure collaboration quality, not just final-answer correctness?"
  • "Your agent run has a long Langfuse trace and a final message — is that a pass? Why not?"
  • "How do hard gates differ from soft quality scores in multi-agent CI?"
  • "How do you keep multi-agent evals from being flaky across trials?"
  • "A specialist claims a tool result without a matching tool call — how does CI catch it?"
  • "How do production failures become golden collaboration cases without inventing a new SaaS?"

Where this actually gets asked

Multi-agent system-design rounds increasingly probe evaluation depth after the candidate sketches an orchestrator graph. Treat this as the Staff+/Principal follow-up to LLM evaluation & observability and agent/tool-use orchestration: panels want to hear that you refuse scalar "quality %" theater. Org grounding: golden-eval-registry collaboration_scorecard

Executive summary

30-second thesis

I'd score collaboration as a vector — CSS, TUE, hard gates, multi-trial — and refuse treating trace length or has_final as quality. Failures promote into golden fixtures; I wouldn't invent a 17th eval SaaS.

2-minute answer

Single-output goldens still matter for "did the mission ship a valid artifact." Multi-agent needs more: did specialists contradict? Did claimed tool results match emitted tool_calls? Did escalation bypass policy? I'd put those in a portable registry kind (collaboration_scorecard) and have the AgentOps runtime hand trajectories, not vibes.

Soft dimensions (CSS / TUE) inform dashboards; hard gates fail the build on contradiction or escalation bypass. Multi-trial CI catches flaky "lucky" passes. Production failures go to failures/ → promote script → versioned GER case — same discipline as ADR-014, deeper surface.

What I'd refuse: quality = 94 if has_final and has_trace. That's theater with a green badge.

What I'd ask them: Which collaboration failures have bitten production? Who owns hard-gate thresholds? How many trials before a mission class is CI-gated?

Quantitative trade-offs

DecisionTrade-off and reversal evidenceEvidence class
Soft vector vs hard gatesSoft scores enable trend/FinOps; hard gates protect safety. Reverse hard-only when every flake pages oncall without a waiver path.H
Multi-trial vs single-shot CIMulti-trial catches non-determinism; reverse if wall-clock CI budget exceeds value for that mission class.H
Registry scorer vs in-app heuristicsRegistry keeps contracts portable; reverse if consumer-specific semantics can't be expressed without bloating GER.O

Numbers and thresholds in interview delivery should be labeled H unless the candidate can defend a measured baseline. Open repositories are O; researched public patterns are R. Do not upgrade O/R into employer P adoption.

Migration and rollout

You're replacing theater green with falsifiable collaboration, not adding a dashboard.

  1. Instrument trajectories at emit time (tool_calls, specialist claims, escalation events) — scoring without instrumentation is fiction.
  2. Shadow-score existing mission CI against the scorecard; publish vector fields without failing merges.
  3. Flip hard gates first (contradiction, escalation bypass); soft CSS/TUE thresholds second with owners.
  4. Multi-trial gate one mission class; expand only when CI minutes and flake rate are owned.
  5. Wire failure→GER promote + weekly drift scrape. Rollback = disable hard gates with audited expiry — not "delete the scorecard."

Org ownership and operating model

  • AgentOps / platform owns trajectory schema and CI wiring.
  • Eval owners own suite versions, hard-gate definitions, and promote-from-failure review.
  • Mission / product owners own which mission classes are multi-trial gated.
  • Safety / policy can veto escalation-bypass gate changes.
  • Exec sponsor owns the rule that waivers expire — otherwise every hard gate becomes optional.

Requirements

Functional

  • Score a multi-agent run from a trajectory: agents, claims, tool calls, escalations, final artifact.
  • Emit a collaboration vector (at least CSS and TUE) plus pass/fail hard gates.
  • Support golden fixtures that assert vector floors and hard-gate outcomes.
  • Promote interesting failures into versioned golden cases.
  • Surface scorecard fields to ops (API / lab UI / spine health), not only CI logs.

Non-functional

  • Deterministic scoring given a trajectory — no live LLM judge required for the core gate.
  • Multi-trial CI must finish inside the repo's CI budget for gated mission classes.
  • Soft scores must not override hard gates.
  • Registry stays dependency-light; consumers own how they reach themselves.

Core entities

  • Trajectory: ordered events for one mission run (agent turns, claims, tool_calls, escalations).
  • Collaboration vector: CSS, TUE, and related soft dimensions with thresholds.
  • Hard gate: boolean fail conditions (e.g. contradiction, escalation bypass) that veto soft pass.
  • Scorecard result: vector + gates + trial aggregate for CI and ops.
  • Failure artifact: persisted failed run eligible for GER promotion.
  • Suite kind collaboration_scorecard: GER contract for fixtures and scoring.

API / interface

Auth: CI service account for gates; humans for promote/waive.

POST /v1/scorecard/evaluate
{"trajectory":{...},"suite_id":"multi_agent_collaboration_v1"}
→ 200 {"css":0.81,"tue":0.74,"hard_gates":{"contradiction":"pass","escalation_bypass":"pass"},
 "passed":true}

POST /v1/scorecard/multi-trial
{"mission_class":"research","n":5,"suite_id":"multi_agent_collaboration_v1"}
→ 200 {"pass_rate":0.8,"trials":[...],"gate":"pass"}

POST /v1/ops/failures/{run_id}/promote
{"suite_id":"multi_agent_collaboration_v1","reason":"contradiction_missed_in_ci"}
→ 201 {"case_id":"case_...","suite_version":"1.1.0"}

GET /v1/ops/scorecard
→ {"latest":{...},"drift_alarms":[...]}

Staff+ callout: soft scores never override hard gates; promotes are versioned and disclosed.

Data Flow

Rendering architecture diagram…

High-level design

Rendering architecture diagram…

Deep dives below target the failure modes panels actually grill: theater metrics, tool fiction, and flake.

Deep dive 1: refuse trace-length quality

Hard constraint: presence of a final message and a trace proves instrumentation, not collaboration.

A heuristic like quality = 94 if has_final and has_trace will pass contradiction, skipped tools, and policy bypass. Principal signal is naming that theater out loud and replacing it with a vector + hard gates before talking about dashboards.

Deep dive 2: TUE requires emit-time tool_calls

Hard constraint: you cannot score tool-use efficiency from prose claims alone.

Specialists that narrate tool results without emitting structured tool_calls invent success. Instrument at emit time; TUE compares claims to calls. Retrofitting from free-text traces is a judge model with its own bias — fine as a shadow signal, not the merge gate.

Deep dive 3: hard gates vs soft CSS

Hard constraint: soft collaboration scores inform; hard gates veto.

CSS can trend down for a week while the product still ships safely. Contradiction or escalation bypass should fail CI immediately. Waivers need owner + expiry or every hard gate becomes a suggestion.

Deep dive 4: multi-trial and failure→GER

Hard constraint: one lucky pass is not a collaboration contract.

Multi-trial aggregates pass rate for a mission class. Failures that slip production (or shadow) promote into GER with a version bump — same honesty as ADR-014's first real RAG fixture bug. Weekly drift alarms catch scorecard rot before the panel narrative does.

Staff+/Principal signal rubric

  • Mid-level: "we'd log traces and spot-check."
  • Senior: versioned golden outputs for the final artifact.
  • Staff+: separates soft vector dimensions from hard gates; requires tool-call instrumentation.
  • Principal: multi-trial + failure→golden loop inside an existing registry — refuses a new eval product and refuses trace-length quality theater.

Follow-up questions to expect

  • "How is CSS defined?" (Answer: org-tuned composite over collaboration dimensions — label H; defend the dimensions, not fake precision.)
  • "What if the judge model disagrees with hard gates?" (Answer: hard gates win for merge; judge is shadow/debug.)
  • "Does every mission type need multi-trial?" (Answer: no — start with the class that burns trust; expand when CI minutes are owned.)

What I'd ask them

  • Which collaboration failure mode cost you the most in the last quarter?
  • Who can waive a hard gate, and do waivers expire?
  • What's your multi-trial budget per gated mission class?
← Design a real-time fraud and risk decisioning systemDesign a real-time wait-time prediction service with an agent and interactive provenance →