Playbook / AI system design / Design an agent skills and harness platform

Design an agent skills and harness platform

Expected question

"Two hundred product teams want agents. How do you stop two hundred slightly different ReAct loops — and how do engineers and agents share capabilities without each team owning policy, evals, and side effects?"

Variant forms

Interviewers often ask the same design with different framing — recognize the archetype:

  • "Design the internal platform so a new agent capability is a package, not a new microservice."
  • "SKILL.md / agent skills vs MCP tools vs the orchestration graph — who owns what?"
  • "How do you version prompts, tools, and runbooks so a bad capability can be pinned or yanked?"
  • "Cursor/Codex skills for humans, runtime skills for agents — same artifact or two planes?"
  • "Design a harness: budgets, traces, eval gates, loop termination — outside the model's head."
  • "Our LangGraph templates forked. Every team has a different 'refund allowed?' How do you recover?"
  • "What's the difference between a skill, a tool, and a workflow?"
  • "How do you prevent a skill from git pushing production because someone pasted a shell recipe?"

Where this actually gets asked

This is the 2026 Principal version of "design an agent platform" after MCP and IDE-agent skills went mainstream. Anthropic's public skills format and MCP spec are R. It is not a sourced "Company X always asks this phrase." Treat it as the missing layer on top of 03 (orchestration ≠ policy) and 24 (tools aren't trust): how capabilities get into the building so you do not operate 200 graphs.

Open proof for the package shape: vpeetla-ai-skills (O) — installable SKILL.md for engineers. That is not a claim that Lucid or any employer runs this as production AgentOps.

Executive summary

30-second thesis

I'd make capabilities a versioned platform product — skills for how to work, MCP for how to call, a gateway for what may happen in the world, a harness for budgets/traces/eval. The graph is a consumer. The wiki of prompts is not a platform.

2-minute answer

Four layers, four owners. Skills are versioned packages (SKILL.md + scripts + eval fixtures): how an engineer or an agent should do a job. MCP standardizes tool calls; it does not authorize side effects. Gateway / HITL is the only path that can refund, push, or notify. Harness is the outer loop: token/$ ceilings, trace run_id, golden-eval gates, loop termination — the model does not get to decide when to stop spending.

What I'd refuse: a shared LangGraph template that each team forks, then copies policy into the prompt. That's how you get seven "refund allowed?" answers and no fleet kill switch. Also refuse treating IDE skills as the production runtime — Cursor loading a skill is O/R developer UX, not AgentOps.

Promote a skill like you promote a library: registry digest, eval slice that can fail the build, pin, rollback. A skill that shells out to git or prod APIs still goes through the gateway. If it can't, it isn't a skill — it's unsanctioned automation.

What I'd ask them: How many teams already have a graph? Which writes are irreversible today? Is the "skill" for humans in the IDE, agents in prod, or both?

Quantitative trade-offs

DecisionTrade-off and reversal evidenceEvidence class
Central skill registry vs per-team reposRegistry gives pin/yank and eval gates; reverse when a BU needs an air-gapped pack and will fund duplicate gates.H
Same artifact for IDE + runtimeOne pack reduces drift; reverse when runtime needs signed digest + gateway that an IDE skill must not bypass.O,H
Harness-in-platform vs in-graphPlatform harness gives fleet budgets; reverse for a research sandbox with no prod tools.H
Fat skill (scripts) vs thin skill (prompt only)Scripts reproduce; prompts drift. Reverse fat skills that hide HTTP around the gateway.H

Numbers in delivery are H unless you can defend a measured baseline. Skills-repo install counts are O, not employer adoption.

Migration and rollout

You're migrating how work is packaged, not swapping a model.

  1. Inventory graphs and prompt wikis. Name which already call irreversible tools.
  2. Stand a registry: skill id + digest + owner + eval fixture. New capabilities only through the registry.
  3. Point the harness at existing graphs in shadow: same traces/budgets, no new tools yet.
  4. Move one high-risk tool path so the skill proposes, gateway decides. Kill switch = yank the skill id, not "tell 40 teams to pull."
  5. Rollback = previous digest pin. IDE packs can lag runtime; never the other way if runtime is stricter.

Org ownership and operating model

  • AI platform owns the registry, harness (budgets, traces, eval gate), and yank.
  • Gateway / security owns side effects — skills do not grow a back door.
  • Skill owners (domain teams) own content, fixtures, and on-call for that capability.
  • Product owns which jobs are skills vs workflows vs "don't agent this" (craft/13).
  • Eval owns the slice that can fail a promote. A README is not a gate.

Requirements

Functional

  • Publish, pin, yank a skill by digest. Agents and (optionally) IDEs resolve the same id.
  • Bind tools only through MCP/registry; irreversible calls still hit the gateway.
  • Harness: budget, timeout, max steps, trace id, eval gate on promote.

Non-functional

  • Yank in seconds, not a deploy train.
  • Fail-closed: unknown skill id or failed digest verify → no tools.
  • Honesty: IDE skill load ≠ production serve.

Core entities

  • Skill pack — id, digest, owner, SKILL.md, optional scripts, eval fixtures
  • Capability grant — which principals/tenants may load which skill
  • Harness policy — $ / tokens / steps / wall-clock; kill on breach
  • Run — run_id, skill digest, graph version, trace
  • Promote record — eval slice, digest, who signed

API / interface

POST /v1/skills
{"id":"aegis-gateway","owner":"ai-platform"} → 201 {"digest":"sha256:..."}

GET /v1/skills/aegis-gateway?pin=sha256:...
→ 200 {"digest":"...","skill_md":"..."} | 404 unknown | 403 principal_not_granted

POST /v1/runs
{"skill_id":"aegis-gateway","digest":"sha256:...","principal":{"tenant_id":"acme"}}
→ 202 {"run_id":"run_..."} | 402 budget_exhausted | 403 skill_not_granted

POST /v1/skills/aegis-gateway/yank
{"reason":"exfil_recipe"} → 204

Staff+ callout: yank is a control-plane call, not a Slack message.

Data Flow

Author pack → eval fixture CI → registry digest → grant to tenant/app → harness starts run (pin digest) → graph may call MCP → gateway on side effects → traces + spend on run_id → promote or yank.

High-level design

Rendering architecture diagram…

Deep dive 1: a skill is not a tool and not a graph

Hard constraint: if you mash them, you cannot yank one without redeploying the product.

  • Skill = how to do a job (procedure, constraints, eval).
  • Tool = a typed call. MCP makes it discoverable; the gateway makes it allowed.
  • Graph = this product's control flow.

A refund skill that embeds curl $ERP is a bypass. The skill should say when to request a refund; the tool + gateway do it.

Deep dive 2: harness vs "the agent will be careful"

Hard constraint: loop termination and $ caps live outside the model.

Harness owns max steps, token/$ ceiling, and "this eval failed — do not promote." The graph may be elegant; without a harness, a stuck tool loop is a blank check. Alerts are not enforcement — same rule as cloud FinOps.

Deep dive 3: IDE skills vs production runtime

Hard constraint: developer UX can be looser; production cannot.

I'd allow the same markdown content in Cursor for humans (O). Production loads only signed digests, with grants and gateway. If a skill tells an IDE agent to push main, that's a documentation smell — the runtime must still refuse. Don't market one install script as AgentOps.

Staff+/Principal signal rubric

  • Mid-level: "We'll share a LangGraph template."
  • Senior: MCP + some prompt repo; policy still in the graph.
  • Staff+: registry + pin; gateway on writes.
  • Principal: skills/harness as the org default other teams inherit; yank-in-seconds; IDE ≠ runtime; eval-gated promote. Who inherits: every team that would otherwise fork a graph. Fail-closed: unknown digest, no tools. Clock: lock four layers + one yank story; skip a SKILL.md format lecture.

Follow-up questions to expect

  • "Isn't this just an internal npm?" Packages without gateway + eval + yank are libraries. Skills that spend money are control plane.
  • "Who writes the skill?" Domain team. Who can yank? Platform + security dual control on high-risk ids.
  • "A2A / Agent Cards?" Discovery later. Don't boil a protocol if 40 teams still paste prompts. 24 if they steer.
  • "What do you skip in 45 minutes?" Reciting YAML frontmatter. Cover registry, harness, gateway, one yank.

What I'd ask them

  • How many teams already have a graph, and which writes are live?
  • Same pack for IDE and runtime, or runtime-stricter?
  • Who can yank a skill at 2am without a meeting?
← Design a human-in-the-loop operating model at org scale