Design durable execution for long-running AI agents

Expected question

"Design durable execution for long-running AI agents. How do you checkpoint state, survive crashes, resume HITL interrupts, and avoid duplicate side effects?"

Variant forms

Interviewers often ask the same design with different framing — recognize the archetype:

• "Design agents that run for hours — research reports, multi-file refactors — with resume."
• "How do you implement interrupt_before human approval without losing partial progress?"
• "Design idempotent tool execution when an agent retries after a network timeout."
• "Our orchestrator OOM-killed mid-mission — architect externalized state and replay."
• "Design workflow engine vs hand-rolled state machine for agent missions."
• "How do you version agent graph definitions while in-flight runs use old code?"
• "Design saga/compensation when step 4 fails after steps 1–3 already emailed a customer."

Where this actually gets asked

No company-specific interview attribution was found for this exact topic — like the previous entry, this one wasn't produced from a dedicated research pass, so treat it as a well-reasoned architectural extension of confirmed real patterns rather than a sourced interview question. What's real and directly relevant: agentic products are visibly shifting from chat-turn-shaped interactions (seconds) toward long-horizon, task-shaped ones (minutes to days) — deep-research agents, autonomous coding agents, and multi-step workflow agents are real, shipped product categories at several of these six companies. The architecture problem this creates — a process that must survive its own restart mid-task without losing reasoning progress — is a genuine, current systems problem distinct from anything else in this repo's ai-system-design/ folder, which otherwise addresses synchronous or short-lived agent interactions.

Executive summary

30-second thesis

I'd put run state outside the worker first — checkpoint plus a side-effect log — because the model context dies with the process. Every consequential tool call needs an idempotency key before I touch prompt quality. HITL pauses release compute entirely; cancel is a first-class path, not an afterthought.

2-minute answer

I'd start by separating two kinds of "memory": what the model thinks, and what the system has already done. Conversation history is not a resume plan. After each meaningful node I'd persist a checkpoint, and before any tool that emails, writes, or charges, I'd check a durable side-effect record keyed by (tenant, run, tool, args_hash).

What I'd refuse: replaying the whole reasoning trace from scratch on crash. That re-fires side effects and, on long tasks, can even get a different plan if the world moved. The pattern I'd name is durable execution — Temporal-style: replay is cheap, side effects return cached results when the log already has them.

For humans: interrupt_before means dump state, free the worker, wait hours or days on a queue row, then resume from the checkpoint. I've seen this work at smaller scale with LangGraph + Redis checkpointers; the scar is the gap where the tool succeeds and the process dies before the log write — so I'd push idempotency into the downstream system too, not just our ledger.

Budgets and poison pills bound retries. Mid-flight events stay queryable so cancel isn't a blind kill. Graph versioning for in-flight runs is a real question I'd ask them early.

What I'd ask them: What's the longest allowed HITL pause before auto-cancel? Which tools are non-idempotent today? Do in-flight runs pin the old graph version on deploy?

Quantitative trade-offs

Numbers and thresholds in interview delivery should be labeled H unless the candidate can defend a measured baseline. Open repositories are O; researched public patterns are R. Do not upgrade O/R into employer P adoption.

Migration and rollout

You're moving from best-effort workers to durable runs. Side effects are the landmine.

1. Inventory non-idempotent tools (email, charge, write). Require idempotency keys before enabling resume-after-crash for those tools.
2. Dual-write checkpoints + side-effect log while still running the old fire-and-forget path for low-risk missions.
3. Chaos drill: kill the worker after tool success / before log write. If you double-send, you're not ready.
4. Enable HITL pause/resume on a small cohort; prove workers are released during multi-hour waits.
5. Pin graph version for in-flight runs on deploy. Rollback = stop new durable runs, drain with old graph. Kill switch = cancel + freeze side-effect tools.

Org ownership and operating model

• Agent runtime / platform owns checkpoint store, side-effect ledger, cancel, and graph versioning.
• Tool owners own idempotency at the downstream API — the ledger alone isn't enough.
• Ops / approvers own HITL pause SLAs and auto-cancel timeouts.
• Product owns which missions are allowed to run multi-hour vs must finish interactive.
• SRE owns worker crash drills and storage durability for checkpoints.

Requirements

Functional

• An agent working on a long-horizon task (e.g., a multi-hour research task, a multi-step coding task) needs to survive a process restart, a deployment, or a worker crash mid-task without losing its reasoning progress or repeating already-completed work.
• Support human-in-the-loop checkpoints that can pause an agent's execution for an indeterminate amount of time (hours to days, not seconds) waiting on human input, then resume exactly where it left off.
• Support cancellation and inspection of an in-progress long-running task — a human should be able to see what an agent has done so far and stop it, not just wait for a final result.

Non-functional

• Every external side effect (an API call, a file write, a message sent) an agent takes during a long-running task must not be repeated on resume — a naive "replay the whole reasoning trace from the start" approach would re-execute every tool call, including ones with real consequences.
• Resuming after a restart should not require the agent to re-derive its own state from scratch (e.g., by re-reading everything it's already processed) — that's both slow and, for sufficiently long tasks, potentially non-deterministic if the underlying data has changed.
• Checkpointing itself must be cheap and frequent enough that a crash never loses more than a small, bounded amount of progress.

Core entities

• Task: a long-running unit of agent work, with a current status (running, paused-for-human, completed, failed) and an execution history.
• Checkpoint: a durable snapshot of the agent's reasoning state (not just its conversation history, but its internal plan/progress state) at a specific point, sufficient to resume execution from exactly that point.
• Side-effect record: a log of every external action the agent has actually taken, keyed by an idempotency identifier, checked before re-attempting any action on resume.
• Human checkpoint: a specific pause point awaiting human input, which can remain open for an arbitrarily long, unbounded duration without holding any compute resource idle.

API / interface

Auth: user token for start/resume; workers use run-scoped credentials.

POST /v1/runs
{"graph_id":"content_pipeline","input":{...},"checkpoint_ns":"tenant_acme"}
→ 201 {"run_id":"run_...","status":"running"}

GET /v1/runs/{run_id}
→ {"status":"interrupted","interrupt":{"node":"publish","reason":"hitl_required"},"checkpoint_id":"ckpt_..."}

POST /v1/runs/{run_id}/resume
{"decision":"approve","payload":{...}} → 200 {"status":"running"}

POST /v1/runs/{run_id}/cancel → 200 {"status":"cancelled"}

GET /v1/runs/{run_id}/events?after=120
→ {"events":[{"ts":"...","node":"research","type":"completed"},{"ts":"...","type":"interrupt"}]}

GET /v1/runs/{run_id}/checkpoints/{checkpoint_id}
→ {"state_uri":"redis://...","created_at":"...","nodes_completed":["research","draft"]}

Staff+ callout: interrupt/resume/cancel + checkpoint fetch are the durability contract — not “retry the HTTP call”.

Data Flow

Run starts, checkpoints after nodes, interrupts for HITL, resumes from checkpoint — cancel is first-class.

Rendering architecture diagram…

High-level design

Maps to functional requirements from step 1 — the component architecture that makes the API and data flow real.

Rendering architecture diagram…

The core design principle: checkpointing happens after every meaningful step (not just at coarse task boundaries), and every side effect is idempotency-checked against a durable log before execution — so a crash-and-restart at any point resumes from the last checkpoint and never re-executes an already-completed side effect, regardless of how long ago the checkpoint was taken.

Deep dives below target non-functional requirements (latency, scale, failure, cost, security).

Deep dive 1: durable execution engines — the real, named pattern this maps to

Hard constraint: a multi-hour agent must survive process death without redoing irreversible side effects — that means an idempotent event log, not a bigger context window.

This is not a novel problem invented by agentic AI — it's the same problem durable-execution workflow engines (Temporal, AWS Step Functions, and similar systems) were built to solve for long-running business processes, applied to agent reasoning instead of business logic. The core mechanism these systems use, and the one a Staff+/Principal answer should name explicitly: the workflow's code re-executes from the beginning on every resume, but every side-effecting operation is wrapped so that its result (not just whether it ran) is durably recorded — on replay, if a wrapped operation's result already exists in the durable log, the engine returns the cached result instantly instead of re-running the operation. This means the "checkpoint" isn't a single blob of frozen state; it's the side-effect log itself, and the agent's reasoning code can be re-run cheaply and deterministically as long as every consequential action it takes is recorded and replayed-from-cache rather than re-executed.

Deep dive 2: human-in-the-loop pauses that don't hold resources hostage

Hard constraint: waiting on a human for hours can't pin a GPU worker or open DB transaction — externalize wait state and free capacity.

A pause waiting on human input might last minutes or might last days — holding a compute process, a database transaction, or a locked resource open for an indeterminate duration is a real operational hazard (resource exhaustion, and a crash during a long pause loses the pause state entirely if it isn't itself durably persisted). The correct design treats a human-input pause as a fully durable, resource-free state: the task's state is checkpointed and the compute process is released entirely, with the pause represented purely as a database/queue record awaiting an external event (a human's response) to trigger resumption — this org's own real ai-content-factory build applies exactly this principle at smaller scale: its interrupt_before=["hitl"] LangGraph pattern plus a Redis checkpointer specifically exists to resume long pipelines after an indeterminate human-approval wait, without holding a process open for that entire duration.

Deep dive 3: observability into an in-progress, not-yet-complete task

Hard constraint: ops needs "where is this mission stuck?" without waiting for completion — partial traces and step status are mandatory.

Unlike a short synchronous request, a long-running agent task needs real mid-flight observability — a human should be able to inspect what an agent has done so far, not just wait for a terminal result. This connects directly to ai-system-design/07's trace/eval distinction: a long-running task's trace needs to be queryable while the task is still executing, not only after completion, so a human deciding whether to cancel a task partway through has real evidence (what's been done, what side effects have already occurred) rather than a black box they can only kill blindly.

Deep dive 4: tenant blast radius on checkpoints

Hard constraint: one tenant's giant checkpoint store or stuck workflow can't exhaust shared workflow capacity for everyone.

Checkpoint and event stores are tenant-namespaced (and encrypted at rest). Workers must refuse cross-tenant checkpoint loads. Side-effect idempotency keys are scoped to (tenant_id, run_id, tool, args_hash). Compare to Temporal in one minute; do not invent a workflow DSL in 45 minutes.

Staff+/Principal signal rubric

• Mid-level: proposes storing conversation history in a database for resume, without addressing side-effect idempotency or the cost of re-running reasoning steps.
• Senior: identifies the need for periodic checkpointing and idempotency keys on side effects, at a coarse (e.g., per-task-stage) granularity.
• Staff+: designs the durable-execution pattern explicitly — checkpoint via a side-effect log checked before every consequential action, not a single frozen-state blob — and treats human-input pauses as fully resource-free durable states.
• Principal: additionally connects this to observability requirements for in-progress tasks (queryable mid-flight, not just post-completion) and can name the real trade-off between coarse checkpointing (simpler, bounded progress loss) and full durable-execution replay (near-zero loss, more implementation complexity) against a stated task-criticality bar.

Follow-up questions to expect

• "What happens if the agent's tool-call side effect succeeds, but the process crashes before recording that it succeeded?" — That's the real scar. I'd tighten record-then-execute or execute-then-verify-on-resume as much as I can, and for emails and money I'd insist the downstream API honors our idempotency key so a duplicate attempt is a no-op on their side.
• "How long should a human-input pause be allowed to last before the task is considered stale?" — I'd refuse "forever." Pick a policy: auto-cancel or escalate after N days so abandoned approvals don't leave zombie runs sitting open.

Related

• ai-system-design/03: Agent/tool-use orchestration platform — the synchronous-orchestration counterpart this entry extends to long-horizon tasks
• ai-system-design/07: LLM evaluation and observability platform — the trace/observability discipline this entry's mid-flight inspection requirement depends on
• ai-content-factory — the real, shipped interrupt_before + Redis-checkpointer pattern this entry's human-pause design generalizes from