Failure Recovery and Retry Contracts in Large-Scale Scheduled Agent Fleets

Persistent and scheduled autonomous agents at scale need four explicit recovery contracts, namely retry, validation, persistence, and escalation. Each must be specified, tested, and owned separately, because waiting, correcting output, restoring state, and handing control to a human solve different failures. Reliable infrastructure enforces those distinctions rather than treating every failure as another reason to retry.

Why retry logic alone cannot make a scheduled agent fleet reliable

An agent demos flawlessly, then pages someone on Monday morning. What changed? The demo showed that the agent could complete a task but left recovery when a response disappeared or a worker died untested. “Adding retries” treats those gaps as temporary interruptions. In a scheduled fleet, that assumption can turn a contained failure into a shared outage.

Fleet’s May 2026 postmortem shows the amplification mechanism. SentinelOne quarantined the orbit process on many hosts, but Fleet Desktop kept polling with tokens that had expired. Server errors made Desktop poll faster, and every request reached the database before its token was checked, which overwhelmed the instance. The only mitigation was blocking the endpoint at the load balancer. Recovery behavior had become the source of load.

The corrective work covered three communication paths, including Fleet Desktop, orbit’s own API, and osquery TLS. Each needed its own backoff approach, with osquery TLS limited to best-effort mitigation because osquery controlled its scheduling. One policy was already too coarse. Scheduled agents add another problem because nobody is necessarily present to notice when a misclassified failure drops work or corrupts state.

The four independent failure domains every agent fleet encounters

Transient transport failures include rate limits, timeouts, and 5xx errors that can clear even if the request stays the same. They need bounded backoff, a retry budget, and circuit-breaker state. Output validation failures occur when the model produces the wrong format, invents a field, or violates a schema. They need a validator and corrective feedback.

State loss occurs when a process dies, a pod restarts, or a workflow times out between steps. It needs durable checkpoints to keep work that is already done. Structural failures occur when an API contract changes, a tool disappears, or the task is impossible in its current context. They need escalation.

These distinctions must exist before production starts. If a log line cannot identify the failure domain, recovery code cannot reliably choose its response either. A planner that keeps sending the same broken handoff to different workers shows what this costs. It looks like parallelism, but it is fan-out failure, spending the fleet’s retry budget on a problem that no worker can fix.

The retry contract: specifying backoff, idempotency, and circuit-breaker state before the first tool runs

A retry contract declares transient exception classes, maximum attempts, the backoff curve, idempotency semantics, and circuit-breaker thresholds. Backoff alone is not enough. Jitter must vary per request rather than per host-session, so agents do not reconnect in lockstep. Each communication path also needs its own consecutive-failure count, because success on one path says nothing about another’s recovery.

Backoff protects the dependency, but idempotency protects business state. Microsoft Foundry’s recovery analysis illustrates the distinction with a procurement agent that creates PR-10482 and loses the response when the network drops. A bare retry creates a duplicate request. Did the operation fail, or only its response? A durable Action Ledger keeps the evidence needed to tell which of those outcomes occurred.

That protection requires three related but distinct values. The trusted orchestrator generates a stable operation identity from durable run and step state, never from the model. A canonical intent hash binds that identity to validated, normalized request fields, while a durable stored result lets a retry recover a lost response. Concurrent attempts also need serialization to prevent races. If equal hashes do not make replaying the stored outcome acceptable, the canonical form is missing a meaningful field.

Circuit breakers coordinate calls across the fleet. Their Closed, Open, and Half-Open states govern whether calls proceed, stop, or probe recovery, rather than letting every agent independently hammer a failing dependency. The contract must also name exclusions. Approval-gated actions forbid automatic retry after an ambiguous response, and deterministic validation errors stop transport retries immediately.

The validation contract: turning model output failures into corrective signals rather than bare retries

Waiting can help a recovering server, but it does not explain a malformed answer to a model. Validation therefore needs its own recovery contract. A useful specification test is blunt. “If your output validator cannot tell the model why it failed, you don't have self-correction.” Incrementing a counter without improving the prompt does not meet that test.

The contract says which fields are required, which values are allowed, what normalization rules apply, and how to reject unknown fields. Instead of “validation failed,” it can say “the 'priority' field must be one of [low, medium, high]; you returned 'urgent'”. A separate validation budget keeps the model from using up the transport budget when it makes errors again and again. When the same failures keep occurring past the re-prompt threshold, they trigger escalation.

Validation also defines what counts as repeated intent. A tool that silently ignores unknown fields or changes its coercion rules cannot reliably tell whether two requests mean the same thing. Tool version must therefore fall within the canonicalization scope used by idempotency. At fleet scale, schema versioning and staged rollout belong in this contract too, because a schema change can otherwise send every agent into corrective re-prompting at once.

The persistence contract: separating execution state from memory so both survive a restart

Persistence operates on the run. Without it, a failure during a run can restart the whole workflow, redo work that is already done, and trigger side effects that have already committed. Execution state needs checkpoints between steps, and agent memory needs durable storage to last from one session to the next.

The durable record includes workflow and task identifiers, task state, assigned agent, attempt count, and input and output artifact references. It also holds dependency state, compensation state, and the timestamp of the last heartbeat. These fields make recovery decisions possible. Across runs lasting hours or days, memory versioning, provenance metadata on retrieved knowledge, and an append-only execution record provide more than a snapshot can.

The phleet fleet coordinator specification makes the detection gap concrete. Its GitHub issue #476 distinguishes a running-only in-memory reconciler from an independent observer that polls closed workflow history and delivers evidence to a durable failure inbox. Running leases help detect workers that stop sending heartbeats. The observer catches failures that workflow-local final notifications can miss, including external termination and worker loss.

Delivery alone does not finish the job. The inbox contract includes exact failure and cause evidence, acknowledgment, and an owned disposition or unresolved escalation. Callbacks are still secondary evidence because the worker may die before it can send them. Persistence thus preserves not only progress, but also the proof needed for someone else to take responsibility.

The escalation contract: making human handoff a first-class, owned path rather than a last resort

Acknowledging a failure does not authorize a retry. Nor does it authorize approval, deployment, cancellation, financial activity, or credential use. The phleet specification spells out that boundary and leaves the existing human gates and grants as they are. Without it, merely acknowledging an alert can become an unintended approval path for an action the agent lacked authority to perform.

Escalation therefore needs a specification before the work starts. Structural failures always call for escalation, and there are set triggers for semantic failures above a confidence threshold and for the same validation failures occurring again and again. A named coordinator owns recovery decisions, distinct from the task author and task target. Durable delivery and acknowledgment prove the handoff occurred. The coordinator either owns a recovery action or leaves the case in an unresolved escalation queue.

Recovery may require compensation. An incorrectly opened support ticket can be closed with a linked audit note, and a bad content draft can be unpublished. A booking or refund can require a second controlled transaction. Those actions need durable compensation state that the control plane can execute, and actions that are irreversible and have no compensation must be named ahead of time.

Some risks require a gate before dispatch. Fleet-level admission control accounts for the cumulative irreversibility risk of agents acting in parallel before an unsafe attempt fails. When escalation does occur, the coordinator still needs persistent evidence of what committed to support human handoff.

How the four contracts depend on each other and why each one needs its own owner

The contracts are distinct, but they are not optional pieces. Escalation needs durable proof, persistence needs a way to distinguish completed side effects from incomplete ones, and idempotency needs intent with a stable, validated shape. Ambiguous output can corrupt the canonical hash. That weakens replay decisions and leaves a coordinator without trustworthy evidence for recovery.

Ownership must follow those dependencies across team boundaries. Without a named owner, a contract exists only through behavior and can change silently when either team changes its implementation. Semantic health checks expose those breaks. Can a planner claim and route a task, can a verifier validate a recent artifact, and can recovery logic reload and resume interrupted work? Those checks test both whether the processes are alive and whether the agreements still hold.

Orchestration topology changes where that responsibility sits. Centralized orchestration gives one coordinator control over distribution, results, and workflow state, making ownership clearer but creating a single point of failure. Decentralized orchestration increases fault tolerance through peer discovery and task negotiation, while making ownership and escalation audits harder. Neither settles the problem by itself. The trade-off requires an explicit architectural decision.

Testing each contract independently before combining them in a fleet

An end-to-end failure does not reveal which contract broke. Testing therefore starts with each specification on its own. For retry, injected rate limits, timeouts, and 5xx errors test backoff, per-request jitter, breaker transitions, and reset behavior on the affected path. Deterministic validation errors must stop transport retries, and a request with an operation identity used before must return the stored result without repeating side effects.

Validation tests need malformed outputs with specific repairable defects. Validation tests must check both whether the validator rejects the failed field and whether feedback explains it. Its budget must exhaust independently of the transport budget. Exhaustion must also reach an owned escalation path.

Persistence tests kill a worker mid-task and check that recovery resumes from the correct step without repeating committed work. External termination needs its own tests because it may skip workflow-local notifications. The tests must exercise lease expiry and the independent observer’s delivery of durable evidence.

Escalation tests trigger each specified failure class and check delivery, acknowledgment, and authority boundaries. Compensation paths must carry out their named actions, and irreversible actions that have no compensation must reach the human coordinator. Only then do fleet-level tests examine the dependencies between contracts. If those tests discover a contract defect first, the isolation coverage has a gap.

How do you run scheduled agents without rebuilding persistence and recovery?

Infrastructure built for this job puts the runtime in charge of these contracts instead of leaving the agreements spread through application code. That is the buying criterion. A queue or HTTP client does not, by itself, know an agent’s durable run and step identity. Independent cloud functions also provide no natural home for fleet-wide irreversibility checks or control-plane execution of compensation.

Cloudflare Agents provides built-in retry support for scheduled tasks, queued tasks, and general-purpose calls, with configurable exponential backoff and jitter. That is a concrete fit for the transport-recovery requirement. It does not, on that evidence alone, establish coverage of every contract. The distinction helps separate a useful capability from a complete answer to long-running agent recovery.

A purpose-built runtime represents execution state and memory as first-class concepts. It integrates triggers, planning, tools, memory, approvals, and logging, with identity, isolation, audit, and rollback controls in the execution environment. Durable failure delivery and acknowledgment are still separate from ordinary notifications. Compensation actions are represented where the control plane can execute them.

The operating-system analogy explains the role of this layer. Concurrent programs needed a shared way to enforce isolation, memory protection, durability, and signals instead of each application having its own conventions. Scheduled agent fleets need a place like that to enforce recovery contracts. A platform must preserve enough state, meaning, and authority to recover safely when an agent it wakes on schedule fails.