Interrupt and Preemption Semantics for Agentic Workloads

Semantic differences between interrupt and preemption versus ordinary error handling

A LangGraph agent kicks off a 30-minute fine-tuning job with no checkpointer attached. At minute 29, a network blip severs the connection. There's no replay, no saved state, nothing to resume: the job restarts from zero, and 29 minutes of GPU time evaporate. That's the default behavior of naive agent loops running in production today, and it exposes a design gap most agent frameworks still haven't closed. Interrupt and preemption semantics need to be built for autonomous, long-horizon work, not borrowed wholesale from request-response software, and the frameworks that keep borrowing are the ones losing GPU budget to silence.

The distinction matters because agents don't fail the way ordinary processes fail. A crashed web server drops a request, and the client retries at no real cost. An agent crashed mid-task has usually already called external tools, written rows to a database, spent GPU budget, or advanced a reinforcement learning rollout, and none of that undoes itself when the process restarts. Long-horizon agent tasks accumulate state, and much of that state is irreversible. Treating a restart as "starting fresh" isn't a neutral default, it's an active decision to throw away work, and the price tag isn't abstract: a multi-agent research run can burn somewhere between $5 and $20 in tokens per task, money spent on work a properly designed system could have resumed instead of discarded.

Traditional error handling, the try/catch block, the retry policy, the circuit breaker, all assume the failed operation can be safely re-run from its beginning. That assumption holds for a stateless API call and collapses the moment an operation has already changed the world outside the process, which is what agent actions tend to do.

An interrupt is a structured, asynchronous signal delivered into a running agent, something the agent has to catch, interpret, and act on deliberately, rather than something thrown at it by malformed input or a system fault. A preemption is a related but distinct idea: an externally imposed suspension of execution, usually driven by resource contention rather than any fault in the agent itself. Spot instance eviction, GPU scheduler reassignment, a shift in training job priority: these are preemptions. The agent didn't do anything wrong, and its intent, along with often its partial state, remains entirely valid.

This is where the pause/cancel boundary carries real weight. A pause is resumable and state-preserving: the agent is expected to continue later, so anything it's holding, a lock, a connection, a reserved GPU slice, may need to stay reserved. A cancel is terminal: the agent won't continue, and those same held resources need releasing immediately. Conflate the two and the failure modes multiply in both directions, resources leak because a cancel got treated like a pause, or a legitimately resumable job gets torn down because a pause got treated like a cancel. Temporal's documentation on Activity operations calls this out directly: Activity code may need to handle Pause and Cancellation differently, releasing held resources on Pause while preserving them on Cancellation, or the reverse, depending on what the activity actually does. That's a semantic requirement baked into how the system expects developers to write activity code.

Heartbeat-based preemption detection: how Temporal makes silence into a structured signal

Most systems wait for failure to announce itself. Temporal inverts that logic. Instead of waiting for a crashed worker to report its own death, which it obviously cannot do, the system treats the absence of a signal as the signal itself, and that inversion is the more defensible design.

The mechanism is the activity heartbeat. A running activity is expected to call heartbeat at regular intervals while it works. If enough intervals are missed, Temporal records a timeout event directly into workflow history, turning silence into a structured, recoverable failure instead of an indefinite hang. Activities that heartbeat get interrupted cleanly on their next heartbeat call: the SDK raises a pause-specific error the activity can catch, clean up after, and exit in an orderly way. Activities that skip heartbeating don't get this treatment. If they fail, Temporal has no visibility into whether they're still alive or just slow, which undermines reliable retry and recovery. Omitting the heartbeat isn't a shortcut, it's a decision to give up recoverability.

Consider a GPU worker topology built on this pattern. The heartbeat_timeout gets set well under the expected duration of the activity it's guarding. The GPU worker might die from spot preemption, an out-of-memory crash, or something else entirely, and Temporal notices the missing heartbeat within one timeout window and reschedules the activity onto any available worker. Pydantic AI's Temporal integration follows this pattern by default, registering heartbeats in the background for every activity, but only model request activities get an explicit heartbeat_timeout, set at 30 seconds. That calibration is deliberate: a long but perfectly healthy model call shouldn't get mistaken for a dead worker just because it's slow.

The replay guarantee is what makes this worth building. When a failure occurs, workflow execution resumes from the last event recorded in history instead of failing. Partial progress survives. That's the entire point.

State checkpointing and context preservation as prerequisites for safe preemption

Preemption without checkpointing is just a crash wearing a nicer name. The agent loses its position in the task and has no choice but to restart, which defeats the purpose of distinguishing preemption from failure.

Resuming correctly requires holding onto several distinct things at once. Execution state has to capture which steps finished, what they returned, and which are still mid-flight. Tool call history has to survive too, not as a nicety but as a necessity, since resuming a task without it risks re-firing external calls that already happened, duplicating side effects that can't be undone twice. The agent's reasoning trace, its accumulated observations, and any partial plan it had built all need to survive the interruption as well. For agents running inside isolated sandboxes, the filesystem, the process tree, and the network state of that sandbox may need preserving too, because the sandbox is functionally part of the agent's working memory.

Restate handles this with a journal approach: every ctx.run() call gets journaled before it executes, and on resume, the journal replays, skipping steps that already ran. That gives exactly-once semantics without pushing idempotency-key bookkeeping onto the application layer. LangGraph takes a comparable approach for human-in-the-loop interrupts, saving graph state through its persistence layer so execution can pause and pick back up later. In production, that persistence layer has to be something durable, an AsyncPostgresSaver or a MongoDBSaver. The in-memory checkpointer LangGraph ships for testing disappears the moment the process does, and treating it as production-ready is the single most common way teams rediscover this problem the hard way.

DeepSeek DSec's decoupling of agent loops from preemptible GPU pods for safe interruption

DeepSeek's arXiv paper 2609.22978, "DeepSeek Elastic Compute (DSec): A Sandbox Infrastructure for Effective Agentic Training at Scale," submitted September 19, 2026 and credited to Jialiang Huang alongside 130 co-authors, lays out a concrete answer to this problem at a scale where guesswork stops being an option. The paper describes DSec's architecture in detail.

The numbers explain why the architecture had to change. A single production-scale unit spans roughly 160 nodes, serves around 3 million sandboxes, supports more than 380,000 concurrent sandboxes, and sustains over 5,000 sandbox creations. At that volume, preemption isn't an edge case, it's a routine, constant event, and the architecture either absorbs it systematically or bleeds data every time it happens.

Earlier versions of DeepSeek's pipeline ran the agent loop directly inside the preemptible GPU training pod, alongside model-serving and the reinforcement learning framework. That was the wrong place for it: when the GPU job got preempted, the agent loop died with it, even though the sandbox holding the actual task state was still intact and untouched. Recovery was a brittle process that worked in limited cases but never scaled gracefully.

Starting with DeepSeek-V4.1, the fix was structural rather than incremental. The agent loop got pulled out of the GPU pod. It now runs independently inside DSec, split across an agent sandbox and a worker container, both living outside the preemptible GPU pool and no longer tied to that pod's lifecycle. When a GPU gets preempted now, the sandbox state is preserved through DSec's infrastructure. Once the GPU comes back, the sandbox picks up where it left off, and the training framework no longer needs to carry its own breakpoint recovery logic. Interrupted training steps resume without re-running tool calls, which is the preemption-safe trajectory replay the decoupling was built to enable. Separating the agent's loop from the hardware it happens to run on turned preemption from a data-loss event into a routine suspend-and-resume cycle, and this separation is worth copying even outside DeepSeek's specific stack.

Programmatic interrupt injection for human-in-the-loop control: LangGraph's interrupt() primitive

LangGraph frames the problem differently from the start. It models an agent as a directed cyclic graph, with conditional branching, persistent checkpoints, and interruption points built in as first-class parts of the graph itself, not bolted-on error paths.

Earlier versions of the framework handled this with interrupt_before and interrupt_after, set at defined node boundaries before execution begins. That worked fine for simple approval gates, where a human just needs to sign off before a particular step runs, but it falls apart for anything resembling a complex reasoning chain, where the need for a pause depends on what the agent actually decided, not on which node it happens to be sitting in. Static interrupt points are a poor fit for dynamic reasoning, full stop. The modern interrupt() function solves this by letting a node pause itself programmatically, based on runtime conditions evaluated as the logic executes. Under the hood, interrupt(payload) raises a GraphInterrupt exception, designed to be handled by the framework rather than to crash execution.

This underlies the human-in-the-loop middleware pattern many LangGraph deployments use: every tool call gets checked against a configurable policy before it's allowed to run, and if intervention is warranted, an interrupt halts execution and saves the graph's state through the persistence layer. A human then has to make a call, and the framework offers four distinct outcomes: approve the action as written, edit it before it runs, reject it with feedback attached, or respond directly, which covers "ask user" style tools where the human's answer is the actual output. That four-way branching is the detail that separates a real interrupt primitive from a binary stop switch. It has to carry enough context for a human to make a real decision, and the system has to know how to resume correctly no matter which of the four paths gets taken.

Policy-layer governance as a higher-order interrupt mechanism

The industry has converged on where policy enforcement belongs: at the point where an action actually executes, not somewhere inside the LLM's context window where a clever prompt could talk its way around it. Checking policy once at startup and trusting it from then on is a mistake, and a joint CISA/NSA advisory, cited in the AgenticRei paper (arXiv:2606.19464), arrived independently at the same conclusion, recommending per-invocation policy evaluation instead.

Most existing policy engines only know how to say yes or no, and that binary is too thin for real governance. They can't express an obligation, a requirement that the agent do something after taking an action, like notifying a CISO once a sensitive file gets touched. They can't express a dispensation either, the conditions under which a standing obligation gets waived. They have no way to resolve a conflict when two policies apply to the same action at once, and they can't reason across a domain's own structure, healthcare class hierarchies or data lineage graphs, the kind of thing a strict permit/deny binary simply can't hold.

AgenticRei's answer is a deontic policy language, built on the Rei framework and expressed in OWL, evaluated at runtime by a logic engine sitting entirely outside the LLM. Permissions, prohibitions, obligations, and dispensations all become first-class objects the engine reasons over directly, rather than instructions the model has to infer and hope to honor. A2AS, a framework led by Wallarm working alongside AWS, Bytedance, Cisco, Elastic, Google, JPMorganChase, Meta, and Salesforce, names the failure mode this architecture exists to close off: "security reasoning drift." A codified policy still has to pass through a model's interpretation to get enforced, and variation in how that model reasons produces misinterpretation or partial compliance, quietly, often unnoticed until the wrong action has already gone through. Taking the decision out of the model's hands and putting it somewhere that reasons the same way every time is the only fix that actually closes the gap, rather than just narrowing it.