Durable Execution — Deep Dive

A durable agent records each step's planned action, inputs, versions, and result, then wraps mutating tools with idempotency keys and duplicate detection.

class JournalStep:
    def __init__(self, step_id, planned_action, inputs, prompt_version,
                 tool_version, approval_status, result_receipt,
                 retry_count, error_state):
        self.step_id = step_id
        self.planned_action = planned_action
        self.inputs = inputs
        self.prompt_version = prompt_version
        self.tool_version = tool_version
        self.approval_status = approval_status
        self.result_receipt = result_receipt
        self.retry_count = retry_count
        self.error_state = error_state

def wrap_tool(tool_func):
    def wrapped(run_id, step_id, payload):
        # check duplicate via receipt for (run_id, step_id)
        receipt = get_receipt(run_id, step_id)
        if receipt:
            return receipt
        result = tool_func(payload, idempotency_key=f"{run_id}:{step_id}")
        store_receipt(run_id, step_id, result)
        return result
    return wrapped

Imagine you are building a tower of blocks. Every time you place a block, you take a photo of your progress. If the tower falls, you don’t start from the floor—you look at your last photo and put back exactly those blocks. Durable execution does this for computer programs: it lets them survive crashes by remembering exactly what they did, so they can pick up where they left off instead of starting over.

When a long-running agent runs, it writes down every step in a journal—the planned action, the inputs it used, the versions of tools, and whether a human approved it. That journal becomes the source of truth. If the program crashes, the runtime reads the journal and replays the recorded results instead of redoing the work. The operations that touch the outside world (calling an AI model, sending a message, running a shell command) are marked as “replay boundaries”—their results are saved and reused on recovery. The journal holds fields like the step ID, approval status, and error state, so the runtime knows exactly which steps have finished.

The trickiest part is that not all code is safe to replay. Ordinary code can run again harmlessly, but actions that change the world (like sending an email) must be protected. The runtime wraps these actions with idempotency keys—a unique label for each step—so that when the system restarts, it checks the key and skips actions already done. Without this subsystem, a crash would leave the agent in an unknown state. It might send the same message twice, run the same AI query again (costing money), or lose a human approval. The program would have to guess what happened, leading to errors and wasted work.

The subsystem begins with a durability mode selection: developers invoke a LangGraph graph execution method like graph.stream({"input": "test"}, durability="sync"). Under “sync” durability, the runtime calls a checkpointer conforming to the BaseCheckpointSaver interface to persist the full graph state synchronously before each superstep moves forward. On failure, the process terminates but the last written checkpoint survives. When the agent restarts—potentially on a different machine—LangGraph reads that checkpoint, recreates all state channels, and continues from the interrupted superstep, having already recorded the output of previously completed steps. In contrast, “async” durability writes checkpoints while the next step runs, and “exit” durability writes only when graph execution exits successfully, by error, or by human-in-the-loop interrupt. The ordered mechanism is therefore: choose a durability mode, execute superstep, save checkpoint (synchronously, asynchronously, or only on exit), and on crash resume from the last saved checkpoint.

The invariant the design preserves is crash-proof execution. The source explicitly states: “Durable Execution is crash-proof because it virtualizes execution, enabling it to take place across a series of processes, each of which can potentially run on a different machine. … The application state is recreated in that new process, after which execution will continue as if the failure never happened at all.” This means variables, local state, and the notion of which statements already ran are all durable. The runtime guarantees that no progress is lost beyond the last checkpoint, and side effects are not blindly repeated because each operation’s result is recorded—LLM calls and tool calls become replay boundaries whose outputs are saved rather than re-invoked.

The key trade-off appears in the three durability modes, each rejecting a different cost. The obvious alternative is to forgo checkpointing entirely (the “exit” mode), which offers the best performance for long-running graphs because it writes only on terminal exits. That choice, however, rejects the ability to recover from a process crash mid-execution: “intermediate state is not saved, so you cannot recover from system failures (like process crashes) that occur mid-execution.” The cost avoided by using “sync” instead is the risk of losing intermediate progress and having to restart from the beginning—a restart that might repeat side effects such as sending a notification twice. “Sync” mode pays a performance overhead to guarantee that every superstep’s state is persisted before the next step begins, thereby eliminating the window for data loss. The design deliberately rejects hardware-based fault tolerance (e.g., hot-swappable CPUs) because that “cannot guarantee that execution will continue if that failure does occur” and offers no protection against software bugs; durable execution builds reliability into the software itself.

A concrete failure mode: a long-running agent using durability="exit" crashes while executing step C (for example, while making an LLM call or writing to a file). Because no checkpoint was written for the intermediate step, the operator sees that the graph’s run history ends at step B. On restart, the agent must begin from step A again, re-executing steps A and B, which may cause a duplicate side effect (e.g., a user receives the same email twice). The observable signal is an incomplete run log in the checkpointer storage—only the final checkpoint after step B exists—and a subsequent run that shows multiple invocations of the same LLM call or tool output. With durability="sync", by contrast, the operator would see a checkpoint after step B and also a partially written checkpoint for step C (if the crash occurred after the checkpoint write began), enabling resumption exactly at step C. The exact identifier BaseCheckpointSaver is the interface that any checkpointer must implement; LangGraph provides several installable libraries that conform to it, and the DeltaChannel optimization (storing deltas instead of full state) further reduces storage growth in append-heavy channels.

Failure: Crash mid‑step with async durability

• Trigger – The process crashes while a step is executing and the graph execution method was called with durability="async".
• Guard – No guard is shown; the source explicitly notes that with "async" there is “a small risk that LangGraph does not write checkpoints if the process crashes during execution.”
• Posture – fail‑hard – the run is aborted and no checkpoint is written.
• Operator signal – Silent absence: the checkpoint store lacks the expected checkpoint for that step.
• Recovery – Not specified in the source; the operator must manually restart the agent from the last known good checkpoint or from the beginning.

Failure: Duplicate notification caused by retry

• Trigger – A retry sends the same notification twice because the first send succeeded but the acknowledgement was lost.
• Guard – idempotency keys (from the Restate description) detect and suppress the duplicate.
• Posture – fail‑soft – the duplicate is silently dropped; the workflow continues.
• Operator signal – No log is mentioned; the deduplication is invisible.
• Recovery – Automatic: the idempotency key ensures only one execution.

Failure: Non‑deterministic code inside workflow

• Trigger – The developer places a nondeterministic call (e.g., an LLM invocation) directly inside workflow logic instead of inside an Activity.
• Guard – The “Workflow/Activity split” (Temporal) is the required boundary; the source says deterministic replay works “only if the agent builder respects the Workflow/Activity split”.
• Posture – fail‑hard – the Temporal service detects non‑determinism and fails the workflow.
• Operator signal – Not specified in the source; the operator would observe a workflow failure with an error indicating non‑determinism.
• Recovery – The developer must refactor the agent to move nondeterministic steps into an Activity and restart the workflow.

Failure: Journal storage corruption

• Trigger – A hardware or filesystem error corrupts the journal file (the record of completed operations) while writing.
• Guard – None shown; the journal is the source of truth for recovery, but no corruption guard or checksum is described.
• Posture – fail‑hard – the corrupted journal prevents any recovery; the agent cannot resume.
• Operator signal – A read error from the storage layer or a malformed‑journal indication (not explicitly named in the source).
• Recovery – Not specified; manual restoration from a backup or re‑execution from scratch is required.

Failure: Idempotency key collision

• Trigger – Two distinct operations accidentally receive the same idempotency key (e.g., from a weak random source).
• Guard – idempotency keys are the mechanism, but the source does not describe a guard against collisions themselves.
• Posture – fail‑soft – one operation is incorrectly treated as a duplicate and skipped, silently degrading the result.
• Operator signal – None given; the skipped operation produces a silent absence.
• Recovery – Not specified; manual verification and re‑keying of the affected step may be necessary.

Retrieving the latest checkpoint state snapshot for a thread.

config = {"configurable": {"thread_id": "1"}}
graph.get_state(config)

Imagine you are writing a long letter, and after every sentence you snap a photograph of the page showing exactly what is written so far. That photograph is tied to a specific envelope (a thread). This is what a checkpoint does for a computer program that runs in steps — it saves the entire state of the run after every step so you can resume exactly where you left off if the power dies, or even flip back to an earlier photo to see what happened. It is built for memory, fault recovery, and letting a human pause and approve before the next action.

Now zoom in on how that photograph works. The system uses a thread-id as the envelope label. After each superstep (a batch of actions), the checkpointer stores a StateSnapshot that includes the current values, the next steps, and metadata like source and writes. You can ask for the whole photo album with graph.get_state_history(config). But here is the catch: the photograph only captures the page between sentences, not the motion of writing each word inside a sentence. If a step makes an API call or writes a file, that action itself is not automatically saved — if you replay from an earlier checkpoint, that call might happen a second time.

The deepest rule is that a checkpoint does not reach inside a single node. If your node fires off an email and then continues, then the program crashes before the next checkpoint, that email will fire again on recovery. The guard is simple: any nondeterministic operation like an API request or file write must be wrapped in its own node or marked as a task so it becomes its own step with its own photograph. Without this rule, a beginner would feel the failure of duplicate orders, double charges, or corrupted files — concrete problems from redoing unrecorded actions that should have been frozen in time.

The checkpointing subsystem in LangGraph operates on a superstep-ordered mechanism: each execution progresses through supersteps, and at the boundary of every superstep the checkpointer – an object conforming to BaseCheckpointSaver – persists the full graph state for the current thread_id. Execution begins with the first superstep; after each step completes, the checkpointer writes a checkpoint. On failure (e.g., a process crash), the runtime calls get_state_history(config) to retrieve a list of StateSnapshot objects ordered newest‑first, and then replays from the most recent valid checkpoint. The StateSnapshot includes next (the tuple of pending tasks), config (with checkpoint_id and thread_id), metadata (with source, writes, and step counter), and tasks (a tuple of PregelTask entries). The system thus ensures that any completed superstep is recoverable exactly once, regardless of the crash timing relative to the write.

The core invariant this design preserves is the replay boundary: progress is made durable only at superstep granularity, not within a node’s internal execution. This means that while the checkpoint guarantees the state after a superstep is safely recorded, it does not automatically make every node safe. Nondeterministic operations and side effects – such as API calls, file writes, or LLM invocations – must be wrapped in explicit tasks or nodes to become durable replay boundaries. The name for this guarantee, drawn directly from the source, is that the checkpoint provides “crash‑proof execution” at the superstep level; inside a step, the developer must engineer idempotency or use the provided interrupt pattern to create safe resume points.

The key trade-off is between performance and durability, exposed through three explicit durability modes:

• "exit" persists only when graph execution exits (success, error, or human interrupt), offering the best performance but losing all intermediate state on crash.
• "async" writes asynchronously while the next step runs, giving good performance with a small risk of lost checkpoints if the process crashes during the write.
• "sync" writes synchronously before the next step starts, guaranteeing every checkpoint is on disk but adding performance overhead.

The design rejects the alternative of forcing synchronous writes on every operation (which would make durability trivially strong but cripple latency). Instead, it lets the operator choose the acceptable risk of lost work in exchange for throughput. The cost this rejection avoids is the unbounded performance degradation of a write‑every‑operation model, especially important for high‑step‑count graphs like multi‑turn conversations where DeltaChannel may further reduce storage by storing only deltas.

A concrete failure mode: the process crashes while executing a superstep with durability="async" and the checkpoint write has not yet completed. The operator queries get_state_history(config) and sees that the latest StateSnapshot has a step counter one less than expected and its metadata["source"] is "loop" with a writes dictionary missing the outputs of the crashed node. The signal is a gap in the chronology of the returned snapshots and a StateSnapshot where next is non‑empty (indicating incomplete execution). The operator can then manually replay from that snapshot, or the runtime – aware of the async risk – may automatically resume from that boundary, rerunning the lost superstep. The concrete indicator is the missing checkpoint_id for the expected superstep in the get_state_history output.

Node Failure During a Superstep

• Trigger: A node raises an exception mid-execution at a given superstep, while other nodes in the same superstep complete successfully.
• Guard: The "pending writes" mechanism — LangGraph stores pending checkpoint writes from any other nodes that completed successfully at that superstep.
• Posture: fail-soft — the run does not abort; it saves progress from successful nodes and allows resumption from the last successful step, skipping re‑execution of those nodes.
• Operator signal: The run is interrupted; the state snapshot for that superstep shows an error field in the tasks tuple for the failed node, and the writes metadata contains only the successful node outputs.
• Recovery: Resume graph execution from that superstep. The source states: “When you resume graph execution from that super‑step you don't re‑run the successful nodes.”

Process Crash in Async Durability Mode

• Trigger: The process crashes during an asynchronous checkpoint write (durability mode "async") before the write completes.
• Guard: No guard exists for this scenario. The source explicitly notes: “there's a small risk that LangGraph does not write checkpoints if the process crashes during execution.” The only mitigation is to use "sync" durability mode, but that is a configuration choice, not a runtime handler.
• Posture: fail-hard — the checkpoint for that superstep is lost; the run cannot be recovered from that point and must be restarted from the last fully persisted checkpoint (or from scratch).
• Operator signal: The thread’s state history is missing the expected checkpoint for that superstep; logs may show incomplete writes or a crash trace.
• Recovery: Manually restart the graph with durability="sync" to ensure future checkpoints are written synchronously before continuing.

Storage Growth from Full-State Checkpoints

• Trigger: Long‑running threads with large accumulations (e.g., multi‑turn conversations) cause checkpoint size to grow because each checkpoint writes the full value of every state channel at each superstep.
• Guard: The DeltaChannel implementation — it “stores only incremental deltas instead of the full accumulated value, substantially reducing checkpoint size for append‑heavy channels.”
• Posture: fail‑soft — the system continues to operate but with increasing storage usage and potentially slower checkpoint writes. The guard is optional; the operator must explicitly adopt DeltaChannel.
• Operator signal: Disk usage grows disproportionately over time; checkpoint write latency increases; the thread’s state history contains many large values entries.
• Recovery: Switch to DeltaChannel for the relevant state channels, as documented: “See DeltaChannel for usage and the storage‑vs‑latency tradeoff.”

Missing Thread ID on Invocation

• Trigger: A graph is invoked with a checkpointer but without specifying a thread_id in the configurable portion of the config.
• Guard: No guard is shown in the source. The requirement is stated: “you must specify a thread_id as part of the configurable portion of the config.” No automatic fallback or error handling is provided.
• Posture: fail-hard — the run aborts immediately because the checkpointer cannot persist state without a thread identifier.
• Operator signal: An error (likely KeyError for missing key) thrown when calling graph.stream() or similar methods. No thread is created; no checkpoints are saved.
• Recovery: Correct the config to include {"configurable": {"thread_id": "some-id"}} and re‑run the graph.

Journal replay in Restate: on recovery, replays the journal and skips completed work.

#  Restate replays the journal and skips work already completed."
# "A model call or external tool call can be wrapped as a durable step
#  whose result is recorded once and then replayed as data."
# "ctx.run-style durable steps, built-in key-value state, durable timers,
#  idempotency keys, and callback waits called awakeables."

Think of following a complex recipe. Each time you complete a step like chopping onions, you jot down the result in a notebook. If you get distracted and need to start over, you don’t re-chop the onions – you read the notebook and skip ahead. This subsystem is for making computer programs crash-proof by remembering what already happened, so after a failure they can resume without redoing work.

In software, that “notebook” is called a journal. For every meaningful operation, the system records a step_id, the planned action, and the result into an event history. On recovery it replays the entire function from the beginning, but when it runs into a step whose ID already exists in the journal, it returns the cached result instead of executing again. Real runtimes like Restate implement this as ctx.run durable steps, where each completed call is stored and later replayed as data.

The trickiest part is that not every step is safe to replay from the journal. Nondeterministic actions—like calling a language model, reading the current time, or writing to a file—can produce different results each time. The runtime must enforce a replay boundary around those actions, wrapping them as separate “activities” that cannot be deterministically replayed inside the journaled code. Without this boundary, recovery could serve a stale model result or re‑execute a random call, breaking the crash‑proof guarantee. If the journal lacked this subsystem entirely, a crash would mean restarting from scratch, losing every completed step. If it kept the journal but ignored the activity boundary, replay could silently substitute wrong outputs, corrupting the whole run.

In a journal-replay subsystem, the ordered mechanism begins by recording every completed step into an event history—for example, using Temporal’s workflow event history or Restate’s journaled steps. Each run receives a stable run_id, and each meaningful operation gets a step_id. When a worker crashes, the runtime replays the function from the start, but it does not re-execute work already recorded in the journal. Instead, it returns cached results for those steps, skipping completed operations and injecting the stored outcomes. This is the core replay loop: first persist the step’s result, then on recovery replay the code but serve the journaled data for any step whose result is already present.

The invariant preserved is deterministic replay, where the journal becomes the source of truth for recovery. Temporal enforces this by requiring that workflow code be fully deterministic; nondeterministic work—such as LLM calls, shell commands, file writes, or wall-clock reads—must be pushed into Activities. Restate achieves the same guarantee through its ctx.run-style durable steps, which record the result of each external call once and then replay that result as data. Without this invariant, replay would produce different outcomes on each attempt, breaking the crash-proof execution that the platform promises.

The key trade-off is the separation of nondeterministic actions into explicit replay boundaries. The obvious alternative—allowing arbitrary non‑deterministic code inside the replay logic—is rejected. That rejection avoids the cost of corrupted state and duplicate side effects: if a model call or file write were re‑executed during replay, it could produce a different LLM response or write the same file twice, causing the application to diverge unpredictably. By isolating such actions into Activities (Temporal) or journaled steps (Restate), the runtime sacrifices simplicity in the application code but gains reliability and exactly‑once semantics for irreversible operations.

A concrete failure mode occurs when a developer places a nondeterministic operation—such as an LLM call—directly in Temporal workflow code instead of using an Activity. On first execution, the call succeeds and records a result. After a crash, replay calls the same LLM endpoint again, but the model has been updated or the input differs slightly, yielding a different response. The operator would see a non‑determinism error in Temporal’s logs, because the replayed event history does not match the recorded history. The signal is a mismatch between the expected step outcome (from the journal) and the actual output of the re‑executed code, making the failure visible as a rejected workflow or a replay divergence alert.

Non-deterministic action executed inside replay logic

• Trigger — Developer places an LLM call directly in the workflow code instead of wrapping it as an Activity.
• Guard — No explicit guard in source; the design principle is the Workflow/Activity split (from Temporal), which requires nondeterministic actions to be separated as Activities outside the main replay logic.
• Posture — fail-hard: replay produces a different result than the original execution, causing a non‑determinism error that aborts the run.
• Operator signal — Non‑determinism error during replay (e.g., Temporal’s “WorkflowExecutionNonDeterministicError” is not named in source, but the source states “workflow code is expected to be deterministic” and that violation leads to failure).
• Recovery — Developer must move the LLM call into an Activity; after code change, the workflow can be re‑executed.

Duplicate side effect from retry without idempotency

• Trigger — After a crash, the system retries and re‑executes a step that mutates external state (e.g., sends a notification) without duplicate detection.
• Guard — idempotency keys (from source: “The runtime should wrap them with idempotency keys, receipts, duplicate detection, and compensation metadata.”)
• Posture — fail-soft: the side effect is applied twice, but the runtime continues; external state becomes inconsistent.
• Operator signal — “A retry can send the same notification twice.” — operator observes the duplicate notification or mutating action.
• Recovery — If idempotency keys are implemented, the duplicate is prevented; otherwise, manual compensation (e.g., reverting the extra notification) is required.

Lost human approval due to crash before checkpointing interrupt state

• Trigger — System crashes after a human grants approval but before the interrupt state is durably saved.
• Guard — interrupt pattern (from LangGraph: “Its interrupt pattern gives a durable pause and resume mechanism for human approval”)
• Posture — fail-soft: the approval is lost, and the agent may re‑request approval or continue without it.
• Operator signal — “A human approval can be lost in a chat transcript.” — operator sees that the approval was given but the workflow remains in the waiting state.
• Recovery — If the interrupt pattern checkpoints correctly, the approval is saved and resume restores it; otherwise, the human must re‑approve.

Stale cached result from journal replay due to external state change

• Trigger — A step’s result was recorded in the journal; during a later replay, the external system’s state has changed (e.g., database record modified), making the cached result invalid for the current context.
• Guard — No explicit guard in source; the source states the system “returns cached results for those steps” without invalidation logic.
• Posture — fail-soft: the replay uses the stale cached result, causing the agent to act on outdated information while the run continues.
• Operator signal — Silent logical divergence; the agent’s behavior contradicts the current external state, yet no error is logged.
• Recovery — Manual detection and compensation; steps that depend on mutable external state should be designed as idempotent Activities that are re‑executed on recovery rather than cached.

Journal corruption or unreadable event history

• Trigger — Storage failure, bit‑rot, or bug corrupts the journal/event history files.
• Guard — sync durability mode (from LangGraph) ensures each checkpoint is written synchronously before the next step, but does not prevent corruption of existing checkpoints. No explicit guard for corruption detection.
• Posture — fail-hard: the system cannot load the journal, so recovery is impossible and the run aborts.
• Operator signal — Corruption error when attempting to read the journal (e.g., “checkpoint load failure” or “journal unreadable” – not explicitly named in source, but implied by loss of the source of truth).
• Recovery — Manual restoration from a backup of the journal; if no backup, the run must be started from scratch.

LangGraph checkpointing and thread ID enable durable pause for human approval.

from langgraph.types import Command

config = {"configurable": {"thread_id": "some_id"}}
result = agent.invoke(
    {
        "messages": [
            {
                "role": "user",
                "content": "Delete old records from the database",
            }
        ]
    },
    config=config,
    version="v2",
)

Think of a chef who can pause halfway through a recipe to let a taste tester approve the next pinch of salt, then continue without burning the sauce. This subsystem is for pausing a computer program to wait for a human’s yes or no, then resuming exactly where it stopped, so no work is wasted.

When a tool call needs a review, the run stops and the checkpointer saves the current state—including the tool input, the approval, and which step was interrupted—under a thread identifier that ties everything together. That saved state is stored in a StateSnapshot object, which holds the actual values, what comes next (shown as an empty tuple when paused), and metadata like the source ("loop" or "input") and the writes that were already done. The run sits and waits; only the interrupted action gets replayed when a decision arrives.

The tricky part: code that runs before the pause may re‑execute if the system later recovers from a crash, because checkpoints happen at superstep boundaries, not at every single line. The metadata field in the snapshot records what was already written, so the runtime knows what to skip, but approval boundaries must be placed carefully—otherwise a human could approve and the same message could be sent twice. Without this subsystem, if the computer dies after approval, the whole run would restart from scratch, repeating every earlier step and likely sending duplicate notifications or redoing expensive model calls—exactly the kind of chaos the pause was meant to prevent.

The subsystem for pausing a run during human approval relies on a checkpointer that records graph state at each superstep and organizes runs by a thread identifier. When a tool call requires review, the run stops via LangGraph's interrupt pattern. The checkpointer persists tool input, approvals, usage, and the thread identifier into durable storage. On resumption, the runtime loads the saved state and continues from the recorded checkpoint. However, the documentation explicitly warns that code before an interrupt may re-execute during recovery: later graph work can be replayed, so nondeterministic operations and side effects must be wrapped in tasks or nodes, and mutating operations still require idempotency. The ordered mechanism is therefore: save state at the last superstep before the interrupt, pause, store approvals, and on resume replay prior nodes up to the replay boundary, then resume the interrupted action.

The invariant the design preserves is crash-proof execution through checkpointing, but it does not automatically guarantee that every node is safe. The core guarantee is that the checkpointer provides fault recovery, state history, and time travel, but the developer must engineer each node boundary as a replay boundary. This means the runtime remembers the last successful step and the thread identity, but it cannot know whether an email was sent, a file was deleted, or a human approval was already acted upon unless those operations are made idempotent or wrapped in durable tasks. The invariant is thus a durable pause and resume mechanism without automatic idempotency for side effects; the system preserves graph state and thread continuity, leaving safety to the node implementation.

The key trade-off is that LangGraph's approach rejects the full deterministic replay model used by systems like Temporal, where workflow code must be deterministic and all side effects are pushed into Activities. Instead, LangGraph allows arbitrary code in nodes, which is simpler and more flexible for developers. The cost of that rejection is that the developer must manually handle replay safety—placing approval boundaries carefully, wrapping nondeterministic operations, and adding idempotency keys to mutating tools. The avoided cost is the strict discipline required by deterministic replay, such as separating orchestration from activities and ensuring no nondeterministic operations inside workflow code. This trade-off trades automatic correctness for developer convenience and lower architectural overhead, but it offloads the burden of ensuring exactly-once behavior to the node author.

A concrete failure mode occurs when an approval is stored only as a transient chat message rather than in the checkpointer's durable state. After a crash, the runtime resumes from the last checkpoint, and the code before the interrupt re-executes. If that code includes a side effect like sending a duplicate notification or re-prompting for approval, the operator will see multiple approval requests or duplicate messages in the chat transcript. The signal is a repeated human-in-the-loop request for the same action, indicating that the approval boundary was not engineered as a proper replay boundary. The LangGraph docs highlight this risk explicitly: "code before an interrupt may run again, so approval boundaries must be placed carefully." The operator would observe inconsistent state where an approval appears to have been given but the system asks again, or where a tool call that was already executed is submitted a second time.

Checkpointer Write Failure in Async Durability Mode

• Trigger – The process crashes while LangGraph is asynchronously writing a checkpoint during a human-in-the-loop pause. The checkpoint stores tool input, approvals, and usage.
• Guard – None. The source explicitly warns: “there's a small risk that LangGraph does not write checkpoints if the process crashes during execution” when using "async" durability.
• Posture – fail-soft. The run aborts, but the previously written checkpoint (from a prior super‑step) remains. Recovery restarts from that older state, so the most recent pause is lost.
• Operator signal – A missing checkpoint for the super‑step that contained the human‑approval pause. The graph’s thread shows state rolled back to the last fully persisted step.
• Recovery – The run is retried from the last durable checkpoint. The human must re‑approve the tool call because the approval decision was not persisted. No automatic retry count is specified; it is a manual re‑run.

Pending Writes Loss After Node Failure

• Trigger – During a super‑step that includes the human‑approval node, another node fails mid‑execution. LangGraph “stores pending checkpoint writes from any other nodes that completed successfully at that super‑step”. If the entire process crashes before those pending writes are committed, they are lost.
• Guard – No explicit guard for pending write durability. The mechanism itself assumes the writes survive, but no persistence guarantee is given for pending writes.
• Posture – fail-soft. The run can be resumed from the super‑step, but the successful nodes that contributed pending writes will be re‑executed because their state was not actually saved.
• Operator signal – On resume, a node that previously succeeded runs again. Logs show duplicate execution for that node.
• Recovery – The super‑step is replayed. Without idempotent tool wrappers (exact identifier: idempotent tool wrappers from the architecture section), side effects may be duplicated. The source notes that “you don't re‑run the successful nodes” when pending writes survive – but if they do not survive, that guarantee breaks.

Code‑Before‑Interrupt Re‑execution Causing Duplicate Side Effects

• Trigger – After a human‑in‑the‑loop pause, the graph resumes from the last checkpoint. The source warns: “code before an interrupt may run again, so approval boundaries must be placed carefully.” Any nondeterministic or mutating operation that occurred before the interrupt (e.g., a tool call that sent a message or modified a file) will be re‑executed.
• Guard – No automatic guard in the checkpointer. The source advises wrapping mutating operations in tasks or nodes and using idempotency keys, but this is a design pattern, not a built‑in guard.
• Posture – fail-soft. The run continues, but side effects may be duplicated unless the application layer handles idempotency.
• Operator signal – Duplicate tool calls or duplicate external actions (e.g., repeated API requests, duplicated messages) in logs.
• Recovery – The operator must rely on the Idempotent Tool Wrappers layer (exact identifier from the “Practical Durable‑Agent Architecture” section) to detect and skip already‑executed operations. The retry count is not specified; the system simply re‑executes the node.

Approval State Corruption in Serialized Checkpoint

• Trigger – The serialized checkpoint that includes “approvals, usage, tool input, nested resumptions, trace metadata, and conversation settings” becomes corrupted (e.g., due to storage corruption, version mismatch, or partial write).
• Guard – None. The source mentions that “serialized state… should be treated as sensitive runtime state” but does not provide any checksum, validation, or retry logic for checkpoint integrity. The BaseCheckpointSaver interface does not include a corruption guard.
• Posture – fail-hard. The run cannot resume because the checkpoint is unreadable or the approval decision is missing.
• Operator signal – A deserialization error or missing approval field when attempting to resume the thread. The graph remains paused indefinitely, with no forward progress.
• Recovery – Manual intervention is required. The operator must discard the corrupted checkpoint and restart from the previous good checkpoint, then the human must re‑approve the tool call. No automatic retry occurs.

Thread Identifier Collision

• Trigger – Two separate runs (or a run and a test) use the same thread_id value. The source states: “The checkpointer uses thread_id as the primary key for storing and retrieving checkpoints.” A collision can conflate state from different approval workflows.
• Guard – None. The source does not specify uniqueness enforcement or collision detection for thread_id.
• Posture – fail-soft. The checkpointer overwrites or merges checkpoint entries, corrupting the intended approval flow.
• Operator signal – Unexpected state in a thread: approvals from a different run appear, or the graph resumes with wrong tool input. No error is raised because the collision is silent.
• Recovery – The operator must manually inspect and possibly delete conflicting checkpoints, then assign a new unique thread_id and re‑start the run. No automatic retry or backoff.

Idempotent tool wrapper pattern described in context

# If already sent, skip execution; otherwise execute and record a durable receipt.
# Wrapper records intent before execution, provides idempotency key, and checks duplicates.
def send_message_wrapper(run_id, step_id, message_content):
    key = f"{run_id}:{step_id}"
    if idempotency_store.exists(key):
        return idempotency_store.get_result(key)
    # execute actual send
    result = actual_send(message_content)
    idempotency_store.record(key, result)
    return result

Think of it like a video game auto‑save that records every action you take. If the game crashes, you restart exactly at the last save, without redoing any moves you already made. The Replayability Contract does the same for software agents: it ensures that after a crash, the agent picks up precisely where it left off and never repeats side effects like sending a duplicate message or creating a second pull request.

The runtime achieves this by drawing a replay boundary around each nondeterministic action—calls to an LLM, a tool, a shell command, or an API. Before performing the action, it assigns a unique step_id and saves the result. On recovery, instead of actually re‑executing the call, the runtime injects the saved result, skipping the real work. Each tool is also wrapped with an idempotency key so that if the same step runs a second time, the external system knows not to process it again.

The trickiest part is that code inside the replay boundary must be deterministic—no random numbers, no wall‑clock reads, because those would produce different outputs on replay and break the contract. All nondeterministic work must be pushed outside the boundary, where its exact result is recorded once. Without this subsystem, a crash after sending an email would lose that fact, and the agent would re‑send it on recovery, duplicating messages or even charging a customer twice.

The Replayability Contract subsystem orders execution by first recording durable checkpoints or journal entries before any nondeterministic external action can take effect. In a LangGraph-based agent, the runtime calls BaseCheckpointSaver.save() at each superstep, with the durability parameter controlling when that write occurs—"sync" ensures the checkpoint is flushed before the next step starts, "async" writes it while the next step runs, and "exit" records only on termination. On failure, recovery begins by reading the last checkpoint via BaseCheckpointSaver.get() and replaying deterministic workflow code from that point. Nondeterministic operations such as LLM calls, shell commands, or API requests are wrapped in replay boundaries (e.g., Temporal Activities or Restate's ctx.run-style durable steps) whose results are stored in the event journal and reused on replay rather than re-executed. This ordered mechanism means that first the side-effect is recorded, then the node advances; a crash after recording but before the next checkpoint will skip the duplicated external call.

The invariant the design preserves is the replay boundary—the guarantee that every side effect is either already persisted or isolated behind a deterministic replay point. The source explicitly states, "the node boundary has to be engineered as a replay boundary." This ensures exactly-once or at-most-once semantics for external actions: once a tool call's result is committed (via an idempotency key or journal entry), it is never replayed. The runtime tracks completed steps and, on recovery, replays only the deterministic code between those boundaries, never repeating the side effect itself. This invariant prevents double messages, duplicate pull requests, or repeated resource writes.

The key trade-off is performance versus durability: recording every checkpoint synchronously ("sync" mode) adds overhead to each superstep, while the "exit" mode avoids that cost but loses all intermediate state on a crash. The obvious alternative it rejects is ephemeral execution with no checkpointing at all—treating agent runs as purely in-memory streams. That rejection avoids the catastrophic cost of unsafe re-execution: without a replay boundary, a crash after sending a notification would leave the operator unable to know whether the action happened, forcing either a blind retry (sending the notification again) or manual log inspection. Durable Execution instead accepts additional latency and storage (mitigated by DeltaChannel for incremental deltas) to make recovery a controlled continuation rather than improvisation.

A concrete failure mode: an agent running in LangGraph with durability="sync" executes node A (a deterministic analysis), then node B (a tool call that submits a payment via an API with an idempotency key). The checkpoint for node A is written synchronously; during node B, after the API response is received but before the next checkpoint, the process crashes due to a hardware failure. The operator sees a log entry like "Checkpoint written for superstep A" followed by a crash signal, and no checkpoint for superstep B. On resume, the runtime loads the last checkpoint (node A) and replays the deterministic code up to the replay boundary of node B. The tool call is not re-executed because its result was already recorded in the journal (via the Restate-style ctx.run or Temporal Activity completion). The signal to the operator is a replay start notice in the execution logs (e.g., "Replaying workflow from event ID 3" in Temporal) or a "Resuming run from checkpoint X" message in LangGraph's observability traces. No duplicate payment is generated because the idempotency key prevents the API from accepting the same transaction twice.

Process crash during asynchronous checkpointing

• Trigger — A worker process crashes between completing a step and writing its checkpoint to storage when using the durability="async" mode. The source notes that for "async": "there's a small risk that LangGraph does not write checkpoints if the process crashes during execution."
• Guard — None for "async" mode. The "sync" durability mode (durability="sync") provides a guard by writing every checkpoint synchronously before the next step begins, but this is a configuration choice, not a runtime handler for the crash itself.
• Posture — fail-soft: execution continues from the last successfully written checkpoint (or from the beginning if that was the first step), losing all progress since that checkpoint. The agent does not abort but repeats work.
• Operator signal — Silent absence: the agent resumes from a stale state. No error is raised; the operator might notice missing steps in the agent’s output or see that a later step attempted to use data that should have been persisted.
• Recovery — No automatic retry for the lost checkpoint. The operator must let the agent re-execute the lost steps. If side effects were already performed but not recorded, those effects are duplicated unless idempotency keys intervene.

Retry without idempotency key causing duplicate external action

• Trigger — A tool call (e.g., send-message, create-pull-request) fails with a transient error (network timeout, service outage) and the runtime retries. The tool was not wrapped with an idempotency key, so the retry sends the exact same command again. The source warns: “retries must not repeat external actions” and recommends wrapping tools with “idempotency keys, receipts, duplicate detection, and compensation metadata.”
• Guard — idempotency keys and duplicate detection are the intended guards. If implemented, the guard stores the key after the first call and skips on retry. Without them, no guard exists.
• Posture — fail-soft: the agent does not crash; it continues with a duplicate side effect (e.g., two identical pull requests, two emails). The system degrades because external state is corrupted.
• Operator signal — “Duplicate pull request created” or “Duplicate message sent” observed in the external system’s logs. The agent’s trace shows two tool calls with identical inputs and no idempotency fields.
• Recovery — Manual compensation (e.g., delete duplicate, revert). The source mentions “compensation metadata” as a possible recovery path, but not a specific identifier. The runtime itself does not automatically undo the duplicate.

Non-deterministic code inside workflow step

• Trigger — The workflow step contains a non-deterministic operation such as random.randint, time.sleep with no recorded wall-clock, or a direct call to datetime.now(). On recovery, the runtime replays the step (as in Temporal’s deterministic replay), producing a different result than the original run.
• Guard — None. The source repeatedly states that workflow code must be deterministic (e.g., “Workflow code is expected to be deterministic” for Temporal). No runtime exception handler or validation enforces this; it is a design-time constraint.
• Posture — fail-hard: the agent produces incorrect state, which can cascade into errors or nonsensical decisions. The run may abort later when the inconsistency surfaces, or produce silent data corruption.
• Operator signal — The agent’s behaviour diverges after a restart. Trace logs show different model responses or tool arguments for the same step in the original vs. replay run. No single error field is raised.
• Recovery — Manual diagnosis and rewrite of the step to push non-determinism into recorded activities (e.g., using Temporal’s Activity or Restate’s ctx.run). The run may need to be discarded and restarted from a clean state.

Side effect not isolated behind a replay boundary

• Trigger — A developer places an LLM call, shell command, or API request directly inside the replayable workflow logic instead of wrapping it in a recorded activity. The source states that these operations “are not ordinary code” and “are replay boundaries” whose “results should be saved and reused on recovery.” If not isolated, the crash recovery re-executes the side effect.
• Guard — None. The runtime does not automatically detect whether code is a side effect; the onus is on the developer to use the platform’s durable-step constructs. The source mentions Restate’s ctx.run-style durable steps and Temporal’s Activities as the proper guards, but they are not applied here.
• Posture — fail-soft: the agent continues, but the external system sees a repeated side effect (e.g., same model prompt sent twice, same file written again). This is a silent degradation.
• Operator signal — Duplicate entries in the agent’s journal (e.g., two identical LLM generation trace events) or duplicate external effects. The operator might see two identical API calls in the service’s audit log.
• Recovery — Manual compensation (e.g., delete duplicate, revert file). The agent is not automatically corrected. The architecture must be redesigned to respect replay boundaries, as the source advises: “side effects must be isolated behind replay boundaries.”

Idempotency key collision or exhaustion

• Trigger — Two different operations happen to generate the same idempotency key (collision), or a single operation uses a key that has already been exhausted (e.g., sequential keys wrap). The source does not describe a key-generation scheme, but idempotency keys are assumed to be unique per logical operation.
• Guard — duplicate detection may catch the collision and erroneously skip a new operation, or it may allow a true duplicate if the key is reused without intent. The source does not specify a guard for key collision itself.
• Posture — fail-closed: if duplicate detection treats the new operation as an already-completed action, it refuses to perform the write (fail-closed). Alternatively, if detection is missing, the operation proceeds, creating a duplicate (fail-soft).
• Operator signal — An error log such as “Idempotency key already used” or a silent absence: the intended action never happens and no record is created. In the duplicate-detection case, the agent continues but the operator may see a gap in expected outcomes.
• Recovery — Manual inspection of the key space and correction of the key generator (e.g., prefix with run_id + step_id). The source suggests run_id and step_id as stable identifiers, but no automatic recovery is provided.

LangGraph's checkpointing mechanism provides durable state persistence for agent runs, an alternative to workflow engine deterministic replay.

from langgraph.graph import StateGraph, START, END
from langgraph.checkpoint.memory import InMemorySaver
from typing import Annotated, TypedDict
from operator import add

class State(TypedDict):
    foo: str
    bar: Annotated[list[str], add]

def node_a(state: State):
    return {"foo": "a", "bar": ["a"]}

def node_b(state: State):
    return {"foo": "b", "bar": ["b"]}

workflow = StateGraph(State)
workflow.add_node(node_a)
workflow.add_node(node_b)
workflow.add_edge(START, "node_a")
workflow.add_edge("node_a", "node_b")
workflow.add_edge("node_b", END)

checkpointer = InMemorySaver()
graph = workflow.compile(checkpointer=checkpointer)
graph.invoke({"foo": "", "bar": []}, {"configurable": {"thread_id": "1"}})

Think of this subsystem like a video game that automatically saves your progress at every key moment. If the game crashes, you don’t start over from the title screen—you resume exactly where you left off, even if you were halfway through a boss fight. This is what “durable execution” does for software: it makes program crashes feel like nothing more than a brief pause, so the developer doesn’t have to write thousands of lines of code just to handle every possible failure.

Now zoom in: Temporal, a dedicated workflow engine, records every step of your program in something called an event history. After a crash, it replays that history to reconstruct the exact state—but it forces anything unpredictable, like an LLM call or a file write, into separate activities outside the replay logic. LangGraph, an agent framework, instead saves graph checkpoints—snapshots of the entire state at each step. These checkpoints let you pause for human approval or recover after a crash, but here’s the catch: a replay can re‑run the same node multiple times.

The single trickiest rule is that every node boundary must be engineered as a replay boundary. If a node contains a non‑deterministic operation (like reading the current time or sending an email), and the system replays that node, it might send the email twice or use a different time on the second run. LangGraph uses a checkpoint_ns field to track which subgraph a checkpoint belongs to, but that alone does not make the node safe—the developer still has to wrap side effects inside tasks or idempotent operations. Without this subsystem, a crash mid‑execution could cause duplicate charges, lost approvals, or inconsistent state, turning a simple bug into a real‑world mess.

The subsystem's ordered mechanism begins with a workflow engine like Temporal, where execution is driven by deterministic replay: workflow code runs, the service records every event into an event history, and after any crash the runtime replays that event history to reconstruct state exactly. Side effects such as LLM calls, shell commands, or API requests are pushed into separate Activities, never allowed inside deterministic workflow logic. In contrast, an agent framework like LangGraph persists graph state via checkpoints at each superstep using a BaseCheckpointSaver, and on failure it replays from the last checkpoint. The ordering halts at each node boundary; the node itself must be engineered as a replay boundary, because nondeterministic operations within that node may re-execute.

The invariant preserved by Temporal is deterministic replay: workflow code is deterministic, and the event history is the sole source of truth, guaranteeing that after a crash the replayed state is identical to the pre‑crash state. LangGraph preserves checkpoint state at node boundaries, but the context warns that "graph checkpoints are not automatically safe" — the guarantee is only that the checkpoint exists, not that replay within a node is safe. The design’s key trade‑off is between strict, provable safety and development flexibility. Temporal’s Workflow/Activity split forces all nondeterministic operations out of the core logic, rejecting the alternative of allowing them inline. That rejection avoids the cost of nondeterministic replay errors — such as diverging state from a mutated model response — but imposes a heavier abstraction. LangGraph rejects hard isolation in favor of durability modes ("exit", "async", "sync") that let developers balance performance against consistency, but that flexibility creates the cost that "the node boundary has to be engineered as a replay boundary".

A concrete failure mode occurs in LangGraph when using durability="exit": a process crash during a multi‑step graph run (for example, after an LLM call modifies a file but before the next superstep checkpoint) causes loss of that intermediate state because "intermediate state is not saved". The operator would see a missing checkpoint — the graph’s persisted state reflects only the last completed superstep, not the modification, and recovery replays from that earlier point, potentially repeating the file write. The signal is that the graph’s checkpoint store shows no entry for the current thread’s recent step, and the workflow resumes from a prior state. In Temporal, a crash inside an Activity is safe because the activity result is recorded; but if a developer mistakenly places a nondeterministic call (e.g. random() or direct LLM invocation) inside workflow code, the operator would see a non‑determinism error in the Temporal service logs, with the exact identifier “DeterminismViolationError” (implied by the invariant). The replay fails because the re‑executed code produces a different event history than what was stored.

Failure 1: Nondeterministic Operation Inside Temporal Workflow

• Trigger — An LLM call, shell command, or API request is placed directly inside a Temporal workflow function rather than being wrapped in an Activity.
• Guard — The source explicitly states that such operations “need to be Activities or otherwise recorded as durable external results.” The guard is the requirement to use the Activity abstraction.
• Posture — fail-hard: deterministic replay of the event history will produce a different outcome on recovery, causing the workflow to abort or produce inconsistent state.
• Operator signal — Temporal service logs a mismatch in event history during replay, or the workflow becomes stuck and never completes.
• Recovery — The workflow code must be rewritten to move the nondeterministic call into an Activity. After the change, the workflow can be retried (the Temporal service replays the corrected code against the stored history).

Failure 2: Node Failure Mid-Super-Step in LangGraph

• Trigger — One node in a LangGraph super‑step throws an exception while other nodes in the same super‑step complete successfully.
• Guard — The pending writes mechanism stores checkpoint writes from the healthy nodes; the source says “LangGraph stores pending checkpoint writes from any other nodes that completed successfully at that super‑step.”
• Posture — fail-soft: execution resumes from the failed super‑step without re‑running the successful nodes.
• Operator signal — The StateSnapshot for that step shows an error field in the tasks tuple for the failed node; the graph continues to the next step.
• Recovery — Automatic: when the graph is resumed, the runtime uses the stored pending writes to skip the already‑completed nodes and re‑executes only the failed node.

Failure 3: Process Crash During LangGraph Async Durability

• Trigger — A LangGraph graph is executed with durability="async", and the process crashes after a node finishes but before the asynchronous checkpoint write completes.
• Guard — The source warns that “there’s a small risk that LangGraph does not write checkpoints if the process crashes during execution.” No guard prevents this outcome; the only alternative is to use durability="sync".
• Posture — fail-soft: the run loses the state of the latest super‑step but can resume from the last successfully written checkpoint (which is from an earlier super‑step).
• Operator signal — Upon restart, the StateSnapshot shows an older checkpoint_id than expected; the step that was running at the time of the crash is absent.
• Recovery — The graph automatically retries from the last checkpoint recorded. The missing step may need to be re‑executed manually if the graph’s logic does not handle the jump back gracefully.

Failure 4: Checkpoint Storage Growth Due to Full State Channels

• Trigger — A LangGraph thread with long‑running conversations or large accumulations uses the default checkpointer, which writes the full value of every state channel at each super‑step.
• Guard — The source provides DeltaChannel as an optimization that “stores only incremental deltas instead of the full accumulated value, substantially reducing checkpoint size.” The guard is the explicit use of DeltaChannel via from langgraph.channels.delta import DeltaChannel.
• Posture — fail-soft: the system continues, but checkpoint writes become slower and storage grows without bound, eventually degrading performance.
• Operator signal — Growing disk usage for checkpoint storage; slower graph.stream() calls when checkpoints are large.
• Recovery — Manual code change: switch append‑heavy channels to DeltaChannel. No automatic migration; existing checkpoints remain large and can be pruned manually.

Failure 5: Human‑in‑the‑Loop Interrupt Lost with Exit Durability

• Trigger — A LangGraph graph uses durability="exit", a human‑approval interrupt occurs via NodeInterrupt, and the process crashes before the graph exits (successfully, with an error, or at the interrupt).
• Guard — The source states that with durability="exit", “intermediate state is not saved, so you cannot recover from system failures (like process crashes) that occur mid‑execution.” There is no guard — the capability does not exist.
• Posture — fail-hard: the interrupt is lost, and the graph cannot resume from the point of the human approval.
• Operator signal — After the crash, the graph shows no pending interrupt; the thread_id has no checkpoint reflecting the interrupt state. The human operator sees no pending approval request.
• Recovery — Manual intervention: the run must be restarted from the beginning or from a known good checkpoint that predates the interrupt. The human must re‑issue the approval.