Human-in-the-Loop

Every agent framework demo runs start to finish without stopping. Almost no production agent is allowed to. The moment an agent can send an email, move money, edit a file, or dismiss a job posting, someone asks the only question that matters to a reviewer: where does a human get to say no?

That question has a name — human-in-the-loop (HITL) — and it is not one feature. It is four distinct interaction shapes:

• Approval gate — the agent proposes an action and waits for yes/no before executing it. The classic "confirm before sending".
• Tool-call review — a specific tool is marked dangerous; every call to it is intercepted, shown to a human, and executed only on approval (possibly with edited arguments).
• State edit — a human inspects and corrects the agent's intermediate state (a wrong plan, a bad extraction) before the run continues.
• Multi-turn input — the agent discovers mid-run that it is missing information and asks for it, then continues with the answer.

Every framework on this page implements some subset of those shapes, and every one of them hides the same iceberg below the waterline: a pause that must survive until a human shows up. A human might answer in four seconds or four days. If your pause lives in a Python stack frame, it dies with the process — which turns human-in-the-loop into a durability problem wearing a UX hat.

This page is the framework-by-framework drill-down. For where on the autonomy spectrum to put a human — which decisions deserve a gate at all — see the autonomy field guide. For what it takes to make any pause survive a deploy, see durable execution. This page covers the third question: what primitive does each framework actually give you, and what does it quietly leave on your plate?

Why agents ask permission

Autonomy is not a binary, and the interesting engineering lives exactly at the seam where it stops. An agent that must ask before everything is a chat UI with extra steps; an agent that never asks is a liability with an API key. The useful middle — act freely on reversible things, pause on consequential ones — is what human-in-the-loop machinery implements.

The four shapes from the intro are worth keeping distinct, because frameworks support them unevenly:

Approval gates sit between steps: the plan is drawn, the draft is written, and the run parks until someone blesses it. This is the shape compliance teams ask for by name, and the one where the pause is most likely to last hours or days.

Tool-call review sits inside a step: the model has already decided to call delete_file(path="/prod/db") and the framework intercepts the call itself. The reviewer sees the exact arguments, not a summary — which is why several newer frameworks (the OpenAI Agents SDK, Pydantic AI, the Claude Agent SDK) chose the tool boundary as their only HITL surface. It is the narrowest gate with the highest information content.

State edits are the underrated shape. A reviewer who can only approve or reject a bad plan sends the agent back to square one; a reviewer who can fix the plan and resume converts a failed run into a cheap one. Only frameworks that expose mutable, persisted state (LangGraph most explicitly) support this well.

Multi-turn input inverts the direction: the agent asks, the human answers. Mechanically it is the same pause/resume machinery — which is the first hint that all four shapes reduce to the same underlying problem.

Here is the approval gate as a sequence — deceptively simple, and the diagram smuggles in the hard part as a single dashed arrow labelled "hours pass":

Everything between "persist" and "rehydrate" is the actual feature. The next chapter names its four mechanics; the framework chapters then grade each implementation against them.

Pause, persist, resume, identify

Strip away the API names and every human-in-the-loop implementation must solve the same four mechanics:

1. Pause. Stop a run mid-flight — between steps or mid-step — without losing what has been computed so far, and without burning a worker. A pause that holds a thread hostage (a blocking input() call, a spinning await with no persistence) only works while the human is already at the keyboard.

2. Persist. Write the paused run somewhere a different process can find it: message history, intermediate state, and a marker saying exactly where the run stopped and what it is waiting for. This is the mechanic that separates demo HITL from production HITL, and it is where most frameworks quietly hand the job back to you.

3. Resume. Take the human's payload — an approval, a rejection with a reason, edited arguments, a free-text answer — and inject it at the pause point, not at the beginning. Resume-by-replaying-the-whole-run and resume-from-the-exact-line are very different guarantees, and frameworks that replay (LangGraph re-executes the interrupted node from its top) turn that difference into a footgun you must design around.

4. Identify. When the approval arrives — from a Slack button, an email link, a web form — which of the thousand paused runs does it belong to? Every framework needs a durable handle: LangGraph's thread_id, LlamaIndex's serialized Context, the OpenAI SDK's stored RunState string. Your approval UI carries this handle in every link it renders.

Grade any framework's HITL story by asking four questions in order: can it stop? does the stop survive a restart? where does the answer re-enter? and what names the waiting run? The chapters that follow do exactly that, framework by framework — and the comparison table at the end is just this rubric in grid form.

LangGraph — the pause is a save point

LangGraph has the most complete pause story of any framework here, and it falls out of a decision covered in the framework comparison: LangGraph checkpoints graph state after every superstep. Once every step is a save point, "pause here and wait for a human" stops being a feature and becomes a one-liner — the run is already persisted; interrupt() just declines to continue it.

from langgraph.checkpoint.sqlite import SqliteSaver
from langgraph.graph import StateGraph, START, END
from langgraph.types import Command, interrupt


def human_review(state: State) -> Command:
    # Parks the WHOLE graph here. The value passed to interrupt() is
    # surfaced to the caller; the return value is the human's payload,
    # delivered on resume.
    decision = interrupt({
        "question": "Send this outreach email?",
        "draft": state["draft"],
    })
    if decision["action"] == "edit":
        return Command(goto="send", update={"draft": decision["draft"]})
    if decision["action"] == "approve":
        return Command(goto="send")
    return Command(goto=END)


builder = StateGraph(State)
builder.add_node("draft", draft_email)
builder.add_node("human_review", human_review)
builder.add_node("send", send_email)
builder.add_edge(START, "draft")
builder.add_edge("draft", "human_review")

graph = builder.compile(checkpointer=SqliteSaver.from_conn_string("state.db"))

config = {"configurable": {"thread_id": "outreach-42"}}
result = graph.invoke({"topic": "intro email"}, config)


# Hours (or a deploy) later, in a different process:
graph.invoke(Command(resume={"action": "approve"}), config)

Walk the rubric. Pause: interrupt(), callable anywhere inside a node, mid-logic, even in a loop. Persist: the checkpointer — swap SqliteSaver for the Postgres saver and the pause survives restarts, deploys, and horizontal scaling. Resume: Command(resume=payload); the payload becomes interrupt()'s return value. Identify: the thread_id in config — that string is what your approval email links carry. Four for four, and the same machinery covers all four HITL shapes: Command(goto=..., update={...}) on resume is the state-edit shape, and calling interrupt() with a question is multi-turn input.

The footgun: resume re-executes the node from its top. LangGraph does not freeze a Python stack frame — on resume it replays the interrupted node from its first line, and this time interrupt() returns the human's payload instead of parking. Every line above the interrupt() call runs twice. An LLM call or a charge placed before the interrupt in the same node will happen again on resume. The discipline: nodes that interrupt should do side effects after the interrupt line, or split the side effect into its own node so the checkpoint boundary protects it.

Two design consequences worth naming. First, because the interrupt payload passes through the checkpointer's serializer, it must be plain data — no live objects, no callbacks. Second, multiple interrupts can be pending at once (two parallel branches each asking for review); the __interrupt__ entry is a list, and resume payloads map to interrupts by id. The decision rule from the comparison page — durability, pause/resume, or human-in-the-loop → LangGraph — is really one rule, not three: HITL is durability, demanded by a user instead of a crash.

LlamaIndex Workflows — the pause is an event

LlamaIndex Workflows route typed events between steps, so its HITL primitive is — inevitably — a pair of events. A step that needs a human emits InputRequiredEvent and awaits ctx.wait_for_event(HumanResponseEvent); whoever drives the workflow catches the first on the event stream and sends the second back. The pause is just an unresolved await on the event queue.

from llama_index.core.workflow import (
    Context, HumanResponseEvent, InputRequiredEvent,
    StartEvent, StopEvent, Workflow, step,
)


class OutreachFlow(Workflow):
    @step
    async def draft(self, ctx: Context, ev: StartEvent) -> StopEvent:
        draft = await write_draft(ev.topic)
        response = await ctx.wait_for_event(
            HumanResponseEvent,
            waiter_id="approve-draft",
            waiter_event=InputRequiredEvent(
                prefix="Send this outreach email?", payload=draft,
            ),
        )
        if response.response == "approve":
            await send_email(draft)
        return StopEvent(result=response.response)


workflow = OutreachFlow()
handler = workflow.run(topic="intro email")

async for event in handler.stream_events():
    if isinstance(event, InputRequiredEvent):
        answer = input(event.prefix)          # stand-in for your real UI
        handler.ctx.send_event(HumanResponseEvent(response=answer))

result = await handler

The rubric. Pause: ctx.wait_for_event() — the step coroutine suspends; other steps keep running, so one pending approval does not freeze unrelated work. Persist: opt-in and manual. The waiting shown above lives in process memory. To survive a restart you serialize the workflow context — ctx.to_dict(serializer=JsonSerializer()) — store the dict, and later rebuild with Context.from_dict(workflow, data) and re-run; the waiter_id deduplicates the re-emitted request so the human is not asked twice. Resume: handler.ctx.send_event(HumanResponseEvent(...)) — typed, so arbitrary payloads ride in event fields. Identify: the serialized context blob is the handle; keying blobs by your own run id is your job.

So LlamaIndex scores four for four on mechanics but leaves persistence as an exercise — the exact exercise the deployed version solves with a Postgres-backed context store, which is the /llama-agents page's whole subject. Read that page for the ops half (server pauses on InputRequiredEvent, context snapshots in Postgres, resume across pod restarts); this chapter is the primitive underneath it.

One sharp edge mirrors LangGraph's, inverted. LangGraph replays the node, so pre-interrupt side effects run twice. LlamaIndex suspends a real coroutine, so nothing replays — but an in-process pause dies silently with the process unless you snapshotted the context before parking. LangGraph makes you defend against double execution; LlamaIndex makes you defend against forgetting to save. Pick which failure mode you'd rather design around.

CrewAI — the pause is a console prompt

CrewAI's headline HITL feature is one flag:

from crewai import Agent, Crew, Task

reviewer = Agent(
    role="Outreach writer",
    goal="Draft outreach emails worth sending",
    backstory="Writes short, specific, non-cringe cold emails.",
)

draft_task = Task(
    description="Draft an outreach email about {topic}",
    expected_output="A subject line and a 120-word body",
    agent=reviewer,
    human_input=True,   # <- the whole feature
)

crew = Crew(agents=[reviewer], tasks=[draft_task])
result = crew.kickoff(inputs={"topic": "intro email"})

When the task completes, the process blocks on stdin: the human reads the output in a terminal, types feedback, and the agent revises. As a feedback loop for a developer babysitting a crew locally, it is genuinely pleasant. As production HITL, it fails the rubric almost everywhere: the pause is a blocking read on a live process (pause: barely), nothing about the wait is written anywhere (persist: no), resume means typing into the same terminal (resume: same-process only), and there is no handle because there is no externally visible waiting state (identify: no). If the process dies while waiting, the run is gone. There is no web-app story in the primitive itself.

CrewAI's real answer for approval gates lives a level up, in Flows — the event-driven orchestration layer around crews. A flow can run a crew as one step, stop at a decision point, persist its state with @persist (SQLite by default), and route on the stored decision:

from crewai.flow.flow import Flow, listen, router, start, persist


@persist()
class OutreachFlow(Flow[OutreachState]):
    @start()
    def draft(self):
        self.state.draft = run_draft_crew()
        self.state.status = "awaiting_review"
        # stop here; your app notifies a human and later re-kicks the
        # flow with the decision recorded in state

    @router(draft)
    def route_on_review(self):
        return "send" if self.state.approved else "revise"

Note what happened: the framework did not pause for you — you restructured the workflow so it ends at the decision point and a later kickoff continues it. That is a fine pattern (it is essentially how you'd hand-roll HITL with a job queue), but it is your pattern to build and test, not a primitive you inherit. The honest summary: human_input=True is a developer-loop tool; production approval gates in CrewAI are a Flows architecture you design.

OpenAI Agents SDK — the pause is serialized state

The framework comparison found that the OpenAI Agents SDK has no durability story: the run lives in the process, full stop. Human-in-the-loop is the one place the SDK carves out an exception — because tool approval forced it to. You cannot intercept a tool call for review without somewhere to put the half-finished run while the human thinks.

The shape: mark a tool with needs_approval, and the run — instead of finishing — returns with interruptions:

from agents import Agent, Runner, function_tool
from agents.run_state import RunState


@function_tool(needs_approval=True)   # or an async callable for per-call logic
async def cancel_order(order_id: int) -> str:
    return f"Cancelled order {order_id}"


agent = Agent(name="Support", tools=[cancel_order])

result = await Runner.run(agent, "Cancel order 123")

if result.interruptions:
    state = result.to_state()
    serialized = state.to_string()        # -> your database, keyed by run id
    # ... a different process, after the human decides ...
    state = RunState.from_string(agent, serialized)
    for interruption in state.interruptions:
        # interruption carries tool_name + arguments — show them verbatim
        state.approve(interruption)       # or state.reject(interruption)
    result = await Runner.run(agent, state)

print(result.final_output)

Against the rubric: pause is the run returning early with interruptions — cooperative, clean, no blocked thread. Persist is state.to_string(): the entire run — history, pending tool calls, approval flags — becomes a JSON string you store wherever you like. Resume is Runner.run(agent, state) with the decisions recorded; approved calls execute, rejected ones return a rejection message to the model, which reacts accordingly. Identify is whatever key you filed the string under. It's a genuinely tidy design — the pause is a value, and values are easy to store, inspect, and ship across processes.

The limits are the mirror image of the design. Only the tool boundary pauses: this is tool-call review (and, via approve(..., always_approve=True)-style options, a lighter approval memory), not arbitrary mid-graph pauses, not state edits, not "stop between planning and execution" unless you make those things tools. And the serialized state is a snapshot you manage — there is no checkpointer writing every step, so a crash between pauses still loses the run. HITL got a durability carve-out; the rest of the run didn't.

AutoGen and AG2 — the human is another agent

AutoGen predates every other framework on this page, and its HITL design shows the era: instead of pausing machinery, the human is modeled as a participant in the conversation. A UserProxyAgent sits in the chat and, depending on human_input_mode, forwards messages to a real person:

from autogen import AssistantAgent, UserProxyAgent

assistant = AssistantAgent("assistant", llm_config={"model": "gpt-4o"})

user_proxy = UserProxyAgent(
    "human",
    human_input_mode="ALWAYS",   # ALWAYS | TERMINATE | NEVER
    code_execution_config=False,
)

user_proxy.initiate_chat(assistant, message="Draft an outreach email.")

The three modes are the whole dial. ALWAYS asks the human before every reply — maximum oversight, chat-with-extra-steps. TERMINATE runs autonomously and only asks when the conversation would otherwise end — a review gate at the finish line. NEVER is full autonomy. In the newer Microsoft AutoGen (AgentChat, the 0.4+ rewrite), the same idea survives as a UserProxyAgent with a pluggable input_func — swap the default console prompt for a callback into your UI. AG2 — the community fork that kept the original autogen package name and API after the projects split in late 2024 — keeps the classic shape above; Microsoft's own line has since folded into the Microsoft Agent Framework alongside Semantic Kernel. Two lineages, one design idea.

Conceptually the framing is lovely — approval, correction, and extra input are all just messages, so every HITL shape collapses into "the human says something". Mechanically, the rubric grades it like CrewAI: the default input function blocks a live process on the console (pause: blocking), the wait is not persisted (persist: no), and resume means the same process is still alive when the human answers (identify: the open socket). A custom input_func can bridge to a web UI, but making that durable — parking the conversation state while the human is away — is again an exercise for your infrastructure. The human-as-agent framing solved the interaction design; it never claimed to solve the durability problem underneath it.

Pydantic AI and Claude Agent SDK — approval at the tool boundary

Two newer SDKs land on the same conviction from different directions: the tool call is the only boundary worth gating, so make that gate excellent.

Pydantic AI makes the pause a typed output. Tools marked requires_approval=True don't fire — instead the run ends, returning a DeferredToolRequests object listing the calls that need sign-off. There is no suspended coroutine to keep alive: the run is over, and the "pause" is whatever you do with the returned value. A later run carries the decisions back in:

from pydantic_ai import (
    Agent, DeferredToolRequests, DeferredToolResults, ToolApproved, ToolDenied,
)

agent = Agent("anthropic:claude-opus-5", output_type=[str, DeferredToolRequests])


@agent.tool_plain(requires_approval=True)
def delete_file(path: str) -> str:
    return f"deleted {path}"


result = agent.run_sync("Clean up the temp directory")

if isinstance(result.output, DeferredToolRequests):
    messages = result.all_messages()      # persist these + the requests
    decisions = DeferredToolResults()
    for call in result.output.approvals:
        decisions.approvals[call.tool_call_id] = (
            ToolApproved() if is_safe(call) else ToolDenied("Not allowed")
        )
    result = agent.run_sync(
        message_history=messages, deferred_tool_results=decisions,
    )

Because the first run genuinely ends, persistence is trivially honest: the message history and the requests are plain data; store them anywhere, resume from any process, days later. Approve-with-edited-args (ToolApproved(override_args=...)) covers the state-edit shape at the tool level, and ToolDenied's message goes back to the model as the tool result, so a rejection steers rather than crashes the run.

The Claude Agent SDK ships the permission system that Claude Code runs on, as a library. HITL is a callback: can_use_tool fires when no allow/deny rule, hook, or permission mode has already settled a tool call, and your code decides — allow (optionally with modified inputs), or deny with a reason the model sees:

from claude_agent_sdk import ClaudeAgentOptions, query


async def can_use_tool(tool_name, input_data, context):
    if tool_name == "Bash" and is_destructive(input_data["command"]):
        decision = await ask_human(tool_name, input_data)   # your UI
        if not decision.approved:
            return {"behavior": "deny", "message": decision.reason}
    return {"behavior": "allow", "updatedInput": input_data}


async for message in query(
    prompt="Tidy up the repo",
    options=ClaudeAgentOptions(
        permission_mode="default",      # default | acceptEdits | plan | bypassPermissions
        can_use_tool=can_use_tool,
    ),
):
    ...

What is distinctive is the layering: hooks run first, then deny rules, ask rules, the permission mode, allow rules — and only unresolved calls reach your callback. That yields graduated autonomy out of the box (auto-approve reads, gate writes, forbid the deploy command everywhere, loosen the mode as trust builds mid-session via set_permission_mode), with the well-documented sharp edge that an allow rule or permissive mode silently bypasses your callback — approval logic that must always run belongs in a PreToolUse hook instead. The callback itself is in-process and the session is live while it waits, so long absences are handled by the session-resume machinery rather than by parking the callback: durable multi-day gates are Pydantic AI's strength; rich, layered, interactive permissioning is this SDK's.

The comparison and the production checklist

One table, rubric against framework:

The decision rule, said out loud: if the human might be away longer than the process, you need durable HITL — LangGraph if the pause can land anywhere in a graph, Pydantic AI or the OpenAI Agents SDK if gating tool calls is enough (Pydantic AI's run-ends design is the cleanest to persist). If a human is present at a terminal, CrewAI's and AutoGen's console loops are honest, useful tools. If you need graduated permissions richer than one yes/no — rules, modes, hooks, then a callback — the Claude Agent SDK is the only one that ships that layering.

And then the part no framework solves, because it lives in your product, not their runtime:

• Timeouts and escalation. Every pause needs an answer to "what if nobody responds?" — expire and abandon, escalate to another approver, or proceed with the safe default. A pause without a timeout is a leak.
• Notification. interrupt() does not send the Slack message. The pause is only as good as the ping that tells a human it exists, and the deep link that carries the run handle back.
• Audit. Who approved what, when, seeing which exact arguments? If the gate exists for compliance, the record of the gate is the deliverable — store decision, identity, and payload next to the run.
• Idempotent resume. Approval UIs get double-clicked, email links get re-opened, Slack retries deliveries. Resuming the same pause twice must be a no-op, not a second wire transfer. LangGraph's replay footgun is one face of this; de-duplicating the resume call itself is another. This is where HITL hands back to durable execution — same iceberg, next chapter of it.

The one-sentence takeaway: every framework can stop; the ones worth trusting in production are the ones that can still find the run after everyone has gone home.