Computer Use — Deep Dive

Error handling middleware for tool calls catches exceptions and returns a ToolMessage.

@wrap_tool_call
def handle_tool_errors(
    request: ToolCallRequest,
    handler: Callable[[ToolCallRequest], ToolMessage],
) -> ToolMessage:
    """Convert tool exceptions into ToolMessages the model can handle."""
    try:
        return handler(request)
    except Exception as e:
        return ToolMessage(
            content=f"Tool error: Please check your input and try again. ({e})",
            tool_call_id=request.tool_call["id"],
        )

Imagine a master chef in a kitchen with hundreds of utensils hanging on the wall. To pick the right one, the chef doesn't fumble through every drawer—instead, a single mental “handle” flashes in their mind for each tool, guiding their hand directly. This is what modern language models do when they choose which external tool to call. The system’s purpose is to let a model reliably pick the correct tool from a huge set—like opening a specific file, running a calculation, or sending a message—without guessing blindly.

Here’s how it actually works step by step. Inside the model’s internal state—its activation space—each pair of tools has a unique direction, like a labeled arrow. Researchers can read which tool the model is about to choose by using a technique called cosine readout: they compare the model’s current internal state against stored directions for each tool, achieving 61–82% accuracy on the BFCL benchmark, even though the model’s natural output gets only 2–10% right. That means the model already “knows” the right tool deep inside before it speaks. Even more, by adding that direction during generation—adding a tiny nudge—the model switches to the new tool, and its JSON arguments automatically adjust to the new tool’s format. This is like gently nudging the chef’s mental handle so they reach for the whisk instead of the spoon, and then they automatically use the whisk correctly.

The trickiest part is why this works so cleanly. The direction is not just a vague topic signal; random vectors at the same strength give zero switch rate. Instead, the direction is tied to the specific pair of tools—it’s a precise representation learned during pretraining, long before any fine-tuning. Even base models carry these directions, and instruction tuning only teaches the model to say the tool name out loud. Without this internal subsystem, a model might confidently output a wrong tool call—like ordering a money transfer when it meant to check a balance—and the mistake is invisible until it executes, causing real damage. With the readout, we can catch that uncertainty before it happens: queries where the model wavers between two tools fail twenty-one times more often.

The subsystem operates through a layered mechanism that begins with the model’s internal representation of tool choice. Evidence from the “From Text to Voice” study shows that even before a model emits a tool call, the chosen tool is already encoded as a single direction in its internal state; reading this direction via a cosine readout recovers 61–82% accuracy on BFCL, while standard generation yields only 2–10%. This representation is then acted upon by the Activation Steering Adapter (ASA), a training-free, inference-time controller that performs a single-shot mid-layer intervention. ASA uses a router-conditioned mixture of steering vectors with a probe-guided signed gate to amplify the correct tool intent while suppressing spurious triggers. On failure—for example, when the model remains too conservative and fails to enter tool mode, known as the Lazy Agent failure mode—the coupling to the Model Context Protocol (MCP) becomes critical. MCP provides a session-oriented, JSON-RPC framework that negotiates capabilities and invokes external tools under OAuth 2.1‑compliant access control. If the internal steering does not result in a valid tool call, MCP’s negotiation layer can detect the mismatch and fall back to a safe default, though the context does not specify an explicit recovery procedure beyond the access-control enforcement.

The design preserves the invariant that the model’s internal representation of tool necessity is accurate and decodable, even when the behavioral output is unreliable. This is captured in the representation-behavior gap identified by the Lazy Agent failure mode: tool necessity is nearly perfectly decodable from mid-layer activations, yet the model remains conservative in entering tool mode. The guarantee is that ASA can bridge this gap without modifying the model’s weights, so the internal representation remains untouched while the output behavior is corrected. Additionally, MCP’s session-oriented architecture ensures that every tool invocation is bound to a specific context and access rights, providing a write boundary enforced by the OAuth 2.1 framework—any tool call outside the negotiated capabilities is blocked at the protocol level.

The key trade-off is the decision to reject parameter‑efficient fine‑tuning (PEFT) in favor of a training‑free, inference‑time intervention. The ASA paper explicitly dismisses PEFT because it requires ongoing training, maintenance, and risks catastrophic forgetting. Instead, ASA uses only about 20KB of portable assets with no weight updates, achieving a strict tool‑use F1 improvement from 0.18 to 0.50 on MTU-Bench with Qwen2.5-1.5B while reducing the false positive rate from 0.15 to 0.05. The obvious alternative—fine‑tuning the model—would require full‑model or adapter training, incurring high computational cost and the danger of overwriting pre‑learned representations. By rejecting that cost, the design gains portability and avoids forgetting, but it accepts that ASA works only when the internal representation is already correct, which is largely true (cosine readout accuracy is high) but not guaranteed in all multi‑turn loops, where the same intervention shows inconsistent results (matched‑baseline gain or loss of up to 30 percentage points).

A concrete failure mode in this subsystem is a false positive tool call that triggers an irreversible action, such as transferring money or deleting data. The operator would see this as an increase in the false positive rate metric tracked by benchmarks like MTU-Bench. Without ASA, the false positive rate sits at 0.15; after ASA intervention, it drops to 0.05. An operator monitoring the system would observe logs from the MCP client showing an invocation of a tool (e.g., delete_data) that was not intended by the task, accompanied by a confidence score from the probe‑guided signed gate. The signal would be a spike in the “false positive” count in the MTU-Bench evaluation dashboard or, in a production setting, an alert from the OAuth 2.1 layer if the call violates the session’s capability scope—though here the failure is that the call is permitted but wrong. The mismatch between the high internal cosine readout accuracy (61–82%) and the low generation accuracy (2–10%) would be visible in ablation logs, indicating that the representation was correct but the output was not steered.

False Positive Tool Call (Lazy Agent Mode)

• Trigger — Distribution shift, strict parsers, or fragile prompt engineering cause the model to spuriously trigger a tool when none is needed, as identified in the Lazy Agent failure mode — the model remains conservative in entering tool mode yet false positives still occur.
• Guard — The probe-guided signed gate from the ASA controller amplifies true intent while suppressing spurious triggers; it operates as a single-shot mid-layer intervention.
• Posture — Fail-closed: the gate is designed to block the spurious call. If the gate functions correctly the tool is never invoked, refusing the execution. If the gate itself fails (e.g., due to an out-of‑distribution input), the false‑positive call may still proceed — the source does not provide a second layer of protection.
• Operator signal — The false positive rate metric drops from 0.15 to 0.05 when the gate is active; an unexpected increase in that rate signals the gate’s failure.
• Recovery — No retry or backoff is specified in the source. After a false positive call executes, the irreversible action (e.g., sending email, transferring money) cannot be undone automatically; manual rollback is required.

False Negative Tool Call (Failure to Enter Tool Mode)

• Trigger — The model’s conservative behavior (the representation‑behavior gap) prevents it from entering tool mode even when a tool is necessary. Conditions include ambiguous input or domain shifts that the base model’s activation space does not easily decode.
• Guard — The router-conditioned mixture of steering vectors in ASA applies a domain‑targeted intervention to push the model into tool mode. It is a training‑free, inference‑time controller that performs a single‑shot activation steering.
• Posture — Fail‑soft: if the steering does not succeed, the model remains in text‑only mode and tries to answer without the tool. The task may degrade (e.g., produce a low‑quality or non‑functional output) but the run continues.
• Operator signal — The tool‑use F1 on MTU‑Bench jumps from 0.18 to 0.50 when the guard is applied; a low or stagnating F1 score indicates that the steer failed to activate the correct tool‑calling behavior.
• Recovery — No retry is specified. Because ASA is a one‑shot intervention, a failed steer leaves the model in its base state; the operator must manually re‑prompt or apply a different steering vector (not detailed in the source).

Wrong Tool Selection (Picking an Incorrect Tool from Many)

• Trigger — With 188 tools across 33 MCP servers, the model’s internal tool‑choice direction becomes ambiguous. The source shows that queries where the model is unsure between two tools fail 21× more often than unambiguous queries.
• Guard — The cosine readout technique reads the intended tool off the model’s hidden states (achieving 61‑89% top‑1 accuracy across multiple models on a 14‑tool airline domain). When the readout indicates low confidence, the system can flag the call before execution.
• Posture — Fail‑closed (if the guard is triggered): the source does not specify an automatic block, but the readout can be used to refuse the call. If the guard is not consulted, the wrong tool is invoked — a fail‑hard scenario because the irreversible action (e.g., booking the wrong flight) may already be executed.
• Operator signal — The τ‑bench airline accuracy sits at 77‑94% for 4B+ models; a drop below that threshold or a high frequency of tool‑name mismatches in logs would indicate the failure.
• Recovery — No retry is embedded. The operator must manually revert the erroneous action (e.g., cancel the booking) and re‑submit the query with an explicit tool hint.

Uncertainty‑Gated Execution Without Semantic Token Filtering

• Trigger — An LLM calls a function with low confidence but the uncertainty quantification method used does not filter out semantically meaningless tokens, leading to a falsely high confidence score and an incorrect call.
• Guard — The logit‑based uncertainty scores improved by selecting only semantically meaningful tokens (as described in the UQ for FC paper). In contrast, Semantic Entropy (a multi‑sample method) shows no clear advantage in the function‑calling setting.
• Posture — Fail‑soft (if the guard is applied): the system can refuse to execute the call when the uncertainty score exceeds a threshold. If the guard is missing or the token filter is not used, the unconfident call goes through — a fail‑hard outcome.
• Operator signal — An abnormally high uncertainty score that was ignored, or a spike in false positive rate after execution. The source also notes that multi‑sample UQ benefits from clustering on abstract syntax tree parsing — absence of that clustering is another silent signal.
• Recovery — The source does not specify a retry mechanism. The operator must manually inspect the executed call and, if erroneous, perform a compensating action (e.g., void a transaction).

Steering Instability in Multi‑Turn Agent Loops

• Trigger — The single‑shot activation steering that works well in single‑turn settings becomes unstable in multi‑turn loops, causing a matched‑baseline gain or loss of up to 30 percentage points with no consistent direction.
• Guard — No guard is provided in the source. The paper simply reports the instability and notes that the same intervention is less stable on multi‑turn scenarios.
• Posture — Fail‑soft: the system continues to operate, but task performance may degrade unpredictably (gain or loss). There is no mechanism to fall back to a stable mode.
• Operator signal — Large, random fluctuations in metrics such as tool‑use F1 or τ‑bench accuracy across turns, with no consistent improvement over the baseline.
• Recovery — No recovery logic exists. The operator must disable the steering intervention for multi‑turn tasks or rely on the base model’s self‑perceived baseline, which the source says yields weaker task performance for most open‑source models.

An agent tool that uses ToolRuntime to access state and returns a Command to update conversation state, with a unique tool call ID for tracking.

from langchain.messages import ToolMessage
from langchain.tools import ToolRuntime, tool
from langgraph.types import Command


@tool
def set_language(language: str, runtime: ToolRuntime) -> Command:
    """Set the preferred response language."""
    return Command(
        update={
            "preferred_language": language,
            "messages": [
                ToolMessage(
                    content=f"Language set to {language}.",
                    tool_call_id=runtime.tool_call_id,
                )
            ],
        }
    )

Think of the screenshot action loop as a person navigating a new city with a live-updating map. Every time they take a step, the map refreshes to show exactly where they are now and what’s around them. That map is the agent’s current state, and it also remembers where they’ve been through a history of past messages. The person uses these two things to decide their next move—turning left, entering a shop, or hailing a cab. In this system, the agent does the same: it first looks at the current state and the message history, then picks a tool from a set of available actions, like moving a cursor or calling a simple function. That tool call is sent to a runtime that gives it access to the state, the conversation context, and user information. After the tool runs, the state updates, and the agent sees the new picture before choosing its next step. This loop keeps going, with the state acting as short‑term memory that holds everything the agent needs to make coherent decisions.

Going deeper, the agent doesn’t just guess which tool to use—it systematically transforms tool documentation into callable functions, checking syntax and runtime correctness before integrating them into executable programs. This is how the runtime knows exactly what each tool does and how to run it. The agent then maps its natural‑language reasoning directly onto these functions, so a phrase like “click the submit button” becomes a precise call to a click(x, y) function. The state update after every action is what keeps the map fresh: the runtime modifies the state object so the agent never repeats a stale decision.

The trickiest part is understanding how the agent’s brain actually chooses which tool to call. Research shows that inside the model, the decision is carried by a single direction in its activation space—like a hidden pointer that says “use tool A, not tool B.” Adding that direction during generation can flip the selection with over 80% accuracy, and reading it from the model’s internal state can predict the tool before it’s even spoken. This means the agent can catch its own uncertainty early: when it’s unsure between two tools, those queries fail far more often, and clustering tool outputs by their structural parse helps measure that confidence. Without this loop, the agent would have no way to actually execute actions—it would only talk about clicking buttons without ever moving the cursor. The map would never update, so it would keep staring at the same screen, repeating the same ineffective decision, stuck in a loop that never touches the real world.

The Screenshot Action Loop is built on the MCP-AgentBench framework, where an agent iteratively selects and executes tools via Model Context Protocol (MCP) servers. The ordered mechanism begins with the agent consuming the current state and message history. It then chooses a tool from the 188 distinct tools spread across 33 operational servers. The tool call is dispatched to the runtime, which injects state, context, and user information. After execution, the state updates and the agent re-reads it to decide the next tool. On failure—for instance, if the tool call returns an error or an incorrect output—the loop does not retry automatically; instead the agent observes the new state and may attempt a different tool or reattempt the same call. This matches the outcome-oriented evaluation methodology introduced as MCP-Eval, which prioritizes real-world task success over intermediate metrics.

The invariant the design preserves is task success as the primary guarantee, as captured by MCP-Eval’s focus on whether the final state meets the user’s goal. This implicitly enforces a boundary on write semantics: a tool call is considered effective only if the resulting state advances the task toward completion. There is no explicit exactly-once or idempotency mechanism; the loop relies on the agent’s ability to reason from the updated state and, if needed, correct prior mistakes. The Activation Steering Adapter (ASA) from a related paper further reinforces this invariant by conditioning the model’s tool selection on mid-layer activations, ensuring that the agent does not lazily refuse tool entry or spuriously call tools. ASA’s probe-guided signed gate amplifies true intent while suppressing spurious triggers, thereby reducing the false positive rate from 0.15 to 0.05 on MTU-Bench with Qwen2.5-1.5B.

The key trade-off is between standardization and flexibility. By adopting MCP as the mediation protocol, the design rejects ad-hoc, per-tool integration that would require custom runtime adapters for each new service. This rejection avoids the cost of maintaining fragile, non-interoperable interfaces and the development overhead of continuous schema engineering. However, the price is a rigid protocol layer that can introduce latency and restrict the kinds of tools that can be exposed—especially those requiring real-time feedback or low-level hardware interaction. The alternative, direct function-calling without a uniform context protocol, would offer lower overhead for a single tool but fail to scale across the 33 servers and 188 tools that MCP-AgentBench orchestrates.

A concrete failure mode occurs when the agent’s internal representation ambiguously reads multiple tools, a phenomenon detailed in the Tool Calling is Linearly Readable work. There, per-tool directions in activation space become confused during generation—queries where the model is unsure between two tools fail 21× more often than single‑choice queries (e.g., on Gemma 3 27B with τ-bench airline). The operator would see a logged tool call that names a wrong function (e.g., “book_flight” instead of “check_availability”) while the agent’s confidence is low. The cosine readout probe, which recovers 61–82% accuracy on BFCL from hidden states, would flag the mismatch before execution, but the loop itself proceeds uncorrected until the state update reveals the error. An operator monitoring logs would observe the incorrect tool name and a subsequent state that does not match the expected outcome, triggering a manual rollback or re‑attempt.

Lazy Agent Failure Mode

• Trigger — Distribution shift or strict parsers cause the model to remain in text mode despite having internally decoded that a tool is necessary (representation–behavior gap).
• Guard — No guard identified in the source. The ASA technique (activation steering) is a proposed solution, not a runtime handler in the described loop.
• Posture — Fail‑soft: the agent continues generating text but omits the needed tool call, degrading task performance without aborting.
• Operator signal — A low strict tool‑use F1 (e.g., 0.18 on MTU‑Bench) and a high false positive rate (e.g., 0.15) that signal missed tool calls.
• Recovery — No automatic recovery; the loop proceeds to the next turn without the tool output. Manual inspection or re‑prompting is required to insert the missing call.

Wrong Tool Selection

• Trigger — The model is unsure between two tools (queries where it is unsure fail 21× more often), often because activation‑space directions for the two tools are close.
• Guard — No guard identified in the source. The paper proposes per‑tool activation directions to flip the choice, but these are not part of the described runtime loop.
• Posture — Fail‑soft (if the wrong tool’s action is non‑critical) or fail‑hard (if the executed action is irreversible, e.g., transferring money or deleting data — effects described in the UQ paper). The runtime does not intercept the call.
• Operator signal — Silent until execution: the failure is invisible until the tool returns unexpected results or causes damage. No log line is emitted beforehand.
• Recovery — No retry or fallback in the loop. Manual rollback or undo is needed; the UQ paper notes that incorrect function calls can have “severe implications” when effects are irreversible.

Unnecessary/Redundant Tool Call

• Trigger — The model’s self‑perceived need or utility is misaligned with the true need; it calls a tool when the internal knowledge already suffices or when the tool’s output would be noisy.
• Guard — No guard identified in the source. The paper trains “lightweight estimators of need and utility” from hidden states and drives “simple controllers” to improve the decision, but those are not deployed in the baseline loop described.
• Posture — Fail‑soft: the tool call executes anyway, wasting time or injecting noise, but the loop continues.
• Operator signal — Increased latency and degraded final task performance; no explicit error metric for unnecessary calls. The absence of a signal that the call was redundant.
• Recovery — No automatic recovery; the loop processes the (potentially harmful) tool output and moves on.

Incorrect Tool Arguments (Argument Schema Mismatch)

• Trigger — When the agent switches to a different tool (e.g., due to activation steering or uncertainty), the JSON arguments may fail to adapt correctly to the new tool’s schema.
• Guard — No guard identified in the source. The paper notes that “arguments automatically adapt” when the tool name is flipped, but no validation or retry mechanism is described.
• Posture — Fail‑hard if the runtime rejects malformed arguments (e.g., parser failure) and aborts the call, causing the agent to stall.
• Operator signal — A runtime error message (e.g., JSON parse error or invalid parameter) or a silent hang if no exception is raised.
• Recovery — No retry or fallback in the described loop; the agent may loop indefinitely or require manual correction of the argument string.

Error handling middleware catches tool call mistakes, addressing the gap between internal knowledge and actual output.

@wrap_tool_call
def handle_tool_errors(
    request: ToolCallRequest,
    handler: Callable[[ToolCallRequest], ToolMessage],
) -> ToolMessage:
    """Convert tool exceptions into ToolMessages the model can handle."""
    try:
        return handler(request)
    except Exception as e:
        return ToolMessage(
            content=f"Tool error: Please check your input and try again. ({e})",
            tool_call_id=request.tool_call["id"],
        )


agent = create_agent(
    model="ollama:devstral-2",
    tools=[],
    middleware=[handle_tool_errors],
)

Picking up a phone to dial a friend, your brain knows the number but your finger taps the wrong digit—the intention is correct, the action is not. This subsystem exposes that hidden intention so we can catch the mistake before it happens: it reads what a language model truly knows inside, even when the model outputs something else.

Inside the model, each possible tool is stored as a specific activation direction, like a mental pointer to the right action. Researchers read that pointer using a cosine readout—a simple measurement of the model’s internal state—and found it predicts the correct tool with 61–82% accuracy even when the model’s final output hits only 2–10% accuracy. When the model is torn between two tools, the internal signal gets confused, and errors happen 21 times more often. The subsystem watches for that confusion and flags likely failures before the wrong call is executed.

The non-obvious rule is that this internal direction is linearly separable—a single vector that, when added during generation, forces the model to switch to the correct tool. The hard part is that even when the model knows which tool to use, it can still mess up the argument values (like the recipient or amount in an email or payment), causing a performance drop of nearly five points when switching from text to voice input. Without this subsystem, a model that internally knows to call a search tool might instead call a delete function, silently erasing data or sending wrong messages—failures you only discover after the damage is done.

The subsystem operates through a two-phase mechanism that first probes internal representations and then intervenes before output generation. Initially, the system reads the model’s hidden-state activations using a cosine readout or a probe-guided signed gate to detect which tool the model internally “knows” is correct. If this internal signal conflicts with the model’s tendency to remain conservative—a failure mode termed Lazy Agent—the system applies a single-shot mid-layer intervention via an Activation Steering Adapter (ASA). This adapter uses a router-conditioned mixture of steering vectors to shift the activation toward the correct tool domain. On failure, if the steering vector does not reliably flip the tool choice (e.g., when per-tool directions yield a switch rate below 83%), the model proceeds with its original output, leading to an incorrect tool call.

The invariant preserved by this design is the improvement in strict tool-use F1 while simultaneously reducing the false positive rate. ASA demonstrates this guarantee by raising strict tool-use F1 from 0.18 to 0.50 and dropping the false positive rate from 0.15 to 0.05 on the MTU-Bench benchmark. Additionally, the use of per-tool directions provides a strong guarantee: adding the correct direction during generation switches which tool the model picks with 83–100% accuracy on instruction-tuned models of 4B parameters and above. This ensures that the internal knowledge is reliably translated into the correct output, as long as the steering vector is properly calibrated.

The key trade-off is between training-free inference-time control and continual parameter-efficient fine-tuning. ASA explicitly rejects fine-tuning because of its training, maintenance, and potential forgetting costs. Instead, the system chooses a lightweight, portable intervention that uses only about 20KB of portable assets and no weight updates. The obvious alternative—continual fine-tuning—would require retraining on each domain shift, storing multiple adapter weights, and risking catastrophic forgetting of previously learned tool behaviors. By steering activations at inference time, ASA avoids this overhead while still achieving substantial gains in reliability.

A concrete failure mode occurs in the Lazy Agent scenario: the model’s internal representation decodes the correct tool from mid-layer activations with high accuracy, yet the model refuses to enter tool mode and outputs a different action. An operator monitoring the system would see a false negative—the model does not call a needed tool—manifesting as a low recall component in the tool-use F1 metric. If the steering vector is too aggressive, the opposite can happen: a false positive where the model calls a tool unnecessarily. However, ASA’s probe-guided signed gate is specifically designed to suppress such spurious triggers by amplifying true intent signals while filtering out noise, and the empirical reduction of the false positive rate from 0.15 to 0.05 demonstrates this mitigation. The operator would detect this failure by observing that the false positive rate remains above the expected threshold after steering is applied.

Lazy Agent Failure Mode

• Trigger — The model’s internal representation (mid‑layer activations) indicates a tool is necessary, but the model stays conservative and does not emit the tool call.
• Guard — Activation Steering Adapter (ASA) – a single‑shot mid‑layer intervention with a probe‑guided signed gate that amplifies true intent and suppresses spurious triggers.
• Posture — fail‑soft – the model continues without the correct tool call, producing degraded task completion; the guard partially recovers correctness.
• Operator signal — strict tool‑use F1 at 0.18 (the baseline reported before ASA intervention, improving to 0.50).
• Recovery — ASA applies its steering vectors automatically at inference time; no retry or backoff is described.

False Positive Tool Calls

• Trigger — The model outputs a tool call when the tool is not actually needed (spurious trigger), often because of distribution shift or strict parsers.
• Guard — ASA’s probe‑guided signed gate suppresses these spurious triggers.
• Posture — fail‑soft – the false tool call may be executed, but the system continues; the guard reduces occurrence.
• Operator signal — false positive rate (reduced from 0.15 to 0.05 after ASA).
• Recovery — ASA automatically filters the spurious call; no manual step or retry is mentioned.

Misunderstanding of Argument Values in Speech

• Trigger — The model transitions from text to speech; it correctly identifies the tool to call but misinterprets argument values in the spoken utterance.
• Guard — None shown. The source only reports that “degradations most often reflect misunderstandings of argument values in the speech” and does not provide a handler, retry, or validation.
• Posture — fail‑soft – the model calls the wrong argument values, producing an incorrect but non‑aborting effect.
• Operator signal — text‑to‑voice gap (e.g., 4.8 points drop for GPT‑Realtime‑1.5 on Confetti).
• Recovery — No automatic recovery; manual re‑prompting or verification is required.

Irreversible Incorrect Function Call

• Trigger — The LLM calls a function (e.g., transferring money, deleting data) with high confidence but the call solves the task incorrectly, and the effect is irreversible.
• Guard — Semantic Entropy (multi‑sample UQ method) quantifies confidence; it can be used to prevent execution of the call. The source notes that “UQ methods can be used to quantify this confidence and prevent potentially incorrect function calls.”
• Posture — fail‑hard – if the uncertainty score exceeds a threshold, the call is blocked and the run is aborted or held.
• Operator signal — An uncertainty score (e.g., from Semantic Entropy) flagged above a threshold, or a blocked call log.
• Recovery — The guard prevents execution; manual operator re‑evaluation or a fallback plan is needed (no automatic retry described).

Single-turn agent using return_direct, representing the simple benchmarks that models saturate.

@tool(return_direct=True)
def fetch_order_status(order_id: str) -> str:
    """Fetch the current status of a customer order."""
    return f"Order {order_id} is shipped and will arrive in 2 days."

agent = create_agent(
    ChatOpenAI(model="ollama:devstral-2"),
    tools=[fetch_order_status],
)

result = agent.invoke({
    "messages": [{"role": "user", "content": "What is the status of order #12345?"}]
})

Think of a chef who can perfectly crack an egg in a one-second test, but when asked to cook a full three-course meal, suddenly grabs a potato peeler to stir the soup. That’s the core idea: a system that works flawlessly on a single, simple job often stumbles when the job stretches across many steps. This subsystem is for measuring and improving how well tool-using AIs hold up when the task gets long and complicated, not just short and easy.

In practice, early checkups only tested the AI on one tool call at a time with a fixed menu of options—like asking the chef to use only a whisk once. Many AIs aced those short tests. But real work—say, booking a flight, changing the time, then canceling a hotel—needs many tool calls in a row. When researchers poked at the AI in those multi-turn loops, they saw it become unstable: performance could swing by thirty percentage points with no clear pattern. A trick that helped in a single call might actually hurt in the longer sequence. That’s because the AI’s internal decision about which tool to pick (a mechanism called "tool calling") is readable as a single direction in its activation space during simple tasks, but in multi-turn loops that reading gets noisy and unreliable.

The trickiest part is why that instability happens. Inside the AI, the choice of tool is carried by a specific direction in its hidden states—a kind of internal pointer. In a single-turn test, adding that direction flips the tool with 83–100% accuracy. But in a multi-turn loop, the same intervention yields a matched-baseline gain or loss of up to thirty percentage points with no consistent direction. The AI’s own confidence also wavers: when the model is unsure between two tools, it fails 21 times more often. Without this subsystem, a beginner would see the AI nail the first step, then on step two it sends the wrong email or deletes the wrong file—a failure invisible until it happens, because the simple benchmark never warned us.

The subsystem operates as follows. First, the LLM processes the input through its forward pass. Second, a cosine readout—decoding the chosen tool directly from mid-layer hidden states—recovers the intended tool with 61–82% accuracy on BFCL, while the model’s own generated output lags at 2–10%. Third, if the readout points to a different tool than the generation, a training-free intervention is applied: either adding a single steering vector from the Activation Steering Adapter (ASA) (which uses a probe-guided signed gate) or adding the per-tool direction identified in the “Tool Calling is Linearly Readable and Steerable” work. On failure—when the intervention is applied in a multi-turn agent loop—the subsystem becomes unstable, producing a matched-baseline gain or loss of up to 30 percentage points with no consistent direction.

The invariant the design seeks to preserve is the linear decodability of tool choice from the model’s internal state. Across 12 instruction-tuned and 6 base models (270M to 27B parameters), the tool selection is carried by a single direction in activation space per tool pair, and a probe within a single domain (14 airline tools) still reads the tool at top-1 accuracy of 61–89%. The guarantee is that the correct tool representation exists internally before generation, as shown by the cosine readout; however, this invariant holds only in single-turn, fixed-menu settings and breaks under the longer, multi-turn loops of realistic benchmarks.

The key trade-off is choosing hidden-state readout and steering over two obvious alternatives: (1) prompt/schema engineering, which is fragile under distribution shift and strict parsers, and (2) continual parameter-efficient fine-tuning, which incurs training, maintenance, and forgetting costs. The rejected alternatives are avoided because the subsystem exploits representations already formed during pretraining—the cosine readout reveals that the model carries the right tool internally before instruction tuning even wires it to the output. This saves the cost of weight updates (ASA uses only ~20 KB of portable assets) and avoids the brittleness of hand-crafted prompts. The penalty is that this internal representation does not guarantee stable behavior when the interaction extends beyond a single turn.

A concrete failure mode appears in the multi-turn agent loop evaluation. An operator monitoring the agent would observe performance swings of up to 30 percentage points—gain or loss from the matched baseline—with no consistent direction across turns. For example, the same steering intervention that boosts accuracy on one turn may collapse it on the next, as the model’s internal tool representation becomes entangled across conversational context. The signal is an erratic, unpredictable accuracy plot that violates the single-turn guarantee, flagging that the subsystem’s invariance to hidden-state linearity does not extend to the temporal dependencies of realistic agent workflows.

Multi-Turn Performance Instability

• Trigger — Same intervention applied in single-turn and multi-turn agent loops, causing performance swings.
• Guard — No guard appears in the source; the system lacks a handler for this instability.
• Posture — fail-soft: the evaluation continues but yields unreliable metrics.
• Operator signal — "matched-baseline gain or loss of up to 30 percentage points with no consistent direction"
• Recovery — No automatic recovery; manual step involves adopting a multi-turn benchmark such as MCP-AgentBench, which comprises 600 queries across 6 categories of varying interaction complexity.

Single-Turn Benchmark Saturation

• Trigger — Early benchmarks that focus on single calls with fixed options allow many models to nearly saturate the evaluation.
• Guard — No guard is provided; the source identifies this as a fundamental challenge without a mitigator.
• Posture — fail-soft: the benchmark produces inflated scores that mask real-world weaknesses.
• Operator signal — "many models nearly saturated those short evaluations" (silent absence of failure indicators).
• Recovery — Manual step: switch to evaluation frameworks that include multi-turn agent loops, as seen in MCP-AgentBench’s 188 distinct tools across 33 servers.

Intervention Inconsistency Across Turns

• Trigger — The same intervention that improves single‑turn performance is applied in multi‑turn loops and instead degrades or fails to improve.
• Guard — No guard is described for this inconsistency; the source only reports the observation.
• Posture — fail-soft: the system continues but with degraded performance relative to baseline.
• Operator signal — "gain or loss of up to 30 percentage points with no consistent direction" (same metric as instability).
• Recovery — No automatic recovery; requires re‑engineering the intervention specifically for multi‑turn contexts.

Lazy Agent Failure Mode

• Trigger — The model is conservative in entering tool mode even though tool need is decodable from mid‑layer activations, creating a representation‑behavior gap.
• Guard — Activation Steering Adapter (ASA) with a probe‑guided signed gate amplifies true intent and suppresses spurious triggers.
• Posture — fail-soft: the model fails to call necessary tools, but ASA intervenes without aborting generation.
• Operator signal — "strict tool‑use F1 from 0.18 to 0.50" on MTU‑Bench with Qwen2.5‑1.5B.
• Recovery — ASA performs a single‑shot mid‑layer intervention; no weight updates or retries.

False Positive Tool Calls in Multi‑Turn

• Trigger — The model calls tools when not needed, particularly in longer sequences with many tool invocations.
• Guard — Same Activation Steering Adapter (ASA) with probe‑guided signed gate reduces spurious triggers.
• Posture — fail-soft: incorrect tool calls occur but do not abort the interaction; ASA afterward mitigates.
• Operator signal — "false positive rate from 0.15 to 0.05" on MTU‑Bench.
• Recovery — ASA’s intervention is applied once per query; no re‑evaluation or fallback.

Tool error handling middleware converts exceptions into ToolMessages the model can process.

@wrap_tool_call
def handle_tool_errors(
    request: ToolCallRequest,
    handler: Callable[[ToolCallRequest], ToolMessage],
) -> ToolMessage:
    """Convert tool exceptions into ToolMessages the model can handle."""
    try:
        return handler(request)
    except Exception as e:
        return ToolMessage(
            content=f"Tool error: Please check your input and try again. ({e})",
            tool_call_id=request.tool_call["id"],
        )

Imagine a chef following a long recipe who sometimes grabs the wrong ingredient without realizing it until the dish is ruined. This subsystem helps a language model decide whether to use an external tool, but in long chains of decisions, mistakes stay invisible until it’s too late to fix them. The model first picks a tool internally before it speaks the choice—like a chef knowing which spice to use without reaching for it yet. A method called cosine readout can peek inside this hidden state and recover the correct tool with 61–82% accuracy in simple, single-turn tasks. But in multi-turn loops—like cooking multiple steps that depend on earlier results—that same peek becomes far less stable. Gains or losses can swing by up to thirty percentage points with no consistent direction, meaning the model might confidently choose a dangerous tool, such as one that sends an email or deletes data, with no warning. The trickiest point is that even though the model’s internal state often holds the right tool, instruction tuning is needed to “wire” that knowledge to its actual output. For single turns this linking works, but for multi-turn workflows the link breaks—the hidden representation becomes unreliable across steps. Without this awareness, a beginner would feel the concrete failure: the model calls a wrong function, an irreversible action like transferring money happens, and there is no way to look inside and catch the mistake beforehand.

The subsystem described in this chapter is the cosine readout intervention for tool-calling, as detailed in the text-to-voice study. The ordered mechanism proceeds as follows: first, the model's mid-layer activations are probed to extract the chosen tool via cosine similarity, before any token generation occurs. This readout recovers the internally encoded tool choice with high fidelity—61–82% accuracy on the BFCL benchmark. On success, the tool is emitted. On failure, particularly in multi-turn agent loops, the same intervention becomes unstable; the matched-baseline gain or loss can swing by up to 30 percentage points with no consistent direction, meaning the readout may yield a correct or incorrect tool unpredictably. The invariant the design preserves is that for single-turn, fixed-menu settings, the internal representation of the correct tool is robustly decodable—the model already carries the right tool internally before it can emit it, a property the chapter identifies as a representation-behavior gap.

The key trade-off is between inference-time representation readout and training-based alignment. The subsystem rejects the obvious alternative of fine-tuning or instruction tuning to wire the internal representation to the output token stream. Building the system this way avoids the cost of continual parameter-efficient fine-tuning, which requires training, maintenance, and risks forgetting previous capabilities. Instead, the cosine readout leverages the pre-existing representation formed during pretraining, requiring no weight updates. However, this efficiency comes at the cost of stability: the intervention is highly effective in single-turn scenarios but fails to maintain reliability in long workflows, where the readout’s accuracy fluctuates wildly. The chapter explicitly notes that on multi-turn agent loops, the same intervention is less stable, with matched-baseline gains or losses of up to 30 percentage points and no consistent direction.

A concrete failure mode is the instability of the cosine readout in multi-turn loops. When an agent interacts over several turns, the operator would see inconsistent tool selection: in one turn the readout might achieve near-perfect accuracy, while in the next it drops dramatically, leading to erroneous function calls. The signal an operator would actually observe is a large swing in the tool-use F1 or accuracy metric across consecutive turns—for example, a drop from 80% to 50% then back to 75% without any clear pattern. This manifests as unpredictable behavior in the agent’s long-horizon tasks, where a wrong tool can cause irreversible actions like money transfers or data deletions, as the chapter warns. The lack of a consistent direction makes debugging and mitigation difficult, as the failure cannot be attributed to a fixed cause.

Wrong Tool Picked (Hidden State)

• Trigger — The agent selects an incorrect tool for the task. Because the decision is invisible inside the model, the mistake is not detected until the tool executes.
• Guard — No guard shown in source. The paper Tool Calling is Linearly Readable and Steerable proposes reading the chosen tool via cosine readout and later steering with per-tool directions, but these are not deployed as runtime exception handlers, retries, or fallbacks.
• Posture — fail-soft. The incorrect tool executes, the agent continues the workflow, and the consequence (e.g., an email being sent to the wrong recipient) only becomes visible after the fact.
• Operator signal — Silent absence during inference; the operator observes the wrong tool use only after execution, when the erroneous action has already taken effect.
• Recovery — No recovery specified in source.

Multi-Turn Loop Instability

• Trigger — The agent enters a multi-turn loop, where the same intervention (e.g., steering vectors) that works reliably in a single-turn setting causes performance to swing unpredictably. Gains or losses of up to 30 percentage points occur with no consistent direction.
• Guard — No guard shown in source. The paper Tool Calling is Linearly Readable and Steerable reports this instability, but offers no retry logic, fallback, or validation to stabilise the loop.
• Posture — fail-soft. The agent continues through the multi-turn workflow despite the erratic quality; the run is not aborted.
• Operator signal — The metric is the observed gain or loss of up to 30 percentage points on the task performance compared to baseline, with no clear trend.
• Recovery — No recovery specified in source.

Lazy Agent Failure Mode

• Trigger — The model’s mid‑layer activations decodably indicate that a tool is needed, yet the model remains conservative and fails to enter tool‑calling mode, creating a representation‑behavior gap.
• Guard — Activation Steering Adapter (ASA). This is a training‑free, inference‑time controller that performs a single‑shot mid‑layer intervention using a router‑conditioned mixture of steering vectors with a probe‑guided signed gate to amplify true intent and suppress spurious triggers.
• Posture — fail‑soft. The agent stays in text‑only mode and continues the conversation without calling the tool, thereby losing the tool’s capability but not crashing.
• Operator signal — The false negative rate: the metric “strict tool‑use F1” improves from 0.18 to 0.50 when ASA is applied, indicating that without it many required tool calls are missed.
• Recovery — No recovery specified in source; ASA is the active intervention and no fallback beyond it is given.

False Positive Tool Activation

• Trigger — The model incorrectly decides to call a tool when no tool is needed, or calls the wrong tool due to spurious triggers in the input.
• Guard — Activation Steering Adapter (ASA), specifically its probe‑guided signed gate that suppresses spurious triggers while amplifying true intent.
• Posture — fail‑soft. The erroneous tool call is executed, the agent continues, but the tool side‑effect (e.g., sending a message, deleting data) may be irreversible.
• Operator signal — The false positive rate metric: ASA reduces it from 0.15 to 0.05 on MTU‑Bench, so without the guard the operator would see a 15% false positive rate.
• Recovery — No recovery specified in source. The guard is intended to prevent the false positive, not to undo it after execution.

Handle tool exceptions by converting them into safe error messages for the model.

@wrap_tool_call
def handle_tool_errors(
    request: ToolCallRequest,
    handler: Callable[[ToolCallRequest], ToolMessage],
) -> ToolMessage:
    """Convert tool exceptions into ToolMessages the model can handle."""
    try:
        return handler(request)
    except Exception as e:
        return ToolMessage(
            content=f"Tool error: Please check your input and try again. ({e})",
            tool_call_id=request.tool_call["id"],
        )

Imagine a security guard watching a bank teller’s hands before they even touch the keyboard—the guard can read which button the teller intends to press by the way their fingers hover. This subsystem does the same for an AI agent that controls a mouse and keyboard: it peeks inside the agent’s internal workings to see which action it is about to take, so a dangerous move—like sending a payment or deleting a file—can be caught before it happens.

Inside the agent, every possible tool choice leaves a faint fingerprint in its “activation space,” a kind of mental map. Researchers found a single direction in that map that corresponds to picking one specific tool. By measuring that direction (called a cosine readout), the subsystem can tell which tool the agent will call with up to 100% accuracy on many models—even before the agent finishes typing the command. If the agent is torn between two tools, that uncertainty shows up as a weaker signal, and those unsure calls fail 21 times more often than confident ones. This gives the guard a reliable early warning.

The deep insight is that these tool-reading directions already exist inside the agent from its initial training, before it ever learned how to answer instructions. Instruction tuning later teaches the agent to output that internal choice, but the underlying direction was already there. Without this subsystem, a wrong tool call is invisible until executed—the guard only learns about the mistake after the money is gone or the data is erased. The failure a beginner would feel: an agent meant to show a bank balance instead sends a transfer, and there is no undo button.

The ordered mechanism begins with a cosine readout applied to the model’s mid-layer activations to decode the intended tool before any output token is generated. This probe recovers 61–82% accuracy on BFCL while base generation lands at only 2–10%, confirming that the representation exists prior to emission. Next, if the probe detects high uncertainty—for example, a query where the model is unsure between two tools (which fail 21× more often on Gemma 3 27B)—the system flags a likely error without executing. On identified uncertainty or a Lazy Agent failure mode (where tool necessity is decodable but the model remains conservative), a single-shot mid-layer intervention is triggered via the Activation Steering Adapter (ASA). ASA applies a router-conditioned mixture of steering vectors with a probe-guided signed gate to amplify the true tool intent and suppress spurious triggers. On successful steering, the model’s generation switches to the correct tool; on failure (e.g., the intervention cannot resolve ambiguity), the call is halted and the operator receives an early-warning signal from the per-tool direction probe.

The design preserves the invariant that tool choice is linearly readable from activation space. This guarantee is named explicitly in the source: “the choice of tool is carried by a single direction in activation space” and “reading the chosen tool off the model’s internal state (cosine readout) recovers 61–82% accuracy”. Because the representation exists independently of the output, the system can check correctness before any irreversible action (e.g., transferring money or deleting data) is executed. The invariant holds across 12 instruction-tuned and 6 base models spanning Gemma 3, Qwen 3, Qwen 2.5, and Llama 3.1 (270M to 27B), with switch accuracy of 83–100% on a 15-tool synthetic benchmark and 77–94% on the real-API τ-bench airline.

The key trade-off sacrifices output‑time flexibility for pre‑execution safety. The obvious rejected alternative is to allow the model to generate the tool call freely and then detect errors only after execution—an approach that is catastrophic when actions are irreversible. By rejecting that alternative, the design avoids the cost of executing a malicious or mistaken tool call that could cause real damage. The cost of the chosen approach is that the intervention is less stable in multi-turn agent loops (matched‑baseline gain or loss of up to 30 percentage points with no consistent direction), and it requires a probe that may miss subtler failures. However, the training-free, inference-time nature of ASA—using only about 20KB of portable assets and no weight updates—makes this trade-off acceptable for safety‑critical deployments.

A concrete failure mode is the Lazy Agent scenario: the model’s mid-layer activations clearly indicate tool necessity, yet the generated output remains conservative and fails to enter tool mode. An operator would see a persistent representation-behavior gap signal from the activation probe—the cosine readout shows high confidence for a tool, but the output never calls it. Alternatively, on a query where uncertainty between two tools is high, the system would flag an error before execution. The operator would observe a false positive rate that the ASA controller reduces from 0.15 to 0.05 on MTU-Bench with Qwen2.5-1.5B, while strict tool-use F1 improves from 0.18 to 0.50. In either case, the operator is warned before any irreversible action takes place.

Failure Mode 1: Tool Misselection due to High Uncertainty

• Trigger — The model’s activation space indicates high ambiguity between two tool options, as shown by the finding that “queries where the model is unsure between two tools fail 21x more often than queries where it is not” (Gemma 3 27B, Tool Calling is Linearly Readable paper).
• Guard — The “probe-guided signed gate” from the Activation Steering Adapter (ASA) suppresses spurious triggers and amplifies true intent, but it is not a retry or exception handler—it is a single-shot inference-time intervention that must have been applied beforehand. The source does not describe an additional runtime guard that catches a misselection after the fact.
• Posture — fail-soft. The wrong tool call is executed because no guard stops it; the agent continues its run, and the damage (e.g., money transfer or data deletion) is visible only after execution.
• Operator signal — A 21× higher failure rate on those uncertain queries; the only observable event is the downstream consequence of the wrong action—no immediate error log.
• Recovery — No automatic recovery is documented in the source. The operator must manually detect the incorrect action and perform a rollback or compensation.

Failure Mode 2: False Positive Tool Call (Unnecessary Invocation)

• Trigger — The model’s “self-perceived need and utility are frequently misaligned with the true need and utility” (To Call or Not to Call paper), causing it to call a tool (e.g., a web search) when the tool is not needed.
• Guard — The “lightweight estimators of need and utility” trained from the model’s hidden states, which drive “simple controllers that improve decision quality.” These controllers can veto unnecessary calls—but they operate at inference time and are fallible.
• Posture — fail-soft. The unnecessary call still executes (unless the controller vetoes it), and the agent continues. If the tool has irreversible side effects (e.g., delete data), the damage proceeds.
• Operator signal — A higher-than-expected tool call count, or an alert from the downstream tool (e.g., an unintended email sent). No dedicated log line identified in the source.
• Recovery — No retry or backoff; the operator can adjust the controller’s threshold after observing the misuse.

Failure Mode 3: False Negative Tool Call (Missed Necessary Tool)

• Trigger — The model exhibits a “Lazy Agent” failure mode where “tool necessity is nearly perfectly decodable from mid-layer activations, yet the model remains conservative in entering tool mode” (ASA paper). This causes the model to skip a necessary tool call.
• Guard — The “Activation Steering Adapter (ASA)” performs a “single-shot mid-layer intervention” with a “probe-guided signed gate” to amplify true intent and push the model into tool mode. If applied, it prevents the false negative. However, the source does not mention a backup guard if ASA fails to activate tool mode.
• Posture — fail-soft. The task proceeds without the tool, potentially producing an incorrect or incomplete result, but the agent does not halt.
• Operator signal — The output lacks expected tool-generated content, or the task objective is not met. No error is raised.
• Recovery — No automatic recovery; the same ASA intervention would have to be re-applied with different parameters, or the operator must rerun the query with adjusted settings.

Failure Mode 4: Activation Readout Detection Error

• Trigger — The “cosine readout” method that “read[s] the chosen tool off the model’s internal state” has an accuracy of 61–82% (Tool Calling is Linearly Readable paper). In 18–39% of cases, the readout misidentifies which tool the model will call.
• Guard — No guard is described in the source that validates or corrects a misread from the cosine readout. The readout itself is a detection tool, not a safety mechanism with fallbacks.
• Posture — fail-soft. If the readout is used to approve or block a tool call, a false readout allows the wrong call to proceed; if used only for monitoring, the agent continues with undetected error.
• Operator signal — A discrepancy between the readout’s prediction and the actually emitted tool call, or a higher error rate on tasks where the readout was relied upon. No explicit log line.
• Recovery — No automatic recovery. The operator may cross-check the readout against the model’s final output, but that is a manual step.