Agent Guardrails — Deep Dive

Input fencing by requiring absolute filepaths in tool commands

def tool_execute(filepath):
    if not filepath.startswith('/'):
        raise ValueError("Only absolute filepaths allowed")
    # proceed with tool execution

Think of input fencing as a security guard at the factory door who checks every package before it reaches the assembly line. Its single job is to stop sneaky instructions—like a crafty message pretending to be a normal internet page or tool reply—from tricking the model into making things up. Researchers saw that when the model was told to output a specific format like JSON, it often filled required fields with fake information because the format felt like it demanded an answer, even with no real facts. This effect, called the C3 anomaly, caused hallucinations to jump by over ten percentage points.

Now, the guard doesn't just watch—it enforces rules. One simple rule is that every tool command must include an absolute filepath, like “/home/user/data.txt” instead of just “data.txt.” That tiny check blocks vague or hidden instructions because the model cannot guess a legitimate path without real evidence. Another mechanism is a fixed vocabulary constraint, which only allows the model to pick from a verified list of tools, preventing it from inventing names. Without these fences, the model would confidently recommend nonexistent tools or fill in plausible but false answers from its training data, leaving users with dangerous, believable errors.

The Input Fencing subsystem is designed as a three-mechanism mitigation stack to prevent mention-level hallucination in the closed-inventory Large Language Model (LLM) system Online-CADCOM. The ordered mechanism operates as follows: first, database-grounded context injection provides the model with a verified list of engineering tools from the platform’s inventory; second, fixed vocabulary constraints restrict the model’s output tokens to only those tool names present in that inventory; third, enforced JavaScript Object Notation (JSON) output forces the final response into a structured schema. On failure of any preceding step—for instance, if the database-grounding mechanism is disabled—the remaining mechanisms become vulnerable. Critically, when only JSON output enforcement is active without any grounding mechanism, the system exhibits the C3 anomaly, where hallucination rates increase by 10.1 percentage points (Generation 1) and 15.1 percentage points (Generation 2) above the unconstrained baseline.

The invariant preserved by this design is that all recommended tools belong strictly to the platform’s verified inventory—a “closed-inventory” guarantee. Under the full three-mechanism architecture, the hallucination rate (HR) drops from 59–74% to 3.3–14.9%, with cross-provider averages stable across generations (6.8% Gen1, 7.5% Gen2). This invariant is maintained even when reasoning-mode models are used, as they provide no statistically significant improvement under the architectural constraints.

The key trade-off lies in rejecting the simpler alternative of relying solely on JSON schema enforcement without grounding mechanisms. That rejected approach would reduce implementation complexity and computational overhead, but it introduces the C3 anomaly: the mandatory entity fields in the JSON schema exert slot-filling pressure, forcing the model to populate tool-name slots from training-data priors when no grounding vocabulary is provided. The design thus accepts the cost of maintaining database-grounded context injection and fixed vocabulary constraints—both requiring periodic updates and synchronization with the inventory—in order to avoid the reliability cost of hallucination spikes observed under the lone-JSON configuration.

One concrete failure mode occurs when the database-grounded context injection mechanism fails (e.g., due to a network outage or stale inventory data). In that scenario, the system falls back to JSON enforcement alone, triggering the C3 anomaly. An operator would observe a sudden, measurable increase in the hallucination rate (HR) metric—specifically, a rise of 10–15 percentage points above the baseline—across all providers, along with a corresponding increase in out-of-inventory tool mentions in system logs. This signal directly indicates that the grounding layer has been compromised and requires immediate attention.

C3 Anomaly — JSON Enforcement Alone Increases Hallination

• Trigger — The mitigation stack uses only JSON output enforcement without any grounding mechanisms (no database context injection, no fixed vocabulary constraints).
• Guard — No guard is active; the source states that under this configuration hallucination rises. There is no explicit exception handler, retry, or fallback defined for this condition.
• Posture — fail-soft — the system continues to generate outputs but with a significantly increased hallucination rate (+10.1 pp Gen1, +15.1 pp Gen2).
• Operator signal — Hallucination rate metric increases; log entries show out-of-inventory tool mentions (e.g., “mention-level hallucination”).
• Recovery — The operator must add database‑grounded context injection and fixed vocabulary constraints to the pipeline. No automatic retry is described; manual reconfiguration required.

Missing Absolute Filepath in Tool Command

• Trigger — A tool reply or internet page contains a command that uses a relative path instead of an absolute filepath.
• Guard — The filter “always require absolute filepaths in tool commands” (the exact identifier from the source). This is presumably a validation check at the system boundary.
• Posture — fail-closed — the filter rejects the command, refusing to write or execute it.
• Operator signal — Log line such as “Command rejected: missing absolute filepath”.
• Recovery — Manual step required: the operator must rewrite the command with an absolute filepath. No automatic retry.

Slot‑Filling Pressure from Mandatory JSON Fields

• Trigger — The output schema contains mandatory entity fields (e.g., tool‑name slots) and no grounding vocabulary is provided. The model populates these slots from training‑data priors.
• Guard — The “fixed vocabulary constraints” component of the mitigation stack (exact identifier from the source). When present, it prevents hallucination.
• Posture — fail‑soft if the guard is absent (system continues with hallucinated outputs); fail‑closed if the guard is present (constrained output).
• Operator signal — Without the guard: “out‑of‑inventory mentions” appear in outputs; a frequency‑weighted audit would show they correspond to real engineering tools absent from the inventory.
• Recovery — Add fixed vocabulary constraints to the schema. Manual intervention required; no automatic fallback.

Injection of Instructions from Internet Page or Tool Reply

• Trigger — The model’s prompt includes content from an external page or tool reply that contains hidden instructions (e.g., “ignore previous instructions and do X”).
• Guard — The source does not show a specific guard for this failure within the Input Fencing subsystem description. The subsystem only mentions the absolute‑filepath filter; no exception handler, retry, or fallback is provided.
• Posture — fail‑soft — the model continues to execute the injected instructions, potentially outputting harmful or incorrect content.
• Operator signal — Unexpected model behavior; no specific log metric from the input fencing itself. Detection would rely on downstream monitoring.
• Recovery — Manual review and sanitization of the input; no automatic recovery described.

Reliance on Reasoning Mode Fails to Mitigate Hallucination

• Trigger — The system enables reasoning‑mode models (e.g., chain‑of‑thought) expecting it to reduce hallucination, but without architectural constraints (database grounding, vocabulary constraints).
• Guard — No guard; the source states “Reasoning‑mode models provide no statistically significant improvement under architectural constraints.” Relying on reasoning mode alone is insufficient.
• Posture — fail‑soft — the model continues to hallucinate at similar rates.
• Operator signal — No decrease in hallucination rate despite enabling reasoning mode; operator sees no improvement in metrics.
• Recovery — Operator must add grounding mechanisms (database context injection, fixed vocabulary constraints) instead of relying on reasoning mode. No automatic recovery.

No source code is present in the provided context.

Think of this screening as a bouncer who checks every partygoer’s story against a reliable guest list before letting them through the door. This subsystem scans AI outputs to catch facts that don’t match real evidence before they reach a user. It first looks up outside sources to back up claims, but that search can stumble—the writing of the query might be sloppy, the document retrieval might grab wrong pages, or the combination of evidence might be botched, introducing new errors. Another guard forces the reply into a fixed structure, like a form with required fields that must be filled. But when no outside facts are checked, this backfires badly: the model starts inventing to complete every slot, and a real study found that error rates jumped by more than ten percentage points. The deep catch is that requiring a strict format without any grounding turns the output into a pressured guessing game. The model treats mandatory fields as slots that must be populated, so it pulls from its own training memories instead of real evidence—this specific failure is called the C3 anomaly. Without this screening, the AI would confidently send out slick yet fabricated answers, and in high-stakes tasks like medical advice, a user could act on a falsehood that leads to real harm.

The Output Screening subsystem in this design enforces a three‑mechanism mitigation stack applied in a strict order. First, database‑grounded context injection retrieves external evidence from the verified inventory and injects it into the prompt. Next, fixed vocabulary constraints restrict the model’s output tokens to the set of known tool names. Finally, enforced JavaScript Object Notation (JSON) output forces the response into a structured schema. If grounding fails—for instance, if the retrieval step provides no relevant context—the system still proceeds to the vocabulary constraint and JSON enforcement. On failure of the vocabulary constraint (e.g., a tool name outside the fixed set is generated), the output is rejected and the system can fall back to a default safe response. The critical invariant preserved by this entire pipeline is the closed‑inventory guarantee: every recommended tool must belong to the platform’s verified inventory, meaning no out‑of‑inventory mentions are allowed.

The design explicitly rejects the alternative of applying JSON output enforcement without any grounding mechanisms. The obvious rejected approach—using only JSON enforcement—was tested and produced the C3 anomaly: a statistically significant increase in the hallucination rate (HR) above the unconstrained baseline for all providers (+10.1 percentage points for Generation 1, +15.1 pp for Generation 2). The cost that this rejection avoids is slot‑filling pressure: mandatory entity fields in the JSON schema force the model to populate tool‑name slots from training‑data priors when no grounding vocabulary is provided, thereby increasing rather than decreasing hallucinations. By insisting on database‑grounded context injection and fixed vocabulary constraints before JSON enforcement, the design prevents that pressure from arising, at the expense of additional retrieval overhead and stricter output‑space limitation.

A concrete failure mode occurs precisely when only JSON output enforcement is active and the grounding mechanisms are absent. An operator monitoring the system would observe a sudden spike in the hallucination rate (HR) : for example, a jump from a baseline of 59–74% to an even higher range (e.g., +15.1 pp for Generation 2). The signal would be an elevated percentage of responses that mention tools absent from the closed inventory, directly contradicting the closed‑inventory invariant. This pattern was named the C3 anomaly in the source study. Under the full architectural stack, the hallucination rate falls to 3.3–14.9%, but the anomaly serves as a clear operational warning that grounding has been bypassed and slot‑filling pressure is degrading factual reliability.

• C3 Anomaly

  • Trigger — Only JSON output enforcement is active without database-grounded context injection or fixed vocabulary constraints.
  • Guard — No guard is active under this configuration; the intended mitigation is the full three‑mechanism stack (database-grounded context injection, fixed vocabulary constraints, JSON output enforcement), but those are absent.
  • Posture — Fail‑soft: the system continues generating output, but with a measurably higher hallucination rate.
  • Operator signal — Hallucination rate (HR) increases by +10.1 percentage points (pp) Gen1 and +15.1 pp Gen2 above the unconstrained baseline.
  • Recovery — Deploy the complete stack, enabling database-grounded context injection and fixed vocabulary constraints alongside JSON output enforcement.

• Remaining Out‑of‑Inventory Mentions

  • Trigger — The full mitigation stack (database-grounded context injection, fixed vocabulary constraints, JSON output enforcement) is active, yet the model still recommends tools that are not in the platform’s verified inventory.
  • Guard — The three‑mechanism stack itself reduces the hallucination rate to 3.3–14.9%, but no further guard handles the remaining out‑of‑inventory mentions.
  • Posture — Fail‑soft: outputs are still produced, and a frequency‑weighted audit confirms that most such mentions are real engineering tools absent from the inventory.
  • Operator signal — Hallucination rate persists in the range 3.3–14.9%, with the majority of errors being out‑of‑inventory mentions of genuine tools.
  • Recovery — Manually update the platform’s inventory with the missing tools, or implement continuous inventory synchronization.

• Coverage Collapse (BERT)

  • Trigger — A BERT‑based Guardrail Classifier is used with a threshold optimized for the toxicity domain.
  • Guard — The Guardrail Classifier (specifically BERT) itself fails to maintain coverage; the framework’s Gaussian Mixture Models (GMM) certificates reveal the collapse.
  • Posture — Fail‑soft: the classifier continues to run, but safety coverage drops sharply, allowing harmful outputs through.
  • Operator signal — Coverage metric drops to 55% at the optimal threshold, a phenomenon described as coverage collapse.
  • Recovery — Replace BERT with GPT‑2 or Llama‑3.1‑8B (which maintain 90% and 80% coverage respectively), or adopt an extremely conservative pessimistic threshold.

• Reasoning‑Mode No Improvement

  • Trigger — Reasoning‑mode models are used under the full architectural constraints (database‑grounded context injection, fixed vocabulary constraints, JSON output enforcement).
  • Guard — The full mitigation stack is already active; no additional guard addresses this failure, as the reasoning mode itself provides no statistically significant improvement.
  • Posture — Fail‑soft: the system operates normally but wastes computational resources without gain.
  • Operator signal — No statistically significant reduction in hallucination rate compared to the standard output mode.
  • Recovery — Disable reasoning mode and revert to the standard output mode to avoid unnecessary latency and cost.

• Slot‑Filling Pressure

  • Trigger — Mandatory entity fields in the JSON schema are present, but no grounding vocabulary is provided; the model must populate tool‑name slots from training‑data priors.
  • Guard — No guard exists; this is a hypothesized cause (slot‑filling pressure) that requires further experimental validation.
  • Posture — Fail‑soft: the system still produces JSON output, but with increased hallucination due to the pressure.
  • Operator signal — Elevated hallucination rate that correlates with the absence of grounding vocabulary, though no direct metric isolates this pressure.
  • Recovery — Provide a grounding vocabulary (e.g., via fixed vocabulary constraints) to prevent the model from falling back on training‑data priors.

Budget and step limits are not directly implemented in the provided context; no source code is present.

Think of it like giving a painter a fixed budget of paint and canvas to finish a portrait. This subsystem is for setting limits on how much resources a model can use, so it doesn’t waste time or money making stupid mistakes. In the gist: you give the model just enough tokens (paint) to think before it writes itself into a corner, like telling a painter “you have one tube of paint and ten brushstrokes—use them wisely.”

Now zoom in: step by step, the system first sets a token budget—the model gets a certain number of words or symbols to complete a task. But crucially, constraints can backfire. One real mechanism from the source is the C3 anomaly: when teams forced the model to always output in strict JSON format without also providing a ground‑truth inventory list, the model started making up tool names just to fill the required slots. That’s like telling the painter “you must paint a face with exactly these seven named features” but giving no reference photo—so they invent features that aren’t real. The fix was to always ground the model with absolute file paths (like a real inventory), which made the tool use flawless.

Deepest level: the non‑obvious edge is that JSON‑only enforcement pushes hallucination rates up by 10–15 percentage points because the required fields act like empty boxes the model feels pressured to fill, even when it has no correct information. Without this subsystem—without both the token budget and the grounded output format—the model wastes the budget by writing gibberish or recommending tools that don’t exist, leaving you with a costly, unusable mess.

The subsystem described in the "Budgets And Step Limits" chapter is the three‑mechanism mitigation stack from the hallucination‑mitigation study. In operation, the ordered mechanism proceeds as follows: first, database‑grounded context injection supplies the model with a verified inventory of permissible tools. Next, fixed vocabulary constraints restrict the model’s output tokens to only those tool names. Finally, enforced JavaScript Object Notation (JSON) output structures the response into a machine‑readable schema. If any step fails—for instance, if the grounding injection is omitted—the system proceeds with the remaining mechanisms, but the protective chain is broken. On failure of the full stack (i.e., when all three mechanisms are absent), the model operates at a baseline hallucination rate of 59–74%.

The invariant the design preserves is that all tool‑name mentions in the model’s output must correspond to entries in the platform’s closed inventory. This “mention‑level hallucination” constraint is guaranteed only when both the grounding injection and vocabulary constraints are active; relying on JSON enforcement alone does not enforce the invariant and, paradoxically, worsens it.

The key trade‑off is between architectural simplicity and factual reliability. A simpler alternative would be to apply only JSON output enforcement, which requires fewer components and lower engineering overhead. That alternative is explicitly rejected because it produces the C3 anomaly—JSON enforcement without grounding increases hallucination above the unconstrained baseline by +10.1 percentage points in Generation 1 and +15.1 points in Generation 2. The cost avoided by using the full stack is the increased rate of false tool recommendations caused by slot‑filling pressure, where mandatory JSON fields force the model to populate tool‑name slots from training‑data priors. Accepting that extra implementation complexity yields a cross‑provider average hallucination rate of 6.8–7.5% instead of double‑digit increases.

One concrete failure mode occurs when the database‑grounded context injection is not applied and only JSON output enforcement is active. In this configuration, the model’s hallucination rate (HR) jumps to a level higher than the unconstrained baseline. An operator monitoring the system would see a +15.1 percentage point rise in HR for Generation 2 models, alongside a frequent‑weighted audit showing that the majority of remaining out‑of‑inventory mentions correspond to real engineering tools absent from the platform’s inventory. The signal is a systematic elevation of cross‑provider hallucination metrics well above the baseline, indicating that the slot‑filling pressure from the JSON schema is driving the model to invent tool names from its training priors rather than the grounding inventory.

1. C3 Anomaly: JSON Enforcement Without Grounding Inflates Hallucination

• Trigger — A constraint (enforced JavaScript Object Notation output) is added to a model’s output format without any accompanying grounding mechanism (no database-grounded context injection, no fixed vocabulary constraints). The JSON schema exerts slot-filling pressure, forcing the model to populate tool‑name slots from training‑data priors.

• Guard — The intended guard is the three‑mechanism mitigation stack (database‑grounded context injection, fixed vocabulary constraints, enforced JSON output). When only JSON enforcement is active, no guard handles the failure; the stack is incomplete. The source does not provide an exception handler or retry for this anomaly.

• Posture — Fail‑soft. The model continues to generate outputs, but the hallucination rate rises from a 3.3–14.9% baseline to 59–74% (the unconstrained baseline is around 59–74% and the full stack brings it down; JSON‑only pushes it above that baseline). The system degrades gracefully—no abort—but reliability is severely impaired.

• Operator signal — A hallucination rate (HR) increase of +10.1 percentage points (Gen1) and +15.1 percentage points (Gen2) above the unconstrained baseline, observable in the configuration where only JSON enforcement is active. The source explicitly reports this as the “C3 anomaly”.

• Recovery — The operator must activate the full three‑mechanism mitigation stack: enable database‑grounded context injection and fixed vocabulary constraints alongside the JSON enforcement. The source shows that doing so drops the HR to 6.8% (Gen1) and 7.5% (Gen2). No retry or backoff is described; the fix is a configuration change.

2. Guardrail Safety Hole: Hyper‑Rectangle Certificates Expose Verifiable Unsafety

• Trigger — A Guardrail Classifier (e.g., toxicity domain) is deployed in a production system without formal verification. When a red‑teaming evaluation attempts to certify the classifier’s safety region using SVD‑aligned hyper‑rectangles, every configuration returns SAT (meaning a safety violation is provably present).

• Guard — The classifier itself is the guard, but it fails. The source does not list any exception handler, retry, or fallback for this condition. The evaluation method (SVD‑aligned hyper‑rectangles) is a verification tool, not a runtime defensive component.

• Posture — Fail‑soft. The system continues to serve requests; the classifier continues to run, but the guarantee is broken. Harmful content may be let through at any time because the classifier’s safety margin is not formally sound.

• Operator signal — The certificate output reads SAT (sat means unsafe). The source states: “every hyper‑rectangle configuration returns SAT, exposing verifiable safety holes across all classifiers, despite seemingly high empirical metrics.” The operator sees SAT in the verification logs.

• Recovery — The operator must replace the deterministic SVD‑aligned hyper‑rectangle certification with a Gaussian Mixture Model (GMM) probabilistic certificate over semantically coherent clusters. The GMM approach yields probabilistic certificates that can expose structural stability issues (like BERT’s coverage collapse) and allow tuning to a conservative pessimistic threshold. No automatic retry is provided; manual reconfiguration is required.

3. BERT Coverage Collapse: Guardrail Safety Margin Becomes Volatile

• Trigger — A BERT‑based Guardrail Classifier is used in a toxicity‑domain system. Under the GMM probabilistic certificate analysis, the classifier’s safety coverage proves uniquely volatile, dropping to 55% at the optimal threshold instead of maintaining robust coverage like GPT‑2 (90%) and Llama‑3.1‑8B (80%).

• Guard — The classifier itself is the guard, but it fails. No exception handler or fallback is described in the source. The GMM certificate is an evaluation technique, not a runtime guard.

• Posture — Fail‑soft. The system continues to run; the classifier still produces scores, but its safety coverage is sparsely populated. Many harmful prompts that should be blocked will be misclassified, allowing harmful outputs to pass.

• Operator signal — The operator observes “coverage collapse” to 55% at the optimal threshold. The source’s exact language: “BERT’s safety guarantees prove uniquely volatile. This ‘coverage collapse’ to 55% at the optimal threshold reveals a sparsely populated safety margin in BERT.”

• Recovery — The operator must adopt an extremely conservative pessimistic threshold (that achieves full coverage) as suggested in the source. This requires manual threshold adjustment and likely trades off precision for recall. No automatic retry exists; the recovery is a manual re‑categorization of the decision boundary.

4. Probabilistic Miscalibration: Model Outputs Remain Confident Despite Inaccuracy

• Trigger — The large language model is asked to produce a probability distribution (e.g., token‑level confidence) but cannot match the requested probability distributions due to underlying probabilistic miscalibration. This is one of the four miscalibration types identified in the position paper.

• Guard — No runtime guard is present. The source calls for calibration metrics to become a standard part of benchmark reporting and for training paradigms that preserve uncertainty, but no exception handler or fallback is provided for production runs.

• Posture — Fail‑soft. The model continues to generate text with high confidence even when wrong, leading to hallucinated or unreliable outputs. The system does not abort; it degrades silently.

• Operator signal — The operator sees no explicit error; the model outputs appear confident but are factually incorrect. The source describes this as “confident hallucinations” and notes that the model’s confidence is misaligned with factual correctness. A calibration error metric (e.g., expected calibration error) would be elevated if monitored, but the source does not name a specific metric.

• Recovery — The recommended recovery is to incorporate calibration metrics during training and to preserve uncertainty in the model’s output (e.g., by using temperature scaling or conformal prediction). The source explicitly states: “We call for calibration metrics to become a standard part of benchmark reporting, for training paradigms that preserve uncertainty.” This is a training‑time fix; at runtime, the operator can only log outputs for later recalibration.

5. Prompt Injection Detection Failure: Attention Tracker Misses Novel Attack Patterns

• Trigger — A malicious user sends a prompt injection that shifts focus from the original instruction to an injected instruction (the distraction effect). The attack uses a pattern not well represented in the training data for Attention Tracker, causing the detection method to miss the attack.

• Guard — Attention Tracker is the detection guard. It is a training‑free method that tracks attention patterns on the instruction. The source reports that it generalizes across models and attack types, but “significant gaps persist against novel attack vectors” (from the guardrails review paper). No fallback or retry is mentioned.

• Posture — Fail‑soft. The LLM continues executing the injected instruction, performing the designated action (e.g., ignoring original safety constraints). The system does not abort; the attack succeeds silently.

• Operator signal — The operator observes no alert from Attention Tracker because the method fails to flag the novel attack. The source’s metric for detection quality is AUROC; a drop in AUROC (or absence of detection) would be the silent signal. The source explicitly notes that “AUROC improvement of up to 10.0% over existing methods” is achieved on known patterns, implying that for novel ones the improvement may vanish.

• Recovery — The operator must manually review logs for unusual behavior or rely on a separate validation layer. The source suggests that architectural defenses (up to 95% protection against known patterns) could be combined, but no specific automatic recovery is provided. The only discussed mitigation is to incorporate attention‑pattern analysis as a diagnostic rather than a blocking guard.

Conditional interrupts on tool arguments allow mistake-proofing by gating approvals only when a predicate fails.

from langchain.agents import create_agent
from langchain.agents.middleware import HumanInTheLoopMiddleware, ToolCallRequest
from langgraph.checkpoint.memory import InMemorySaver

def writes_outside_workspace(request: ToolCallRequest) -> bool:
    """Pause writes to paths outside the workspace directory."""
    path = request.tool_call["args"].get("path", "")
    return not path.startswith("/workspace/")

agent = create_agent(
    model="gpt-5.4",
    tools=[write_file, execute_sql, read_data],
    middleware=[
        HumanInTheLoopMiddleware(
            interrupt_on={
                "write_file": {
                    "allowed_decisions": ["approve", "edit", "reject"],
                    "when": writes_outside_workspace,
                },
            },
        ),
    ],
    checkpointer=InMemorySaver(),
)

Imagine a chef who uses a knife that only cuts in one safe direction, and always pauses to plan each step before cooking, so he never slices a finger or ruins a dish. This subsystem does the same for an AI helper: it builds simple, mistake-proof rules into the tools the AI uses, and gives the AI enough time to think before acting. Its purpose is to prevent errors before they happen, especially when the AI recommends something from a real-world inventory.

One concrete mechanism is making the tool always use absolute filepaths—like forcing the chef to use a knife that can only chop straight down—so a common error is stopped before it starts. Another rule is to give the AI plenty of space to think, letting it reason through a problem before writing itself into a corner. But here’s the catch: if you force the AI to stick to a very strict format, like requiring it to answer only in a fixed JSON template, without also giving it a trustworthy list of allowed options, the AI actually makes more mistakes. In real tests, requiring JSON output alone increased mistakes across every provider tested. So the gate needs the right support—both the safety guard (like the knife guard) and the grounded list of valid choices.

The trickiest detail is why strict formatting without grounding backfires. The AI feels a “slot-filling pressure”: the JSON template demands that every field be filled, so the AI reaches into its memory for tool names, even if those tools don’t actually exist in the inventory. This led to a 10–15 percentage point jump in hallucination (recommending fake tools). Without this subsystem, a beginner would see the AI confidently suggest a tool that doesn’t exist, wasting time and money in a real engineering or medical setting.

The subsystem described in the Menxhiqi and Marinova paper implements a three-mechanism mitigation stack executed in a fixed order. First, database-grounded context injection injects verified inventory data into the model's context to anchor recommendations. Next, fixed vocabulary constraints limit the set of acceptable tool names to those in the closed inventory. Finally, JavaScript Object Notation (JSON) output enforcement forces the model to produce structured outputs. When any earlier mechanism fails (e.g., grounding is omitted), the subsequent mechanisms must compensate. Under the full architecture, the hallucination rate (HR) drops from 59–74% to 3.3–14.9%, but the ordering is critical: if only the last mechanism (JSON enforcement) is active without the preceding grounding steps, the design's guarantee can be violated.

The invariant the design preserves is factual reliability expressed as a low hallucination rate (HR) — specifically, cross-provider averages of 6.8% for Generation 1 (Gen1) and 7.5% for Generation 2 (Gen2) under the full stack. This is a correctness guarantee: every tool name in the model's output must belong to the verified inventory. The system rejects any output that mentions an out-of-inventory tool, and the guardrail is built into the response generation path itself, not as a post-hoc filter.

The key trade-off is accepting the computational cost of retrieval and vocabulary constraints in exchange for avoiding the slot-filling pressure that emerges when JSON enforcement is applied without grounding. The obvious alternative — requiring JSON output alone as a lightweight format constraint — is explicitly rejected because it backfires: under the C3 anomaly configuration, JSON enforcement alone increases hallucination by +10.1 percentage points (pp) in Gen1 and +15.1 pp in Gen2 across all providers (OpenAI, Anthropic, Google). The cost this design avoids is the forced population of mandatory tool-name fields from training-data priors, which would produce confident but incorrect recommendations. By grounding the vocabulary first, the system prevents that pressure from manifesting.

A concrete failure mode is the C3 anomaly itself. An operator monitoring the hallucination rate would observe a sudden spike in out-of-inventory mentions — for example, a +15.1 pp increase in Gen2 when the grounding mechanisms are disabled and only JSON enforcement remains. The signal would be a rising HR above the unconstrained baseline, visible in per-provider dashboards over the 6,912 API calls across the 12 configurations. The operator would trace this to the configuration where grounding is absent, confirming that the invariant (low HR) is violated due to the missing retrieval and vocabulary steps.

Enforcing JSON Output Without Grounding (C3 Anomaly)

• Trigger — The human approval gate requires JavaScript Object Notation (JSON) output from the LLM, but no database-grounded context injection or fixed vocabulary constraints are active. Under this configuration the model faces slot-filling pressure from mandatory entity fields in the JSON schema.
• Guard — No guard exists for this failure; the source explicitly reports that “JSON enforcement alone increases hallucination above the unconstrained baseline for all providers (+10.1 pp Gen1, +15.1 pp Gen2)”. The intended guard is the full three-mechanism stack (database-grounded context injection, fixed vocabulary constraints, and JSON output), but it is absent here.
• Posture — fail-soft. The system continues to produce output, but the hallucination rate (HR) rises from a baseline of 59–74% to 3.3–14.9% only when the full stack is applied; without grounding the HR surges, degrading factual reliability.
• Operator signal — The operator would observe the “C3 anomaly” identifier in the hallucination rate metrics, specifically an increase in HR of +10.1 percentage points (Gen1) or +15.1 pp (Gen2) above the unconstrained baseline.
• Recovery — No automatic recovery is shown. The manual step is to add database-grounded context injection and fixed vocabulary constraints to the JSON output enforcement.

Insufficient Token Allocation for Model Reasoning

• Trigger — The human approval gate does not allocate enough tokens for the LLM to “think before it writes itself into a corner”, as the chapter warns. The model is forced to produce output before fully reasoning, leading to premature decisions.
• Guard — The source provides no guard identifier for this condition. The chapter merely states the rule “give the model enough tokens to think” without specifying an exception handler, retry, or validation.
• Posture — fail-soft. The system continues but the output may be suboptimal or contain errors, because the model writes itself into a corner.
• Operator signal — The operator would see illogical or truncated tool recommendations; no explicit metric or log line is given in the source.
• Recovery — No retry logic is described. The manual fix is to increase the token limit in the API call configuration.

Guardrail Classifier Coverage Collapse (BERT’s Safety Margin Failure)

• Trigger — The human approval gate uses a guardrail classifier (e.g., a BERT-based model) to filter harmful outputs. At the optimal threshold, BERT’s safety guarantees become volatile, with coverage dropping to 55% — a “coverage collapse” as documented in the Beyond Red-Teaming source.
• Guard — The guard is the guardrail classifier itself, but it fails to provide formal guarantees. The source identifies “SVD-aligned hyper-rectangles” and “Gaussian Mixture Models (GMM)” as verification methods, not as runtime guards. No exception handler or fallback is shown.
• Posture — fail-soft. The system continues, but the guardrail classifier lets harmful content through because its safety margin is sparsely populated; the operator sees “coverage” metrics drop from 90% (GPT-2) or 80% (Llama-3.1-8B) to 55% for BERT.
• Operator signal — The operator observes the “coverage collapse” metric; a “SAT” result from hyper-rectangle certificates would reveal verifiable safety holes, even when empirical metrics are high.
• Recovery — The source suggests adopting an extremely conservative pessimistic threshold to achieve full coverage, which would require manual threshold tuning and likely degrade usability.

Missing Absolute File Path (Poka Yoke Violation)

• Trigger — A tool within the human approval gate is implemented using relative filepaths instead of absolute filepaths, violating the poka yoke (mistake-proofing) rule that “a tool was changed to always use absolute filepaths.”
• Guard — The source does not name a specific guard function. The correction described is to “always use absolute filepaths”, but no runtime validation, retry, or exception handler is provided.
• Posture — fail-hard. A file-resolution error (e.g., “File not found”) would abort the current operation, because the tool cannot locate the intended resource.
• Operator signal — The operator would see a file system error message; the source gives no exact log line.
• Recovery — No automatic recovery; the manual step is to replace all relative paths with absolute paths in the tool’s implementation.

Omitted Gate at a Critical Decision Step

• Trigger — The human approval gate is not applied to a step where “a mistake would be an error”, contradicting the chapter’s rule that a gate should only be placed where mistakes are critical. The omission occurs because the designer assumed the step was safe.
• Guard — No guard exists for this failure; the chapter only advises selectively placing gates, but provides no exception handler or validation to detect the omission.
• Posture — fail-soft. The system proceeds without a gate, so an error may occur and degrade reliability, but the run does not abort.
• Operator signal — The operator may observe uncaught tool-recommendation errors or an unexpectedly high hallucination rate (HR) for that specific step; no explicit metric is given.
• Recovery — No automatic recovery; the manual step is to add a human approval gate at the identified critical step after the fact.

Rejection feedback allows guardrails to adapt by instructing the agent not to retry a dangerous tool call.

agent.invoke(
    Command(
        resume={
            "decisions": [
                {
                    "type": "reject",
                    "message": "User rejected this action. Do not retry this tool call.",
                }
            ]
        }
    ),
    config=config,
    version="v2",
)

Think of a coach who watches their team practice, spots the same fumbles happening again and again, and then changes the drills to prevent those mistakes before the big game. Adaptive guardrails do exactly that—they are safety checks that evolve based on real-world failures instead of staying fixed forever, making AI systems steadily more reliable.

The coach runs many test plays, carefully noting every time a player stumbles. In the same way, one team ran countless examples with their AI model, logging exactly where it made mistakes. Then they redesigned the tools the model used—for instance, they required the model to use absolute file paths instead of relative ones, which eliminated a whole class of errors that happened whenever the model changed directories. They also tweaked parameter names and descriptions to be more obvious, making it harder for the model to misuse them. Another study tested different combinations of defenses, building a stack of three layers: database-grounded context, fixed vocabulary, and structured output. They found that using all three together slashed failure rates from over 70% down to under 15%.

The trickiest lesson came when they tried using just one guard—the structured output enforcement—by itself. Without the other two checks, the model actually hallucinated more than it did with no safeguards at all, because the required fields forced it to invent fake tool names from its training data. That is the non‑obvious edge: a single safety rule can backfire if it is not paired with grounding mechanisms. Without adaptive guardrails, you would never catch that hidden trap, and the system would keep delivering false recommendations that look convincing but are completely wrong.

In the adaptive guardrails subsystem, the ordered mechanism begins with adaptive safety check generation, where the system generates context-specific safety checks based on the task and environment. Next, effective safety check optimization refines these checks by iterating on examples and tuning parameters—mirroring the practice of carefully crafting the agent-computer interface (ACI) through thorough tool documentation and testing, such as requiring absolute file paths to eliminate path-related errors. On failure—when a generated check proves ineffective—the system re-enters the optimization loop or falls back to a default guardrail, ensuring continuous improvement rather than a single static decision.

The invariant preserved by this design is lifelong adaptation—the guardrails are never fixed for all time. The source explicitly names this as a lifelong agent guardrail that maintains “strong performance against task-specific and system risks” across evolving scenarios. This guarantees that safety checks remain effective even as the agent’s environment or task requirements change, preventing degradation over time that would otherwise expose the system to unmitigated vulnerabilities.

The key trade-off is between simplicity of static guardrails and the adaptability of dynamic generation. The obvious rejected alternative is a fixed set of safety checks applied uniformly, which avoids the cost of frequent retraining and reduces operational overhead. However, that alternative is rejected because it would fail against novel attack vectors or task-specific risks—as seen in the broader literature where “standardized evaluation frameworks remain limited” and “significant gaps persist against novel attack vectors.” By choosing adaptive generation and optimization, the subsystem incurs the cost of extra iteration but avoids the far greater cost of brittle, outdated guardrails that lead to unsafe agent behavior at scale.

A concrete failure mode arises when the optimization step produces a safety check that only partially covers the input space. For example, if the adaptive check for toxicity relies on a classifier with a sparsely populated safety margin, the system may suffer a coverage collapse—a term from the Beyond Red-Teaming paper applied to BERT, where coverage dropped to 55% at the optimal threshold. An operator would observe a sharp drop in the safety pass rate (e.g., from 90% to 55%) across logged prompts, signaling that the guardrail’s effective protection interval has narrowed and that re-optimization or a fallback mechanism must be triggered.

1. Overfitting to Historical Examples

• Trigger: The team optimizes tool design based on a finite set of observed mistakes, causing the guardrail to miss novel error patterns in unencountered scenarios.
• Guard: No explicit guard in the source; the adaptive process itself (e.g., adaptive safety check generation) is the only mechanism, and it lacks a validation step for unseen failure modes.
• Posture: fail-soft — the system continues operating but with degraded detection accuracy for new attack types.
• Operator signal: Silent degradation; no log or metric indicates reduced coverage, so the operator would observe an unexplained rise in false negatives only after damage occurs.
• Recovery: Manual step: collect a more diverse set of failure examples and re‑run the optimization loop.

2. Incompatible Tool Change (Absolute Path Requirement)

• Trigger: The agent is deployed in a sandboxed environment where write access to absolute paths is forbidden, causing the redesigned tool (which now requires absolute paths) to fail.
• Guard: No guard is shown in the source; the tool modification required absolute file paths does not include a fallback or permission check.
• Posture: fail-hard — the tool throws an error and aborts the current run.
• Operator signal: Error log line containing Permission denied or Path not allowed — exact format unspecified in source.
• Recovery: Manual step: either adjust the tool to accept relative paths when absolute paths are unavailable, or reconfigure the sandbox to grant write permission.

3. Defense Combination Interaction Failure

• Trigger: The study tests individual defenses but does not validate interactions; when multiple defenses are stacked, they interfere (e.g., one guardrail classifier misclassifies benign inputs due to a preceding filter).
• Guard: No guard in the source addresses combined effects; the evaluation was limited to separate runs.
• Posture: fail-soft — the system continues, but safety is compromised (e.g., increased false positives or undetected attacks).
• Operator signal: Anomaly in safety metrics (e.g., rise in false positive rate or missed jailbreak alerts) without an explicit error log.
• Recovery: Manual step: conduct a new study that tests the full set of combined defenses and revert or reorder components accordingly.

4. Performance Degradation from Frequent Optimization

• Trigger: The teamʼs habit of repeatedly optimizing tools (e.g., changing parameter names and descriptions) introduces latency overhead without a throttling mechanism.
• Guard: No guard in the source limits the frequency or cost of optimization; the system runs the optimizations continuously.
• Posture: fail-soft — the system continues but with increased inference time, degrading user experience.
• Operator signal: Observed latency spike in transaction logs (e.g., average response time increases by a factor of 2–3); no explicit error.
• Recovery: Manual step: impose a rate limit on optimization runs or schedule them during low‑load periods.

5. Single‑Defense Insufficiency (Incomplete Context)

• Trigger: The study tested different combinations of defenses and concluded that using a single guardrail (the truncated phrase using a single g suggests a single guardrail) leaves systemic risks unaddressed.
• Guard: No guard in the source compensates for the missing defenses; the architecture relies on the tested combination but the combination was incomplete for certain attack vectors.
• Posture: fail-soft — the system continues but is vulnerable to attacks that were not covered by the chosen single defense.
• Operator signal: Silent vulnerability; no metric or log would alert the operator until an actual compromise occurs.
• Recovery: Manual step: design and integrate additional complementary guardrails (e.g., using Attention Tracker or Constitutional Classifiers as cited in other sources) and re‑evaluate.

A human-in-the-loop guardrail rejects tool calls to enforce strict boundaries before granting broader autonomy.

agent.invoke(
    Command(
        # Decisions are provided as a list, one per action under review.
        # The order of decisions must match the order of actions
        # in the interrupt request.
        resume={
            "decisions": [
                {
                    "type": "reject",
                    # Optional: explain why the action was rejected
                    # and whether the agent should retry a different approach.
                    "message": "User rejected this action. Do not retry this tool call.",
                }
            ]
        }
    ),
    config=config,  # Same thread ID to resume the paused conversation
    version="v2",
)

Think of a young assistant learning to run errands. First, they must show they can follow a precise set of instructions, like using a full street address instead of a vague “over there.” That simple rule prevents getting lost. This subsystem is for making sure a language model can be trusted with more control only after it proves it can stick to strict guidelines, so it doesn’t make costly mistakes.

The model’s first test is a guardrail: it must use absolute file paths, not relative ones, which eliminates errors from ambiguous references. Once it passes that, it earns the right to handle more tasks. Next, the system adds database-grounded context injection and a fixed vocabulary of allowed tool names. These mechanisms cut the rate of made-up suggestions—called hallucinations—from over fifty-nine percent down to between three point three and fourteen point nine percent, a huge jump in reliability. The model now only recommends items from a verified list.

The trickiest part is that even simple rules can backfire. The source reveals a surprising anomaly: if you only force the model to output in a specific format without giving it a list of allowed terms, the hallucination rate actually goes up by ten to fifteen percentage points. This happens because the required format pressures the model to fill empty slots with whatever it remembers from training, even if wrong. So the protective boundaries must be carefully combined—just one piece without the others can make things worse. Without this full subsystem, the model would confidently suggest tools that don’t exist, leading to broken workflows and wasted time.

The subsystem implements a three-mechanism mitigation stack consisting of database-grounded context injection, fixed vocabulary constraints, and enforced JavaScript Object Notation (JSON) output. The ordered mechanism applies these mechanisms in sequence: first, the model is injected with verified database context to ground its knowledge; second, its output vocabulary is constrained to a fixed set of allowed entities; and third, the output format is forced into a strict JSON schema. If any of these mechanisms fail—for instance, if the database injection produces no matching context or the JSON schema cannot be populated with valid entries—the system rejects the output entirely or falls back to a restricted mode. Under the full architecture, the hallucination rate (HR) drops from 59–74% to 3.3–14.9%, demonstrating the combined effectiveness of the ordered stack.

The invariant preserved by this design is factual reliability with respect to the closed inventory. The system guarantees that every recommended tool is present in the verified inventory, enforced by the fixed vocabulary constraint that prevents the model from generating tool names outside that set. This is a strict write boundary: the model cannot introduce any entity that has not been pre-audited. The architecture rejects the alternative of unconstrained generation, which would allow the model to freely recommend tools from its training priors. The cost avoided by this rejection is the 59–74% hallucination rate observed in the unconstrained baseline, which would make the system untrustworthy for high-stakes engineering tool selection.

The key trade-off is between strict output enforcement and the risk of forcing erroneous completions when grounding is missing. This emerges starkly in the C3 anomaly: under the configuration where only JSON output enforcement is active without any grounding mechanisms, the hallucination rate increases above the unconstrained baseline for all providers (+10.1 percentage points for Generation 1, +15.1 points for Generation 2). The system is built this way despite that cost because mandatory entity fields in the JSON schema exert slot-filling pressure, forcing models to populate tool-name slots from training-data priors when no grounding vocabulary is provided. The obvious alternative—relying solely on JSON enforcement without grounding—would avoid the overhead of database injection and vocabulary constraints, but it would incur the much higher cost of increased hallucination under the C3 anomaly, as the model defaults to plausible but incorrect entries.

A concrete failure mode is the C3 anomaly itself, where an operator would see the hallucination rate spike compared to the unconstrained baseline—for example, a +10.1 percentage point increase in Gen1 or +15.1 pp in Gen2 across all providers. The signal an operator would observe is a sudden rise in out-of-inventory mentions, confirmed by a frequency-weighted audit showing that the majority of remaining out-of-inventory mentions correspond to real engineering tools absent from the platform’s inventory. This indicates that the JSON schema has forced the model to invent plausible-sounding but unverified tool names, a sign that the grounding mechanisms have been bypassed or omitted.

Residual Hallucination After Full Mitigation

• Trigger — The model encounters a request for a real-world tool that is either missing from the grounding database, outside the fixed vocabulary, or that elicits a generation that deviates despite the architectural constraints. The paper reports a final hallucination rate (HR) between 3.3% and 14.9% even when all three mechanisms are active.
• Guard — No additional guard beyond the full mitigation stack of database‑grounded context injection, fixed vocabulary constraints, and enforced JSON output is shown in the source. These mechanisms reduce the failure rate but do not eliminate it.
• Posture — fail‑soft. The system continues to generate tool recommendations; some of them are hallucinated (out‑of‑inventory mentions) but the overall pipeline runs to completion.
• Operator signal — In an audit, the operator observes a hallucination rate (HR) in the range 3.3–14.9%. The paper also notes that the majority of remaining out‑of‑inventory mentions correspond to real engineering tools absent from the platform’s inventory.
• Recovery — Manual inspection of the output batch; periodic inventory updates to add missing tools. No automatic retry or fallback is specified.

Incomplete Grounding Database

• Trigger — The platform’s verified inventory does not contain all real‑world engineering tools. When the model is queried for a tool that exists but is not in the database, database‑grounded context injection cannot supply the correct entry, and the model may fall back to its training‑data prior or generate a hallucinated name.
• Guard — The paper mentions a frequency‑weighted audit that identifies these missing tools, but this is a manual, post‑deployment validation step, not an automatic guard. No exception handler, retry, or fallback is described for this condition.
• Posture — fail‑soft. The system still returns a recommendation (possibly a hallucinated name); the error is only discovered during auditing.
• Operator signal — Audit output showing out‑of‑inventory mentions that correspond to real engineering tools not yet added to the platform’s inventory.
• Recovery — Manual addition of the missing tools to the inventory database. No automatic recovery path.

C3 Anomaly (JSON Enforcement Without Grounding)

• Trigger — The enforced JavaScript Object Notation (JSON) output guard is activated, but database‑grounded context injection and fixed vocabulary constraints are not. Mandatory entity fields in the JSON schema create slot‑filling pressure, forcing the model to populate tool‑name slots using training‑data priors, which drastically increases hallucination. The source reports a +10.1 percentage‑point increase for Generation 1 and +15.1 percentage‑point increase for Generation 2 over the unconstrained baseline.
• Guard — This failure is prevented by the full stack, specifically by adding database‑grounded context injection and fixed vocabulary constraints. Those two mechanisms provide the grounding necessary to fill the slots correctly.
• Posture — fail‑soft. The model continues to generate tool recommendations, but the hallucination rate is worse than if no guard were applied.
• Operator signal — The metric hallucination rate (HR) shows a sharp increase above the unconstrained baseline; the C3 anomaly is named in the source.
• Recovery — Reconfigure the system to include database‑grounded context injection and fixed vocabulary constraints. No automatic retry is implemented.

Slot‑Filling Pressure from Mandatory JSON Fields

• Trigger — When enforced JSON output is active (without grounding) and the JSON schema includes mandatory entity fields, the model experiences slot‑filling pressure. It is forced to produce a value for every required field, even when no correct value is known, leading to insertion of training‑data priors.
• Guard — The guard that prevents this is fixed vocabulary constraints, which restricts the possible values for tool‑name slots to a verified list. The source explicitly hypothesizes that providing a grounding vocabulary avoids the slot‑filling pressure.
• Posture — fail‑soft. The model still generates JSON output, but the tool‑name slots may contain hallucinated or irrelevant entries.
• Operator signal — Elevated hallucination rate; the operator may notice that tool names are plausible but incorrect, matching training‑data patterns rather than the platform inventory.
• Recovery — Add fixed vocabulary constraints as a guard. No automatic fallback or retry is defined.

Guardrail Classifier Safety Holes

• Trigger — A harmful prompt (e.g., a universal jailbreak) is sent to the system. The Guardrail Classifier fails to block it because the representation of that prompt lies outside the convex hull used for certification, or because the classifier’s coverage collapses at certain thresholds. The source (Kezins et al.) shows that every hyper‑rectangle configuration returns SAT (exposing safety holes) and that BERT’s coverage drops to 55% at the optimal threshold.
• Guard — The Guardrail Classifier itself is the intended guard, but the formal analysis reveals it provides no strict guarantee. The source identifies hyper‑rectangle and Gaussian Mixture Model certificates as evaluation tools, not operational guards.
• Posture — fail‑soft (or fail‑open if the classifier misses the attack). The system continues processing the harmful prompt; the safety violation occurs without interruption.
• Operator signal — A post‑deployment red‑team test or audit would find that a harmful prompt bypassed the classifier. No immediate runtime alert is specified.
• Recovery — Retrain or retune the Guardrail Classifier, possibly using updated adversarial data or switching to a classifier with more robust coverage (e.g., GPT‑2 or Llama‑3.1‑8B, which maintain 90% and 80% coverage respectively). Manual intervention to block the harmful output after detection.