Self-Healing Agents — Deep Dive

Reflection and self-critique are encoded in the agent's output schema via the 'criticism' field, enabling iterative self-improvement.

{
    "thoughts": {
        "text": "thought",
        "reasoning": "reasoning",
        "plan": "- short bulleted\n- list that conveys\n- long-term plan",
        "criticism": "constructive self-criticism",
        "speak": "thoughts summary to say to user"
    },
    "command": {
        "name": "command name",
        "args": {
            "arg name": "value"
        }
    }
}

Imagine a student writing an answer to a question and then checking it against a textbook to catch mistakes. This is what self-reflection does for a language model—it turns a single guess into a process of verifying and fixing its own output, making it more reliable by grounding answers in trusted knowledge. The model first generates an initial response, then retrieves relevant evidence from a stored database to check each claim. In one study, grounding the model in a verified inventory cut hallucination rates from over fifty percent down to under fifteen percent, using mechanisms like database-grounded context injection and fixed vocabulary constraints. A multi-layer framework breaks the answer into atomic claims, retrieves evidence for each, and verifies them with a smaller model before refining uncertain cases. This step-by-step verification acts like the student flipping through the textbook page by page, ensuring each part of the answer is supported. The trickiest part is that without proper grounding, forcing the model to follow a strict format can backfire—the context notes a "C3 anomaly" where enforcing JSON output without providing a verified vocabulary actually increased hallucination, as the model filled required fields with training-data guesses rather than external evidence. This shows that self-reflection requires not just checking but a reliable external source to check against, or the process degrades. Without this subsystem, the model would confidently produce false answers like a student insisting on a wrong answer without ever consulting the textbook, leading to unreliable outputs that undermine trust.

The subsystem operates as an iterative loop. After each action (a_t), the agent computes a heuristic (h_t) that evaluates trajectory quality. If the heuristic detects inefficiency or hallucination, the agent may reset the environment to start a new trial. Self‑reflection is triggered by providing two‑shot examples—pairs of a failed trajectory and an ideal reflection—to the Large Language Model (LLM). These reflections are then added to the agent’s working memory, up to a maximum of three, forming context for subsequent LLM queries. Finally, the model is fine‑tuned using Chain of Hindsight (CoH): given a sequence of past outputs ranked by reward and accompanied by human‑provided hindsight feedback, the model predicts only the best output (y_n) conditioned on the prefix. This turns a single generation attempt into an iterative, self‑correcting process.

The design preserves the invariant that the trajectory will not contain a sequence of consecutive identical actions leading to the same observation. Specifically, the heuristic function defines hallucination as exactly such a pattern. When it occurs, the agent resets the environment, guaranteeing that no infinite loop of repeated, unproductive actions can persist. This invariant is enforced by the (h_t) computation at every step and the optional reset decision, providing a clear bound on execution progress.

The key trade‑off is between latency and factual reliability. By adding iterative self‑reflection steps, the system increases total response time compared to a one‑shot generation. However, this rejects the obvious alternative of a single pass without external grounding, which the source shows yields hallucination rates above 50%. The cost avoided is the high error rate—one study documents that grounding in a verified database cut hallucination from over 50% to under 15%. The iterative mechanism therefore exchanges speed for the guarantee that outputs are checked against stored knowledge and historical feedback, reusing external evidence rather than relying on the model’s parametric memory alone.

A concrete failure mode occurs when the model repeatedly outputs the same action across consecutive steps, leading to unchanged observations. The heuristic (h_t) detects this pattern as a hallucination, and the agent signals the operator by resetting the environment to begin a new trial. The operator would see the environment restart without the requested output, accompanied by the heuristic’s internal flag that a self‑reflection reset was triggered. Without this signal, the system would stall silently, but the design ensures the failure is visible and actionable.

Retrieval Failure in Database-Grounded Context Injection

• Trigger — The external knowledge base query returns no relevant documents or incorrect documents, e.g., the query formulation fails or the database lacks coverage. This failure is documented as retrieval errors and incomplete or selective integration of evidence (Barua et al., 2026).
• Guard — The source does not specify an exception handler, retry, or fallback for a failed retrieval. The mitigation stack only assumes successful retrieval; no guard identifier such as a try‑except or fallback variable is provided.
• Posture — fail‑soft: the model continues generation without external evidence, relying on internal training priors, degrading factual reliability.
• Operator signal — "retrieval errors" and "incomplete or selective integration of evidence" are the reported failure modes. The operator would observe unsupported claims or missing citations in the output.
• Recovery — No automatic recovery. The operator must manually correct the query, expand the database, or re‑run the pipeline; the system does not retry.

C3 Anomaly from JSON Output Enforcement Without Grounding

• Trigger — When only enforced JSON output is active and neither database‑grounded context injection nor fixed vocabulary constraints are present, the mandatory entity fields in the JSON schema exert slot‑filling pressure, forcing the model to populate tool‑name slots from training‑data priors (Menxhiqi & Marinova, 2026).
• Guard — The intended guard is the full three‑mechanism mitigation stack: database‑grounded context injection, fixed vocabulary constraints, and enforced JSON output. Under the anomalous configuration (only JSON enforcement), no guard is active.
• Posture — fail‑soft: the system continues to generate outputs, but with increased hallucination rate (observed +10.1 pp in Gen1, +15.1 pp in Gen2).
• Operator signal — The hallucination rate (HR) metric rises above the unconstrained baseline. A frequency‑weighted audit reveals remaining out‑of‑inventory mentions.
• Recovery — No automatic recovery. The operator must activate the missing grounding mechanisms and re‑run the system; retry is manual.

Self‑Reflection Loop Inefficiency or Indefinite Iteration

• Trigger — The heuristic function ( h_t ) fails to detect a stalled trajectory (e.g., the model outputs consecutive identical actions leading to the same observation). The source defines hallucination as exactly that pattern, and inefficient planning as trajectories that take too long without success.
• Guard — The guard is the heuristic function itself, which decides when to stop and optionally reset the environment. However, the source does not specify a timeout, retry limit, or fallback if the heuristic never triggers.
• Posture — fail‑soft: the system continues iterating, consuming resources without progress. The task may eventually be interrupted externally, but the framework itself does not abort.
• Operator signal — "Inefficient planning refers to trajectories that take too long without success." The operator would observe prolonged response times or excessive API calls without task completion.
• Recovery — The environment may be reset if the heuristic triggers; otherwise manual intervention is required. No automatic retry count or backoff is given.

Overfitting and Shortcutting in Chain of Hindsight

• Trigger — During fine‑tuning of Chain of Hindsight (CoH), the presence of many common words in the feedback sequences causes the model to learn shortcuts and copy past outputs rather than genuinely self‑reflecting.
• Guard — Two guards are defined: a regularization term to maximize log‑likelihood of the pre‑training dataset, and a randomly mask 0% - 5% of past tokens during training procedure. These reduce but do not eliminate the risk.
• Posture — fail‑soft: the model’s self‑reflection quality degrades; training continues, but downstream performance suffers from redundant or copied outputs.
• Operator signal — The paper notes "shortcutting and copying (because there are many common words in feedback sequences)". The operator would observe a lack of genuine improvement in produced sequences during evaluation.
• Recovery — No runtime recovery. The issue is addressed during training by adjusting the masking rate or regularization strength; manual re‑training is necessary.

Probabilistic Miscalibration Leading to Confident Hallucinations

• Trigger — The model outputs a factually incorrect statement with high confidence because its internal probability estimates are misaligned with correctness – one of the four miscalibration types (probabilistic miscalibration) described in Hu et al. (2026).
• Guard — The source discusses temperature scaling and domain‑conditional Platt scaling as post‑hoc calibration techniques (Reed & Mason, 2025), but the self‑reflection subsystem does not incorporate any calibration check at inference time. No guard is present in the described reflection loop.
• Posture — fail‑soft: the model continues to generate confident‑ but‑wrong outputs, potentially misleading downstream decision‑makers. The system appears reliable while being incorrect.
• Operator signal — "confident hallucinations" and "semantic miscalibration" are the identified failure indicators. The operator would observe statements that sound plausible but are factually unsupported.
• Recovery — No automatic recovery. The source calls for interaction designs that surface model confidence to downstream decision‑makers, but no concrete recovery mechanism is implemented.

Agent response format with criticism field for feedback.

{
    "thoughts": {
        "text": "thought",
        "reasoning": "reasoning",
        "plan": "- short bulleted\n- list that conveys\n- long-term plan",
        "criticism": "constructive self-criticism",
        "speak": "thoughts summary to say to user"
    },
    "command": {
        "name": "command name",
        "args": {
            "arg name": "value"
        }
    }
}

Imagine you’re a student who writes an answer, but instead of just a pass or fail, your teacher gives you a numerical score out of 100. If the score is too low, you have to rewrite the answer until it improves. This subsystem is for that: a retry loop that uses a critic to judge how reliable each part of an answer is, so the system can decide whether to regenerate it rather than trust a weak response.

In practice, the system first breaks a generated answer into tiny, factual claims—think separate sentences that each state a specific fact. For each claim, it searches for supporting evidence from trustworthy sources. Then a lightweight critic, called a small language model verifier, scores how well the evidence backs up each claim. If the score is too low, the system retries that part from scratch. The critic uses a calibrated confidence score, and in tests the area under the ROC curve was 0.87—meaning the score is a useful indicator of reliability, not random guessing.

The tricky part is picking the right threshold for what counts as “too low.” The team found that a stricter threshold (around 0.60) cut down on falsely supported claims, even though a looser threshold (0.40) gave better raw scores. Without this critic and retry loop, the system would often accept confident but wrong answers, leaving you with believable nonsense instead of fact‑checked truth.

The retry‑with‑feedback subsystem begins with the LLM generating an answer against a user query and an input context. That answer is first passed through a lightweight hallucination detector: a linear SVM that operates on 30 k unigram TF‑IDF features augmented with 11 overlap/length features. The detector outputs a raw score, which is then transformed into a calibrated confidence via domain‑conditional Platt scaling. If the calibrated confidence falls below an operator‑set threshold, the system regenerates the answer and cycles the same critic over the new output. This ordered mechanism—generation, detection, calibration, conditional retry—ensures that only outputs whose confidence meets the threshold proceed to the next stage.

The invariant the design preserves is calibrated confidence: the score produced by domain‑conditional Platt scaling is intended to match the true probability that the output is factually grounded. The source explicitly names this calibration target and validates it by reporting a ROC‑AUC of about 0.87, which “are useful confidence behaviour.” Because the critic outputs a continuous, calibrated score instead of a binary pass/fail, the operator can choose any threshold, enabling fine‑grained control over the precision‑recall trade‑off. The guarantee is that the system will only accept outputs whose confidence reflects a known (calibrated) reliability level.

The key trade‑off is lightweight, interpretable verification versus accuracy. The design rejects the obvious alternative of using a larger, more capable LLM as the critic. That alternative would likely yield higher detection accuracy but at prohibitive computational cost and, as the context notes in a separate finding, may itself suffer from “a potential problem with using LLM to evaluate its own performance on domains that requires deep expertise.” By choosing a linear SVM on hand‑crafted features, the system avoids the expense and opacity of a large‑model judge while still achieving useful discrimination (AUC 0.87). The cost of this rejection is a modest false‑positive and false‑negative rate, which the retry loop partially absorbs.

A concrete failure mode is a false negative: a hallucinated output receives a high calibrated confidence (e.g., 0.95) and passes the retry loop’s threshold, even though the answer contains an unsupported claim. The signal an operator would see is the final confidence score, misleadingly high, accompanied by the output text. Without manual inspection or a downstream verification step, the operator has no indication that the score is miscalibrated for that particular instance. The source acknowledges that such residual hallucinations persist despite the critic: evaluation practices remain retrospective, and “residual hallucinations despite retrieved context” are a recurrent failure mode. Observing a high‑confidence output that later proves unsupported would be the operator’s actionable signal to investigate the critic’s calibration on that specific domain.

1. Probabilistic Miscalibration of the Critic’s Confidence

• Trigger — The critic’s raw confidence scores (from the linear SVM with TF‑IDF features) do not match the empirical error rate, causing over- or under‑confidence across the whole output distribution.
• Guard — domain‑conditional Platt scaling (trained on a held‑out validation set) post‑processes the scores to re‑align them with the true error frequency.
• Posture — fail‑soft: the critic continues to produce scores, but calibration error degrades the reliability of the retry decision; the system still accepts or rejects outputs.
• Operator signal — a rising expected calibration error (ECE) on the HaluEval benchmark’s held‑out split, or a ROC‑AUC value that drops substantially below the reported 0.87.
• Recovery — no automatic retry; the operator must retrain the domain‑conditional Platt scaling or switch to temperature scaling using a fresh validation set.

2. Semantic Miscalibration — Confident Yet Factually Wrong Score

• Trigger — The critic assigns a high calibrated score to an output that is actually a hallucination (e.g., a factually unsupported claim). This happens when the linear SVM’s TF‑IDF unigrams and 11 overlap/length features fail to capture deeper semantic flaws.
• Guard — No guard from the source. The system relies solely on the calibration step, which cannot correct a poor underlying feature representation.
• Posture — fail‑soft: the high score passes the retry threshold, so the hallucination is accepted as final output; downstream use continues with incorrect information.
• Operator signal — a silent false negative: no log line or metric flags the hallucination; the operator only sees a post‑deployment complaint.
• Recovery — manual: the operator must inspect misclassified examples from the HaluEval test set, augment the TF‑IDF vocabulary, or add new overlap/length features.

3. Distributional Miscalibration — Collapsed Diversity of Critic Scores

• Trigger — The critic’s calibrated scores cluster in a narrow range (e.g., all near 0.5 or all near 0.9), so the retry decision becomes nearly deterministic regardless of actual output quality.
• Guard — domain‑conditional Platt scaling can amplify differences per domain, but the source does not explicitly guard against global score plateauing.
• Posture — fail‑soft: the system continues to retry or accept based on a nearly constant score, effectively degrading the value of the feedback loop.
• Operator signal — a histogram of critic scores (from periodic logging) shows extreme kurtosis or a single mode; the ROC‑AUC remains high but the score distribution is uninformative.
• Recovery — operator must retune the scaling parameters (e.g., adjust temperature in temperature scaling) or re‑train the linear SVM with richer features.

4. Feature Representation Gap for Out‑of‑Domain Inputs

• Trigger — A user query or model output contains terminology, paraphrases, or structure never seen in the HaluEval training data. The 30k unigram TF‑IDF features cannot represent the novel pattern, so the critic’s score is unreliable.
• Guard — No guard from the source. The system has no fallback for out‑of‑vocabulary or low‑frequency n‑grams.
• Posture — fail‑soft: the critic produces a score anyway (often a default low‑confidence output), leading to either needless retries or undetected hallucinations.
• Operator signal — a spike in skip‑gram warnings (if logged) or a sudden divergence between the critic’s score and human‑judged quality in a new domain.
• Recovery — manual: the operator must expand the TF‑IDF feature set (e.g., add bigrams or use subword tokenization) and re‑train the linear SVM on a broader corpus.

5. Retry Loop Without Maximum Attempt or Backoff Control

• Trigger — The critic persistently returns a low calibrated score (e.g., below a fixed threshold) for a particular input, causing the system to regenerate output indefinitely with no retry limit or exponential backoff.
• Guard — No guard from the source. The description mentions “a retry loop” but does not specify a maximum retry count, fallback, or timeout.
• Posture — fail‑hard: the loop never terminates, blocking the system from processing further requests or exhausting API call quotas.
• Operator signal — a monotonically increasing retry counter in the metrics dashboard; a growing queue of pending requests or rising latency; finally a timeout exception from the calling process.
• Recovery — manual: the operator must kill the process, set an explicit maximum retry count, and implement exponential backoff.

No relevant code excerpt is available in the provided context.

Think of this subsystem as a librarian who checks every book reference before allowing it on the shelf, making sure each fact has a solid source. It is meant to catch when an AI invents details that sound true but are not, stopping false information from reaching you. The librarian starts by splitting the AI's answer into tiny claim-sized pieces. For each piece, she looks up official documents to find support. She uses a detailed checklist with thirty thousand small word patterns and eleven rules about how sentences overlap and are structured. A special calculator adjusts her certainty based on the topic, making her trust level more accurate. This process has proven reliable about eighty-seven percent of the time according to the ROC AUC score. When her certainty is low, she starts a second round of checking that strictly forces the answer to only use facts from the approved inventory. The trickiest part is that she deliberately sets a very high bar for what counts as supported, even though this means many correct statements get flagged. This is because the system is calibrated with a threshold near sixty percent to avoid approving any false claim, prioritizing safety over convenience. Without this subsystem, the AI could confidently give you wrong facts, leading you to make mistakes in important decisions like choosing a medicine or an engineering tool.

The subsystem operates in a staged detection pipeline. First, a lightweight linear support vector machine (SVM) processes the model output using exactly 30,000 unigram TF‑IDF features augmented with 11 overlap and length features. This binary classifier scores the output for hallucination risk. Next, domain‑conditional Platt scaling is applied to the raw SVM score to produce a calibrated confidence value—this step ensures that the confidence level reliably reflects the probability of hallucination. If the calibrated confidence falls below a predetermined threshold, the system triggers a post‑generation verification phase that explicitly checks for unsupported claims. Only outputs that pass both the detector and the verification step are released.

The invariant the design preserves is calibrated confidence, as evidenced by the reported ROC‑AUC value of ≈ 0.87. The domain‑conditional Platt scaling mechanism guarantees that the model’s confidence scores are aligned with actual hallucination probabilities, enabling downstream policies (such as abstention or verification) to behave predictably. This calibration invariant is the foundation for the entire detection‑before‑release strategy—without it, threshold‑based decisions would be arbitrary and unreliable.

The key trade‑off is the deliberate choice of a lightweight linear SVM over a more powerful deep‑learning classifier. The obvious alternative is a large neural network that could achieve higher raw accuracy but would incur substantial computational cost and opacity. This design rejects that alternative to avoid the expense of GPU‑bound inference and the lack of interpretability in high‑volume or resource‑constrained deployments. By keeping the detector lean, the subsystem remains computationally practical while still providing useful confidence behaviour—the 0.87 AUC demonstrates that the trade‑off does not sacrifice essential discriminative power.

A concrete failure mode is a residual hallucination that passes both the detector and the verification step due to incomplete evidence integration. For example, if the external retrieval returns stale or topically irrelevant documents, the post‑generation verification may find no unsupported claims because the LLM internally fabricated a plausible‑sounding answer that matches no retrieved paragraph. An operator monitoring system logs would see a sustained rate of outputs flagged as “confident and verified” yet later contradicted by the inventory—manifesting as a gradual rise in complaints or a spread between the detector’s confidence histogram and actual hallucination counts. The signal would be a mismatch between the calibrated confidence distribution and the observed failure rate, indicating a breakdown in the calibration invariant.

Retrieval Error (Incomplete or Missing External Evidence)

• Trigger: The RAG retrieval process fails to return relevant, complete documents due to query formulation errors, index gaps, or low document quality.
• Guard: post generation verification — this step checks for unsupported claims, which can catch some outputs built on absent evidence, but it relies on the generation itself rather than on retrieval quality.
• Posture: Fail‑soft — the system continues to produce an answer, but grounding is weakened; the detector and verification may still filter the output, but retrieval failures degrade overall factuality.
• Operator signal: No explicit error field for retrieval misses is logged; the operator would observe an increased rate of hallucinated flags during downstream verification or a mismatch between the generated answer and the sparse retrieved snippets.
• Recovery: No automatic retry or backoff is described; the system proceeds with the imperfect context. Manual steps would involve updating the retrieval index, re‑engineering query formulation, or adding fallback to a more thorough retrieval pass.

Lightweight Detector False Negative (High‑Confidence Hallucination Missed)

• Trigger: A hallucinated output that the linear SVM on 30k unigram TF‑IDF features augmented with 11 overlap/length features scores with high confidence, causing it to pass the detector threshold.
• Guard: No guard exists for this case. post generation verification is only invoked when confidence is low; here confidence is incorrectly high, so verification is skipped.
• Posture: Fail‑soft — the hallucinated output proceeds through the pipeline and is released to users, degrading trust without aborting the run.
• Operator signal: Silent absence — no log line, metric, or alert indicates the miss. The operator would only observe the failure later via user complaints or downstream accuracy audits.
• Recovery: No automatic recovery; manual inspection of detector outputs and retraining of the SVM on harder negatives would be required.

Calibration Miscalibration Under Domain Shift

• Trigger: The domain‑conditional Platt scaling is applied to a new domain or distribution not represented in the held‑out validation set, producing miscalibrated confidence scores (e.g., overconfidence in hallucinations).
• Guard: No dedicated guard exists. The same post generation verification could theoretically catch some cases if confidence drops, but miscalibration can inflate confidence, bypassing the verification trigger.
• Posture: Fail‑soft — the system continues with degraded calibration; confidence scores no longer reliably reflect factuality, but the pipeline does not halt.
• Operator signal: The operator would see a discrepancy between reported confidence values and actual accuracy during periodic evaluations, or note that the ROC‑AUC measure declines on new data.
• Recovery: No automatic re‑calibration is described; manual retraining of the Platt scaler with representative data from the new domain is necessary.

C3 Anomaly (JSON Output Enforcement Without Grounding)

• Trigger: The mitigation stack is configured with only JSON output enforcement active, while database‑grounded context injection and fixed vocabulary constraints are disabled. The schema’s mandatory fields exert slot‑filling pressure, forcing the model to hallucinate tool names from training priors.
• Guard: The intended guard is the database‑grounded context injection mechanism; when present, it provides valid slot fillers and prevents the anomaly. The source explicitly notes that the anomaly disappears under the full three‑mechanism architecture.
• Posture: Fail‑hard — hallucination rate spikes (+10.1 pp to +15.1 pp over unconstrained baseline), and the system produces out‑of‑inventory tool mentions. The run continues but produces unreliable output.
• Operator signal: A clear rise in hallucination rate (HR) metric, especially for tool‑name fields; the operator would observe JSON‑output failures dominating error logs.
• Recovery: No automatic fallback; the operator must re‑enable database‑grounded context injection (and optionally fixed vocabulary constraints) to re‑establish grounding. The system does not retry or degrade gracefully when only JSON enforcement is active.

Post‑Generation Verification Misses Subtle Unsupported Claims

• Trigger: A hallucinated output that contains plausible‑sounding yet unsupported claims that the post generation verification (which checks for unsupported claims) fails to flag—either because the verification logic is too coarse, the claim is outside its scope, or the evidence comparator is incomplete.
• Guard: No further guard exists beyond the verification step itself. The mitigation stack (database‑grounded context injection, fixed vocabulary constraints, JSON output enforcement) applies before generation and does not re‑run after verification.
• Posture: Fail‑soft — the unsupported claim passes through to the user; the system has no abort or rejection mechanism at this stage.
• Operator signal: Silence — no error is raised. The operator would only detect the failure through downstream manual audits or a post‑release analysis of factuality scores.
• Recovery: No automatic recovery. Manual enhancement of verification logic (e.g., expanding the set of checks or adding attribution‑aware generation) is needed.

CONVOLVE method for agentic abstention

def convolve(trajectories):
    # Distills full interaction trajectories into reusable stopping rules
    # Aim: improve timely abstention without updating model parameters
    pass

Imagine a chef who keeps adding random spices when the correct ingredient is missing, instead of simply saying "I don't have that." This subsystem is for making an AI assistant know when to stop and admit it doesn't have the right information, rather than making things up.

In practice, the system works like a three‑step safety check. First, it looks up a trusted inventory (database‑grounded context injection) to see what tools or facts are actually available. Then it restricts the assistant to only use names from that approved list (fixed vocabulary constraints). Finally, it forces the output into a strict format (enforced JSON output) so the assistant cannot slip in extra words. When all three guards are active, the rate of made‑up answers drops from a shocking 59–74% down to 3–14%. But there’s a hidden trap: if only the JSON‑output rule is turned on without the grounding steps, the assistant actually increases its lies by over ten percentage points. This is called the C3 anomaly — the mandatory fields in the JSON schema pressure the model to fill them with whatever pops out of memory, even if it’s wrong.

The deepest reason for that backfire is slot‑filling pressure: when the model must provide a tool name but has no real inventory to draw from, it invents one from its training data. Without the subsystem, the assistant never learns to abstain; it barrels ahead with 35–63% of responses still containing unsupported claims, leaving users with confident but false answers they cannot trust.

In a closed-inventory system such as Online-CADCOM, where the agent must recommend only from a verified set of engineering tools, the mechanism for knowing when to stop is implemented as an ordered three-step mitigation stack. First, database-grounded context injection supplies the model with verified inventory data at inference time. Next, fixed vocabulary constraints restrict the set of allowable tool names to those present in the inventory. Finally, enforced JavaScript Object Notation (JSON) output forces the model to produce a structured response that can be validated against the schema. If at any point the model generates a tool name that does not appear in the inventory—a mention-level hallucination—the system detects the failure and should cause the agent to abstain rather than produce an unsupported recommendation. This ordering ensures that grounding is applied before structural enforcement, avoiding the pitfall of the C3 anomaly, where JSON enforcement alone (without prior grounding) increases the hallucination rate above the unconstrained baseline by over ten percentage points.

The core invariant maintained by this design is that all recommended tool names belong to the platform’s closed-inventory. The paper reports that under the full three-mechanism architecture, the hallucination rate (HR) drops from a baseline of 59–74% to a cross-provider average of approximately 6.8–7.5%, demonstrating that the design preserves the guarantee of inventory adherence. However, even with the full stack in place, 35–63% of responses still contain at least one out-of-inventory mention, indicating that the invariant is not absolute and that remaining failures represent real engineering tools absent from the inventory rather than fabricated names.

The key trade-off is between the reliability gained by layering all three mechanisms and the computational and latency cost of doing so. The obvious alternative—rejected by the design—is to rely solely on enforced JSON output without any grounding mechanisms. That approach is rejected because it triggers the C3 anomaly: mandatory entity fields in the JSON schema exert slot-filling pressure, forcing the model to populate tool-name slots from training-data priors when no grounding vocabulary is provided. By rejecting that simpler alternative, the design avoids the cost of increased hallucination (an extra +10.1 to +15.1 percentage points) and the subsequent need for manual verification of fabricated tool names. The cost it accepts instead is the engineering overhead of maintaining a verified inventory and the runtime required for database lookups and vocabulary constraints.

A concrete failure mode an operator would observe is an out-of-inventory mention in a response generated still under the full architecture. For instance, the agent might recommend a real engineering tool that happens not to be listed in the platform’s inventory. The operator would see the tool name appear in the JSON output field and could compare it against the inventory list to confirm the mismatch. The paper notes that a frequency-weighted audit showed the majority of remaining out-of-inventory mentions correspond to such real tools, meaning the agent failed to stop despite having grounding context and constraints. The signal to the operator is thus a response containing a tool name that does not appear in the curated inventory, indicating that the agent continued generating output when it should have abstained.

Failure 1: No Grounding Leading to High Hallucination Rates

• Trigger — The agent operates without any retrieval or database grounding, i.e., no external evidence is injected. The source reports hallucination rates of 59–74% under this condition.
• Guard — No guard is shown in the source for this failure. The heuristic heuristic $h_t$ exists but does not address the absence of grounding; it only detects a narrow pattern of consecutive identical actions.
• Posture — Fail-soft: the agent continues to generate outputs with degraded factual reliability; the run is not aborted.
• Operator signal — The metric hallucination rate (HR) would be observed at 59–74%.
• Recovery — No automatic recovery is specified. A human operator must add grounding mechanisms (e.g., retrieval-augmented generation) to the system.

Failure 2: Full Mitigation Stack Still Leaves Residual Hallucinations

• Trigger — Even when the full architecture (database-grounded context injection, fixed vocabulary constraints, and enforced JSON output) is active, 35–63% of responses still contain at least one out‑of‑inventory mention.
• Guard — The full architecture itself is the mitigation layer, but it is not sufficient to eliminate all failures. No additional guard for residual errors is shown.
• Posture — Fail-soft: the system continues to operate despite a substantial fraction of unsupported outputs.
• Operator signal — The fraction of responses with at least one unsupported mention remains 35–63%.
• Recovery — No automatic recovery; manual audit or human‑in‑the‑loop verification is required.

Failure 3: JSON Enforcement Without Grounding (C₃ Anomaly)

• Trigger — Only the enforced JSON output mechanism is active, with no grounding vocabulary or context injection. This alone increases hallucination above the unconstrained baseline by +10.1 pp (Gen1) and +15.1 pp (Gen2).
• Guard — No guard is shown in the source for this specific configuration. The full architecture is not applied.
• Posture — Fail-soft: hallucination worsens; the agent continues to produce outputs under slot‑filling pressure.
• Operator signal — A hallucination rate (HR) increase relative to the baseline, named the C3 anomaly in the source.
• Recovery — No automatic recovery; the system must be reconfigured to include grounding mechanisms.

Failure 4: Heuristic Fails to Detect Non‑Identical Hallucination Patterns

• Trigger — The heuristic heuristic $h_t$ defines hallucination only as “a sequence of consecutive identical actions that lead to the same observation.” Any other hallucination (e.g., factually incorrect but varied actions) is not detected.
• Guard — The heuristic $h_t$ is present but insufficient; it does not cover all hallucination types.
• Posture — Fail-soft: the agent continues on a trajectory that is actually hallucinated but not flagged.
• Operator signal — No immediate signal; the hallucination may be discovered later during evaluation or by external validation.
• Recovery — No automatic recovery from this failure. The agent continues until the trajectory is either stopped by another mechanism (e.g., timeout from inefficiency) or completed.

Failure 5: Self‑Reflection Memory Exhaustion

• Trigger — The agent can hold at most three reflection entries in its working memory. When a new failure occurs and memory is full, the oldest reflection is overwritten (as inferred from “up to three”) or the new reflection is discarded—the source does not specify the behavior.
• Guard — The limit up to three acts as a capacity guard, but there is no guard to handle overflow or preserve important reflections.
• Posture — Fail-soft: the agent continues, but may repeat past mistakes because the relevant reflection is lost.
• Operator signal — Silent; no error or log line is described.
• Recovery — No automatic recovery; a human might need to manually reset or expand the memory.

Failure 6: Retrieval Grounding Errors (RAG‑Induced Hallucination)

• Trigger — Errors arise during query formulation, document retrieval, evidence aggregation, or answer grounding. The source explicitly states: “Errors may arise during query formulation, document retrieval, evidence aggregation, and answer grounding.”
• Guard — The source mentions lightweight verification as a potential mitigation in the Epistemic Field Theory paper, but no exact guard identifier is given for this specific failure mode. No retry or fallback is described.
• Posture — Fail-soft: the agent uses incorrect or irrelevant evidence and may produce a hallucinated output.
• Operator signal — Factual errors or unsupported claims in the final output; no dedicated signal is defined.
• Recovery — No automatic recovery. The operator would need to inspect and correct the retrieval pipeline or rely on post‑generation verification.

Built-in checks enforce a fixed vocabulary and database grounding to reduce hallucination, as demonstrated by the GPT-Engineer clarification system.

[
    {
        "role": "system",
        "content": (
            "You will read instructions and not carry them out, only seek to clarify them.\n"
            "Specifically you will first summarise a list of super short bullets of areas that need clarification.\n"
            "Then you will pick one clarifying question, and wait for an answer from the user.\n"
        )
    },
    {
        "role": "user",
        "content": "We are writing {{a Super Mario game in python. MVC components split in separate files. Keyboard control.}}\n"
    },
    {
        "role": "assistant",
        "content": "Summary of areas that need clarification:\n1. Specifics of the Super Mario game (e.g. level design, characters, gameplay mechanics)\n2. Details about the MVC components (e.g. which components are in each file)\n3. Keyboard control implementation (e.g. ..."
    }
]

Think of this subsystem like a librarian who only lets you borrow books that are on the library’s official inventory list. It is for keeping a language model from recommending tools or facts that do not actually exist in its verified database.

The model first suggests something, like a tool name. Then a built-in check compares that suggestion against a fixed vocabulary list of approved items—only names already in that list pass. The librarian also gives the model a relevant database entry before it answers, so it knows which books are allowed. This grounding and the list together cut errors from over half down to just a few percent. But there is a catch: when the librarian forces you to fill out a checkout slip (a JSON output format) without showing you the inventory list first, you are more likely to write down a made‑up book. Researchers call this the C3 anomaly—it happens because the model feels pressured to put something in every required field, so it invents a title from memory instead of sticking to reality.

Without this subsystem, the model would confidently recommend imaginary tools nearly three‑quarters of the time, leaving users frustrated and unable to trust its suggestions.

The subsystem enforces a three-mechanism mitigation stack in ordered application: first, database-grounded context injection supplies the model with verified inventory data at inference time; second, fixed vocabulary constraints restrict the set of admissible tool names to those present in the closed inventory; third, enforced JavaScript Object Notation (JSON) output mandates that the model’s response conform to a structured schema. If any mechanism is omitted, the downstream steps operate on a degraded foundation—most critically, when only JSON enforcement is active without the preceding grounding steps, the model falls into the C3 anomaly where hallucination rates increase above the unconstrained baseline.

The design preserves the invariant of mention-level hallucination reduction: the full stack guarantees that the hallucination rate (HR)—the fraction of responses recommending tools absent from the inventory—drops from 59–74% to 3.3–14.9% across providers and generations. This guarantee is achieved by ensuring every tool-name slot in the JSON output is populated exclusively from the fixed vocabulary, so that outputs are grounded in the verified inventory rather than the model’s unconstrained training priors.

The key trade-off is against the simpler alternative of using only enforced JSON output without grounding. That alternative is rejected because it triggers the C3 anomaly: JSON enforcement alone increases hallucination by +10.1 percentage points (pp) for Gen1 and +15.1 pp for Gen2 above the unconstrained baseline. The cost avoided is a sharp rise in unsupported tool mentions, hypothesized to result from slot-filling pressure where mandatory JSON fields force the model to hallucinate tool names from training-data priors. The chosen architecture accepts the overhead of retrieval and constraint injection to keep errors low, even though 35–63% of responses under the full stack still contain at least one out-of-inventory mention, indicating that handling unseen tools remains an open challenge.

A concrete failure mode is the C3 anomaly itself. An operator monitoring system logs would observe that when enforced JSON output is deployed in isolation—i.e., without database-grounded context injection or fixed vocabulary constraints—the hallucination rate spikes significantly above the unconstrained baseline. For example, in Gen2 the HR rises by +15.1 pp, visible as a dramatic increase in the frequency of out-of-inventory tool names appearing in the JSON responses. This signal is directly measurable via automated audits of tool-name mentions, confirming that the slot-filling pressure from an ungrounded schema actively degrades factual reliability.

Failure 1: JSON-Only Enforcement C3 Anomaly

• Trigger — The configuration where only enforced JavaScript Object Notation (JSON) output is active without any grounding mechanisms (neither database-grounded context injection nor fixed vocabulary constraints).
• Guard — No guard exists in the source for this condition. The paper explicitly states that this anomaly “requires further experimental validation” and does not describe any handler or retry.
• Posture — Fail-soft: the system continues to produce output, but the hallucination rate increases above the unconstrained baseline (+10.1 pp for Gen1, +15.1 pp for Gen2). No abort or write refusal occurs.
• Operator signal — Observed hallucination rate (HR) higher than the baseline; specifically, the C3 anomaly manifests as a spike in out‑of‑inventory mentions when JSON schema fields force slot‑filling pressure.
• Recovery — The operator must manually add grounding mechanisms (database context injection and fixed vocabulary constraints) and re‑run the configuration. No automatic retry or fallback is provided.

Failure 2: Residual Out‑of‑Inventory Mentions Under Full Architecture

• Trigger — Even when all three mechanisms (database‑grounded context injection, fixed vocabulary constraints, enforced JSON output) are active, 35–63% of responses still contain at least one mention of a tool not in the verified inventory.
• Guard — The three‑mechanism stack itself serves as the primary guard, but it is insufficient. No additional exception handler or retry logic is described for this residual case.
• Posture — Fail‑soft: the system continues to generate recommendations, and the out‑of‑inventory mentions are not blocked or aborted. The run proceeds with degraded factual reliability.
• Operator signal — A frequency‑weighted audit reveals that the majority of remaining out‑of‑inventory mentions correspond to real engineering tools absent from the platform’s inventory. The exact metric is the percentage of responses containing such mentions.
• Recovery — No automatic recovery is specified. The operator must manually audit the unseen mentions and either update the inventory or add additional retrieval grounding. The paper notes this as “an open challenge”.

Failure 3: Unconstrained Baseline Hallucination Before Mitigation

• Trigger — The system operates without any of the three mitigation mechanisms (no database grounding, no vocabulary constraints, no JSON enforcement). The initial hallucination rate (HR) is between 59–74%.
• Guard — No guard is present in this baseline condition. The entire mitigation stack is the intended guard.
• Posture — Fail‑soft: the system runs and produces output, but with a very high rate of unsupported claims. No abort or write refusal.
• Operator signal — The hallucination rate (HR) is observed to be in the 59–74% range. The paper reports this as the starting point for the ablation study.
• Recovery — The operator must deploy the full architecture (database‑grounded context injection, fixed vocabulary constraints, enforced JSON output) to reduce the HR to 3.3–14.9%. No automatic fallback exists.

Failure 4: Reasoning‑Mode No‑Improvement Under Architectural Constraints

• Trigger — The system uses reasoning‑mode models (e.g., from providers like OpenAI, Anthropic, Google) while the full mitigation architecture is active. The source states that “reasoning‑mode models provide no statistically significant improvement under architectural constraints.”
• Guard — No guard is implemented to detect or handle this lack of improvement. The system proceeds with reasoning mode regardless.
• Posture — Fail‑soft: the system continues to operate, but the reasoning mode does not improve factual reliability beyond the standard mode. The run degrades in efficiency (extra compute without benefit).
• Operator signal — No statistically significant difference in hallucination rate (HR) or other metrics between reasoning and standard output modes when the architecture is applied.
• Recovery — The operator must explicitly select standard output mode instead of reasoning mode. The source does not describe any automatic detection or fallback.

Failure 5: Slot‑Filling Pressure from Mandatory JSON Fields

• Trigger — When JSON output enforcement is active without a grounding vocabulary, the mandatory entity fields in the JSON schema exert “slot‑filling pressure”, forcing the model to populate tool‑name slots from training‑data priors. This is the hypothesized cause of the C3 anomaly.
• Guard — No guard exists; the paper calls for “further experimental validation” of this hypothesis and does not provide a handler.
• Posture — Fail‑soft: the system continues generating JSON output, but with hallucinated tool names inserted into required fields. No abort occurs.
• Operator signal — Increased hallucination rate (HR) above baseline, specifically in the tool‑name slots of the JSON output. The operator would see tool names that are not in the inventory.
• Recovery — The operator must add database‑grounded context injection or fixed vocabulary constraints to provide a valid vocabulary for the mandatory fields. No automatic recovery is described.

Forced JSON output format that can increase hallucination when applied without grounding.

{
    "thoughts": {
        "text": "thought",
        "reasoning": "reasoning",
        "plan": "- short bulleted\n- list that conveys\n- long-term plan",
        "criticism": "constructive self-criticism",
        "speak": "thoughts summary to say to user"
    },
    "command": {
        "name": "command name",
        "args": {
            "arg name": "value"
        }
    }
}

Imagine someone confidently filling out a form that demands specific fields—like "item name" and "quantity"—but without the actual shopping list. That is what this subsystem reveals: large language models can produce confident but wrong answers, and attempts to force them into correct behavior can actually make the errors worse. Its purpose is to show why self-repair fails.

Now add step‑by‑step detail. A system trying to guarantee accurate output might require the model to return results only in a fixed format, such as JSON. But if the model is not given the real facts—what the source calls database-grounded context injection—it simply fills the required slots from its training‑data memories, inventing plausible‑sounding but false entries. In one study, this JSON output enforcement without any grounding increased the mistake rate by 10 to 15 percentage points (the C3 anomaly). Even switching to a reasoning mode offered no help. The problem extends to retrieval: pulling the wrong document or using evidence poorly also injects errors.

The trickiest point is the slot‑filling pressure: when mandatory fields exist, the model feels compelled to put something in each slot, so it guesses rather than admitting uncertainty. The source explicitly notes this hypothesis: “mandatory entity fields in the JSON schema exert slot‑filling pressure, forcing models to populate tool‑name slots from training‑data priors.” The result is that the very mechanism meant to fix hallucinations makes them worse. Without this subsystem’s insight, users would trust a confidently formatted output that is actually more unreliable than an unconstrained one—a concrete failure where a system that appears polished delivers quiet but damaging falsehoods.

The subsystem operates as a multi-stage detection pipeline grounded in the Epistemic Field Theory (EFT) framework. First, after a large language model produces an output, a set of multiple models generates responses to the same query, and the system computes a consensus field σ ∈ [0,1] over that query space. Next, the predictor P(H) = (1 − σ)·η is applied, where η is a model-specific noise coefficient derived from calibration data; if P(H) exceeds a threshold, the system flags the output as likely hallucinated. On failure—when a hallucination is detected or when the model’s own generation is unreliable—the designed fallback is to trigger retrieval-augmented generation (RAG), which supplements parametric knowledge with external evidence. However, the position paper by Hu et al. (2026) notes that such self-repair attempts can paradoxically worsen errors because the model suffers from metacognitive miscalibration: it cannot assess its own competence, so forcing a specific output format, for example, may cause the model to fill in required fields from training data rather than real facts. Reasoning modes offer no statistically significant improvement under architectural constraints, meaning the ordered mechanism of detect-then-repair frequently cycles without resolving the underlying miscalibration.

The invariant that the design preserves is calibration in the sense defined by the miscalibration framework: the model’s confidence must be aligned with factual correctness (semantic calibration) and with its own awareness of its limitations (metacognitive calibration). The EFT-based predictor explicitly targets the probabilistic and semantic dimensions by modeling consensus, but the overarching guarantee sought is that the system does not silently produce confident incorrect outputs. This aligns with the call for calibration metrics to become standard in benchmark reporting, so that the system always surfaces a well-calibrated estimate of its own uncertainty to downstream decision-makers. The invariant fails if the model’s self-reported confidence (e.g., token probabilities) remains high while σ is low, a condition that EFT captures but that simpler schemes ignore.

The key trade-off is between the cost of multi-model consensus and the rejection of the obvious alternative: single-model self-reported confidence or majority voting. Majority voting implicitly assumes error independence across samples, but EFT empirical results show that hallucination counts exhibit systematic overdispersion (ρ = 1.50) and majority-failure rates are 2.96× higher than independence would predict. Rejecting majority voting avoids the hidden cost of overconfident aggregation: an operator who sees a majority-consensus answer may incorrectly trust it, leading to downstream decisions based on fabricated information. Instead, EFT pays the computational overhead of running multiple frontier models (four models across three professional domains) to derive σ, gaining an AUC of 0.787 versus 0.518 for majority voting. This cost is justified because the avoided failure mode—confidently wrong answers that bypass detection—has much higher stakes in high-assurance enterprise contexts.

One concrete failure mode occurs when the retrieval step itself introduces errors: the system pulls the wrong document or uses evidence poorly, as described in the review by Yang (2026). In such a case, even though the generated output may appear fluent and cite sources, the citations are spurious or the claims are unsupported by the retrieved evidence. An operator monitoring the system would see outputs that are internally plausible but factually wrong, together with a high self-reported confidence from the model and a moderately high consensus σ (because multiple models may similarly misuse the same flawed evidence). The operator’s signal is the mismatch between high confidence and low factuality—exactly the semantic miscalibration that the Hu et al. (2026) framework identifies—and the EFT predictor would only partially mitigate this because σ can be high even when all models share the same retrieval error, as the model-specific noise coefficient η may not capture correlated failures in the retrieval stage.

1. Confidence Miscalibration Leading to Factual Hallucination

• Trigger — The model generates a confident but factually incorrect output because its confidence is not aligned with correctness. This is the core failure described in the position paper by Hu et al. (2026) where "confidence is misaligned with factual correctness" (semantic miscalibration).
• Guard — No explicit runtime guard identifier is provided in the source. The paper calls for calibration metrics (e.g., Expected Calibration Error) to become a standard part of benchmark reporting and for training paradigms that preserve uncertainty, but no function, retry, or exception handler is implemented in the described system.
• Posture — Fail‑soft. The model continues to produce fluent output; the failure degrades factual reliability but does not abort the run.
• Operator signal — A high confidence score paired with a factual error. The metric that would reveal this is the calibration error (ECE) reported during evaluation; in production, the operator would observe a silent absence of any error signal because the model does not flag its own mistake.
• Recovery — Manual fact‑checking is required. The source suggests adopting calibration‑aware training and post‑hoc calibration (e.g., temperature scaling or Platt scaling as in Reed & Mason, 2025) as mitigation, but these are not runtime recovery steps in the current subsystem.

2. Retrieval Error in a RAG‑Augmented System

• Trigger — The retrieval step pulls the wrong document, an incomplete snippet, or evidence that is poorly integrated, leading to unsupported claims. This failure is documented across several sources: Barua et al. (2026) note "retrieval errors, incomplete or selective integration of evidence", and Yang (2026) states "errors may arise during query formulation, document retrieval, evidence aggregation".
• Guard — No explicit guard identifier is given in the source. Mitigation strategies such as "retrieval optimization" and "evidence‑grounded generation" are described as categories, not as implemented functions or exception handlers.
• Posture — Fail‑soft. The system continues to generate an answer, but the factual grounding is degraded.
• Operator signal — A drop in the "retrieval relevance" metric (Yang, 2026) or an increase in "unsupported claims" as counted in evaluation (Barua et al., 2026). In a production system, the operator would see a low factual grounding score or a high hallucination rate.
• Recovery — Manual intervention to improve the retrieval pipeline (e.g., query rewriting, reranking, or switching to a different retrieval strategy). The source does not specify an automated retry or fallback.

3. Schema‑Induced Hallucination (C3 Anomaly)

• Trigger — Enforced JSON output format activated without any grounding mechanisms. The model exerts slot‑filling pressure and populates required fields from training‑data priors, increasing hallucination above the unconstrained baseline. This is the exact C3 anomaly observed by Menxhiqi et al. (2026): "JSON enforcement alone increases hallucination above the unconstrained baseline for all providers (+10.1 pp Gen1, +15.1 pp Gen2)".
• Guard — database‑grounded context injection and fixed vocabulary constraints (both identifiers appear verbatim in the paper as part of the three‑mechanism mitigation stack). When both are active, the failure is prevented.
• Posture — Fail‑soft. When the guard is absent, the system continues to run but produces worse outputs (higher hallucination rate). When the guard is present, the system still runs with reduced but non‑zero hallucination (3.3–14.9%).
• Operator signal — The hallucination rate (HR) metric increases compared to the unconstrained baseline. The operator would observe an increase of roughly 10–15 percentage points in HR.
• Recovery — Add the missing grounding mechanisms: database‑grounded context injection and fixed vocabulary constraints. The source suggests the full three‑mechanism stack is required.

4. Residual Hallucination Despite Correctly Retrieved Evidence

• Trigger — Even when the retrieval step fetches the correct document, the model still produces an unsupported claim or incomplete integration of the evidence. This is explicitly reported by Barua et al. (2026): "residual hallucinations despite retrieved context".
• Guard — No explicit guard identifier is provided in the source. The paper identifies this as a recurrent failure mode and calls for "attribution‑aware generation" and "conflict‑sensitive reasoning" as future work, not as existing runtime guards.
• Posture — Fail‑soft. The system continues to output; the failure is a factuality error that does not halt processing.
• Operator signal — The presence of "unsupported claims" in the output, measurable by a factual‑grounding metric or a human audit. The operator would see that the output contradicts or goes beyond the retrieved context.
• Recovery — Manual redesign of the evidence integration step. No automated retry or fallback is specified in the source.

5. Reasoning Mode Ineffectiveness Under Architectural Constraints

• Trigger — The system is configured to use a reasoning‑mode model (e.g., chain‑of‑thought) while the full mitigation stack (grounding + vocabulary constraints + JSON enforcement) is active. The reasoning mode provides "no statistically significant improvement" over standard generation, as reported by Menxhiqi et al. (2026).
• Guard — No guard; the reasoning mode is an optional configuration, not a failure‑handling mechanism. The paper states "reasoning‑mode models provide no statistically significant improvement under architectural constraints".
• Posture — Fail‑soft. The system runs but gains no benefit from the reasoning mode; performance is essentially unchanged.
• Operator signal — An A/B comparison shows that the hallucination rate is not significantly different between reasoning and standard modes. The operator would see overlapping confidence intervals or a non‑significant p‑value.
• Recovery — Disable the reasoning mode for this task. The source does not provide an automatic fallback; the change is manual.

6. Vulnerability Misclassification in Security Reasoning

• Trigger — The model correctly detects the presence of a software vulnerability but assigns it to the wrong CWE (Common Weakness Enumeration) category. DeepSeek‑AI et al. (2025) found "high detection rates and markedly poor classification accuracy, with frequent overgeneralization and misclassification".
• Guard — No explicit guard identifier is provided. The paper evaluates the models in a closed‑world classification setup; no runtime guard for classification errors is described.
• Posture — Fail‑soft. The system continues to output a CWE label, but the classification is incorrect; the security‑sensitive output is degraded.
• Operator signal — The CWE misclassification rate is high. The operator would observe a low classification accuracy metric despite high detection accuracy.
• Recovery — Manual review of the classification by a security expert. The paper suggests that "ensemble or rule‑based post‑processing" could help, but it is not implemented in the evaluated subsystem.