Token Economics — Deep Dive

The CascadingRouter implements the model cascade lever by trying a cheap model first and escalating only if quality checks fail.

class CascadingRouter:
    async def generate(self, request: LLMRequest) -> LLMResponse:
        # Try the cheap model first
        cheap_response = await self.call_model("claude-haiku", request)

        # Quality check (can be rule-based or LLM-based)
        if self.passes_quality_check(request, cheap_response):
            return cheap_response

        # Escalate to the expensive model
        expensive_response = await self.call_model("claude-sonnet", request)
        return expensive_response

    def passes_quality_check(self, request, response) -> bool:
        if response.finish_reason == "length":
            return False
        if response.confidence_score and response.confidence_score < 0.7:
            return False
        if len(response.content) < 50 and request.expected_length == "long":
            return False
        return True

Think of it like ordering a custom pizza. You want the best value without overpaying, so you first try a basic version and only upgrade if it falls short. This subsystem is for cutting the cost of getting answers from a large language model by using three simple levers: reuse what you already have, shorten what you ask, and start with a cheaper helper.

Specifically, it reuses past answers through caching so you don’t pay for the same request twice. It shortens your question with prompt compression and asks for briefer outputs. Most importantly, it first tries a cheap, fast model and has a small distilled model score that answer. If the score is high enough, you accept the cheap answer and skip the expensive model. This cascade, called FrugalGPT in the research, avoids paying full price for every query.

The trickiest part is the scorer itself. The small distilled model may not perfectly align with the expensive model’s judgment—a distribution discrepancy. It could approve a wrong answer or reject a good one, costing either accuracy or money. Without this subsystem, every request would blindly use the most costly model, wasting your budget on simple questions. You would feel the overspend immediately in inflated bills.

The subsystem’s ordered mechanism begins with Hindsight Budget Relabeling (HBR), which retrospectively simulates recorded trajectories under a spectrum of hypothetical initial budgets, producing over 2.38 million budget-annotated transitions without additional API cost. These transitions feed Conservative Q-Learning (CQL) to learn a safe lower-bound Q-function that respects budget constraints. At deployment, a λ-sweep mechanism enables zero-shot Pareto navigation: by varying the single hyperparameter λ, a single policy continuously traces the cost-quality frontier, allowing operators to dial between minimal cost and maximal safety without retraining. If a turn’s budget is depleted, the system falls back unconditionally to a weak model, which may produce inadequate answers—this failure is immediately visible in the next turn’s response quality.

The core invariant the design guarantees is the explicit budget state encoded in the BC-MDP formulation. By treating each model-selection decision as part of a finite-horizon Markov Decision Process whose state includes the remaining session budget, the agent learns delayed gratification—suppressing expensive model usage during budget-tight early turns and strategically deploying it on decisive final queries. This invariant ensures resource-awareness across the whole session: the router never spends beyond its remaining allowance, because the Q-function is conditioned on that exact quantity. Without this explicit state, behavior cloning (BC) suffers a bankruptcy rate exceeding 31.8% under realistic budgets, confirming that the budget state is essential for preventing premature exhaustion.

The key trade-off is offline safety versus data richness. The system rejects the obvious alternative of online exploration because it is “financially prohibitive and risks catastrophic user experience.” Instead, it stays offline and synthesizes budget-aware trajectories from unconstrained logs via HBR. The cost this rejection avoids is twofold: real API spend during exploration and the risk of delivering terrible answers to live users while learning. Offline learning is cheaper by avoiding trial-and-error consumption of expensive reasoning models, but it requires the theoretical validity guarantee that HBR’s spliced transitions are unbiased—a proof the authors provide. This trade-off is why the system can afford to generate 2.38 million transitions for free rather than spending even a fraction of that on live budget-stressed rollouts.

A concrete failure mode is budget bankruptcy, where the router exhausts the session allowance before a genuinely hard follow-up question arrives. An operator monitoring the system would see that after a budget-depleted early turn, the agent “either fails entirely or falls back to a weak model that produces an inadequate answer.” The signal an operator can track is the bankruptcy rate—the fraction of sessions where budget hits zero before the last turn. For a greedy behavior-cloning baseline that lacks the explicit budget state, that rate is above 31.8% ; for the SeqRoute agent using the full mechanism, it is near zero. The operator would observe this as a sudden spike in low-quality answers on final turns, even when the early turns appeared to cost little.

Budget Bankruptcy

• Trigger — The router makes per-query decisions without reasoning about future consequences, depleting the session budget before hard follow-up turns. By the third turn, when a genuinely hard follow-up question arrives, the budget is already exhausted.
• Guard — SeqRoute, the finite-horizon MDP agent that treats model selection as a long-horizon planning problem and learns delayed gratification to preserve resources for high-stakes turns.
• Posture — Fail-hard: the router either fails entirely or falls back to a weak model that produces an inadequate answer; the system cannot continue the session effectively.
• Operator signal — “the router either fails entirely or falls back to a weak model that produces an inadequate answer” at the third turn, or a bankruptcy rate exceeding 31.8% under greedy behavior cloning.
• Recovery — Manual deployment of SeqRoute as a replacement for the greedy router; if SeqRoute is already in use, the failure should not occur because the agent explicitly reasons about remaining budget.

Severe Under‑Allocation of Compute

• Trigger — A high‑consequence task (class‑2: silent data corruption, security bypass, irreversible operation) is routed to the lowest compute tier because the scheduler underestimates its risk, either due to misprediction or a difficulty‑only policy.
• Guard — The issue-only predictor (Qwen3-8B in issue-only mode), which on the SWE-bench Lite dataset never predicts class 0 for a class‑2 task, giving a severe under‑allocation rate of 0/44 = 0%. This guard prevents the scheduler from seeing low‑consequence predictions for high‑consequence tasks.
• Posture — Fail‑soft: the system produces a fix, but the fix is incorrect and the error (e.g., silent data corruption) persists downstream, degrading reliability without immediate crash.
• Operator signal — The eventual manifestation of “silent data corruption, security or permission bypass, irreversible operations (delete/overwrite/migration), or incorrect results consumed downstream without visibility” as defined in the consequence label for class‑2.
• Recovery — No automatic retry is described; operator must detect the corruption manually and roll back the erroneous fix. The guard itself cannot recover from an allocation error that already occurred, but it prevents future occurrences on similar tasks.

Data Starvation

• Trigger — Standard interaction logs exist but lack budget annotations, depriving the agent of the bankruptcy signals essential for learning constraint satisfaction.
• Guard — Hindsight Budget Relabeling (HBR), which retrospectively simulates recorded trajectories under a spectrum of hypothetical initial budgets, projecting unconstrained rollouts into a dataset of over 2.38 million transitions rich with bankruptcy signals, without additional API costs.
• Posture — Fail‑hard: without HBR the agent cannot learn from budget‑bankruptcy examples, making it prone to budget bankruptcy and ultimately failing to produce adequate answers in multi‑turn sessions.
• Operator signal — “standard interaction logs lack budget annotations” – no bankruptcy signals are present in the training data, and the resulting agent exhibits high bankruptcy rates after deployment.
• Recovery – Apply Hindsight Budget Relabeling to the existing logs to synthesize the missing transitions; no manual annotation required.

Difficulty‑Aware Routing Underperformance

• Trigger — The scheduler uses only task difficulty (e.g., hardest tasks get most compute) while ignoring the varying cost of failure. “The hardest tasks often have the lowest marginal gain from extra compute, because they remain unsolved at every available tier.”
• Guard — Consequence-aware routing (the cost‑weighted objective) or its priority-aware variant that combines predicted consequence with estimated marginal gain. The priority‑aware variant reduces cost‑weighted loss by over 30% while retaining over 90% of the oracle gain.
• Posture — Fail‑soft: the scheduler continues to allocate compute, but its policy is suboptimal, sometimes performing worse than random (“difficulty‑aware routing can underperform random”).
• Operator signal — “difficulty‑aware routing can underperform random” – cost‑weighted loss is higher than expected, possibly even exceeding a uniform baseline.
• Recovery — Replace the difficulty‑only policy with the priority-aware router; no manual rollback needed because the system does not crash, only wastes resources.

Consequence Misprediction by Issue‑Only Predictor

• Trigger — The deployment‑time issue-only predictor (Qwen3-8B in issue‑only mode) outputs an ordinal label that disagrees with the with‑patch reference, potentially routing a task to an inappropriate compute tier. The cross‑model Claude Sonnet 4.5 issue-only predictor shows a lower Cohen’s κ = 0.379.
• Guard — The cross-model Claude issue-only predictor is used as an independent robustness check, though it does not directly override the primary predictor’s decisions. Also, the primary predictor on SWE‑bench Lite avoided misclassifying high‑consequence tasks as low (0/44), so the guard is the predictor’s own accuracy property rather than an explicit fallback.
• Posture — Fail‑soft: misclassification that is not severe (e.g., class‑2 → class‑1) leads to suboptimal allocation (over‑ or under‑spending) but not catastrophic under‑allocation; the system continues operating.
• Operator signal — A low Cohen’s κ (e.g., κ=0.379 for Claude) or confusion‑matrix entries showing class‑2 tasks assigned to class‑0 (if that ever occurs). The source reports that on the 300 SWE‑bench Lite tasks neither predictor misclassifies a high‑consequence task as low‑consequence, so this signal would be absent on that dataset.
• Recovery — No automatic retry is described; operator would need to audit predictor outputs against gold labels or switch to the cross‑model predictor’s labels if its performance on other datasets proves superior.

The cost model uses the formula c_m(p) = n_in * c_in + n_out * c_out to compute per-request cost from input/output tokens and their prices.

def compute_cost(m, p, n_in, n_out, c_in, c_out):
    """
    c_m(p) = n_in(m,p) * c_in(m) + n_out(m,p) * c_out(m)
    """
    return n_in * c_in + n_out * c_out



# Frontier model: input $12/MTok, output $50/MTok
# Reasoning query: 200 answer tokens + 2000 thinking tokens
n_in_ex = 1000
n_out_ex = 200 + 2000   # output includes thinking tokens
c_in_ex = 12.0 / 1_000_000
c_out_ex = 50.0 / 1_000_000
cost = compute_cost("claude-opus", "reasoning", n_in_ex, n_out_ex, c_in_ex, c_out_ex)

Imagine stepping into a coin-operated photo booth. You drop in coins to tell it which background you want (your question), and it prints your picture (the answer). But the printing costs more than the selection because the booth builds the photo pixel by pixel. That is how API pricing works: you pay for every word you feed the model and every word it writes back, but the written words cost more because they are generated one at a time.

Now zoom in. When you ask a simple question, the booth prints two hundred pixels of answer. But if you switch to an extended thinking mode, the booth first runs an internal test—thinking tokens—that takes over two thousand invisible steps before printing the final picture. This is like the booth adjusting lighting inside; you pay for those invisible steps even though you never see them, making your total cost ten times higher. Real mechanisms at work are output tokens and thinking tokens, the latter being a hidden layer that silently inflates expenses.

The trickiest point is that these thinking tokens are the model's internal chain-of-thought reasoning, burned during complex math or coding. The system charges for this hidden labor through a rule called "token budgeting," where models like those with extended thinking force you to pay for every internal step. Without understanding this, a beginner might ask a deep question expecting a normal price, only to receive a bill that exploded to ten times the expected cost because the model silently reasoned for thousands of tokens before answering.

The subsystem operates as a token-cost-aware allocation pipeline that begins with a budget estimator, which predicts an optimal token budget for each incoming question. Using that estimate, the system constructs a token-budget-aware prompt by combining the question with the budget, then submits it to the LLM for final answer generation. If the estimated budget proves infeasible, the system falls back to a greedy search (embodied in Algorithm 2) that evaluates both correctness and cost reduction via the isFeasible condition: a candidate budget is accepted only if the answer is correct and its token cost is lower than that of the previously searched budget. The search terminates when either condition fails, ensuring the pipeline always lands on a budget that has been validated.

The invariant the design preserves is correctness-preserving cost reduction. This is formalized in the feasibility function of Algorithm 2, which mandates that any accepted budget must simultaneously produce a correct answer and reduce token cost relative to earlier candidates. The underlying principle, drawn from the observation of token elasticity, holds that there exists a budget range where both objectives are achievable; the system searches within that range to guarantee that reduced token spend never comes at the expense of answer correctness.

The key trade‑off is balancing token cost against answer quality. The obvious alternative—using a fixed budget or no budget at all (as in vanilla chain‑of‑thought)—would either waste tokens on easy tasks or risk errors on hard ones. The design rejects that approach in favor of a per‑task search because the cost of failure in production is not symmetric: an uncapped reasoning model on an easy task can silently inflate output costs by 3‑20x via thinking tokens. By optimizing the budget per query, the system avoids this silent cost blowout while preserving the accuracy that a fixed budget would sacrifice on complex queries.

A concrete failure mode occurs when latency‑sensitive traffic is inadvertently routed into batch processing: the mixing causes timeouts on service‑level agreements. The signal an operator would see is a spike in request latency in monitoring dashboards, accompanied by a sudden increase in token cost per request with no corresponding quality improvement. This alerts the team to adjust the thinking budget cap or the routing logic (e.g., via the model routing gate described for cost optimization) to ensure that latency‑critical tasks bypass batch endpoints.

Uncapped Reasoning Model on Easy Task

• Trigger — An easy task is assigned to a reasoning model (e.g., Claude Sonnet with extended thinking) without configuring a thinking budget cap.
• Guard — max_new (the configured token budget from Table 1) or thinking budget cap; both are the runtime control that caps total output tokens including thinking tokens.
• Posture — fail-soft. The system continues to return correct answers, but the uncapped model silently generates thousands of extra thinking tokens, inflating cost by up to 10×.
• Operator signal — Silent increase in the thinking/answer ratio and total cost metric; no error or warning is emitted.
• Recovery — Set an appropriate max_new value and route only genuinely complex tasks to reasoning models; wire the cost delta from this change into the merge gate to catch future budget overruns before shipping.

Routing Easy Tasks to Expensive Model Tiers

• Trigger — Low-complexity queries (e.g., simple questions) are routed to expensive models like GPT‑4o instead of cheaper tiers like Gemini Flash.
• Guard — AI gateway (cited in the summary as the centralized routing component that enforces model selection).
• Posture — fail-soft. The system functions correctly but pays 50–100× more per request than necessary.
• Operator signal — Elevated cost per request; the cost delta metric (if fed into the merge gate) would highlight the price disparity.
• Recovery — Reconfigure the AI gateway routing rules to default easy tasks to the cheapest suitable model, reserving expensive models for hard tasks.

Not Using Server‑Side Prompt Caching

• Trigger — Application with stable, repeated prompt prefixes does not enable server‑side caching.
• Guard — server-side prefix caching (the mechanism offered by providers that gives 50–90% discounts on cached tokens).
• Posture — fail-soft. No disruption to service, but every request redundantly processes the prefix, wasting 50–90% of potential input‑token savings.
• Operator signal — No direct signal; higher‑than‑expected token usage on repeated queries with identical prefixes is the only indication.
• Recovery — Enable server-side prefix caching and restructure prompts (e.g., keep system instructions as a stable prefix) to maximise cache hits.

Not Compressing Long Input Prompts

• Trigger — Input prompts contain verbose few‑shot examples, conversation histories, or large JSON schemas that are fed without compression.
• Guard — LLMLingua (the tool explicitly named for prompt compression, achieving 2–5× input‑token reduction).
• Posture — fail-soft. The model still responds correctly, but input‑token count is 2–5× higher than necessary.
• Operator signal — Consistently high input‑token counts per request with no compression tool active; no error is raised.
• Recovery — Integrate LLMLingua or manual techniques (selective retrieval, conversation summarization, schema pruning) into the pipeline to compress prompts before sending.

Mixing Latency‑Sensitive Traffic into Batch API

• Trigger — Real‑time requests are mistakenly routed to the Batch API, which is designed for non‑latency‑sensitive workloads.
• Guard — None. The source only states the failure mode: "The failure mode is mixing — routing latency‑sensitive traffic into batch and timing out SLAs." No explicit guard is provided.
• Posture — fail‑hard. The batch API cannot meet the required latency, causing request timeouts and SLA violations.
• Operator signal — timing out SLAs — a concrete alert that requests are exceeding their deadline.
• Recovery — Separate traffic into latency‑sensitive (real‑time) and latency‑insensitive (batch) channels, using the AI gateway to enforce correct routing based on user‑facing latency requirements.

This function structures messages in order of volatility to maximize the cacheable prefix, the core economic principle of prompt caching.

def build_messages_for_caching(
    system_prompt: str,        # Layer 0 + 1: static
    few_shot_examples: str,    # Layer 2: semi-static
    conversation_history: list,  # Layer 3: per-session
    current_query: str,        # Layer 4: unique
) -> dict:
    """Structure messages for maximum cache efficiency."""

Imagine you are at a food court where the first time you order a drink you pay an extra fee to get a special cup that the server remembers. After that, every time you hand back the exact same cup for a refill, the drink costs a fraction of the normal price. This subsystem does the same thing for computer programs that answer questions: it lets the first request pay a small extra amount to save the unchanging part of the question in a short‑term memory, so that later identical questions can reuse that memory and pay much less.

That memory is called a cache, and it works like this. The first time you ask a question, the program writes the opening part—the prefix—into the cache and charges a write premium: for one provider that extra cost is a twenty‑five percent surcharge, while another charges nothing extra for writing. Then when you ask the exact same prefix again within a few minutes, the program reads from the cache instead of recomputing from scratch, giving you a steep discount—about ninety percent off with the first provider and half the normal rate with the other. Every correct reuse resets the clock, but the cache only lasts a short time; the first provider keeps it for about five minutes, the second for minutes to hours.

The trickiest rule is that the prefix must be completely stable. If even one word changes between requests, the whole cached memory becomes useless and you pay the full write premium again. Without this subsystem, every single request—even ones that ask identical opening parts—would pay the full price, making your bill multiply quickly when you repeat the same work over and over.

The prompt caching subsystem operates as a two-phase economic engine. On the first request, the provider computes the KV cache for the prefix and charges a write premium — for Anthropic this is a twenty‑five percent surcharge, while OpenAI imposes no write surcharge. Subsequent requests that share an identical, contiguous prefix from the start of the prompt pay only a cached read discount: Anthropic reads are about ninety percent cheaper, and OpenAI cached tokens cost half the normal rate. Each cache hit resets the TTL (time to live) timer — five minutes for Anthropic, minutes to hours for OpenAI — so that a steady stream of identical prefixes keeps the cache alive. On failure — a cache miss caused by a changed token or an expired TTL — the request falls back to the full prefill cost and the write premium reapplies if the provider uses it. The operator thus observes a sequence of write‑premium, then discounted reads, then potentially full‑cost reads again when the cache empties.

The invariant the subsystem preserves is that the cached prefix must be byte‑identical and contiguous from the start of the prompt. As the mental model states, “one changed token, or dynamic content placed before static content, busts the entire cache.” This guarantees that the KV tensors reused are exactly the ones the model would have computed for that prefix, ensuring correctness without any approximate recovery. The runtime enforces this by checking the token sequence from the beginning — any deviation forces a full prefill and invalidates the cached entry for that variant.

The key trade‑off is paying a write premium on the first request in exchange for drastically cheaper reads on all subsequent identical prefixes. The obvious alternative rejected is treating every request independently, which avoids the write surcharge but forfeits the 50–90% cost reduction that caching delivers. The subsystem is built this way because it exploits the fundamental property that “the KV cache computed during the prefill phase for a given token prefix is identical regardless of what follows it.” The write premium covers the provider’s cost of storing and managing the cache, while the large read discount rewards users who stabilize their prefixes. Rejecting the no‑caching alternative avoids the cost of recomputing those expensive KV tensors from scratch on every call.

A concrete failure mode is cache busting due to dynamic content. If any token in the prefix changes — a timestamp, a user‑ID, or a slight variation in the system prompt — the entire cache for that variant is invalidated. The operator would see a sudden drop in cache hit rate, a corresponding rise in per‑request latency (since each request now runs a full prefill), and an increase in total cost because every request pays the write premium or full price instead of the cached read discount. This is exactly the “subtle bug” analogous to retry storms from the broader gateway discussion: a small, overlooked dynamic element in the prefix can silently undo all caching benefits, and the only signal is a degraded cost and latency baseline in monitoring dashboards.

Cache Miss from Unstable Prefix

• Trigger – Any change in the prompt prefix (e.g., a dynamic timestamp, user‑ID, or session token) breaks the exact match required for provider‑side prefix caching.
• Guard – None. The source only notes the condition “stable prompt prefixes” as a prerequisite for the discount; no exception handler, retry, or fallback is described.
• Posture – Fail‑soft. The request proceeds at full price with no caching savings.
• Operator signal – Higher‑than‑expected token cost per request; no explicit cache‑hit/miss log is mentioned in the source.
• Recovery – Manual stabilization of the prefix (moving dynamic components to the suffix) before re‑deployment.

Cache Eviction by Other Agents

• Trigger – A burst of non‑critical requests from other agents evicts a critical agent’s KV cache from the provider’s cache store. The source states “A burst of non‑critical agents can therefore evict a critical agent’s cache regardless of the schedule”.
• Guard – None. The source explains that “the memory allocator underneath still treats every agent’s cache as an independent object” and provides no eviction‑prevention mechanism.
• Posture – Fail‑soft. The evicted agent’s next request incurs the full write premium and loses the read discount.
• Operator signal – Sudden increase in latency and cost for previously cached prefixes, observable in aggregated latency and token‑usage metrics.
• Recovery – No automatic retry; the agent recomputes from scratch on the next request. Repeated evictions may require external cache‑space reservation (not provided in the source).

Accuracy Degradation from Cache‑Reuse Perturbations

• Trigger – Use of advanced prefix‑caching methods such as CacheBlend’s PIC (“CacheBlend’s PIC recovery introduces small numerical perturbations at selectively recomputed positions”). This degradation can cascade through multi‑agent rounds.
• Guard – None. The source reports that TokenDance “does not introduce any additional accuracy degradation beyond what the underlying PIC method already produces”, but no fallback to a slower, exact path is mentioned.
• Posture – Fail‑soft. The system continues with potentially inaccurate outputs, which may lead to downstream costs from incorrect decisions.
• Operator signal – Divergence in simulation round counts (the source shows relative differences of 3.3%–11.9% in some scenarios).
• Recovery – The operator must manually rerun the affected tasks without caching enabled.

Cache Persistence Failure (SQLite)

• Trigger – The optional SQLite disk‑cache persistence fails (e.g., disk full, write permission error, or I/O failure). The source notes the caching layer “optionally persists cache entries to SQLite on disk”.
• Guard – None; no error handling, retry, or fallback logic is described for the persistence operation.
• Posture – Fail‑soft. The in‑memory cache continues to operate, but cross‑run persistence is lost, reducing long‑term savings.
• Operator signal – Absence of expected cache hits in subsequent runs; no explicit error log is specified.
• Recovery – Manual intervention to resolve the disk issue (free space, fix permissions) and re‑trigger persistence.

A trained ML classifier routes requests based on feature extraction and quality targets to save costs.

class MLModelRouter:
    def __init__(self, classifier_path: str):
        self.classifier = load_model(classifier_path)
        self.model_configs = {
            "simple": ProviderConfig(provider="openai", model="gpt-4o-mini"),
            "moderate": ProviderConfig(provider="anthropic", model="claude-haiku-3-5"),
            "complex": ProviderConfig(provider="anthropic", model="claude-sonnet-4-20250514"),
            "frontier": ProviderConfig(provider="anthropic", model="claude-opus-4-20250514"),
        }

    async def select_provider(self, request: GatewayRequest) -> ProviderConfig:
        features = self._extract_features(request)
        tier_probabilities = self.classifier.predict_proba(features)
        selected_tier = self._select_tier(tier_probabilities, request.quality_target)
        return self.model_configs[selected_tier]

    def _extract_features(self, request: GatewayRequest) -> dict:
        prompt_text = " ".join(m["content"] for m in request.messages)
        return {
            "prompt_length": len(prompt_text.split()),
            "num_messages": len(request.messages),
            "has_system_prompt": any(m["role"] == "system" for m in request.messages),
            "has_tools": bool(request.tools),
            "num_tools": len(request.tools or []),
            "contains_code": bool(re.search(r"```|def |function |class ", prompt_text)),
            "contains_math": bool(re.search(r"\d+[\+\-\*/]\d+|equation|calculate", prompt_text)),
            "question_complexity": self._estimate_complexity(prompt_text),
            "embedding": self._get_prompt_embedding(prompt_text),
        }

    def _select_tier(self, probabilities: dict, quality_target: float) -> str:
        # Select the cheapest tier that meets the quality target
        ...

Imagine a restaurant where a junior cook handles every order first—simple dishes like fries or salads—and only calls the head chef when the junior is unsure or the dish is too complex. This subsystem does the same for language models: it decides which model should answer each question to save money without sacrificing quality.

Here is the real shape. Every incoming query is first sent to a cheap, fast model. A learned scoring mechanism—like a tasting panel—checks whether that answer is reliable. If the score is high enough, the answer is accepted; if not, the query is escalated to a larger, more expensive model. The source names this approach a “cascade” and describes a concrete system called BEST-Route, which uses a small network called DeBERTa-v3-small to estimate query difficulty and decide when to escalate. The junior cook (small model) handles about twenty-two percent of all orders, and the quality loss is less than one percent—a huge cost saving.

The trickiest detail is that the scoring panel itself is a small, efficient model, not an all-knowing oracle. It must be fast enough to avoid slowing down every order, yet accurate enough not to send easy dishes to the head chef or let bad ones through. BEST-Route also uses “best-of-n sampling,” generating multiple answers from the junior cook and picking the best one, before deciding to escalate. Without this careful cascade, a beginner would experience budget bankruptcy: you might waste the head chef’s expensive time on easy questions early, leaving no budget for the genuinely hard ones later—resulting in burnt orders and angry customers.

The subsystem implements a cascade-based routing strategy, not a single-shot decision. Ordered mechanism: each query first goes to a small, fast model (e.g., a 13B-parameter model). A learned quality estimation mechanism—exemplified by approaches such as FrugalGPT and RouteLLM—evaluates the initial response. If the estimation deems the answer insufficient or unreliable, the query is escalated to a large, expensive model. On failure (i.e., the quality check triggers escalation), the system proceeds to the fallback model; otherwise the cheap response is accepted.

The invariant the design preserves is the avoidance of budget bankruptcy—a systemic failure where the session budget is exhausted prematurely, causing the router to collapse on critical later turns. This is enforced through the BC-MDP formulation, which explicitly includes a budget state in the MDP, and through policies trained to learn delayed gratification—suppressing expensive model usage early to reserve resources for high-stakes queries. These constructs are named explicitly in the source and together guarantee that the system does not overspend its finite resource budget over the session horizon.

The key trade-off is between single-turn optimization and long-horizon budget-aware planning. The design rejects the obvious alternative of greedy behavior cloning (BC), which treats each query independently and ignores future budget constraints. That rejection avoids the cost of budget bankruptcy: BC suffers a bankruptcy rate exceeding 31.8% under realistic budgets, because it inevitably exhausts resources on early turns, leaving nothing for decisive late-session queries. Instead, the system adopts offline reinforcement learning with Hindsight Budget Relabeling (HBR) to synthesize bankruptcy-annotated trajectories from unconstrained logs, and Conservative Q-Learning (CQL) to learn a safe lower-bound Q-function. This allows a single policy to trace the cost-safety Pareto frontier via a λ-sweep mechanism at deployment, enabling smooth navigation between cost efficiency and bankruptcy avoidance.

A concrete failure mode observed in the greedy baseline is budget bankruptcy: by the third turn, when a genuinely hard follow-up question arrives, the budget is already depleted. The operator would see that the router either fails entirely or falls back to a weak model that produces an inadequate answer. The signal is a high bankruptcy rate—for example, 31.8% in the behavior‑cloning baseline—visible as a systematic collapse in response quality on late-session queries, even when earlier turns appeared successful. This confirms that without an explicit budget state and HBR-driven training, the system cannot reason about future spending consequences.

Budget Bankruptcy

• Trigger — Session-level routing without sequential planning under resource constraints; the subsystem uses a simple cascade (cheap model → strong model) without considering future turns, leading to budget exhaustion before high-stakes queries. Source: “greedy behavior cloning suffers a bankruptcy rate exceeding 31.8% under realistic session budgets… consistently worse than our SeqRoute agent across all budget levels.”

• Guard — None in the described subsystem; the source proposes “SeqRoute” (formalizing multi-turn LLM routing as a finite-horizon MDP) and “Hindsight Budget Relabeling (HBR)” as guards, but no existing exception handler or fallback is present.

• Posture — Fail-hard: the system collapses exactly when capability matters most (“collapsing exactly when capability matters most”), unable to answer subsequent high-stakes turn.

• Operator signal — Observed “bankruptcy rate exceeding 31.8%” in the deployment population; per-session metric “budget exhausted” with dropped or weak-model fallback.

• Recovery — No automatic recovery; operator must restart the session with a fresh budget or manually re-route failed high-stakes queries. No retry or backoff is defined.

Consequence Prediction Inaccuracy

• Trigger — The learned scorer (issue-only predictor) sees only “the GitHub issue text and the affected file path” and no gold patch or diff. The source reports Cohen’s κ = 0.572 for the primary Qwen3-8B issue-only predictor and κ = 0.379 for the cross-model Claude issue-only predictor, indicating imperfect agreement.

• Guard — None. The source does not specify a validation, retry, or fallback guard for prediction failures. It only notes that for the SWE-bench Lite tasks, “neither predictor misclassifies a high-consequence task as low-consequence” (class-2 recall 88.6%, zero severe under-allocation), but that is an empirical observation, not a coded guard.

• Posture — Fail-soft: a misprediction leads to suboptimal compute allocation; cost-weighted loss increases, but the system continues operating. The source describes “dangerous error for scheduling … severe under-estimation of risk” as a risk, not a hard abort.

• Operator signal — “Cohen’s κ = 0.572” or “κ = 0.379” from offline audit; online signal could be a rise in cost-weighted loss or an observed high-consequence error on a misclassified task.

• Recovery — None automatic. Operator may adjust the predictor threshold or fall back to a manual review of high-consequence predictions. The source does not specify a recovery step.

Thinking Length Insensitive to Consequence

• Trigger — Model (e.g., Qwen3-8B hybrid) allocates compute (thinking length) with no correlation to task consequence. Source Table 1: “Qwen3-8B (hybrid) … ρ = +0.002, p = n.s., failure mode: no signal.” The thinking length does not distinguish high- from low-consequence tasks.

• Guard — None. The model itself lacks any guard to correlate thinking with consequence; the system simply uses the model as-is.

• Posture — Fail-soft: compute is wasted on low-consequence tasks and potentially under-allocated on high-consequence tasks, but the system does not crash.

• Operator signal — “ρ = +0.002, p = n.s.” from a model audit (as in Table 1); also the average difference in thinking length between high- and low-consequence tasks is negligible (the “High vs Low” column shows +0.002 tokens).

• Recovery — None automatic. Operator must replace the model with one that shows a significant correlation (e.g., Claude Sonnet 4.5 with ρ = +0.203) or implement a separate consequence predictor to override the model’s allocation.

Saturated Compute Allocation

• Trigger — The model (e.g., Qwen3-VL-8B-Thinking) always uses near the maximum token budget regardless of consequence. Source Table 1: “Qwen3-VL-8B-Thinking … 99.3% saturated.”

• Guard — None. The model has no budget-limiting guard or cost-aware logic.

• Posture — Fail-soft: compute is heavily wasted because each task consumes the full budget of 8,192 tokens (configured max_new) no matter how easy or low-consequence the task is. No system failure, but cost is high.

• Operator signal — “99.3% saturated” from a model audit; per-task token usage is always near max_new = 8,192.

• Recovery — Operator must reconfigure the token budget max_new to a lower limit or choose a different model (e.g., Qwen3-8B hybrid) that uses tokens more efficiently. No automatic fallback.

Difficulty-Aware Routing Underperformance

• Trigger — Routing decisions based solely on difficulty (accuracy gain) rather than consequence-weighted error cost. Source: “difficulty-aware routing can underperform random: the hardest tasks often have the lowest marginal gain from extra compute, because they remain unsolved at every available tier.”

• Guard — None in the described subsystem. The source proposes a “priority-aware variant that combines predicted consequence with estimated marginal gain” but does not implement it as a guard in the simple cascade.

• Posture — Fail-soft: cost-weighted loss exceeds that of random routing; overall allocation is suboptimal but the system continues.

• Operator signal — “cost-weighted loss” higher than the baseline random or oracle; the source states consequence-only and priority-aware variants reduce loss by over 30% relative to difficulty-aware.

• Recovery — Operator must switch to a priority-aware router that uses both predicted consequence and marginal gain. No automatic retry or fallback in the original design.

LLMLingua-2 uses a small BERT-class model to score token informativeness and drops low-perplexity tokens.

compressor = PromptCompressor(
    model_name=model_name,
    use_llmlingua2=True,
)

result = compressor.compress_prompt(
    context=[context],
    question=question,
    rate=target_ratio,
    condition_in_question="after",
    reorder_context="sort",
    dynamic_context_compression_ratio=0.3,
)

return result["compressed_prompt"]

Imagine you are packing a suitcase for a trip. You want to take only what you really need to avoid heavy bag fees and save space. This is exactly what prompt compression does: it trims down the text you send to a language model so the model can answer faster and at lower cost, while keeping the core message intact.

The system works like a smart packing assistant that scans every item in your bag. It uses a small language model, such as XLM-RoBERTa, to score each word based on how predictable it is. Predictable words, like common filler, are seen as low risk to leave behind—similar to leaving out a second pair of socks you could always buy later. The assistant then applies a method called token classification to decide which words to keep, preserving only the most informative ones in their original order. This step-by-step filtering reduces the prompt length by two to five times, making the model's job quicker and cheaper.

The hidden challenge is that predictability can be deceptive. A word like "not" might seem easy to predict from the surrounding text, but deleting it could flip the entire meaning of your request. The system avoids this by using bidirectional context: it looks at words both before and after each candidate to judge its true importance, much like checking how a shirt fits with the entire outfit before tossing it out. Without this two-way view, the compressor might accidentally remove a small word that completely breaks your instruction, causing the language model to give a wrong or even opposite answer. You would then waste time debugging why your carefully crafted prompt failed, instead of enjoying the speed you paid for.

The LLMLingua subsystem operates as a coarse-to-fine pipeline: first, the Budget Controller assigns token-level budgets across segments using perplexity from a small language model, then Iterative Token-level Compression drops low-perplexity tokens while protecting anchor spans (entities, numbers, instructions) from removal, and finally Alignment refines the compressed prompt to retain semantic integrity. On failure—such as when the compression ratio overshoots the accuracy cliff—the system must fall back to a lower ratio via empirical task evaluation, because performance degrades non-linearly: flat up to ~50% reduction, then a sharp drop. The failure mode is not a crash but silent quality loss; operators detect it by monitoring task accuracy on a held-out eval (e.g., drop from 0.84 to 0.62) rather than output fluency.

The invariant this design preserves is faithful anchor retention: every span classified as critical (a negation, a number, an instruction) must survive compression verbatim. This guarantee is enforced by the token-level dropping logic, which excludes those spans from the low-perplexity removal set. The underlying principle is that the loss function is downstream answer quality, not text reconstruction—dropping a single “not” cheats the metric but destroys the answer. Thus the system deliberately sacrifices compression ratio on anchors to maintain the invariant, and validates success on task metric rather than on “how natural the output looks.”

The key trade-off is compression ratio versus task accuracy. LLMLingua rejects the obvious alternative of generation-based compression (e.g., using GPT-4 to summarize prompts) because generation yields uncontrollable content and length—it often loses multi-step reasoning paths or produces unrelated text, requiring multiple iterations to meet ratio constraints, and incurs high computational overhead (small LMs cannot handle complex generation; large LMs amplify cost). By instead using a small Transformer encoder (XLM-RoBERTa-large, mBERT) to treat compression as token classification with bidirectional context, the subsystem avoids the cost of extra LLM calls and preserves reasoning chains exactly as they appear in the original prompt. It also rejects the unidirectional perplexity approach of Selective-Context, which ignores interdependence between dropped tokens, and replaces it with iterative coarse-to-fine dropping that considers context.

A concrete failure mode is negation deletion: the Iterative Token-level Compression drops a low-perplexity “not” token because the small LM scores it as uninformative, while the remaining sentence reads fluently. The signal an operator sees is a sudden collapse in task accuracy—for example, accuracy on GSM8K dropping from 0.84 at 25% compression to 0.62 at 75%—without any change in compression ratio reporting. The operator would also notice that the compressed prompt, though grammatically intact, consistently produces wrong arithmetic answers because the required negation is missing. This forces rollback to a less aggressive budget via the Budget Controller.

JSON Overflow Beyond ~494 Tools

• Trigger – The number of tool definitions in the schema exceeds roughly 494, causing the JSON representation to produce more tokens than the 200K context window can hold.
• Guard – The source references Tscg as the compression method that extends the operational range beyond 803 tools. However, no exception handler or fallback is shown for when Tscg is not used; the system simply fails.
• Posture – Fail‑hard: the source reports “JSON produces 0% EM (total failure).” No degraded output is produced; the run is effectively aborted.
• Operator signal – The operator observes a 0% exact‑match (EM) rate on all tasks that trigger the overflow.
• Recovery – No automatic recovery is described. The operator must manually reduce the number of tools below the ~494 threshold or switch to the Tscg compression scheme to keep schema tokens within the 200K budget.

Critical Token Deletion (Negation / Number Loss)

• Trigger – The LLMLingua token‑level compressor drops low‑perplexity tokens that carry essential meaning (e.g., a “not” or a specific number), causing the compressed prompt to change the intended instruction.
• Guard – The source mentions that “production systems protect key spans like numbers and instructions,” but provides no concrete guard identifier (function, variable, or exception handler). No explicit guard is shown in the source.
• Posture – Fail‑soft: the system continues to generate a response, but that response is likely incorrect because the meaning was altered.
• Operator signal – No dedicated log line or metric is specified. The operator would see anomalous or counter‑intuitive model outputs without any immediate alert.
• Recovery – No automatic recovery is given. The operator might rerun the task with compression disabled or with an explicit key‑span protection list (though that list is not identified in the source).

Accuracy Cliff Beyond 50% Compression

• Trigger – The compression ratio (percentage of tokens removed) exceeds approximately 50%. The source states “performance stays flat up to about fifty percent reduction. Then it drops sharply.”
• Guard – The source does not present any guard that detects or prevents crossing the 50% threshold. There is no retry, fallback, or validation tied to the compression ratio.
• Posture – Fail‑soft: the model still produces outputs, but with a sharp drop in accuracy.
• Operator signal – The operator would see a measurable decline in benchmark accuracy or downstream task success once the compression ratio passes the 50% mark. No specific alert is described.
• Recovery – No automatic recovery exists. The operator must reduce the compression ratio below 50% to restore expected performance.

Distractor Dilution Effect in Small Models

• Trigger – Small language models (not further specified in the source) receive a compressed schema (likely from Tscg) that contains irrelevant or distracting context tokens. The source labels this a “distractor dilution effect in small models (Section 5.3).”
• Guard – The source does not show any guard targeted at this effect. No retry, fallback, or validation is linked to model size or context noise.
• Posture – Fail‑soft: the small model continues to execute but is confused by the extra, irrelevant information, leading to degraded tool‑calling accuracy.
• Operator signal – No specific log line is given. The operator would observe higher error rates for tasks involving many tools when using small models, compared to large models.
• Recovery – No automatic recovery is described. The operator may need to use a larger model or reduce the number of tools to limit the distracting content.

Compound Format–Budget Interaction

• Trigger – The combination of a specific schema format (JSON vs. Tscg) and a particular token budget leads to a compound negative effect. The source references “a compound format–budget interaction (Section 5.2)” as an underlying mechanism.
• Guard – The source does not identify any guard for this interaction. The analysis only describes it as a mechanism, not a failure that the system handles.
• Posture – Fail‑soft: performance degrades gracefully, but no catastrophic failure occurs.
• Operator signal – No direct signal is specified. The operator might notice that certain format‑budget pairs give worse accuracy than others, without an obvious cause in the input.
• Recovery – No automatic recovery is shown. The operator must experiment with different schema formats or adjust the token budget to avoid the harmful interaction.

Batch API usage with asynchronous polling for non-real-time workloads.

import anthropic

client = anthropic.Anthropic()

# Submit a batch of requests
batch = client.messages.batches.create(
    requests=[
        {
            "custom_id": f"request-{i}",
            "params": {
                "model": "claude-sonnet-4-20250514",
                "max_tokens": 1024,
                "messages": [{"role": "user", "content": prompt}]
            }
        }
        for i, prompt in enumerate(prompts)
    ]
)

# Poll for completion (batch SLA is typically 24 hours)
while batch.processing_status != "ended":
    await asyncio.sleep(60)
    batch = client.messages.batches.retrieve(batch.id)

# Retrieve results
results = client.messages.batches.results(batch.id)

Think of a delivery driver who can carry many packages in one trip. That trip saves fuel per package, but each package must wait until the truck is full before leaving. This is exactly what batching does for language model requests: it groups multiple requests together so the computer works more efficiently, lowering the cost per request.

The computer has two stages per request: quickly reading the whole prompt (prefill) then slowly generating tokens one by one (decode). During decode, the computer is mostly idle waiting for memory. Batching, called continuous batching in the source, combines several requests at each decode step so that while one request fetches its next token from memory, another request uses the computer's compute units. A scheduler decides which requests belong together each step, balancing how many are batched versus how long each waits. This raises the total throughput — more tokens produced per second — which directly reduces cost per token, just as the driver’s single trip saves fuel.

The trickiest part is that not all packages can wait. The source describes that for mixed workloads — some with very long prompts (like document analysis) and some with short ones (like chat) — the system uses chunked prefill to interleave them. But if a short, urgent request gets batched with many long, slow ones, it will miss its deadline. The scheduler must keep urgent requests separate, never mixing them into the batch built for delay-tolerant work. Without this careful separation, a user expecting an instant chat reply would instead wait seconds for the batch to finish, and the system would violate its latency promise — a failure that feels like paying express shipping and getting standard delivery.

In the batch-and-budget subsystem, the ordered mechanism begins with the client selecting a configuration (\mathbf{c}=(m_1,\dots,m_N)) from the candidate model set (\mathcal{M}), a choice that specifies which model is assigned to each role in the pipeline (the “model combination”). That configuration is then submitted to the LLM serving stack, where requests enter a queue and the scheduler applies continuous batching: each step arbitrates between throughput and per-request latency by grouping requests for the compute‑heavy prefill phase and then processing the memory‑bandwidth‑bound decode phase token by token. On failure—such as mixing real‑time traffic into a batch workload—the server risks timing out its service‑level agreement, while in the budget‑constrained routing component a premature budget depletion causes the router to collapse exactly when capability matters most, either failing entirely or falling back to a weak model that produces an inadequate answer.

The design preserves the explicit budget state as the core invariant, a guarantee named in the source’s BC‑MDP formulation. This state, encoding both conversational context and remaining session budget, ensures that the router learns resource‑aware planning rather than myopic per‑query decisions. Additionally, the safe lower‑bound Q‑function learned by Conservative Q‑Learning provides a conservative guarantee against overestimation of future rewards, and the (\lambda)-sweep mechanism preserves a zero‑shot Pareto frontier that lets operators navigate between cost and safety without retraining. For batching, the invariant is that latency SLOs are not violated by mixing traffic; the scheduler’s per‑step arbitration maintains this boundary.

The key trade‑off is throughput versus latency in batching, and cost versus safety in budget routing. The subsystem rejects the obvious alternative of greedy behavior cloning (BC), which routes each query independently without considering future turns. That rejection avoids the cost of budget bankruptcy: the source shows that BC suffers a bankruptcy rate exceeding 31.8% under realistic session budgets, while sequential planning with explicit budget state reduces cost by up to 73.5% with near‑zero bankruptcy. The batch API similarly rejects mixing real‑time workloads because doing so would incur the cost of repeated SLA timeouts and loss of the fifty‑percent discount. Building this way shifts the challenge from per‑query difficulty estimation to sequential planning under resource constraints – a qualitatively different problem.

One concrete failure mode is budget bankruptcy, where the router exhausts its session budget by the third turn and cannot answer a genuinely hard follow‑up question. An operator would see the signal of a high bankruptcy rate (source reports 31.8% for BC), along with increased per‑session cost as the router falls back to weak models. For the batch side, mixing real‑time traffic into the batch API triggers SLA timeout events; the operator would observe latency metrics exceeding the agreed threshold, a direct violation of the service‑level agreement. Both failures are diagnosed through the explicit identifiers—budget bankruptcy for routing, SLA timeout for batch—and are preventable by the subsystem’s core mechanisms: Hindsight Budget Relabeling (HBR) for data synthesis, Conservative Q‑Learning for safe policy learning, and the scheduler’s per‑step arbitration to keep batch and real‑time workloads separated.

1. Real‑Time Traffic Mixed Into the Batch Queue

• Trigger — A pipeline sends both real‑time and batch requests through the same API, violating the explicit rule “Never mix real‑time traffic into batch.”
• Guard — None shown in source. The paragraph warns against mixing but implements no check, queue partition, or exception handler.
• Posture — Fail‑soft. The system continues processing all requests, but real‑time responses are delayed and may time out, degrading the service‑level agreement without aborting the entire run.
• Operator signal — “time out your service‑level agreement” — the operator observes SLA breach alarms, increased latency dashboards, or user complaints about slow responses.
• Recovery — Manual re‑architecture: separate real‑time and batch endpoints/pipelines. No automatic retry or fallback is defined.

2. Batch API Used for Time‑Sensitive Workloads

• Trigger — A workload that requires real‑time answers is submitted as a batch request, ignoring the statement “Use them only for workloads that are not sensitive to delay.”
• Guard — None shown in source. No validation of latency requirements is present.
• Posture — Fail‑soft. Batch responses eventually arrive, but the workload misses its deadlines, causing downstream components to wait, fail, or produce stale results.
• Operator signal — Late or missing responses; downstream tasks time out or emit “deadline exceeded” errors; user‑facing features appear unresponsive.
• Recovery — The workload must be re‑submitted via the real‑time API. No automatic switchover or backoff exists in the source.

3. Pipeline Impatience: No Waiting Logic for Batch Responses

• Trigger — A pipeline expects immediate responses from the batch API and does not incorporate asynchronous polling or long timeouts, ignoring the guidance “You must be patient for responses.”
• Guard — None shown in source. There is no retry loop, fallback handler, or timeout adjustment for batch operations.
• Posture — Fail‑hard. The pipeline treats the delayed or absent response as an error and aborts its execution, halting the entire run.
• Operator signal — “Batch request timed out” or “No response received” in pipeline logs; the run exits prematurely.
• Recovery — Manual code change to implement polling or increase timeout; no automatic retry count or backoff is provided.

4. Misapplying the Fifty Percent Discount Without Considering the Cost of Delay

• Trigger — A user selects the batch API solely for the fifty percent discount, assuming the speed trade‑off is negligible, even though the workload has implicit time constraints (e.g., daily reports that must finish before opening hours).
• Guard — None shown in source. No cost‑benefit check or deadline estimator is present.
• Posture — Fail‑soft. The batch runs complete, but delays cause downstream processes to start late, potentially missing business‑critical windows. The system does not abort but delivers results past the acceptable time.
• Operator signal — “Batch completion time exceeded SLA threshold” alerts; business reports appear after their required deadline.
• Recovery — Re‑evaluate workload priority and switch to real‑time API for future runs. No automatic escalation or priority override is implemented.

An algorithm checks token cost reduction and correctness.

def isFeasible(current_budget, previous_budget, question):
    # compute actual token cost for current and previous budgets
    current_cost = compute_token_cost(current_budget, question)
    previous_cost = compute_token_cost(previous_budget, question)
    # check correctness of the answer with current budget
    correct = check_correctness(current_budget, question)
    # greedy strategy: must reduce token cost
    reduced = current_cost < previous_cost
    # return feasibility based on both criteria
    return correct and reduced

Imagine you're packing a suitcase for a trip, and you only have a bag of a certain size. If you throw in too many clothes, the zipper won't close, and you'll have to pay for a second bag. This is what happens when a reasoning model thinks through a problem: it can produce a huge internal commentary, consuming many tokens and making the cost skyrocket. The subsystem called Token-Budget-Aware LLM Reasoning, or TALE, is for giving that model a "suitcase" of a specific size—a token budget—so it knows exactly how much internal thinking it can use to arrive at the answer efficiently.

The system first asks the model itself to guess how many tokens its reasoning will need, using a zero-shot estimation prompt called TALE-EP. That estimate becomes the budget, which is then added to the prompt like a travel checklist that must fit in the suitcase. The model is trained using a loss function that encourages it to stay within that budget. For a simple question, the suitcase is small; for a complex proof, it's larger. On average, this cuts token use by about sixty-nine percent while keeping accuracy high, dropping less than five percent.

The trickiest detail is a phenomenon called token elasticity: the model's actual thinking can still far exceed the given budget if left unchecked, like clothes expanding to fill extra space. To enforce the constraint, the system uses budget forcing—it truncates the reasoning at a predefined maximum, or appends the word "Wait" to make the model refine its answer. Without this enforcement, a trivial question could consume as many thinking tokens as a complex proof, and a beginner would feel the shockingly high cost on their bill.

The Reasoning Token Tax subsystem, as implemented by TALE, operates as a deterministic three-stage pipeline. First, a token budget estimator—implemented as a zero-shot estimation prompt for the reasoning LLM itself—computes a per-question token budget. Second, the system crafts a token-budget-aware prompt by combining the original question with the estimated budget. Third, this prompt is submitted to the target LLM for generation. If a budget must be validated, the system invokes Algorithm 2, which defines a feasibility function that checks two criteria on the candidate budget: the correctness of the answer and a reduction in token cost relative to the previously searched budget. If either condition fails—if the answer is incorrect or the cost is not strictly lower—the search process terminates, and that budget is discarded. This ordered mechanism ensures that only budgets passing both gates proceed to final generation.

The central guarantee the design preserves is the feasibility condition—a joint invariant of correctness and monotonic cost reduction. This invariant is codified in Algorithm 2’s feasibility function: “feasibility is assessed based on two criteria: the correctness of the answer and a reduction in token cost following a greedy strategy.” The design enforces that each successive budget not only maintains output quality but also strictly lowers token consumption, thereby preventing the silent waste identified by the token elasticity observation: “while a minimal budget may keep the correctness of the answer, it does not necessarily minimize the token cost.” The system thus guarantees that every accepted budget is Pareto-superior to its predecessor on both dimensions.

This architecture explicitly rejects the obvious alternative of using a minimal token budget directly, which would appear to minimize cost by brute force. The rejection is grounded in the token elasticity phenomenon: a minimal budget can preserve correctness yet fail to minimize the actual token bill, because the model may still exceed the cap or produce suboptimal token usage patterns. By instead performing a greedy search via Algorithm 2, TALE finds a budget that genuinely minimizes cost while preserving accuracy, avoiding the silent cost blowout that occurs when an uncapped reasoning model overgenerates on an easy task. The cost avoided is twofold: unnecessary token expenditure from overthinking, and wasted search iterations on budgets that pass correctness but not cost reduction.

A concrete failure mode is silent cost blowout, where the reasoning model, lacking a thinking budget cap, emits hidden “thinking” tokens that dwarf the visible answer. The operator would observe a high token cost per response on the billing dashboard, accompanied by an elevated thinking/answer ratio in the logs, particularly for tasks that appear simple. As the source warns, “an uncapped reasoning model on an easy task is the most common silent cost blowout, structurally similar to the over-generation problem.” Monitoring this ratio and capping the thinking budget are the runtime controls needed to detect and mitigate this failure mode.

Token Elasticity Overflow

• Trigger – The reasoning model ignores the token budget in the prompt and generates far more thinking tokens than allocated, as described by the "token elasticity" phenomenon where actual token use can far exceed the given budget.
• Guard – None explicitly in source; the TALE method relies on the LLM internalizing the constraint via fine-tuning, but no runtime enforcement mechanism exists.
• Posture – fail-soft (the system continues to produce a full answer, but token consumption and cost escalate).
• Operator signal – "actual token use far exceeds the given budget" (the observed metric of token usage versus the estimated budget).
• Recovery – No automatic fallback; operator must manually apply a hard thinking budget cap at the provider level or refine the prompt to enforce stricter adherence.

Token Budget Undershoot

• Trigger – TALE's zero-shot estimator assigns an insufficient token budget for a complex mathematical question, causing the LLM to produce an incomplete or incorrect answer.
• Guard – None explicitly in source; the two-stage process is designed to encourage concise responses but provides no run‑time validation or exception handler for under‑budget cases.
• Posture – fail-soft (accuracy degrades; the output may be present but wrong).
• Operator signal – "accuracy drops more than five percent" (evaluation metric on GSM8K, GSM8K‑Zero, or MathBench).
• Recovery – No automatic retry; operator must increase the estimated budget or route the question to a reasoning model without budget constraint.

Model Transfer Failure

• Trigger – TALE's zero-shot estimator is applied to a state‑of‑the‑art LLM (e.g., GPT‑4o, GPT‑4o‑mini, Yi‑lightning) that was not represented in the estimator's training data, leading to poor budget estimates.
• Guard – None explicitly in source; research question RQ3 investigates generalization but no runtime guard is implemented.
• Posture – fail-soft (both token savings and accuracy may degrade relative to the original model).
• Operator signal – "TALE effectiveness drops across different state‑of‑the‑art LLMs" (observed variance in token reduction percentage and accuracy between models).
• Recovery – Manual step: retrain the zero‑shot estimator on data from the target LLM, or fall back to a fixed budget strategy.

Fine-tuning Constraint Evasion

• Trigger – The two‑stage fine‑tuning process fails to internalize the token budget constraint, so the LLM reverts to verbose reasoning even when the budget is included in the prompt.
• Guard – None explicitly in source; the training objective (Equation 2) and instruction prompt (p_i) are used offline, but no inference‑time guard checks constraint adherence.
• Posture – fail-soft (the token reduction benefit of TALE is lost, but the system continues with normal operation).
• Operator signal – "token consumption does not decrease" (actual token count remains near uncapped levels; silent absence of savings).
• Recovery – No automatic recovery; must re‑run fine‑tuning with stronger regularization or additional reinforcement steps.

Extrapolation to Real-World Scenarios

• Trigger – TALE is evaluated only on mathematical datasets (GSM8K, GSM8K‑Zero, MathBench) but deployed on production tasks with different reasoning characteristics, causing budget mismatches.
• Guard – None explicitly in source; the chapter does not mention any cross‑domain validation or fallback mechanism.
• Posture – fail-soft (accuracy may drop or token savings may not hold in production).
• Operator signal – "accuracy drop or cost increase on production tasks" compared to benchmark performance (metric discrepancy).
• Recovery – Manual step: collect domain‑specific data and retrain the zero‑shot estimator; or disable TALE for unknown domains.

PagedAttention uses fixed-size blocks and a page table to allocate KV-cache memory on demand.

class KVCacheManager:
    def __init__(self, num_blocks, block_size, num_heads, head_dim):
        self.block_size = block_size
        self.free_blocks = list(range(num_blocks))
        self.kv_cache = torch.zeros(num_blocks, 2, num_heads, block_size, head_dim)
        self.page_tables = {}

    def allocate_block(self, request_id):
        if not self.free_blocks:
            raise RuntimeError("OOM")
        block_id = self.free_blocks.pop()
        self.page_tables.setdefault(request_id, []).append(block_id)
        return block_id

Imagine a warehouse where each customer used to claim a whole section, but because their stuff grows and shrinks unpredictably, most sections had empty wasted space, so only a few customers could use the warehouse at once. vLLM’s PagedAttention is like dividing the warehouse into small, identical bins that can be given to any customer as needed, allowing many more customers to be served simultaneously. This system is for deciding whether to build your own LLM server – it dramatically reduces memory waste so you can pack far more requests onto one GPU.

Step by step, the system works like this: Each LLM request needs a key-value cache that expands as it generates tokens. Previously, the cache was allocated in one contiguous block, leading to fragmentation – like forcing each customer to take a whole section even if they only fill part. PagedAttention instead carves the cache into fixed-size pages (the small bins). A request’s cache is stored across many non-contiguous pages, tracked by a page table, just like virtual memory in operating systems. This eliminates internal and external fragmentation, achieving near-zero waste. The source shows that for a 13B parameter model, KV cache can consume over 30% of GPU memory; PagedAttention reclaims that wasted space.

The trickiest trade-off is that scattering cache pages across memory can slow down the attention kernel, because it must read from non-contiguous addresses. The source notes a trade-off in kernel speed. But the huge gain in usable batch size – often 2–4× higher throughput than systems like FasterTransformer – more than compensates. Without this subsystem, fragmentation would cap your batch size, forcing you to idle GPUs or buy more hardware, making self-hosting economically unfeasible compared to buying API access. The concrete failure: you could only serve a handful of concurrent users per GPU, leading to high per-request latency or skyrocketing costs.

The subsystem begins with the first-come-first-serve (FCFS) scheduling policy: new requests join a queue, and the system processes them in arrival order, with the earliest arrived served first. When a request is admitted, it enters the prefill phase, where the entire input prompt is processed in parallel, producing the initial KV cache blocks. Then the decode phase proceeds iteratively, generating one token per step. At each decode step, the scheduler must decide which requests to batch together, arbitrating between throughput and per-request latency. If the GPU runs out of physical blocks for new KV cache entries—detected when the GPU block allocator cannot satisfy a block request—the engine triggers preemption. Preemption always evicts the latest-arrived requests first, following the FCFS ordering. The evicted sequence’s blocks are handled by one of two recovery mechanisms: either they are copied to CPU RAM via swapping (managed by the CPU block allocator) or the KV cache is recomputed later by concatenating the already-generated tokens with the original prompt.

The design preserves the all-or-nothing eviction policy: when a sequence must be evicted, all of its blocks are evicted together, never a subset. This guarantee is enforced because all blocks of a sequence are accessed together during decode, and multiple sequences within the same request (e.g., beam search candidates) are gang-scheduled as a sequence group, so they must be preempted or rescheduled as a unit. This invariant avoids the complexity of partial eviction and ensures that when a preempted sequence is rescheduled, its full context is either swapped back or recomputed atomically, maintaining correctness without dangling references.

The key trade-off is between swapping and recomputation as the eviction recovery method. Swapping copies evicted KV cache blocks to CPU RAM and later brings them back to GPU memory, which is bandwidth-bound and works well when GPU–CPU transfer is fast. Recomputation recalculates the KV cache from scratch by feeding the original prompt plus all generated tokens through a single prefill pass; it is compute-bound and can be faster if the GPU has ample compute relative to bandwidth. The obvious alternative is to avoid preemption entirely by over-provisioning GPU memory, but that would waste resources and cap utilization. The system rejects static memory allocation, instead using PagedAttention’s block-level paging to achieve near-zero waste in the KV cache, enabling far more concurrent requests per GPU and boosting throughput 2–4× over systems like FasterTransformer and Orca. The cost this rejection avoids is internal and external fragmentation of KV cache memory, which would otherwise reduce batch sizes and leave the GPU idle.

A concrete failure mode occurs when request traffic exceeds the total physical block capacity of the GPU. The operator would observe repeated preemption cycles: the system logs entries like “preempting sequence group X, swapping blocks to CPU” or “recomputing KV cache for sequence Y.” Over time, cumulative preemption overhead degrades throughput and latency, and the GPU block allocator reports “out of free physical blocks” repeatedly. The operator sees that the effective tokens-per-second drops while CPU RAM utilization rises, indicating the system is thrashing between swap and recompute rather than serving requests directly. The underlying signal is the rapid growth of the swap space on CPU RAM—which, by design, never exceeds the GPU’s KV cache memory budget—combined with stalled decode iterations as the engine waits for blocks to be returned.

1. GPU Physical Block Exhaustion and Request Preemption

• Trigger — Request traffic (number of concurrent requests or sequence length) exceeds the GPU’s physical KV‑cache blocks.
• Guard — The all-or-nothing eviction policy that evicts all blocks of a sequence (or sequence group) when memory is full, combined with the first-come-first-serve (FCFS) scheduling policy that decides which requests to preempt (latest requests preempted first).
• Posture — fail‑soft. The system continues processing earlier requests; preempted sequences are dropped or must be restarted, but the server does not abort.
• Operator signal — No dedicated log line is given in the source; the operator would observe increased tail latency, incomplete responses, or timeouts for the preempted requests.
• Recovery — Evicted blocks may be recovered later by the recomputation method to recover the evicted blocks, but no automatic retry is shown; manual intervention (reduce concurrency or add GPUs) is required.

2. Automatic Prefix Cache (APC) Miss after KV‑Block Eviction

• Trigger — A request arrives with a common prefix (system prompt, few‑shot examples) whose KV‑cache blocks were evicted under memory pressure before another request with the same prefix was issued.
• Guard — No guard is shown for this specific miss. The APC (--enable-prefix-caching) transparently hashes and reuses blocks, but eviction is handled by the general all-or-nothing eviction policy – there is no separate protection for cached prefixes.
• Posture — fail‑soft. The server recomputes the prefix KV‑cache, degrading throughput and increasing latency, but still serves the request.
• Operator signal — No metric is specified; the operator would see higher latency for requests that share long prefixes compared to previous runs, without an explicit error.
• Recovery — The recomputation method automatically recreates the evicted blocks when needed; no retry count or backoff is described.

3. Consequence‑Aware Scheduler Misconfigured to Difficulty‑Only Routing

• Trigger — The scheduler uses only a difficulty- or thinking‑length‑based routing signal (as current models do) instead of the consequence‑weighted objective. The source shows that thinking‑length (e.g., of Qwen3‑8B) is uncorrelated with consequence (ρ = 0.002).
• Guard — No guard is shown for this misconfiguration. The source explicitly proposes a cost‑weighted compute allocation objective but does not define a runtime validation or fallback if the scheduler ignores consequence.
• Posture — fail‑soft. The system continues to serve tasks, but it under‑allocates compute to high‑consequence tasks and over‑allocates to low‑consequence tasks, producing higher cost‑weighted loss.
• Operator signal — The deployer would observe worse cost‑weighted error than expected under the same compute budget; no alert is defined in the source.
• Recovery — Manual reconfiguration to use the predictor (Qwen3‑8B in issue‑only mode) that estimates consequence labels, and applying the minimizing cost‑weighted loss objective.

4. Consequence Predictor Adjacent‑Class Error Leading to Sub‑Optimal Tier Assignment

• Trigger — The predictor (Qwen3‑8B issue‑only) mislabels a task by one class (e.g., class‑2 task predicted as class‑1, or class‑1 as class‑0). The source confirms a Cohen’s κ = 0.572 and that the predictor never outputs class‑0 for a class‑2 task, so the only errors are adjacent.
• Guard — No runtime guard corrects the prediction; the scheduler accepts the predicted label as is. The error pattern is built into the predictor.
• Posture — fail‑soft. The task is routed to a compute tier that is one step too cheap (or too expensive), but the most severe under‑allocation (class‑2 → class‑0) is avoided.
• Operator signal — No dedicated log line; the operator might notice that some high‑consequence tasks are resolved with medium budget instead of high budget, but no immediate failure.
• Recovery — No automatic retry; the deployer can re‑train the predictor or use the cross‑model Claude 4.5 issue‑only predictor as a robustness check (κ = 0.379), though no automatic fallback is implemented.

Tool-schema compression reallocates context budget from overflow to retrieval.

Think of an AI agent as a worker at a tiny desk. The desk has a fixed area—that's the agent's working memory and budget. The worker needs to fit both instruction manuals (tools) and reference books (retrieved information) onto that desk, and pay only for the space used. To make everything fit, the worker first compresses the thick manuals into short labels instead of keeping the full text—the source calls this "tool-schema compression," saving up to half the space. Next, the worker reuses sticky notes from previous tasks even if the notes are moved to a different spot, relying on "cached key-value segments" that avoid rewriting. When conversations get long, the worker writes a tiny summary instead of keeping every word, known as "conversation summarization." The worker only pulls out relevant sentences from books ("selective context retrieval") and plans the desk's whole usage against "one global budget" rather than charging per glance. The trickiest piece is why compression matters at the tightest budgets. The source reveals a "binary enablement effect": when the desk has only 8K tokens, full manuals overflow and accuracy drops to around 2.6%, but compressed labels restore functionality with a 20+ percentage point exact-match lift. Yet for small workers (≤8B), too many compressed labels cause "distractor dilution"—the clutter overwhelms attention. Without these strategies, whenever the desk is tiny the worker either cannot function at all or wastes so much money that using the agent becomes pointless.

The subsystem first performs round-level KV Cache optimization under the All-Gather prompt structure: agents in a round share output blocks from the previous round, but each agent has a private history of different length, so shared blocks appear at different absolute positions. Despite this positional shift, the cache reuse avoids redundant recomputation. Next, tool-schema compression (the Tscg conservative profile, which saves ~50% context while preserving descriptions) reduces the per-tool footprint so agents with many tools can fit within the context budget. On the client side, a model combination assignment is selected from candidate set ℳ, trading off task performance, latency, and cost. Finally, session-level routing (the SeqRoute agent) formalizes multi-turn LLM routing as a finite-horizon MDP whose state includes the remaining session budget. The ordered mechanism thus proceeds: (1) reuse KV caches across agents in the same round; (2) compress verbose tool schemas to recover budget; (3) choose a client-side model combination; (4) route each query based on the MDP policy learned via Hindsight Budget Relabeling (HBR). On failure (e.g., budget depletion), the system either falls back to a weak model or fails to answer adequately.

The invariant the design preserves is budget solvency across turns, enforced by treating the remaining session budget as a first-class state variable in the MDP. SeqRoute’s agent learns delayed gratification—deliberately routing early queries to cost-effective models so that resources remain for high-stakes turns later. This guarantee prevents budget bankruptcy, the catastrophic failure mode where the router exhausts the budget prematurely and cannot answer a genuinely hard follow-up question.

The key trade-off is per-query difficulty estimation versus session-level planning under resource constraints. The obvious rejected alternative is greedy per-query routing, which optimizes each turn independently based on predicted difficulty or confidence. That approach suffers a budget bankruptcy rate exceeding 31.8% even with behavioral cloning, because it cannot reason about the consequences of today’s spending on future turns. The cost avoided by the session-level design is catastrophic collapse on hard follow-up queries—the very situation where capability matters most. The system is built this way because session-level routing is a qualitatively different problem: the core challenge shifts from per-query difficulty estimation to sequential planning with a finite resource, and no amount of single-turn optimization can prevent budget bankruptcy.

A concrete failure mode is budget bankruptcy. The operator would see that after the third turn of a multi-turn session, when a genuinely hard follow-up question arrives, the router either fails entirely or falls back to a weak model that produces an inadequate answer. The visible signal is a sudden drop in accuracy or a high rate of incomplete or incorrect responses on later turns, contrasting sharply with earlier turns that were correctly handled. This pattern indicates that the remaining session budget was depleted before the high-stakes turn, exactly the outcome the SeqRoute design is intended to avoid.

Budget Bankruptcy

• Trigger — A router that cannot reason about future consequences of today’s spending, causing premature budget depletion when genuinely hard follow‑up questions arrive. This is explicitly observed for greedy behavior cloning under realistic session budgets.
• Guard — The source does not show a guard for this failure. The proposed SeqRoute agent and Hindsight Budget Relabeling (HBR) are training‑time remedies, not deployed exception handlers.
• Posture — fail‑soft: the router either fails entirely or falls back to a weak model that produces an inadequate answer, but the session continues in degraded mode.
• Operator signal — The bankruptcy rate metric, quantified as “exceeding 31.8% under realistic session budgets”.
• Recovery — No automatic recovery is specified. The failure would require manual intervention or system redesign (e.g., adoption of SeqRoute).

Distractor Dilution

• Trigger — Additional retrieval chunks or tool descriptions are added to the context, reducing accuracy in small models. The source explicitly names this as “distractor dilution as a failure mode where additional chunks reduce accuracy in small models”.
• Guard — The source does not show a guard for this failure.
• Posture — fail‑soft: the system’s accuracy degrades (EM drops), but execution continues.
• Operator signal — Measured accuracy reduction; in the evaluation the effect is observed via EM points lost when extra chunks are present.
• Recovery — No recovery mechanism is specified in the source.

No Thinking–Consequence Correlation (“no signal”)

• Trigger — Using the Qwen3‑8B (hybrid) model, which produces thinking lengths that are uncorrelated with the actual consequence of the task (Spearman ρ = 0.002, p n.s.). The failure mode is labelled “no signal” in Table 1.
• Guard — The source does not show a guard for this failure.
• Posture — fail‑soft: the model still produces a response, but compute is not allocated by consequence, potentially wasting resources.
• Operator signal — The metric “ρ = 0.002” and the “n.s.” p‑value; also the “no signal” label in the failure‑mode column.
• Recovery — No recovery is specified. The operator would need to switch to a different model or a consequence‑aware scheduler.

Thinking Saturation (“99.3% saturated”)

• Trigger — Using the Qwen3‑VL‑8B‑Thinking model, which saturates its configured thinking budget on nearly every input (99.3% of tokens used regardless of consequence). The failure mode is labelled “99.3% saturated” in Table 1.
• Guard — The source does not show a guard for this failure.
• Posture — fail‑soft: the model produces a response, but it consumes the maximum allowed compute on every task, ignoring varying need.
• Operator signal — The “99.3% saturated” metric from the table; also the fact that maximum_new tokens are always fully used.
• Recovery — No recovery is specified. Mitigation would require adjusting the model’s budget or selecting a different model.

A file-backed SQLite cache that stores previous answers and sources to avoid recomputation, keyed by a unique identifier.

class _SqliteCache:
    """File-backed cache. A short-lived connection per call keeps it thread-safe under
    FastAPI's threadpool without a shared-connection lock dance."""

    def __init__(self, path: str):
        self.path = path
        os.makedirs(os.path.dirname(path), exist_ok=True)
        with sqlite3.connect(self.path) as con:
            con.execute(
                "CREATE TABLE IF NOT EXISTS explanations ("
                "  key TEXT PRIMARY KEY,"
                "  query TEXT,"
                "  answer TEXT,"
                "  sources TEXT,"
                "  created_at REAL"
                ")"
            )

    def get(self, key: str) -> dict[str, Any] | None:
        with sqlite3.connect(self.path) as con:
            row = con.execute(
                "SELECT answer, sources FROM explanations WHERE key = ?", (key,)
            ).fetchone()
            if not row:
                return None
            return {"answer": row[0], "sources": json.loads(row[1] or "[]")}

    def put(self, key: str, query: str, answer: str, sources: list) -> None:
        with sqlite3.connect(self.path) as con:
            con.execute(
                "INSERT OR REPLACE INTO explanations"
                " (key, query, answer, sources, created_at)"
                " VALUES (?, ?, ?, ?, strftime('%s','now'))",
                (key, query, answer, json.dumps(sources)),
            )

Imagine you have a fixed budget and a strict deadline for a group project. This subsystem does the same for every LLM request: it sets a hard limit on how much can be spent and how long the work can run, to prevent runaway costs.

The system gives each request its own budget (the per-request ceiling) and a deadline. Every time the LLM is called or a tool step executes, that step is charged against the ceiling. Once the ceiling is reached or the deadline passes, the run stops immediately—no extra charges. It also keeps an exact match cache: it stores previous questions and their answers. If the same question comes again, it reuses the stored answer, avoiding a costly new model call. On top of that, an ingestion pipeline caches work by hashing each node and transformation together, so repeated content is not reprocessed.

The trickiest detail is why the cache stores only exact matches. This prevents accidentally returning a wrong answer for a nearly identical question, but it means even a small wording variation still triggers a full generation. Without these guards, a single request with many tool steps could blow through the per-request ceiling without any limit, or a buggy loop could run indefinitely, racking up unbounded cost and time.

The subsystem described in “Cost Controls In LlamaIndex” is a layered budget enforcement mechanism that sequences operations in a strict, fail-fast order. When a request enters the service, it is first checked against an exact match cache that stores previous question–answer pairs. If a cache hit occurs, the stored answer is reused immediately—no model call is made, and no budget is deducted. On a cache miss, the service proceeds to make model calls and tool steps, each of which is charged against a per request ceiling. The ceiling is a hard budget cap: as soon as cumulative charges reach that value, the entire run stops. In parallel, a deadline independent of the budget halts all processing if wall‑clock time expires before the ceiling is consumed. Ingestion work follows a separate cached path: the pipeline hashes each node and each transformation together; if the same content repeats, the transformation is skipped, avoiding redundant computation. The ordered mechanism is thus: cache lookup → (if miss) deduct budget on each call/step → stop on ceiling hit or deadline expiration.

The central invariant the design preserves is that no request ever exceeds its per request ceiling. This is a hard guarantee enforced by the stop condition; the source explicitly states that once the ceiling is reached “the run stops.” The deadline acts as a complementary invariant—a maximum wall‑clock duration—but the budget ceiling is the primary cost guard. The design does not attempt exactly‑once or idempotency guarantees; instead it ensures that aggregate spend per request is bounded, preventing runaway spending.

The key trade‑off is latency and completeness in exchange for cost safety. The obvious alternative rejected here is an unbounded model‑call loop with no upfront budget limit—a system that simply continues making calls until the task is finished regardless of expense. That alternative would guarantee perfect completion (no premature termination) but incurs the cost of potentially infinite or very high spending per request. The LlamaIndex design deliberately rejects that to avoid the “runaway spending” failure mode. It caps budget at the cost of possibly incomplete responses when the ceiling is hit or the deadline expires. This trade‑off is built this way because, in production LLM services, unbounded costs can silently multiply (e.g., a reasoning model generating thousands of thinking tokens), and a hard stop is the simplest, most reliable guard against financial surprise.

A concrete failure mode occurs when the per request ceiling is exhausted before all intended model calls complete. The operator would see a log signal indicating that the run stopped early—most likely a termination message like “budget exceeded” or “ceiling reached,” together with a partial or empty response to the user. Because the deadline is independent, a second failure mode manifests when processing drags past the deadline; the operator would observe a timeout signal and a halted run even if budget remains. Both failure modes are visible as abrupt stoppages in the request trace, and the only recovery is to either increase the per request ceiling/deadline or reduce the number of steps required for that request.

Budget Ceiling Reached

• Trigger – The cumulative cost or step count in Budgets exceeds the configured step_budget or cost_budget during calls to Budgets.step() or Budgets.cost().
• Guard – The internal checks inside Budgets.step() and Budgets.cost() (methods not shown in source, but implied by the Budgets dataclass and the _budget_reset method). No try/except is visible in the screen_output function that calls these methods.
• Posture – Fail-hard. The unhandled exception (likely a custom BudgetExceeded or similar) propagates upward, aborting the current request.
• Operator signal – An unhandled exception log line (e.g., BudgetExceeded or RuntimeError with a budget-related message).
• Recovery – None automated. The operator must inspect the budget configuration (RAG_COST_BUDGET env var) and adjust the limits or retry the request with a higher ceiling.

Deadline Exceeded

• Trigger – Wall‑clock time since _start exceeds the deadline value in Budgets before the request completes.
• Guard – The time check inside Budgets.step() or Budgets.cost() (no explicit handler shown in the source).
• Posture – Fail-hard. An exception (e.g., DeadlineExceeded) terminates the run.
• Operator signal – A timeout‑related error in the logs, perhaps DeadlineExceeded or a generic TimeoutError.
• Recovery – None. The operator can increase the deadline field (set in _budget_reset) or investigate why the request took too long.

Injection False Positive Overload

• Trigger – A benign user input matches one of the compiled patterns in INJECTION_PATTERNS (or _INJECTION_FAMILIES), causing fence_text to flag it.
• Guard – fence_text still returns the original content verbatim (as stated in its docstring: “NEVER drops the original substring”). The detection does not block the response.
• Posture – Fail-soft. The flagged content is added to a worklist (via build_worklist, not shown in the excerpt), but the answer is served unchanged.
• Operator signal – The worklist contains an entry for the request, listing the detected family (e.g., "override", "code_block").
• Recovery – Manual review of the worklist. The operator can refine the regex patterns or whitelist known false positives.

Injection False Negative (Missed Attack)

• Trigger – A novel injection vector not covered by any pattern in _INJECTION_FAMILIES (e.g., a new prompt‑injection technique).
• Guard – No guard shown in source. The _detect method uses only the fixed set of regex patterns; there is no fallback or anomaly detector.
• Posture – Fail-open. The injection passes through fence_text undetected and reaches the downstream model.
• Operator signal – Silent absence – no alert, no worklist entry, no log line.
• Recovery – After a security incident, the operator must append a new pattern to _INJECTION_FAMILIES and redeploy.

Budget Initialization Failure

• Trigger – The RAG_COST_BUDGET environment variable is missing, malformed, or contains a non‑integer value when Guardrails.__init__ calls _budget_reset.
• Guard – No try/except is shown around the parsing of RAG_COST_BUDGET in the source excerpt. If parsing fails, an exception (e.g., ValueError or KeyError) would propagate at import or init time.
• Posture – Fail-hard. The service fails to start or the Guardrails instance cannot be created.
• Operator signal – A ValueError or KeyError traceback during service startup.
• Recovery – The operator must set a valid RAG_COST_BUDGET (e.g., "5000") and restart the process.

Markdown Redirect Bypass

• Trigger – A crafted href or src attribute that evades the _SAFE_HREF_RE or _SAFE_SRC_RE checks in the MARKDOWN_REDIRECT_HREFS or MARKDOWN_REDIRECT_SRC families (e.g., using a protocol other than http/https).
• Guard – The regex patterns in _SAFE_HREF_RE and _SAFE_SRC_RE (shown as r"^https?://"). If the URL does not start with http:// or https://, it is replaced by fence_text.
• Posture – Fail‑closed for the matched pattern (the dangerous URL is replaced with the safe anchor/source text), but fail‑open for any URL that bypasses the regex entirely.
• Operator signal – If a bypass occurs, no log entry is generated. If caught, the response contains the safe replacement (e.g., <a href="">…</a>).
• Recovery – Manual analysis of the attack and addition of a broader regex to _SAFE_HREF_RE or _SAFE_SRC_RE.

Use a small model first and escalate only when quality checks fail.

class CascadingRouter:
    async def generate(self, request: LLMRequest) -> LLMResponse:
        cheap_response = await self.call_model("claude-haiku", request)
        if self.passes_quality_check(request, cheap_response):
            return cheap_response
        expensive_response = await self.call_model("claude-sonnet", request)
        return expensive_response

    def passes_quality_check(self, request, response) -> bool:
        if response.finish_reason == "length":
            return False
        if response.confidence_score and response.confidence_score < 0.7:
            return False
        if len(response.content) < 50 and request.expected_length == "long":
            return False
        return True

Self-Route uses a RAG prediction if answerable, else uses long context.

# For queries deemed answerable, accept RAG prediction as final answer.
# For queries deemed unanswerable, provide full context to long-context LLM.
if query_is_answerable_by_rag:
    final_answer = rag_predict(query)
else:
    final_answer = lc_predict(query, full_context)