Token Economics — the Papers

A router can go bankrupt before the hard question even arrives. That is the failure mode SeqRoute names. Most routing frameworks treat each query as an independent event, but real users arrive in sessions that share one global budget. A myopic policy burns the expensive model on early small talk, then hands the genuinely difficult turn to whatever capacity is left. SeqRoute formulates multi-turn routing as a finite-horizon Markov Decision Process and solves it with offline reinforcement learning. Remaining budget goes into the state space. Conservative Q-Learning teaches the router delayed gratification: preserve resources for the high-stakes turns later in the session. The result is operational cost down 6.0-73.5% while maintaining or improving quality, with bankruptcy rates suppressed under 1%. What to take from it: • Route the session, not the query. Cost is a resource shared across turns. • Budget bankruptcy is a measurable failure mode, not a vibe. Instrument for it. • Their deployment-time budget sweep navigates the cost-quality Pareto frontier with no retraining. Full map of the token-economics literature: https://ai-engineer-roadmap.xyz/token-economics

#LLMRouting #InferenceCost #ReinforcementLearning #LLMOps #TokenEconomics

Most routers are built to save money. The interesting ones can also buy accuracy. SCOPE (Scalable and Controllable Outcome Performance Estimator) is a routing framework that does not pick a model by name. It predicts, for each query, how accurate a model will be and how expensive it will be, by retrieving how models behaved on similar problems. That one design choice means it keeps working when you swap in a model it has never seen. Because cost and performance are both predicted, routing stops being a fixed choice and becomes a dial. Turn it toward performance and SCOPE boosts accuracy by up to 25.7%. Turn it toward efficiency and it cuts cost by up to 95.1%. Same router, same weights, different constraint. Takeaways: • A router that predicts outcomes generalises to new models. A router that memorises model names does not. • Cost control and quality control are the same lever pulled in opposite directions — build one, expose both. • Your budget is a runtime parameter, not a procurement decision made last quarter. The routing chapter, with every paper cited: https://ai-engineer-roadmap.xyz/token-economics

#LLMRouting #InferenceCost #LLMOps #ModelSelection #CostAwareAI

The cheaper model on the price page is often the more expensive model on your invoice. The Price Reversal Phenomenon evaluates 8 frontier reasoning models across 12 tasks and finds that in 32% of model-pair comparisons, the model with the lower listed price actually incurs the higher total cost. The reversal magnitude reaches up to 28x. The mechanism is thinking tokens. On the same query, one model may use 900% more thinking tokens than another, or 10x more turns of environment interactions. Listed price per token says nothing about how many tokens the model decides it needs. It gets worse for anyone trying to forecast. Repeated runs of the same query yield thinking token variation up to 9.7x — an irreducible noise floor for any per-query cost predictor. Takeaways: • Never select a model from a pricing table. Select it from a measured cost per completed task. • Per-request cost monitoring is not observability polish, it is the only ground truth you have. • If your cost model is a point estimate, it is wrong; the paper argues for predicting cost distributions. Read the full chapter: https://ai-engineer-roadmap.xyz/token-economics

#InferenceCost #LLMOps #ReasoningModels #FinOps #TokenEconomics

Ten open-source models, orchestrated, beat Gemini-3-Pro at 47% of the cost. That is the claim in Beyond Gemini-3-Pro: Revisiting LLM Routing and Aggregation at Scale, and it is a direct shot at the assumption that the frontier model is the ceiling. The framework, JiSi, argues that routing alone has been leaving value on the table. Train-free routers match queries on textual similarity and miss difficulty. Aggregation methods are static and pick the same aggregator regardless of task. And nobody was combining the two. JiSi adds query-response mixed routing that captures semantics and difficulty, support-set-based selection of the aggregator, and a switch that decides per task whether to route or aggregate. Across nine benchmarks, that collective surpasses Gemini-3-Pro with only 47% costs. Takeaways: • The unit of capability is the ensemble, not the checkpoint. Complementary weak models cover each other's failures. • Routing and aggregation are not competitors — knowing when to switch between them is the actual product. • Open weights plus orchestration is a credible frontier strategy, not a budget compromise. More in the routing chapter: https://ai-engineer-roadmap.xyz/token-economics

#LLMRouting #OpenSourceLLM #InferenceCost #LLMOps #ModelSelection

Wiring your strongest model into every node of a multi-agent graph is not rigor. It is waste. CASTER makes that concrete. Graph-based multi-agent systems support complex cyclic workflows, but static model allocation means frontier compute gets spent on trivial sub-tasks a small model would have nailed. The orchestration layer never asks how hard the step actually is. CASTER is a lightweight router that does ask. A Dual-Signal Router fuses semantic embeddings of the task with structural meta-features of the graph to estimate difficulty, then picks the model. During training it self-optimizes from its own routing failures through on-policy negative feedback, so the mistakes become the curriculum. Across software engineering, data analysis, scientific discovery and cybersecurity workloads, it cuts inference cost by up to 72.4% versus strong-model baselines while matching their success rates, and beats both heuristic routing and FrugalGPT in every domain tested. Worth internalising: • Node difficulty inside an agent graph varies enormously. Price accordingly. • Graph structure is signal. Semantics alone underrates it. • Routing failures are free training data, if you bother to log them. Full map of the token-economics literature: https://ai-engineer-roadmap.xyz/token-economics

#MultiAgentSystems #LLMRouting #AgentOrchestration #InferenceCost #TokenEconomics

Beating GPT-5 while spending 58.7% less than GPT-5 is now a published result. That is MTRouter on ScienceWorld. On Humanity's Last Exam it holds competitive accuracy for 43.4% less total cost. The setting is cost-aware multi-turn routing: at every turn of a long-horizon task, choose which model from the pool to invoke, under a fixed budget. The mechanism is a joint history-model embedding. MTRouter encodes the interaction history alongside each candidate model, then learns an outcome estimator from logged trajectories that predicts turn-level model utility. Not "is this query hard" but "is this model worth calling right now, given everything that has already happened." The behavioural analysis is the part most teams will skip and shouldn't. Against prior multi-turn routers, MTRouter switches models less often, tolerates transient errors instead of panicking and escalating, and develops emergent specialization across the pool. The gains carry over to held-out tasks. Takeaways: • Long-horizon agents accumulate cost per turn. So route per turn. • Your trajectory logs are already a routing training set. • Fewer model switches, not more, tracks with better outcomes. Full map of the token-economics literature: https://ai-engineer-roadmap.xyz/token-economics

#LLMRouting #AgenticAI #InferenceCost #LLMOps #TokenEconomics

Every router on the market shares one blind spot: it assumes a model has a single fixed cost per query. It does not. A model's cost, and its quality, both move with how much it is allowed to write. R2-Router builds on exactly that. Existing routers estimate one price per model, then exclude the powerful ones the moment that price crosses the budget line, never noticing the option they just discarded: the strong model, told to be brief, at a price the weak model cannot beat on quality. So R2-Router treats output length budget as a controllable variable and jointly selects both the model and its length budget, enforcing it through length-constrained instructions. The authors also release R2-Bench, the first routing dataset capturing LLM behaviour across diverse output length budgets. State-of-the-art quality follows, at 4-5 times lower cost than existing routers. What this changes: • Cost per model is not a constant. It is a dial you already control. • A constrained strong model can dominate an unconstrained weak one. • Routing becomes reasoning: which model, and at what output budget. Full map of the token-economics literature: https://ai-engineer-roadmap.xyz/token-economics

#LLMRouting #ReasoningModels #InferenceCost #PromptEngineering #TokenEconomics

In production, nobody rates your responses. They just leave. That sparse, one-sided feedback is the exact signal most cost-aware routers cannot learn from. Published methods lean on complete feedback signals, offline training and extensive per-workload tuning, and most offer neither an SLA guarantee nor inference-time adaptivity. Then a commercial contract arrives with a quality SLA attached, and the cost-versus-quality tension stops being academic. SLARouter is built for that setting: an online routing algorithm that learns a cost-optimal policy from the thin feedback production systems actually produce, with theoretical guarantees for both cost optimality and strict SLA compliance. Across a wide range of LLM benchmarks it satisfies the SLA constraints without per-benchmark tuning, reducing operating cost by up to 2.2x over existing baselines. The practical read: • Design for the feedback you get, not the labels you wish you had. • An SLA is a routing constraint. Encode it, do not hope for it. • Per-workload tuning is a hidden cost. Methods that skip it win on total cost of ownership. Full map of the token-economics literature: https://ai-engineer-roadmap.xyz/token-economics

#LLMRouting #ServiceLevelAgreements #InferenceCost #OnlineLearning #TokenEconomics

A 121M-parameter model reaching 73.8% of a 7B state-of-the-art model's performance is the least interesting number in this paper. The interesting one is 46.3%: the cut in cloud LLM usage MCCom achieves by cascading that small local model with a cloud LLM for line-level code completion. Latency drops by up to 47.9%, and the LLM's own exact match rate improves by 8.9% through the collaboration. Cheaper, faster, and better. The trick is the escalation signal. Rather than guessing difficulty up front, MCCom watches user actions and triggers the cloud model only when the local model has visibly failed. Your developer's next keystroke is a free difficulty label. Add a two-stage speculative decoding strategy and iterative retrieval, and the two models genuinely cooperate instead of merely taking turns. Evaluated on RepoEval and a new real-world benchmark, StmtEval. Takeaways: • The cheapest classifier of query difficulty is often the user themselves. • Local-first with cloud escalation beats cloud-always on both latency and cost. • Cascades can raise quality, not merely preserve it. Full map of the token-economics literature: https://ai-engineer-roadmap.xyz/token-economics

#CodeCompletion #ModelCascading #EdgeAI #DeveloperTools #TokenEconomics

Cheap answers are worthless if you cannot prove how often they are right. That is the gap BARGAIN attacks in Cut Costs, Not Accuracy. Standard cascades defer to the expensive model using the cheap model's confidence, typically log-probabilities. The paper's diagnosis is blunt: those systems deliver only marginal savings and weak theoretical guarantees, because they estimate the cheap model's output quality badly. BARGAIN replaces the guesswork with an adaptive sampling strategy and a statistical estimation procedure that yields tight guarantees. You state the accuracy you require — or the precision, or the recall — and it uses the affordable model exactly as aggressively as that constraint allows, and no more. Across 8 real-world datasets, BARGAIN reduces cost by up to 86% more than state-of-the-art cascades, while providing stronger guarantees on the accuracy of the output. Takeaways: • A cascade without a quality guarantee is not a cost optimisation, it is an unmeasured quality regression. • Confidence scores are an estimator. Treat them as one, with sampling and error bars. • Pick the metric you actually ship on — accuracy, precision or recall — and constrain the router on that. The chapter: https://ai-engineer-roadmap.xyz/token-economics

#ModelCascades #LLMOps #InferenceCost #DataEngineering #CostAwareAI

Cost down up to 60%, with less than 1% performance drop. The mechanism is almost embarrassingly simple. Ask the cheap model twice. BEST-Route diagnoses why earlier routers underdeliver. They generate exactly one response from the model they select. A single response from a small, inexpensive model is often not good enough to beat a response from a large one, so the router keeps escalating, overusing the expensive model and missing out on the savings it was built to capture. But it is well known that for small models, generating multiple responses and selecting the best enhances quality while remaining cheaper than a single large-model response. BEST-Route turns that into a routing decision: choose the model and the number of responses to sample from it, based on query difficulty and the quality thresholds you set. That reframing matters: • Routing is two decisions, not one: which model, and how many samples. • Test-time compute on a cheap model is frequently the better buy. • A high escalation rate is a symptom. Look at your sampling policy before blaming the router. Full map of the token-economics literature: https://ai-engineer-roadmap.xyz/token-economics

#LLMRouting #TestTimeCompute #InferenceCost #LLMOps #TokenEconomics

The cheapest quality signal in your organisation is one you already collected: human preference data. RouteLLM turns it into a router. The framing is deliberately narrow, which is why it works. Two models, one stronger and expensive, one weaker and cheap. Learn to dispatch each query, at inference time, to whichever will do. Train that decision on human preference data plus augmentation, rather than on difficulty heuristics you invented in a meeting. On widely recognised benchmarks the routers cut cost by over 2 times in certain cases, without compromising response quality. The finding people overlook is transfer. Swap the strong model, swap the weak model, and the trained router keeps performing at test time. It learned something about queries, not about a specific pair of endpoints. In a market where the frontier reprices every few months, that property is worth more than the headline saving. Takeaways: • Preference data you already have beats difficulty heuristics you invent. • A router that survives model swaps is infrastructure. One that does not is debt. • Cost versus quality is a dial. Someone on your team should own where it sits. Full map of the token-economics literature: https://ai-engineer-roadmap.xyz/token-economics

#LLMRouting #PreferenceData #InferenceCost #TransferLearning #TokenEconomics

Up to 40% fewer calls to the large model, with no drop in response quality. That is the entire pitch of Hybrid LLM, and it holds. The asymmetry it exploits is one every deployment already lives with. Large models need expensive cloud servers because of their size. Smaller models run on lower cost edge devices but lag behind on response quality. Most teams resolve this by picking a side and quietly eating the downside. Hybrid inference refuses the choice. A router assigns each query to the small or the large model based on predicted query difficulty and the desired quality level. And that quality level is not baked in during training. It can be tuned dynamically at test time, so you trade quality for cost as the scenario requires: per environment, per customer tier, per incident. What to take away: • Difficulty prediction is cheap. The routing decision is where the money is. • Expose the cost-quality trade-off as a runtime parameter, not a deploy-time constant. • Edge plus cloud is a cost architecture, not only a latency one. Full map of the token-economics literature: https://ai-engineer-roadmap.xyz/token-economics

#LLMRouting #EdgeAI #HybridInference #InferenceCost #TokenEconomics

Up to 98% of an LLM bill is optional. That is the FrugalGPT result, and it still has not been priced into how most teams build. The paper reviews popular LLM APIs and finds fees that differ by two orders of magnitude for work that is often interchangeable. So it stops asking which model is best and starts asking which model is enough. The cascade is embarrassingly simple. Send the query to a cheap model. Score the answer. Escalate only when the score is low. Most queries never reach the expensive tier, because most queries were never hard. The headline: FrugalGPT matches the performance of the best individual LLM, GPT-4, with up to 98% cost reduction. Flip the knob the other way and it improves accuracy over GPT-4 by 4% at the same cost. Takeaways: • Your model choice is a per-query decision, not an architecture decision. • A scorer good enough to rank two answers is far cheaper than a model good enough to write one. • Same-cost accuracy gains mean the cascade is not only a discount, it is an ensemble. The full breakdown, with the papers: https://ai-engineer-roadmap.xyz/token-economics

#LLMRouting #InferenceCost #LLMOps #ModelCascades #TokenEconomics

Performance comparable to GPT-4, at only 40% of GPT-4's cost, across six reasoning benchmarks. No fine-tuning. No new model. The cascade is the product. Simpler questions go to the weaker, more affordable LLM. Only the challenging ones get escalated to the stronger, expensive one. Which collapses the entire engineering problem into a single question: how do you know when the weak model got it wrong? The answer here is answer consistency. Sample the weaker model repeatedly and treat its agreement with itself as a difficulty signal. If it keeps landing on the same answer, trust it. If it wobbles, defer upward. The paper pushes further with a mixture of thought representations, checking consistency across both Chain-of-Thought and Program-of-Thought, so agreement has to survive two different ways of reasoning about the same problem. With GPT-3.5-turbo as the weak model and GPT-4 as the strong one, that deferral rule is enough. Takeaways: • Self-consistency is a free, model-agnostic confidence estimator. • Agreement across representations is stronger evidence than agreement within one. • Deferral logic, not model choice, is where cascade savings actually come from. Full map of the token-economics literature: https://ai-engineer-roadmap.xyz/token-economics

#ModelCascading #ChainOfThought #LLMReasoning #InferenceCost #TokenEconomics

At an 8K context budget, JSON tool schemas overflow the window so completely that exact match collapses to 2.6% on average. The model is not stupid. It never saw the question. This is the first systematic study of the tool-context trade-off. Agentic RAG systems hand the model dozens to hundreds of tool definitions, and those definitions consume the very context window retrieval needs. Two things compete for one budget, and nobody had measured who wins. Applying TSCG conservative-profile compression saves 44-50% of schema tokens and produces what the authors call a binary enablement effect: +20.5 pp average exact match, rising to +24.7 pp among the models that flip fully on. On HotpotQA multi-hop questions the same overflow scenario yields +48 pp. At 32K, where both formats fit, the effect essentially vanishes. It was never about quality. It was about fitting. Takeaways: • Tool schemas are context cost. Budget them like retrieved passages. • JSON schemas overflow at roughly 494 tools. Compressed ones stay operational beyond 800. • Compression buys capability at small budgets, not marginal accuracy at large ones. Full map of the token-economics literature: https://ai-engineer-roadmap.xyz/token-economics

#AgenticRAG #ContextEngineering #ToolCalling #PromptCompression #TokenEconomics

Overapproximating the code context charges you twice: once at the API meter, and once when the noise drowns the bug-fixing signal. SWEzze settles the bill. On SWE-bench Verified it cuts the total token budget by 51.8%-71.3% and improves issue resolution rates by 5.0%-9.2%, holding a stable compression rate of about 6 times across three frontier LLMs. The tension it resolves is real. Generic compressors compromise the semantic integrity of code, mangling the thing you needed intact. Code-specific tools understand syntax but lack awareness of task context, so they cannot tell which fragments the patch will actually require. The method is Oracle-guided Code Distillation: genetic search plus delta debugging, systematically reducing each code context to its minimal sufficient subsequence, retaining only the ingredients required for a successful fix. That distilled data then fine-tunes a lightweight model that learns to do the same filtering at inference time. Takeaways: • Irrelevant context is not neutral. It distracts the model and you pay for the privilege. • Compression is a quality intervention, not only a cost one. • Compress for the task, not for generic English redundancy. Full map of the token-economics literature: https://ai-engineer-roadmap.xyz/token-economics

#SWEbench #ContextCompression #CodeLLM #AgenticCoding #TokenEconomics

Deleting tokens is not the only way to compress a prompt. You can rewrite it into a denser language. Telegraph English does exactly that. Rather than training a classifier to remove low-importance tokens at a fixed ratio, TE performs a full semantic rewrite: decompose the input into atomic fact lines, substitute verbose phrases with a vocabulary of logical and relational symbols, and let the compression ratio adapt to each document's information density instead of a preset target. The elegant consequence of the line-structure rule is that compression and semantic chunking become the same operation. Every output line is an independently addressable fact, so the compressed representation is simultaneously a semantic index. One pass, two artefacts. Evaluated on LongBench-v2 across five OpenAI models, TE preserves 99.1% accuracy on key facts with GPT-4.1 at roughly 50% token reduction, and wins at matched compression ratios on every model and task tested. The gap widens on smaller models. Takeaways: • Fixed compression ratios ignore that documents differ in density. • Explicit relational structure compensates for limited model capacity. • If your compressed form is addressable, it doubles as your retrieval index. Full map of the token-economics literature: https://ai-engineer-roadmap.xyz/token-economics

#PromptCompression #ContextEngineering #SemanticChunking #RAG #TokenEconomics

Give an agent the tools to forget, and it manages its own context better than your summarizer does. Focus tests that claim. The target is context bloat: as interaction history grows across a long-horizon software engineering task, computational cost explodes, latency increases, and reasoning degrades because the model keeps getting distracted by irrelevant past errors. The usual remedies are passive, external summarization mechanisms the agent cannot control. Focus inverts that. Inspired by the exploration strategy of slime mold, the agent decides for itself when to consolidate key learnings into a persistent knowledge block, then actively withdraws the raw interaction history behind it. On context-intensive SWE-bench Lite instances with Claude Haiku 4.5, Focus cuts tokens by 22.7%, from 14.9M to 11.5M, at identical accuracy. It performs 6.0 autonomous compressions per task on average, with savings reaching 57% on individual instances. Takeaways: • Capable models can self-regulate context when given the right tools and prompting. • Compression belongs inside the agent loop, not bolted on outside it. • Consolidate the learnings, then throw away the transcript that produced them. Full map of the token-economics literature: https://ai-engineer-roadmap.xyz/token-economics

#AgentMemory #ContextCompression #SWEbench #AgenticAI #TokenEconomics

Prompt compression can make your system slower. Almost nobody checks. Across thousands of runs and 30,000 queries, spanning several open-source LLMs and three GPU classes, this study separates compression overhead from decoding latency instead of quietly folding them together. The result is uncomfortable for anyone who bolted a compressor on and declared victory. LLMLingua achieves up to 18% end-to-end speed-up, with response quality statistically unchanged across summarization, code generation and question answering. But only when prompt length, compression ratio and hardware capacity are well matched. Outside that operating window, the compression step dominates and cancels out the gains. You paid for preprocessing in order to save on decoding, and lost the trade. The sleeper finding is memory. Effective compression can reduce memory usage enough to offload workloads from data centre GPUs onto commodity cards, for only a 0.3s increase in latency. That is a procurement decision, not a tuning tweak. Takeaways: • Compression has a break-even point. Find yours before you ship it. • Measure end-to-end wall clock, never token counts alone. • The real win may be cheaper hardware rather than faster responses. Full map of the token-economics literature: https://ai-engineer-roadmap.xyz/token-economics

#PromptCompression #InferenceLatency #RAG #GPUEfficiency #TokenEconomics

A fixed compression rate over-compresses your simple queries and under-compresses your complex ones. Every single time. That is the observation behind ACC-RAG, and once stated it is hard to unsee. RAG buys you external knowledge and charges you in inference cost, because retrieved contexts are lengthy. Context compression is the standard mitigation, but existing methods apply one rate to everything. A trivially answerable question gets the same budget as a multi-hop reasoning chain. One is starved, the other is bloated. ACC-RAG dynamically adjusts the compression rate per query, based on input complexity. It pairs a hierarchical compressor producing multi-granular embeddings with a context selector that retains the minimal sufficient information, which the authors liken to human skimming. Evaluated on Wikipedia and five QA datasets, it outperforms fixed-rate methods and unlocks over 4 times faster inference than standard RAG, while maintaining or improving accuracy. Takeaways: • Compression rate is a per-query decision, not a config value. • Complexity estimation is cheap. Uniform budgets are not. • Skim first, then compress what survives. Full map of the token-economics literature: https://ai-engineer-roadmap.xyz/token-economics

#RAG #ContextCompression #AdaptiveInference #RetrievalAugmentation #TokenEconomics

Stop re-reading the same codebase on every single query. That is the argument behind Cartridges (Lightweight and general-purpose long context representations via self-study). Models support 100K-1M token contexts, but serving that is brutal: KV cache memory scales with input length, and you pay it again for every query against the same corpus. Cartridges instead trains a small KV cache offline, once, per corpus — and amortizes that cost across every query that references it. The catch they found: naive next-token training on the corpus is not competitive with in-context learning. Their fix, self-study, generates synthetic conversations about the corpus and trains with a context-distillation objective. Result on challenging long-context benchmarks: Cartridges match in-context learning performance while using 38.6x less memory and enabling 26.4x higher throughput. Effective context stretches from 128k to 484k tokens on MTOB. Three takeaways: • If your corpus is stable and your queries are many, in-context learning is the expensive path. • The cost model matters more than the benchmark. Amortization is the whole trick. • Cartridges compose at inference time without retraining. The economics behind long context: https://ai-engineer-roadmap.xyz/token-economics

#ContextCompression #KVCache #InferenceCost #LongContext #LLMOps

Long context charges you twice. It inflates prefill computation, and it triggers the lost-in-the-middle degradation where the model stops attending to material buried in the sequence. Remove the right tokens and you fix the second problem while solving the first. QwenLong-CPRS reports 21.59 times context compression alongside 19.15-point average performance gains across flagship LLMs. It works through multi-granularity compression guided by natural language instructions, with bidirectional reasoning layers for boundary awareness, token critic mechanisms with language modelling heads, and window-parallel inference. Integration is architecture-agnostic: GPT-4o, Gemini2.0-pro, Claude3.7-sonnet, DeepSeek-v3 and Qwen2.5-max all take the plug-in. It beats RAG and sparse attention on both accuracy and efficiency, across contexts spanning 4K to 2M words. Deployed with Qwen2.5-32B-Instruct, it surpasses leading proprietary LLMs by 4.85 and 10.88 points on Ruler-128K and InfiniteBench. Takeaways: • Long context is a cost problem and a quality problem. Attack both at once. • Instruction-guided compression keeps what this query needs, not what a generic scorer likes. • An architecture-agnostic plug-in beats waiting for a bigger context window. Full map of the token-economics literature: https://ai-engineer-roadmap.xyz/token-economics

#LongContext #ContextCompression #PrefillOptimization #LLMInference #TokenEconomics

Long-horizon agents fail for the same reason overwhelmed engineers do. They cannot tell which of the last hundred things they did still matters. ACON attacks that directly. Unbounded context growth in agentic tasks creates two bottlenecks at once: prohibitive inference memory costs, and reasoning degradation caused by irrelevant information. Existing compression methods either rely on brittle heuristics or require parameter updates, which is impractical for proprietary or large-scale LLMs. ACON optimises in natural language space instead. It iteratively refines the compression guidelines themselves, based on failure analysis of the agent, so the compressor learns which state information is critical without a single gradient step. The optimised compressor is then distilled into smaller models to keep the overhead down. On AppWorld, OfficeBench and multi-objective QA, ACON reduces peak token usage by 26-54% while improving task success over existing compression baselines. Smaller language models gain up to 46%, because removing distraction matters most where capacity is scarce. Takeaways: • Compress observations and history together. They draw on one budget. • Failure analysis can optimise a prompt as effectively as a gradient updates a weight. • Distil the compressor, or its overhead eats your savings. Full map of the token-economics literature: https://ai-engineer-roadmap.xyz/token-economics

#AgentMemory #ContextCompression #LongHorizonAgents #Distillation #TokenEconomics

78% of the tokens in a production prompt template were doing nothing. ProCut (LLM Prompt Compression via Attribution Estimation) is the paper for anyone who has watched a prompt template grow to thousands of tokens because six teams each appended their own task instructions, few-shot examples and heuristic rules — and nobody was ever allowed to delete any of it. ProCut segments a prompt template into semantically meaningful units, quantifies each unit's impact on task performance, and prunes the low-utility ones. Training-free, LLM-agnostic. Across five public benchmark datasets and real-world industrial prompts, it achieves 78% fewer tokens in production while maintaining or even slightly improving task performance — up to 62% better than alternative methods. Its LLM-driven attribution estimator reduces compression latency by over 50%. What this changes: • Prompt bloat is an organizational failure before it is a technical one. Attribution gives you permission to delete. • Measure each block's contribution, not the prompt's overall score. Averages hide the dead weight. • Compression that needs no training and no model access is deployable this quarter. Audit your prompt spend here: https://ai-engineer-roadmap.xyz/token-economics

#PromptCompression #PromptEngineering #InferenceCost #LLMOps #TokenEconomics

Information entropy is the wrong metric for deciding which tokens to delete. That quiet claim sits at the heart of LLMLingua-2, and it reshapes the entire method. Earlier compressors score tokens by information entropy under a causal language model such as LLaMa-7B. Two problems. Entropy only leverages unidirectional context, so it misses information the rest of the sentence makes essential. And it was never aligned with the compression objective to begin with. It is a proxy nobody trained. So LLMLingua-2 reformulates prompt compression as a token classification problem, learned through data distillation from an LLM, on an extractive text compression dataset the authors release. The base architecture is a Transformer encoder, which captures the full bidirectional context. Faithfulness to the original prompt becomes structural rather than hoped for. The economics: compression ratios of 2x-5x, end-to-end latency accelerated by 1.6x-2.9x, and a compressor that runs 3x-6x faster than existing prompt compression methods. Takeaways: • A small bidirectional encoder beats scoring tokens with a large causal one. • Train on the objective you actually want. Proxies leak. • Task-agnostic means it generalises out of domain, and here it does. Full map of the token-economics literature: https://ai-engineer-roadmap.xyz/token-economics

#PromptCompression #TokenClassification #TaskAgnostic #LLMInference #TokenEconomics

Not all compression happens in the prompt. Activation Beacon compresses the activations themselves, achieving an 8x reduction of KV cache memory and 2x faster inference at quality comparable to the uncompressed baseline. The design choice worth stealing is the one it rejects. Soft prompts are a bottleneck: relaying the complex information inside a long context through a handful of learned vectors throws away most of what made that context worth having. So Activation Beacon skips the relay and directly compresses the keys and values at every layer, as a plug-in module for transformer-based LLMs. Two details make it practical. Each fine-grained input unit is compressed progressively, keeping both training and inference efficient. And the compression ratio is randomly sampled at each training step, so a single model supports a wide range of compression configurations at deployment instead of one fixed setting. It generalises well past its training length, handling contexts of 128K after training at 20K. Takeaways: • KV cache memory is the constraint that actually bites in production. • Compress the activations, not a summary of the text. • Randomising the ratio during training buys you a runtime dial for free. Full map of the token-economics literature: https://ai-engineer-roadmap.xyz/token-economics

#KVCache #LongContext #InferenceOptimization #TransformerArchitecture #TokenEconomics

An entire natural language context, compressed into a minimum of one single special token. 500xCompressor (Generalized Prompt Compression for Large Language Models) pushes prompt compression to the point where it stops looking like text at all. It introduces approximately 0.3% additional parameters and achieves compression ratios ranging from 6x up to 480x. The compressed prompt can be used by the original LLM without fine-tuning it. The honest part is the number people skip: at these ratios the LLM retains 62.26-72.89% of its capabilities compared to using non-compressed prompts. This is not free. It is a trade you can now actually price. What to take from it: • Compression has a dial, and the far end of the dial is not lossless. Know which of your workloads can live on 62.26-72.89% of the model. • KV values beat embeddings for preserving information at high compression ratios — the representation you compress into matters as much as the ratio. • Not all compressed tokens are equally utilized. Compression budgets are unevenly spent, and that is exploitable. More on where the ratios actually pay: https://ai-engineer-roadmap.xyz/token-economics

#PromptCompression #InferenceCost #KVCache #LLMEfficiency #TokenEconomics

Soft prompts do not transfer. Compress a context into learned vectors and you have married one model's embedding space, which puts the whole technique out of reach for anyone calling an API-based LLM. Nano-Capsulator takes the harder road: compress lengthy prompts into natural language. The output is a Capsule Prompt, readable text, portable across models, sendable anywhere you can send a string. That buys transferability, and it costs two genuine problems. Natural language prompts are incompatible with back-propagation, and they lack any natural handle for imposing length constraints. The framework answers both with reward functions. One interacts with a semantics-preserving loss to hold the meaning. The other features length constraints to hold the size. The payoff is a Capsule Prompt that reduces 81.4% of the original length, saves 80.1% of budget overheads, and decreases inference latency up to 4.5x, while transferring across diverse LLMs and different datasets. Takeaways: • If you call APIs, natural language is the only compressed format you can actually ship. • Portability across models is worth a premium in a market that reprices quarterly. • Put the length budget in the objective, not in a post-hoc truncation. Full map of the token-economics literature: https://ai-engineer-roadmap.xyz/token-economics

#PromptCompression #SoftPrompts #APILLMs #InferenceLatency #TokenEconomics

Your prompt can lose 20x of its tokens and the model barely notices. That is the finding of LLMLingua (Compressing Prompts for Accelerated Inference of Large Language Models). Chain-of-thought and in-context learning have pushed prompts into the tens of thousands of tokens, and most of those tokens are carrying almost no information. LLMLingua treats that redundancy as an engineering problem: a budget controller decides how much of the prompt each section deserves, then a token-level iterative pass strips the rest while modelling the dependencies between what stays. Across four datasets from different scenarios — GSM8K, BBH, ShareGPT and Arxiv-March23 — it reaches up to 20x compression with little performance loss. What this means in practice: • Prompt length is not the same thing as prompt information. You are paying for both and only one of them is doing work. • Compression is a budgeting decision, not a truncation decision. Deciding what to cut is the whole method. • A small model can do the cutting for a large model. The compressor does not have to be the thing you are compressing for. Token economics is the discipline nobody teaches. Start here: https://ai-engineer-roadmap.xyz/token-economics

#PromptCompression #InferenceCost #LLMLingua #TokenEconomics #LLMOps

Compressing a prompt made the model more accurate, not less. LongLLMLingua (Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression) starts from an uncomfortable fact: LLM performance depends on the density and position of key information in the prompt. Long contexts do not just cost more — they bury the signal. So the compressor is aimed at the model's perception of key information, not merely at byte count. On NaturalQuestions with GPT-3.5-Turbo, it boosts performance by up to 21.4% with around 4x fewer tokens. On the LooGLE benchmark it achieves a 94.0% cost reduction. Compressing prompts of about 10k tokens at ratios of 2x-6x accelerates end-to-end latency by 1.4x-2.6x. Three things worth internalizing: • Stuffing more context into the window can be a downgrade. Position bias is a real tax. • Cost and quality are not always on opposite ends of a slider. Removing noise buys both. • Latency improves as a side effect. Fewer tokens in means less prefill to pay for. The full token-economics map, papers and all: https://ai-engineer-roadmap.xyz/token-economics

#PromptCompression #InferenceCost #LongContext #RAG #TokenEconomics

4 times context compression from a module that introduces about 1% additional parameters. The In-context Autoencoder has one of the best leverage ratios in the compression literature. The idea is to let the LLM compress its own context. ICAE learns to squeeze a long context into short, compact memory slots that the same LLM can then condition on directly. No separate compressor model. No external summarizer. No new format the target model has to be taught to read. Training runs in two stages. First, pretraining with both autoencoding and language modeling objectives on massive text data, so the memory slots accurately and comprehensively represent the original context. Then fine-tuning on instruction data, so the model produces desirable responses when conditioned on those slots. Built on Llama, it improves both latency and GPU memory cost at inference. The authors draw a line to working memory in cognitive science. That is a hint about where context management is heading. Takeaways: • Compression can be a lightweight adapter, not a second model. • Memory slots the LLM already understands remove the translation tax. • Autoencoding is a natural objective for faithful compression. Full map of the token-economics literature: https://ai-engineer-roadmap.xyz/token-economics

#ContextCompression #MemorySlots #Autoencoders #LongContext #TokenEconomics

Prefix caching only helps when the prefix actually matches. Production traffic rarely cooperates. CacheTune, from Adaptive KV Cache Reuse for Fast Long-Context LLM Serving, goes after the case everyone else skips: reusing KV entries in non-prefix positions, where naive reuse shatters cross-chunk attention and quality collapses. Frequency-domain analysis, run offline, identifies the KV pairs that carry the cross-attention structure. Those get recomputed online. Everything else is reused as-is. The payoff is 3.72x-4.86x faster time-to-first-token and 3.93x-6.21x higher throughput, with generation quality close to full recompute. • Prefill is where long-context latency dies, not decode. TTFT is the number your users feel. • Cache-or-recompute is a false binary. Selective recompute of the semantically critical tokens buys back most of the quality for a fraction of the compute. • Even with caches offloaded to I/O-bound SSD or HDD, adaptive recomputation ratios sustain 2.34x-2.36x TTFT speedup. Cheap storage tiers stay viable. Stop treating the KV cache as a hit-or-miss lookup. It is a lossy compression problem with a recompute dial. The full token-economics reading list: https://ai-engineer-roadmap.xyz/token-economics

#KVCache #LLMInference #TTFT #LongContext #MLSys

Not every token in your prompt is worth the GPU memory it occupies. Some are worth 756 times more than others. That is the measurement at the heart of "Not All Tokens Are Worth Caching: Learning Semantic-Aware Eviction for LLM Prefix Caches." System prompts, user queries, tool outputs, model responses, chain-of-thought traces — these token types exhibit up to 756x variation in reuse rates. And LRU, the policy actually running in production, treats every cached block as interchangeable. It throws away the system prompt and keeps a one-off tool dump. SAECache exploits the signal instead: multi-queue routing per token type, an online-learned weighting of reuse value, and fully adaptive parameter updates. • Result: 1.4x-2.7x TTFT improvement over production-style baselines. • The trap: fixed-parameter alternatives degrade by up to 2.7x under workload mismatch. Hand-tuned eviction is a bet on a workload that will change. • The lesson: your eviction policy is a pricing decision. Evicting the wrong block means paying full prefill for something you already computed. Stop treating cache blocks as uniform. Full breakdown: https://ai-engineer-roadmap.xyz/token-economics

#KVCache #PrefixCaching #CacheEviction #LLMServing #LLMInference

Change one field in your prompt and the entire downstream prefix cache is invalidated. That is the tax everyone pays and nobody questions. "Models Take Notes at Prefill: KV Cache Can Be Editable and Composable" asks the obvious follow-up: why not just overwrite the field's own key/value vectors and keep the rest? Because it does not work — the model already wrote the field-conditioned conclusion onto the downstream notes. The paper establishes this causally across four model families: the field's own key/value drives under 1% of the decision. Read the cache as a notebook of memoized conclusions and two things follow. • Editable. An append-only erratum amends the notes. With chain-of-thought, editing the field alone recovers the decision. • Composable. The notes are position-portable, so a precompiled skill can be RoPE-repositioned and spliced into any context — indistinguishable from full recompute, at O(L) instead of O(L^2) time-to-first-token. • Cheap. A unified edit+compose agent stays decision-identical to recompute at up to 14.9x lower latency. In an online vLLM benchmark it holds a 98.5% prefix cache hit-rate and cuts p90 time-to-first-token by 53-398x. The cache is not a blob. It is a data structure. More on cache economics: https://ai-engineer-roadmap.xyz/token-economics

#KVCache #PrefixCaching #LLMServing #vLLM #InferenceOptimization

Your RAG pipeline retrieves the right documents and then throws away the cache hit by putting them in the wrong order. That is the whole insight behind CacheWeaver: Cache-Aware Evidence Ordering for Efficient Grounded RAG Inference. Prefix caching in engines like vLLM only pays out when requests share a literal token prefix. Adjacent queries often retrieve overlapping evidence, but in different orders, so set overlap never becomes prefix overlap. The cache sits there useless. CacheWeaver keeps a prefix tree over recently served evidence sequences and greedily places the most reusable prefix first. The retriever is untouched. The serving engine is untouched. Only the ordering changes. • Median time-to-first-token drops by about 20-33 percent versus retrieval-order prefix caching, across three vLLM configurations, with no hit to answer quality. • The greedy walk captures 97.5 percent of the gain available from oracle ordering. You do not need a clever policy, just any policy. • The cheapest wins in inference are scheduling layers, not kernels. This one lives entirely in the prompt layer. More cache-economics research, distilled: https://ai-engineer-roadmap.xyz/token-economics

#RAG #PrefixCaching #vLLM #KVCache #TTFT #LLMOps

Every agent in your multi-agent system is holding a private copy of the same KV cache. That is the memory bill nobody audits. PolyKV: A Shared Asymmetrically-Compressed KV Cache Pool for Multi-Agent LLM Inference writes the compressed cache once and injects it into N independent agent contexts. The compression is deliberately asymmetric: Keys are quantized at int8 to preserve softmax stability, while Values get a Fast Walsh-Hadamard rotation followed by 3-bit Lloyd-Max quantization. On Llama-3-8B with 15 concurrent agents sharing a 4K-token context, KV memory falls from 19.8 GB to 0.45 GB. That is a 97.7% reduction for a +0.57% perplexity cost and a mean BERTScore F1 of 0.928. • Keys and Values do not deserve the same bit budget. Treating them identically is the default, and the default is wrong. • The perplexity delta does not grow with agent count. Fan-out is nearly free once the pool is shared. • Longer contexts make it better, not worse: the delta inverts to -0.26% at 1,851 coherent tokens. The rest of the KV cache literature: https://ai-engineer-roadmap.xyz/token-economics

#KVCache #MultiAgent #Quantization #LLMInference #GPUMemory

Coupling your prefix cache to your request scheduler feels efficient. It is why your cluster has load imbalance, duplicated caches and fragmented memory. TokenLake: A Unified Segment-level Prefix Cache Pool for Fine-grained Elastic Long-Context LLM Serving argues the two jobs should be separated. Existing systems shuttle ever-longer prefix caches between instances, which makes low-latency memory pooling impossible. TokenLake instead exposes query tensors, prefix caches and cache-aware operations through a declarative interface, then manages the cache at segment granularity with heavy-hitter-aware load balancing. Deduplication, defragmentation and balance become properties of the pool. The scheduler goes back to doing one thing: scheduling. • Throughput rises up to 2.6 times over cache-aware routing and 2.0 times over cache-centric PD-disaggregation. • Hit rate climbs 2.0 times and 2.1 times against those same baselines. Hit rate is the real currency here. • Abstraction beats coupling. Shield the scheduler from cache management and both layers get simpler. Cluster-level caching is the least glamorous, highest-leverage line item in your inference budget. Read the full breakdown: https://ai-engineer-roadmap.xyz/token-economics

#PrefixCaching #KVCache #LLMServing #Scheduling #MLSys

LRU is a confession that your cache has no idea what happens next. In agentic workflows, it does. KVFlow: Efficient Prefix Caching for Accelerating LLM-Based Multi-Agent Workflows makes the case bluntly. Agent systems reuse fixed prompts across repeated invocations, which is exactly what prefix caching is for, but LRU eviction routinely discards an agent's KV cache moments before that agent runs again. The cache miss is not bad luck. It is the policy. KVFlow abstracts the execution schedule as an Agent Step Graph and gives each agent a steps-to-execution value estimating how soon it fires. Eviction then happens at the KV node level, guided by that value. • Against SGLang with hierarchical radix cache, KVFlow is up to 1.83 times faster for single workflows with large prompts. • Under many concurrent workflows, the speedup reaches 2.19 times. Contention is where scheduling knowledge compounds. • Prefetching overlaps fully: tensors move from CPU to GPU in background threads for whichever agents run next, so misses never stall generation. You already have the workflow graph. Your cache should be reading it. The full token-economics research index: https://ai-engineer-roadmap.xyz/token-economics

#AgenticWorkflows #KVCache #SGLang #PrefixCaching #LLMServing

A LLaMA-3.1 8B model running 128K context on a single RTX 4090 is not a memory trick. It is a codebook design choice. CommVQ: Commutative Vector Quantization for KV Cache Compression starts from the obvious bottleneck, the KV cache growing without bound as context grows, and applies additive quantization with a lightweight encoder and codebook that decodes through a simple matrix multiplication. The elegant part is the constraint: the codebook is trained, via Expectation-Maximization, to commute with Rotary Position Embedding. That commutativity is what lets decoding fold directly into self-attention instead of paying a dequantization tax at every step. • 2-bit quantization cuts FP16 KV cache size by 87.5% while beating state-of-the-art KV quantization baselines on long-context benchmarks and GSM8K. • 1-bit KV quantization becomes viable with minimal accuracy loss. That is the regime everyone assumed was broken. • Compression schemes that ignore positional encoding pay for it at decode time. Design the codebook around RoPE, not against it. Consumer hardware is a serving tier, if your cache math is right. Every paper in the token-economics corpus: https://ai-engineer-roadmap.xyz/token-economics

#KVCache #Quantization #LongContext #RoPE #LLMInference

Compressing your distributed KV cache saves bandwidth, then quietly steals it back at decompression time, right out of the GPU's hands. ShadowServe: Interference-Free KV Cache Fetching for Distributed Prefix Caching names that trade-off and refuses it. Distributed prefix caching is meant to accelerate long-context serving by reusing KV entries across common prefixes, but on a constrained network the fetch becomes the bottleneck. Compression helps the wire and hurts the host, because decompression contends with model computation. So ShadowServe splits the planes: control stays on the host, the data plane is fully offloaded to a SmartNIC. No interference with host GPU or CPU. A chunked pipeline parallelizes across the NIC's modest compute, and minimal-copy memory management keeps its limited memory from becoming the new wall. • Loaded time-per-output-token drops up to 2.2x versus state-of-the-art solutions. • Time-to-first-token improves up to 1.38x in low-bandwidth scenarios, translating to up to 1.35x higher throughput. • Interference is a first-class cost. Measure it, or your compression win is imaginary. The full inference-economics reading list: https://ai-engineer-roadmap.xyz/token-economics

#KVCache #SmartNIC #PrefixCaching #LLMServing #Throughput

Uniform KV cache budgets across layers are a rounding error masquerading as a design. CAKE: Cascading and Adaptive KV Cache Eviction with Layer Preferences reframes eviction as a cake-slicing problem. Attention patterns differ sharply layer to layer, yet most eviction schemes hand every layer the same allocation. CAKE profiles each layer's preferences along both spatial and temporal dimensions, sizes its cache accordingly, and enforces the global memory budget in a cascading manner. It also fixes the eviction indicator itself, scoring the shifting importance of tokens over time rather than freezing an early judgment. • Model performance holds on just 3.2% of the KV cache, across LongBench and NeedleBench, with the biggest margins in low-memory settings. • Decoding latency improves over 10x versus full cache at 128K-token contexts with FlashAttention-2. Eviction is a speed technique, not only a memory one. • A global view of allocation beats locally optimal per-layer heuristics. Budget the whole cake before slicing it. If you evict uniformly today, you are spending memory on layers that never asked for it. More KV cache economics: https://ai-engineer-roadmap.xyz/token-economics

#KVCache #CacheEviction #LongContext #FlashAttention #LLMInference

Most serving stacks treat the KV cache as a side effect of inference. Moonshot AI built its platform around it and got a 525% throughput increase. Mooncake is the serving system behind Kimi. Its architecture is KVCache-centric and disaggregated: prefill and decoding run in separate clusters, and the underutilized CPU, DRAM, and SSD sitting idle in every GPU node get conscripted into a distributed KVCache pool. The scheduler's job is not to place requests — it is to maximize effective throughput against latency SLOs, with a prediction-based early rejection policy for overload. What makes this a token-economics paper rather than a systems paper: • Up to a 525% increase in throughput in certain simulated scenarios, while still adhering to SLOs. • Under real production workloads, the architecture lets Kimi handle 75% more requests. Same hardware. Same model. • The gains concentrate in long-context scenarios — exactly where prefill cost dominates your bill. The cheapest token is the one you already computed and did not throw away. Read the rest of the KV-cache economics thread: https://ai-engineer-roadmap.xyz/token-economics

#KVCache #LLMServing #Disaggregation #InferenceOptimization #TokenEconomics

Two ways to get a KV cache: compute it, or load it from storage. The right answer is both, at the same time. Compute Or Load KV Cache? Why Not Both? points out that prefix caching trades a compute bottleneck for an I/O bottleneck and calls that progress. Storage bandwidth is finite. While the GPU sits idle waiting on a disk read, it could have been prefilling the very tokens it is waiting for. The system, Cake, runs a bidirectional scheduling strategy that dynamically balances KV cache computation against loading, so both resources saturate together instead of taking turns. An adaptive mechanism folds in non-prefix-cacheable requests rather than treating them as an exception. • Time to first token falls 2.6x on average versus compute-only and I/O-only methods, across varied hardware, datasets and storage conditions. • Idle compute during a cache fetch is a real cost. Most caching systems never account for it. • The scheduler should treat compute and I/O as one pool of capacity, chosen per request rather than fixed per system. The rest of the token-economics corpus: https://ai-engineer-roadmap.xyz/token-economics

#KVCache #PrefixCaching #TTFT #LLMInference #StorageIO

Hybrid LLMs broke prefix caching, and almost nobody noticed. Models that mix Attention with Recurrent layers (state space models) use in-place state updates. That means cache entries cannot be rolled back for a partial sequence overlap — only exact-match hits count. The result is a flood of large cache entries per sequence, most of which will never be reused. Classic prefix caching quietly stops paying for itself. Marconi, from "Prefix Caching for the Era of Hybrid LLMs," changes what gets admitted in the first place. Its policies judge a candidate entry not on recency but on a forecast of its reuse likelihood across a taxonomy of hit scenarios, and on the compute savings a hit would deliver relative to its memory footprint. • Up to 34.4 times higher token hit rates than state-of-the-art prefix caching systems. • That converts to 71.1% lower TTFT — 617 ms of latency that simply stops existing. • The principle generalizes: admission is a cheaper lever than eviction, because a block you never admit costs you nothing. Architecture changes your cache economics. Details: https://ai-engineer-roadmap.xyz/token-economics

#KVCache #PrefixCaching #HybridLLMs #StateSpaceModels #LLMServing

Attention does not spread evenly through a model. It funnels. Your cache allocation should follow it down. PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling opens with an observation, not a method. Attention scatters widely in lower layers, consolidates within specific contexts as you climb, and finally collapses onto a handful of critical tokens, the attention sinks, in the upper layers. Uniform per-layer cache sizing ignores every bit of that structure. So PyramidKV allocates more cache to lower layers and less to higher ones. • Full-KV performance is matched on LongBench while retaining only 12% of the KV cache. • Pushed to a brutal 0.7% budget, it beats other compression techniques by up to 20.5 absolute accuracy points on TREC. The gap widens exactly where memory is scarce. • In Needle-in-a-Haystack, keeping just 128 KV cache entries lets LLAMA-3-70B hit 100.0 accuracy. Long-context comprehension survives aggressive pruning when the pruning respects layer structure. Measure your model's information flow before you budget its memory. The full research breakdown: https://ai-engineer-roadmap.xyz/token-economics

#KVCache #CacheCompression #LongContext #AttentionSinks #LLMInference

The KV cache was wasting most of its memory to fragmentation, and the fix came from operating systems written long before transformers existed. Efficient Memory Management for Large Language Model Serving with PagedAttention is the paper behind vLLM, and it is the closest thing inference has to a founding document. High-throughput serving demands large batches. Large batches demand KV cache memory. But that memory grows and shrinks per request, so contiguous allocation fragments badly and duplicates aggressively, which caps batch size and starves throughput. PagedAttention borrows classical virtual memory and paging: cache blocks need not be contiguous, so waste approaches zero and blocks can be shared within and across requests. • Throughput rises 2-4 times at the same latency versus FasterTransformer and Orca. • Gains grow with longer sequences, larger models, and more complex decoding algorithms. Exactly where modern workloads live. • Memory layout is a throughput decision. Fragmentation, not FLOPs, was the binding constraint. Almost every serving stack you touch today inherits this design. Worth knowing why. The full token-economics reading list: https://ai-engineer-roadmap.xyz/token-economics

#PagedAttention #vLLM #KVCache #LLMServing #Throughput

Multi-call LLM programs recompute the same prefix over and over, and the runtime happily lets them. SGLang: Efficient Execution of Structured Language Model Programs treats that as a systems bug rather than an application problem. Complex tasks now involve many generation calls, advanced prompting, control flow and structured inputs and outputs, but the tooling stops at a single request boundary. SGLang pairs a frontend language, with primitives for generation and parallelism control, against a runtime built to exploit what that frontend reveals. RadixAttention is the centerpiece: KV cache reuse across calls, not merely within them. Compressed finite state machines accelerate structured output decoding on top. • Throughput reaches up to 6.4x higher than state-of-the-art inference systems, across agent control, logical reasoning, few-shot benchmarks, JSON decoding, RAG pipelines and multi-turn chat. • The reuse opportunity lives between calls. Optimizing a single forward pass leaves that entire surface untouched. • Structured decoding is a cost lever, not only a correctness one. If your agent loops share prefixes, and they do, your runtime should know about it. Every paper in the corpus: https://ai-engineer-roadmap.xyz/token-economics

#SGLang #RadixAttention #KVCache #StructuredGeneration #LLMServing

A handful of tokens at the very start of the sequence are semantically meaningless, and deleting their KV state destroys the model. "Efficient Streaming Language Models with Attention Sinks" is still the most counterintuitive result in KV-cache land. Window attention — cache only the most recent KVs — is the obvious way to bound memory in a long chat. It collapses the moment the text outgrows the cache. The reason is not what you would guess: LLMs dump enormous attention scores onto the initial tokens as a sink, regardless of whether those tokens mean anything. Keep the sink tokens' KV alongside the sliding window, and the collapse disappears. • StreamingLLM lets Llama-2, MPT, Falcon and Pythia model stably over up to 4 million tokens and more, with no fine-tuning at all. • Against the sliding-window recomputation baseline, it delivers up to 22.2x speedup. • Adding a dedicated placeholder sink token during pre-training improves streaming deployment further. Bounded memory, unbounded conversation — the whole economics of multi-turn serving turns on a quirk nobody designed. More: https://ai-engineer-roadmap.xyz/token-economics

#KVCache #AttentionSinks #LongContext #LLMServing #StreamingLLM

Reusing a KV cache across machines sounds free until you try to move the tensors over a network. CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving takes the honest view. Long contexts mean nothing gets generated until the whole context is processed, so caching that context is essential. But fetching those large tensors over the wire introduces a delay that can swallow the savings whole. CacheGen builds a custom tensor encoder that exploits the KV cache's distributional properties, turning it into a compact bitstream with negligible decoding overhead. Then it adapts: compression level varies across parts of the cache as available bandwidth shifts, and when bandwidth collapses it can simply recompute a chunk on the fly. • KV cache size shrinks 3.5-4.3x, and total fetch-plus-process delay drops 3.2-3.7x, with negligible impact on response quality. • The KV cache is a data structure with exploitable statistics, not an opaque blob. • Bandwidth-adaptive compression means degrading gracefully instead of failing over. Treat cache transfer like video streaming and the economics change. The full token-economics research index: https://ai-engineer-roadmap.xyz/token-economics

#KVCache #CacheCompression #LongContext #Bandwidth #LLMServing

A typo in a log message and a migration that corrupts a production database both count as one benchmark failure. Your compute allocator cannot tell them apart. Not All Errors Are Equal: Consequence-Aware Reasoning Compute Allocation goes after the hidden assumption inside every difficulty-based router. Accuracy objectives weight every task equally, so spending more thinking tokens where accuracy is likeliest to improve looks optimal. Deployment disagrees. What matters is the cost of being wrong. The fix is a lightweight predictor that estimates, from the issue text alone, how expensive a wrong answer would be. The scheduler then routes high-consequence tasks to bigger compute tiers under the same total budget. • Consequence and difficulty are approximately orthogonal. Routing on one tells you almost nothing about the other. • Under matched compute budgets, cost-weighted loss drops by 22% to 33% relative to difficulty-aware routing. The priority-aware variant crosses 30%. • The issue-only predictor never misclassifies a high-consequence task as low-consequence across the 300 SWE-bench tasks. Safe failure modes are buildable. Budget your reasoning by blast radius, not by perceived hardness. The full breakdown: https://ai-engineer-roadmap.xyz/token-economics

#TestTimeCompute #ReasoningBudget #SWEBench #LLMRouting #AIEngineering

More thinking made a function-calling agent worse than not thinking at all. Not marginally worse. Catastrophically. Brief Is Better: Non-Monotonic Chain-of-Thought Budget Effects in Function-Calling Language Agents sweeps six token budgets across 200 tasks from the Berkeley Function Calling Leaderboard v3 Multiple benchmark, and the curve is a hill, not a ramp. On Qwen2.5-1.5B-Instruct, a 32-token reasoning budget lifts accuracy from 44.0% to 64.0%. Push to 256 tokens and accuracy falls to 25.0%, well below the no-CoT baseline. The error decomposition explains why. Without CoT, 30.5% of tasks fail because the model selects the wrong function. Brief CoT cuts that to 1.5%, effectively acting as a routing step. Long CoT reverses the gain, yielding 28.0% wrong selections and 18.0% hallucinated functions. • Oracle analysis shows 88.6% of solvable tasks need at most 32 reasoning tokens. • The proposed FR-CoT templates the reasoning phase to commit to a valid function name up front, driving hallucination to 0.0%. • In tool use, reasoning is routing. Past that point, it is drift. More reasoning-budget research: https://ai-engineer-roadmap.xyz/token-economics

#FunctionCalling #ChainOfThought #ToolUse #LLMAgents #ReasoningBudget

Long reasoning traces can crowd out the very answer they were meant to support. The paper calls it the coupling tax, and you pay it whenever traces and answers share one output-token budget. The Coupling Tax: How Shared Token Budgets Undermine Visible Chain-of-Thought Under Fixed Output Limits shows this is no corner case. Across GSM8K, MATH-500 and five BIG-Bench Hard tasks with Qwen3 models, non-thinking mode matches or outperforms thinking mode on GSM8K and MATH-500 at every budget up to 2048 tokens. Harder tasks push the crossover higher, but the mechanism holds: the trace runs long, the output limit hits, the answer never lands. A truncation-waste decomposition predicts that crossover from chain-length and accuracy statistics, and even explains inverse scaling inside the Qwen family. • Decoupling the budgets fixes it. Split-budget generation on full MATH-500 reaches 74.0% accuracy, a strengthened extraction variant 78.8%, and a fixed non-oracle gate 83.6%. • Test-time reasoning is a budget-allocation problem, not a question of whether longer traces are available. • Cap your output tokens and you are silently capping answers. Full research breakdown: https://ai-engineer-roadmap.xyz/token-economics

#ChainOfThought #ReasoningBudget #TestTimeCompute #MATH500 #LLMEvaluation

Routing an entire query to the frontier model because a few tokens in it were hard is a rounding error the size of your inference bill. RelayLLM: Efficient Reasoning via Collaborative Decoding argues that cascades and routers operate at hopelessly coarse granularity. A small model can usually handle the majority of reasoning steps. Offloading the whole query wastes compute on every step it could have done itself. So RelayLLM moves the decision down to the token level. The small model acts as an active controller and issues a special command to invoke the large model only for critical tokens, relaying generation back and forth. Training is two-stage: a warm-up phase, then Group Relative Policy Optimization to teach the model when to ask for help and when to keep going. • Across six benchmarks it averages 49.52% accuracy, bridging much of the gap between the small and large models. • The large model is invoked for only 1.07% of the total generated tokens. • That yields a 98.2% cost reduction compared with performance-matched random routers. Granularity is the lever nobody pulls. Route tokens, not queries. The full token-economics corpus: https://ai-engineer-roadmap.xyz/token-economics

#CollaborativeDecoding #SmallLanguageModels #InferenceCost #GRPO #LLMRouting

Automatic prompt optimizers chase accuracy and hand you back a prompt that inflates your token bill. Nobody put cost in the objective. CROP: Token-Efficient Reasoning in Large Language Models via Regularized Prompt Optimization does exactly that. Standard prompt-optimization frameworks target task accuracy exclusively, and since verbose reasoning traces tend to help accuracy at the margin, the optimizer learns to produce prompts that elicit them. Latency and cost are externalities the loop never sees. CROP adds regularization on response length, generating textual feedback about verbosity alongside the usual accuracy feedback. Optimization is pushed toward prompts that elicit concise responses carrying only the critical information and reasoning. • Evaluated on GSM8K, LogiQA and BIG-Bench Hard, it achieves an 80.6% reduction in token consumption while maintaining competitive accuracy, with only a nominal decline in performance. • The prompt is a cost surface. Treat it as one and savings show up without touching the model. • Whatever you leave out of the objective, the optimizer will happily spend. Cheapest reasoning intervention in the stack, and it lives entirely in text. More token-economics research: https://ai-engineer-roadmap.xyz/token-economics

#PromptOptimization #TokenEfficiency #ChainOfThought #LLMOps #InferenceCost

The shortest thinking chain a reasoning model produces is up to 34.5% more accurate than the longest one it produces for the same question. Same model. Same prompt. More thinking, worse answers. "Don't Overthink it. Preferring Shorter Thinking Chains for Improved LLM Reasoning" breaks the assumption the entire test-time-compute industry is priced on. Long chains are not deliberation. Past a point they are the model wandering, and you are paying output-token rates for every step of the walk. The paper's fix is almost insultingly simple. Run k generations in parallel, stop as soon as the first m finish thinking, majority-vote across those. They call it short-m@k. What that buys: • short-1@k matches or beats standard majority voting while using up to 40% fewer thinking tokens. • short-3@k beats majority voting at every compute budget tested, with up to 33% wall-time reduction. • Fine-tuning on the short chains, rather than the long ones, produces the better model. So the cheap path and the accurate path are the same path. That almost never happens in this field, and when it does you should take it. More on what reasoning tokens actually cost you: https://ai-engineer-roadmap.xyz/token-economics

#ReasoningModels #TestTimeCompute #ChainOfThought #InferenceCost #TokenEconomics

Roughly three quarters of a reasoning model's thinking tokens can be deleted with no accuracy cost. Not compressed. Deleted. FlashThink starts from an observation anyone who has read a raw R1 trace already suspects: at some point mid-thought, the model already knows the answer, and everything after that is ritual. So the authors train a small verification model whose only job is to spot that moment and cut the generation off. The result across four benchmarks: reasoning content reduced by 77.04% on DeepSeek-R1 and 77.47% on QwQ-32B, without reducing accuracy. Read those numbers as an invoice, because that is what they are. Thinking tokens bill as output tokens. • If your reasoning spend is real, assume most of it is buying nothing, and make the burden of proof land on the long trace. • The exit signal is learnable and cheap. A small verifier watching a big model is a better cost lever than swapping to a smaller model. • Measure token length per task, not just accuracy per task. You cannot cut what you never charted. The full breakdown of the reasoning-token line item: https://ai-engineer-roadmap.xyz/token-economics

#ReasoningModels #TestTimeCompute #EarlyExit #LLMInference #DeepSeekR1

Shorter reasoning is purchasable without RL, without fine-tuning, and without touching a prompt. Merge the weights instead. Unlocking Efficient Long-to-Short LLM Reasoning with Model Merging takes the overthinking problem seriously: System 2 models generate redundant reasoning steps with no proportional gain in output quality. The usual fixes, supervised fine-tuning, reinforcement learning and prompt engineering, are either computationally expensive or unstable. Merging is neither. Integrate the quick-thinking capability of a System 1 model into the methodical reasoning of a System 2 model and you get a cheap, robust middle. The study sweeps task-vector-based, SVD-based and activation-informed merging. • Average response length drops by up to 55% while preserving or even improving baseline performance. • Merging efficacy correlates strongly with model scale, across 1.5B/7B/14B/32B evaluations. Bigger models merge better. • The merged model retains self-critique and self-correction, and adapts its response length to task complexity rather than flattening it. Length reduction is usually framed as a decoding or training problem. It is also a weight-space problem, and that is by far the cheapest place to solve it. The full reading list: https://ai-engineer-roadmap.xyz/token-economics

#ModelMerging #ReasoningEfficiency #System2 #LLMTraining #TokenEfficiency

Setting a token budget requires knowing how hard the question is. If you knew that, you would not need the model. e1: Learning Adaptive Control of Reasoning Effort names that circularity. Thinking budgets do raise accuracy, but not every question warrants the same spend, and existing controls demand an absolute token count up front. That is an impossible ask at request time, and it forces dataset-specific and phase-specific tuning that never survives contact with real traffic. Adaptive Effort Control replaces it with a relative dial. The model is trained by self-adaptive reinforcement learning to use a user-specified fraction of tokens relative to the current average chain-of-thought length for each query. • Chain-of-thought length falls 2-3x while maintaining or improving performance against the base model used for RL training. • It holds across model scales ranging from 1.5B to 32B parameters. • The model learns on its own to allocate resources proportionally to task difficulty, and users adjust the cost-accuracy trade-off through a continuous effort parameter at inference time. Expose effort as a knob to your callers. Absolute budgets are a guess. Relative ones are a policy. Full breakdown: https://ai-engineer-roadmap.xyz/token-economics

#TestTimeCompute #ReinforcementLearning #ReasoningBudget #InferenceCost #LLMControl

Grade-school math is where reasoning models bleed the most money, because they refuse to notice it is grade-school math. DART (Difficulty-Adaptive Reasoning Truncation) attacks exactly that. Instead of an unstable RL loop that has to be rewarded into brevity, it distills concise reasoning patterns from stronger models, interpolates them into a continuum of styles, and trains on data curated to balance correctness against compactness. The model learns when to stop thinking. On GSM8K with DeepSeek-R1-Distill-Qwen-7B: 81.2% reasoning truncation and 5.33x computational acceleration, while preserving or improving accuracy. The uncomfortable implication is that the length of a reasoning trace is not a property of the problem. It is a habit the model was trained into, and habits can be trained back out. • Difficulty-conditioned length is a supervised problem, not only an RL problem. Cheaper and more stable to get right. • Easy queries dominate real production traffic. That is where a truncation policy pays, not on your hardest eval. • Accuracy preserved at 5.33x speedup means the long trace was latency you were shipping to users for free. Where your thinking tokens actually go: https://ai-engineer-roadmap.xyz/token-economics

#ReasoningModels #TestTimeCompute #AdaptiveInference #LLMEfficiency #TokenEconomics

Token efficiency and accuracy are supposed to trade off. SwiReasoning improves both at once, and it does it without any training. The idea: a reasoning model does not have to think in words the entire time. It can also reason in latent space, which packs more information per step. But pure latent reasoning spreads probability mass across too many implicit paths, adds noise, and hurts accuracy. And overthinking survives the switch to latent anyway. So SwiReasoning switches. It watches block-wise confidence, estimated from entropy trends in the next-token distribution, and flips between explicit and latent thinking to force timely convergence. Then it caps the number of thinking-block switches, which is what actually kills the overthinking. Across math, STEM, coding and general benchmarks: • Average accuracy up 1.8% to 3.1% across model families and scales. • Under constrained budgets, average token efficiency up 57% to 79% — and the gains get larger as the budget gets tighter. • Training-free. It is a decoding policy, not a fine-tune. The part worth internalising: the tighter you squeeze the budget, the more a smart thinking policy is worth. Budgets are not a tax on quality. They are a forcing function. More: https://ai-engineer-roadmap.xyz/token-economics

#ReasoningModels #TestTimeCompute #LatentReasoning #DecodingStrategies #LLMEfficiency

A small model, given the right test-time compute strategy, can outperform a model 14x its size in a FLOPs-matched comparison. That is the headline result of "Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters," and it quietly reframes the entire buy-a-bigger-model reflex. The paper studies two ways to spend inference compute: searching against process-based verifier reward models, and adaptively revising the model's own answer distribution at test time. Neither wins everywhere. Which one works depends on prompt difficulty, and that dependency is the whole game. So they allocate compute per prompt instead of uniformly. Compute-optimal scaling improves test-time compute efficiency by more than 4x over a best-of-N baseline. • Best-of-N is the default because it is easy, not because it is efficient. It is the flat tax of test-time compute. • Difficulty estimation is the load-bearing component. Get that wrong and adaptive allocation degrades to uniform spend. • The parameter-count reflex is a budgeting decision in disguise. Sometimes the right move is a smaller model that thinks better, not a larger one that thinks the same. Read the full test-time-compute economics: https://ai-engineer-roadmap.xyz/token-economics

#TestTimeCompute #ReasoningModels #ComputeOptimal #InferenceScaling #LLMSystems

Chain-of-thought and tree-of-thoughts both optimize accuracy and pretend the API invoice does not exist. On open-ended tasks with huge solution spaces, that invoice is the whole story. Synergy-of-Thoughts: Eliciting Efficient Reasoning in Hybrid Language Models borrows dual process theory and runs with it. By default, smaller-scale models generate multiple low-cost intuitive thoughts, the parallel intuitions of System 1. A confidence evaluator cross-evaluates them, and a controllable threshold decides whether they conflict. Only when they do does a scaled-up model get invoked to perform reflective reasoning, override the intuitions and rectify the result. The framework is model-agnostic and training-free, so it drops onto off-the-shelf models. • API cost falls by 38.3%-75.1% across six representative reasoning tasks, while reaching state-of-the-art accuracy and solution diversity. • Token cost reduction reaches up to 69.1% on open-ended tasks, exactly where naive reasoning bleeds most. • Disagreement is a free difficulty signal. Escalate on conflict rather than on guesswork. Expensive reasoning should be an exception path, never the default one. The full token-economics corpus: https://ai-engineer-roadmap.xyz/token-economics

#DualProcess #HybridLLM #APICost #ReasoningEfficiency #LLMRouting

A semantic cache is only as good as its definition of close enough, and most of them use a definition that is embarrassingly crude. MVR-cache replaces the single blunt similarity score with Multi-Vector Retrieval. A learnable segmentation model intelligently splits each prompt, enabling fine-grained similarity comparisons via MaxSim rather than one cosine number standing in for meaning. The training objective is derived from a rigorous theoretical analysis, so optimizing it directly maximizes cache hits under strict correctness constraints. The resulting problem is non-differentiable and combinatorial, which is why it is trained with reinforcement learning using that objective as the reward. • Cache hit rates increase by up to 37% over the state of the art while maintaining the same correctness guarantees. Both halves of that sentence are load-bearing. • Prompt segmentation is learned, not heuristic. Where you cut the prompt decides what is allowed to match. • Retrieval quality is a cost lever in its own right. A better matcher is cheaper than a bigger cache. Hit rate is the exchange rate between engineering effort and your inference bill: https://ai-engineer-roadmap.xyz/token-economics

#SemanticCaching #LLMOps #VectorRetrieval #InferenceCost #MLSys

A cache hit on open-web data is a bet that the world has not changed since you last asked. Most semantic caches never model that bet. FreshCache (Risk-Constrained Freshness-Aware Semantic Caching for Open-Web Retrieval-Augmented LLMs) treats reuse as a risk-constrained temporal inference problem: before approving a hit, it estimates the probability that the cached result is stale, using a fitted exponential decay model with a learned MLP on top, and approves reuse only when that probability sits under a per-tier error budget — answers, URL lists, page content. The result is a cache that ages gracefully instead of flipping between a stale hit and a full pipeline run. The numbers, at a 24-hour evaluation window: → FreshCache_MLP: 97% search API savings at 0.1% hash-based stale error. → Baselines it beats: SemanticTTL at 14.9% stale / 72% saved, and vCache at 7.2% stale / 47% saved. → The temporal risk gate alone accounts for an 11.6 point reduction in stale error over similarity-only reuse. And the subtlest finding: an LLM-judge evaluation showed only 34.3% of detected content changes actually affect answer correctness. Most staleness does not matter. You still have to know which kind you have. More token economics: https://ai-engineer-roadmap.xyz/token-economics

#SemanticCaching #RAG #InferenceCost #CacheFreshness #LLMServing

Semantic caching is deployed by AWS and Microsoft. It is also, by construction, attackable, and the flaw is not one anyone can patch. Treat a semantic cache key as a fuzzy hash and the conflict becomes formal: the locality required to maximize cache hit rates fundamentally conflicts with the cryptographic avalanche effect necessary for collision resistance. Performance and collision resilience are the same dial, pointed in opposite directions. Prior work chased side-channel and privacy risks. This is the first systematic study of the integrity risk, which is what happens when an attacker manufactures a key collision so their entry gets served in place of the real answer. • CacheAttack is an automated framework for black-box collision attacks and achieves a hit rate of 86% in LLM response hijacking. • It induces malicious behavior in LLM agents and preserves strong transferability across different embedding models, so swapping encoders is not a mitigation. • A case study on a financial agent shows the real-world cost of a wrong hit. Your similarity threshold is a security control now, not just a performance knob. Every cache saving is a correctness bet. Read what the bet actually costs: https://ai-engineer-roadmap.xyz/token-economics

#SemanticCaching #LLMSecurity #AdversarialML #AgentSecurity #LLMOps

The similarity threshold in your semantic cache is one number doing two incompatible jobs. Krites states the tradeoff exactly. Production deployments are tiered, a static cache of curated, offline-vetted responses backed by a dynamic cache populated online, and both tiers are commonly governed by a single embedding similarity threshold. Conservative thresholds miss safe reuse opportunities. Aggressive thresholds risk serving semantically incorrect responses to real users. One dial, two failure modes. The move is to get the hard decision off the critical path. On the critical path, Krites behaves exactly like a standard static threshold policy. When the nearest static neighbor of a prompt falls just below the static threshold, an LLM judge is invoked asynchronously to verify whether the static response is acceptable for the new prompt. Approved matches are promoted into the dynamic cache, so future repeats and paraphrases reuse curated answers. • The share of requests served with curated static answers rises by up to 3.9 times on conversational and search-style traffic. • Critical-path latency is unchanged. Verification happens beside the request, never in front of it. • Static coverage expands over time instead of freezing at deploy. More cache economics, quantified: https://ai-engineer-roadmap.xyz/token-economics

#SemanticCaching #LLMServing #LLMAsJudge #InferenceCost #LLMOps

Users almost never repeat themselves. Your pipeline repeats itself constantly. Cache that instead. SemanticALLI, built inside PMG's marketing intelligence platform Alli, targets a hidden inefficiency in agentic pipelines: they frequently reconstruct identical intermediate logic, such as metric normalization or chart scaffolding, even when the user's natural language phrasing is entirely novel. Conventional boundary caching cannot see any of it, because it treats inference as a monolithic black box. So stop treating it as one. Generation is decomposed into Analytic Intent Resolution and Visualization Synthesis, and the structured intermediate representations between those stages are elevated to first-class, cacheable artifacts. • Baseline monolithic caching caps at a 38.7% hit rate, hard-limited by linguistic variance. No embedding model rescues that number. • The Visualization Synthesis stage reaches an 83.10% hit rate, bypassing 4,023 LLM calls with a median latency of just 2.66 ms. • Cache at stable, structured checkpoints inside the pipeline, not at its boundary, because that is where reuse is most reliable. The reuse you are missing sits in the middle of your agent, not at its front door: https://ai-engineer-roadmap.xyz/token-economics

#SemanticCaching #AgenticAI #LLMOps #InferenceCost #AIArchitecture

Your semantic cache has a magic number in it, and that number is wrong for almost every query it sees. Every production semantic cache ships the same design: embed the prompt, find the nearest neighbor, and if cosine similarity clears a fixed threshold, serve the cached response. vCache (Verified Semantic Prompt Caching) points out the obvious thing everyone tolerated — a single static threshold gives no formal correctness guarantee at all. It produces unexpected error rates, and it leaves hits on the table, because the similarity score that means "same question" for a SQL prompt means nothing of the kind for an open-ended chat turn. vCache instead learns an optimal threshold per cached prompt, online, with no additional training. You state the error rate you can live with; the cache holds it. The results: → vCache consistently meets the user-defined error bound, which static thresholds cannot promise. → Up to 12.5x higher cache hit rates than static-threshold and fine-tuned embedding baselines. → Up to 26x lower error rates against those same baselines. Stop tuning the threshold. Make it a per-entry decision with a stated error budget. More token economics: https://ai-engineer-roadmap.xyz/token-economics

#SemanticCaching #InferenceCost #LLMServing #VectorSearch #TokenEconomics

Twenty to thirty percent of production LLM traffic is left uncached — not because it cannot be cached, but because the cache miss costs too much. That is the argument in Category-Aware Semantic Caching for Heterogeneous LLM Workloads, and it reframes the whole problem. A remote vector database search costs about 30ms. At that price, a cache only breaks even at a 15-20% hit rate. Any query category below that line is economically irrational to cache — so the long tail gets excluded by policy, forever. But workloads are not uniform. Code queries cluster densely in embedding space and hit 40-60%. Conversational and volatile categories are sparse and hit 5-15%. One threshold, one TTL, one quota across all of them causes false positives in the dense regions and misses valid paraphrases in the sparse ones. The fix is architectural, not statistical: → Separate in-memory HNSW search from external document storage. Miss cost drops from 30ms to 2ms. → That drops break-even from 15-20% down to 3-5% — the long tail becomes viable. → Vary thresholds, TTLs and quotas per query category, not per system. Cheapen the miss and the whole distribution becomes cacheable. More token economics: https://ai-engineer-roadmap.xyz/token-economics

#SemanticCaching #InferenceCost #HNSW #LLMServing #VectorSearch

Exact-match caches ignore meaning. Semantic caches ignore almost everything else. SISO's critique of the field is blunt: exact-matching and prefix caches neglect query semantics, while state-of-the-art semantic caches remain confined to traditional intuitions and offer little conceptual departure. They inherited LRU-era instincts and bolted an embedding on top. The redesign is three ideas that only work together. Centroid-based caching maximizes coverage with minimal memory instead of storing one entry per query. Locality-aware replacement preserves high-value entries rather than evicting on recency. Dynamic thresholding balances accuracy and latency under varying workloads, instead of freezing at a value someone tuned once on a good day. • SISO delivers up to 1.71 times higher hit ratios than state-of-the-art semantic caching systems, across diverse datasets. • SLO attainment is consistently stronger. Hit ratio alone was never the only thing serving cares about. • Memory is the real constraint, so coverage per byte is the metric that should govern eviction. Your cache policy was designed for web pages. Your traffic is not web pages: https://ai-engineer-roadmap.xyz/token-economics

#SemanticCaching #LLMServing #SLO #CacheEviction #MLSys

One embedding model is one opinion about what counts as the same question. This paper's fix for semantic caching is unglamorous and effective: stop trusting a single encoder. Existing frameworks rely on single embedding models for query representation, which limits their ability to capture the diverse semantic relationships present in real-world query distributions. An ensemble of embedding models, combined through a trained meta-encoder, captures more of them. Evaluation runs on the Quora Question Pairs dataset and tracks both errors that matter in a cache: the paraphrase you failed to reuse, and the different question you wrongly served from cache. • The ensemble reaches a 92% cache hit ratio for semantically equivalent queries. • It still maintains an 85% accuracy in correctly rejecting non-equivalent queries as cache misses. A cache tuned only for hits is a bug generator with a savings dashboard. • Ensemble embedding significantly outperforms single-model approaches at distinguishing similar from dissimilar queries. Token savings and response times follow from that, not the other way around. Every cache decision is a precision-recall tradeoff with a dollar sign attached: https://ai-engineer-roadmap.xyz/token-economics

#SemanticCaching #Embeddings #LLMOps #InferenceCost #MLSys

Over 70% of real user requests to LLMs have a semantically similar counterpart. Almost nobody harvests it. The obvious move, caching the old response and replaying it, leads to a big quality drop. IC-Cache does something sharper with the same similarity signal. It leverages historical request-response pairs from larger models as in-context examples, empowering small LLMs to imitate and even exceed the compositional abilities of their larger counterparts. The cache stops being an answer store and becomes a capability transfer channel, which is what makes selective offloading of requests safe. Offloading is where the money actually is. • For a new request, IC-Cache selects similar, high-utility examples, prepends them to the input, and adaptively routes across LLMs of varying capability, accounting for response quality and serving loads. • A cost-aware cache replay mechanism refines example quality offline to maximize online cache utility. • Evaluated on millions of realistic requests: serving throughput up 1.4-5.9x, latency down 28-71%, without hurting response quality. Reuse the retrieval, not the reply: https://ai-engineer-roadmap.xyz/token-economics

#SemanticCaching #InContextLearning #LLMServing #InferenceCost #MLSys

Agents do not repeat answers. They repeat plans. The plan is the cacheable unit. Agentic Plan Caching argues that existing LLM caching techniques, context caching and semantic caching alike, were primarily designed for serving chatbots and are insufficient for agent applications, where outputs depend on external data and environmental contexts. Cache the response and you have cached a snapshot of a world that already moved. So cache the plan. APC extracts, stores, adapts and reuses structured plan templates from the planning stages of agent applications, pulled out of completed executions at test time. Keyword extraction matches new requests against cached plans, and lightweight models adapt a template into a task-specific plan with the current context. Planning and reasoning are the expensive stages, and they are also the ones that generalize across semantically similar tasks. • Across multiple real-world agent applications, cost drops 50.31% and latency drops 27.28% on average, with performance maintained. • The memory is built at test time from the agent's own execution history, not curated in advance. • It complements existing LLM serving infrastructure rather than replacing it. Half your agent bill is re-deriving a plan you already had: https://ai-engineer-roadmap.xyz/token-economics

#AgenticAI #SemanticCaching #LLMAgents #InferenceCost #LLMOps

About 31% of queries to an LLM service are repeats. That is the size of the refund nobody is claiming. MeanCache claims it from an unusual direction: the user's device. Instead of one central semantic cache, every user gets a local cache, and the query similarity model is trained collaboratively with Federated Learning, so nobody ships their prompts anywhere to make matching better. Costs, service provider load and environmental impact fall together. It also encodes context chains for every cached query, which is how it separates a contextual follow-up from a standalone question. Confusing the two is exactly what gives naive semantic caches unacceptable false hit-and-miss rates. • Roughly 17% higher F-score and a 20% increase in precision on cache hit-and-miss decisions than the state-of-the-art caching method. • Storage requirements drop by 83%, and hit-and-miss decisions run 11% faster. • Privacy and cost are not opposed here. Federated training buys both, on hardware you do not pay for. The cheapest token is the one you already paid for once: https://ai-engineer-roadmap.xyz/token-economics

#SemanticCaching #FederatedLearning #LLMOps #PrivacyPreservingAI #InferenceCost

A customer service chatbot asks the same question all day long, in slightly different words, and pays full price every single time. GPT Semantic Cache (Reducing LLM Costs and Latency via Semantic Embedding Caching) is the least glamorous paper in this space and possibly the most immediately useful. The whole method: embed incoming user queries, keep the embeddings in Redis, look for a semantically similar prior question, and return the pre-generated response instead of calling the model again. No new architecture. No retraining. Redis and an embedding model. What that buys you, measured across query categories: → Up to 68.8% fewer LLM API calls. → Cache hit rates ranging from 61.6% to 68.8%. → Positive hit accuracy above 97%, so the reused answers are not garbage. The economics here are brutal in the good way. Two-thirds of the calls in a repetitive workload are redundant, and the bottleneck for most teams is not the sophistication of their retrieval — it is that they never put a cache in front of the model at all. Exact-match caching is table stakes. Semantic matching is where the repeated-query mass actually lives. More token economics: https://ai-engineer-roadmap.xyz/token-economics

#SemanticCaching #InferenceCost #Redis #LLMOps #TokenEconomics

MXFP4 is not a free lunch, and now there is a benchmark that says so out loud. A systematic investigation of post-training quantization under Microscaling Floating-Point formats spans over 7 PTQ algorithms, 15 evaluation benchmarks and 3 LLM families. It lands on an uncomfortable split. MXFP8 consistently achieves near-lossless performance. MXFP4 still introduces substantial accuracy degradation, and no amount of algorithm shopping rescues it. The interesting part is why. The scaling factor of quantization, not the weights themselves, is the critical error source in MXFP4, and a simple pre-scale optimization strategy significantly mitigates its impact. What that means when you pick a serving precision: • MXFP8 is the safe production default. Treat MXFP4 as a research setting, not a cost lever you ship on a Friday. • PTQ effectiveness depends strongly on format compatibility. An algorithm tuned for integer quantization does not transfer to MXFP for free. • In multimodal LLMs, quantization sensitivity is dominated by the language model rather than the vision encoder. Spend your precision budget there. Low precision is the loudest cost lever in serving, and the most oversold. The whole map, paper by paper: https://ai-engineer-roadmap.xyz/token-economics

#Quantization #LLMServing #MXFP #InferenceOptimization #MLSys

Faster inference is not always better inference. On a phone, it is often the wrong objective entirely. EnerInfer starts from a finding most on-device stacks ignore: LLM inference has exploitable configuration slack. Modestly lowering NPU and memory frequencies preserves quality of experience while substantially improving energy efficiency and reducing heat. Existing systems optimize decoding speed and implicitly assume faster execution is always preferable. That assumption is burning battery. The hard part is that the most energy-efficient NPU/DDR setting varies with the model, the inference engine, the platform and runtime conditions, with no stable ranking across configurations. Commercial devices lack component-level power sensing, so you cannot simply measure your way out. • EnerInfer predicts throughput and power for unseen LLMs across frequency settings instead of profiling every model by hand. • Limited-horizon thermal prediction switches between energy-optimized and thermally constrained inference before the shell gets hot. • Energy efficiency improves by up to 65% on phones, 12% on a laptop and 24% on a development board, with no QoE violation. Edge inference economics are denominated in joules, not dollars. The rest of the cost surface: https://ai-engineer-roadmap.xyz/token-economics

#OnDeviceAI #EdgeInference #EnergyEfficiency #LLMServing #MLSys

The bottleneck in LLM serving is no longer arithmetic. It is how much memory sits next to it. FlashAccel takes the capacity wall literally: model weights and KV cache outgrow High-Bandwidth Memory, so it puts High-Bandwidth Flash inside the GPU. HBF offers higher capacity than HBM at comparable bandwidth, but with punishing access latency, low bandwidth utilization and no support for heterogeneous resource management, which is why nobody had made it work. The co-design is the contribution. Architectural support mitigates access latency. Specialized data layouts for both model weights and KV cache raise bandwidth utilization. An HBF-aware storage management layer plus a programming model organizes persistent data and coordinates heterogeneous memory at the system level. • Integrating six HBF stacks into the GPU delivers an average 2.54 times better throughput per GPU than an HBM-only GPU under a 100ms latency constraint. • Energy efficiency improves 1.93 times on the same setup. Capacity relief pays twice. • Your serving cost curve bends on memory capacity, not on arithmetic throughput. Hardware is quietly rewriting token economics. See the full map: https://ai-engineer-roadmap.xyz/token-economics

#LLMServing #GPUArchitecture #KVCache #MemoryBandwidth #MLSys

First-come-first-served is a scheduling policy the way a coin flip is a strategy. Kairos goes after the ugly truth of disaggregated LLM serving: in production, request lengths follow a long-tail distribution. That produces head-of-line blocking on the prefill side and stragglers that idle the decode side. Systems pairing FCFS prefill with continuous-batching decode cannot adapt to the imbalance, and both SLO attainment and throughput pay for it. Two mechanisms, one on each side of the disaggregation. • Urgency-based priority scheduling predicts prefill completion times and dynamically selects requests to maximize time-to-first-token SLO attainment, lifting it by up to 23.9%. • Slack-guided adaptive batching exploits the gap between per-step decode time and the time-per-output-token SLO to greedily pack short requests. TPOT attainment rises by up to 27.1% and decode throughput by up to 19.3%. • End-to-end SLO attainment climbs by up to 33.8%, on the hardware you already own. Scheduling is the cheapest capacity in the building, because the GPUs are already paid for. More levers like this one: https://ai-engineer-roadmap.xyz/token-economics

#LLMServing #Scheduling #SLO #DisaggregatedInference #MLSys

Two of the strongest inference accelerations known cancel each other out. Nobody warns you before you stack them. Speculative decoding and quantization both attack memory-bound inference: one verifies multiple tokens within a single forward pass, the other compresses weights and activations into lower bit-widths. You expect the gains to compound. Instead, applying EAGLE-2 to quantized models shows the memory benefits of 4-bit weight quantization are diminished by the computational load speculative decoding adds. Verifying a tree-style draft incurs significantly more time overhead than a single-token forward pass once the target model is 4-bit. The fix is structural. A hierarchical framework inserts a small model as an intermediate stage that turns tree-style drafts into sequence drafts, so the quantized target model keeps its memory-access advantage. • On the 4-bit weight Llama-3-70B model on an A100 GPU, the hierarchical design achieves a 2.78 times speedup across tasks. • It outperforms EAGLE-2 by 1.31 times. The naive stack was leaving all of that on the floor. • Optimizations compose in theory. Measure them together before you believe it. More compatibility traps, quantified: https://ai-engineer-roadmap.xyz/token-economics

#SpeculativeDecoding #Quantization #LLMInference #LLMServing #MLSys

Every existing batch speculative decoding implementation produces corrupted output. Not slower output. Wrong output. That is the claim in Batch Speculative Decoding Done Right, and the paper backs it. Speculative decoding is only valid if it yields a distribution identical to standard autoregressive generation. In a batch, sequences accept different numbers of draft tokens — the ragged tensor problem — which desynchronizes position IDs, attention masks and KV-cache state. The failure mode is not a crash. It is repetitive tokens and gibberish, quietly, in production. The authors formalize the synchronization invariants a correct implementation must satisfy, then build one. EQSPEC guarantees output equivalence. EXSPEC reduces the cost with cross-batch scheduling that groups same-length sequences. What matters if you serve models: • Correctness is not free: alignment overhead grows superlinearly and consumes up to 40% of computation. • Done right, the win survives anyway — up to 3x throughput improvement at batch size 8. • If you enabled batch speculative decoding and only watched the tokens-per-second dashboard, go read your outputs. More on where the serving speedups actually come from → https://ai-engineer-roadmap.xyz/token-economics

#SpeculativeDecoding #LLMServing #InferenceOptimization #MLSystems #ModelServing

Pipeline parallelism is the cheap way to serve a large model. Pipeline bubbles are why almost nobody does. TD-Pipe names the culprit: prefill and decode have imbalanced workloads and complex data dependencies, so every phase switch blows bubbles through the pipeline and throughput collapses. Its answer is to disaggregate the two phases in the temporal dimension rather than the spatial one, eliminating the bubbles that phase switching creates. Four pieces make it hold. A hierarchy-controller decouples scheduling from execution. A greedy prefill approach predicts output length and simulates memory usage so it can push more prefills through. Inter-batch work stealing rebalances decode workloads across batches. A spatial-temporal intensity comparison decides the exact moment to switch back from decode to prefill. • Throughput rises by up to 1.91 times over the existing tensor parallel approach and 2.73 times over the existing pipeline parallel approach. • The wins land on GPU nodes with only PCIe interconnection. No exotic fabric, no new purchase order. • If your cluster is commodity, your parallelism strategy should be built for commodity. The rest of the serving cost map: https://ai-engineer-roadmap.xyz/token-economics

#PipelineParallelism #LLMServing #Throughput #DistributedInference #MLSys

A single GPU can serve a Mixture-of-Experts model at throughput most teams assume requires a cluster. MoE-Gen's insight is that the batching strategy, not the hardware, is the binding constraint. Model-based and continuous batching were originally designed for interactive inference; applied to MoE they hand the attention and expert modules excessively small batches, and utilization collapses. Module-based batching fixes it by accumulating tokens in host memory and dynamically launching large batches on the GPU, module by module. Batch sizes are then chosen per module so GPU computation and communication fully overlap. • MoE-Gen achieves 8-31x higher throughput than state-of-the-art systems using model-based batching, namely FlexGen, MoE-Lightning and DeepSpeed, and even greater improvements over continuous batching systems such as vLLM and Ollama. • Evaluation runs on the MoE models people actually deploy, DeepSeek and Mixtral, across offline inference tasks. That is exactly where batch economics decide the bill. • Offline and interactive workloads want opposite systems. Stop serving both with one. Batching is a cost decision disguised as a config flag. Where the money really goes: https://ai-engineer-roadmap.xyz/token-economics

#MixtureOfExperts #LLMServing #Batching #Throughput #MLSys

The cheapest GPU in your inference budget is the one somebody else already bought and is not using. DeServe builds an offline LLM serving system on decentralized, idle GPU capacity. The premise is economic before it is technical: open-source models have become strong contenders, but deploying them is constrained by the high costs and limited availability of GPU resources. The spare capacity exists. It is just scattered across high-latency networks, which is precisely the condition conventional serving systems fall apart in. That network environment is the entire engineering problem, and DeServe optimizes serving throughput specifically for it. • DeServe achieves a 6.7x-12.6x throughput improvement over existing serving system baselines in high-latency network environments. • The workload is offline inference, where latency tolerance is the resource you trade away for price. • If a job can wait, it should not be running on on-demand hardware at on-demand prices. Decentralization here is a pricing strategy wearing an architecture costume, and it only works because a batch job never had to answer in real time. More on that cost surface: https://ai-engineer-roadmap.xyz/token-economics

#DecentralizedAI #LLMServing #GPUUtilization #OfflineInference #MLSys

Throughput and latency are supposed to be a tradeoff. Sarathi-Serve shows most of that tradeoff was a scheduling bug. The diagnosis: prefill saturates GPU compute but has high latency; decode has low latency but terrible compute utilization, because it processes a single token per request. Batching helps decodes enormously. But batching multiple requests interleaves prefills into the decode stream, and every new prefill stalls the decodes already running. That stall is where your tail latency goes to die. Sarathi-Serve (Taming Throughput-Latency Tradeoff in LLM Inference) fixes it with chunked-prefills — splitting a prefill into near-equal chunks — plus stall-free scheduling that admits new requests without pausing ongoing decodes. Uniform batches fall out of it, which also flattens pipeline bubbles. What this buys you, versus vLLM: • 2.6x higher serving capacity for Mistral-7B on one A100. • Up to 5.6x for Falcon-180B — the bigger and more pipelined your deployment, the more the stalls were costing you. • Same weights, same hardware, same accuracy. The gain is entirely in when work is scheduled. The rest of the serving levers → https://ai-engineer-roadmap.xyz/token-economics

#LLMServing #ContinuousBatching #vLLM #InferenceOptimization #MLSystems

Your offline batch job is being served by an engine designed for a chatbot, and it shows. BatchLLM starts from a property of real industrial workloads: large batched and offline tasks show prefix sharing, where different prompt inputs partially share a common prefix. Existing engines lean on an LRU-based cache, so KV context that is about to be reused gets prematurely evicted by implicit cache management. Worse, streaming-oriented systems never leverage the request-batch information, so they cannot mix decoding tokens with prefill chunks well enough to saturate the GPU. BatchLLM identifies common prefixes globally, schedules requests sharing a prefix together, reorders so requests with a larger ratio of decoding go first, and applies memory-centric token batching to enlarge token-batch sizes. • It outperforms vLLM and SGLang by 1.3 to 10.8 times across microbenchmarks and a typical industry workload. • The gains hold under different hardware environments. This is a scheduling win, not a kernel win. • Prefix sharing is free money once the scheduler can see the whole batch up front. More offline inference economics: https://ai-engineer-roadmap.xyz/token-economics

#LLMServing #PrefixCaching #BatchInference #vLLM #MLSys

The right quantization format for your model depends on something most teams never consider: whether you batch. "Give Me BF16 or Give Me Death"? ran over 500,000 evaluations across the entire Llama-3.1 family, covering FP8, INT8 and INT4 on academic benchmarks and real-world tasks. It is the least hand-wavy quantization study out there, and it kills two myths at once. Myth one: quantization costs you real accuracy. FP8 (W8A8-FP) is effectively lossless across all model scales. Well-tuned INT8 gives up only 1 to 3% accuracy. INT4 weight-only rivals 8-bit. Myth two: there is a single best format. There is not — it depends on your serving regime. The deployment guidance, straight from the paper: • W4A16 is the most cost-efficient for synchronous setups, where you are memory-bandwidth bound and weight size dominates. • W8A8 dominates under asynchronous continuous batching, where compute is the constraint. • Pick your format after you know your traffic shape, not before. How quantization changes the self-hosting break-even → https://ai-engineer-roadmap.xyz/token-economics

#Quantization #LLMServing #FP8 #InferenceOptimization #ContinuousBatching

2-bit weights sound like a rounding error waiting to happen. Vector quantization says otherwise. VPTQ pushes weight-only quantization into the extreme low-bit regime where traditional scalar-based quantization simply runs out of numerical representation. It compresses vectors into indices using lookup tables, formulates the problem with Second-Order Optimization, and refines weights through Channel-Independent Second-Order Optimization for granular VQ. Residual and outlier quantization extend it further, and a brief codebook initialization algorithm keeps the whole thing tractable. The measurements carry the argument. • At 2-bit, VPTQ reduces quantization perplexity by 0.01-0.34 on LLaMA-2 and 4.41-7.34 on LLaMA-3 over prior state of the art, with average accuracy up 0.79-1.5% on LLaMA-2 and 11-22% on LLaMA-3 on QA tasks. • Inference throughput increases 1.6-1.8 times versus SOTA, because fewer bits per weight means less memory bandwidth burned per token. • It uses only 10.4-18.6% of the quantization algorithm execution time. The compression step itself is cheap. Memory bandwidth is the tax. Quantization is the deduction. The full map: https://ai-engineer-roadmap.xyz/token-economics

#Quantization #VectorQuantization #LLMInference #LowBit #MLSys

Your system prompt is not free. It is the most expensive tensor in your serving stack. Hydragen (High-Throughput LLM Inference with Shared Prefixes) makes the point brutally. In a batch where every sequence shares a prefix — a system prompt, few-shot examples, a retrieved document — attention re-reads that identical KV cache once per sequence, per token. Pure waste, and it scales with your batch size. Hydragen decomposes attention into the shared prefix and the unique suffixes, batching the prefix queries together so the GPU runs matrix multiplications instead of memory-bound matrix-vector products. Same math, exact attention, different hardware story. The result: up to 32x end-to-end throughput on CodeLlama-13b against competitive baselines. Three things to take away: • Growing the shared prefix from 1K to 16K tokens costs Hydragen under 15% throughput. Baselines drop over 90%. Long shared context stops being a tax. • Speedup grows with batch size and prefix length — so the workloads that hurt most benefit most. • If you self-host, prefix sharing is not a micro-optimization. It is a different cost curve. The full serving-economics breakdown → https://ai-engineer-roadmap.xyz/token-economics

#LLMServing #KVCache #InferenceOptimization #GPUEfficiency #SelfHosting

One LLM request is two completely different workloads, and running both on the same GPU wastes the GPU. Splitwise (Efficient generative LLM inference using phase splitting) characterizes it precisely. Prompt computation is compute-intensive and saturates the newest accelerators. Token generation is memory-intensive and leaves that compute idle — it does not need the latest GPU at all, and runs happily at lower power and lower cost. So Splitwise splits the phases onto separate machines, provisions each pool with hardware suited to its phase, and moves the KV state across the fast back-plane interconnect already sitting in your cluster. The numbers are the argument: • 1.4x higher throughput at 20% lower cost, versus keeping both phases on the same machine. • Or, spend the same money and power, and get 2.35x more throughput. Same model, same accuracy, different placement. • Disaggregation is a procurement decision as much as an engineering one — your decode fleet does not need your prefill fleet's silicon. If you own the GPUs, layout is cost-per-token. Read the full serving break-even → https://ai-engineer-roadmap.xyz/token-economics

#LLMServing #Disaggregation #InferenceOptimization #GPUInfrastructure #MLSystems

Everyone repeats that draft-then-verify buys you maybe 1.4 to 2 times. That impression came from measuring the wrong drafter. The original Speculative Decoding paper, SpecDec, was the first to formally study exploiting speculative execution to accelerate autoregressive decoding, and it changed two things nobody else had. Spec-Drafter is an independent model optimized specifically for efficient and accurate drafting, rather than a generic small checkpoint pressed into service. Spec-Verification is a reliable method for verifying drafted tokens efficiently inside the decoding paradigm. Build the drafter for the job and the ceiling moves. • Across seq2seq tasks including machine translation and abstractive summarization, SpecDec reaches around 5 times speedup for popular Transformer architectures, with generation quality comparable to beam search decoding. • That refreshes the impression that the draft-then-verify paradigm introduces only 1.4 to 2 times speedup. • The drafter is a design decision, not a leftover checkpoint. Most speculative stacks still treat it as one. Every speedup you depend on has a lineage worth reading. Start here: https://ai-engineer-roadmap.xyz/token-economics

#SpeculativeDecoding #LLMInference #DecodingSpeedup #Transformers #MLSys

Weights are easy to quantize. Activations are not. That asymmetry is the entire game. SmoothQuant exploits it with a mathematically equivalent transformation that offline-migrates quantization difficulty from activations to weights, smoothing activation outliers until 8-bit weight, 8-bit activation quantization actually holds up. Training-free, accuracy-preserving, general-purpose, and it applies to every matrix multiplication in the model rather than the convenient subset. That coverage is why it stuck. W8A8 works across OPT, BLOOM, GLM, MT-NLG, Llama, Falcon, Mistral and Mixtral, with INT8 quantization of both weights and activations for all the matmuls. • Up to 1.56x speedup and 2x memory reduction, with negligible loss in accuracy. • It enables serving a 530B LLM within a single node. That is a hardware bill deleted, not a benchmark trophy. • The lesson generalizes past quantization: when one tensor class resists compression, migrate the difficulty to the class that does not. Turn-key wins that genuinely reduce hardware cost are rare, and this one aged well. The rest of the map: https://ai-engineer-roadmap.xyz/token-economics

#Quantization #SmoothQuant #INT8 #LLMServing #MLSys

Two agent pipelines can hit exactly the same accuracy and differ 13 to 32 times in cost. That is the headline finding of AgentOpt v0.1, a framework for client-side agent optimization. In a multi-step pipeline, the highest-impact lever is which model gets assigned to which role, and at matched accuracy the gap between the best and worst assignment reaches 13-32x. Almost nobody tunes it, because brute-forcing the combination space is expensive on its own. AgentOpt makes that search cheap. Its UCB-E bandit recovers near-optimal accuracy while cutting the evaluation budget by 62-76% relative to brute-force search. Takeaways: • Model choice is not one decision. It is one decision per pipeline stage, and your planner does not need what your extractor needs. • Treat assignment as a search problem over a small evaluation set, not as a vibe. • Budget your evaluations too. Brute force was spending 62-76% more than it needed to. Server-side tricks — caching, speculative execution, load balancing — cannot save you from putting a frontier model in a role a cheap one handles fine. That decision lives on the client. The full token-economics corpus, every number traced to its paper: https://ai-engineer-roadmap.xyz/token-economics

#LLMOps #AIAgents #InferenceCost #ModelSelection #TokenEconomics

Every agent in your swarm is paying to store the same context, separately. TokenDance names the pathology. Multi-agent systems run in synchronized rounds: a central scheduler gathers every agent's output and redistributes the combined context. That All-Gather pattern means each agent's prompt contains the same shared blocks, and each agent keeps its own KV cache of them. Existing reuse methods fail to exploit it. TokenDance pays the reuse cost once per round instead of once per agent, and encodes sibling caches as block-sparse diffs against a single master copy. Compression on representative workloads: 11 to 17 times. Takeaways: • Per-agent KV cache storage drops by up to 17.5 times. That is memory you were buying over and over. • Under the same SLO, it supports up to 2.7 times more concurrent agents than vLLM with prefix caching. • Prefill runs up to 1.9 times faster than per-request position-independent caching. Concurrency ceilings in agent systems get blamed on the model. Often they are just redundant cache, replicated once per participant, on a workload whose whole design guarantees the redundancy. More like this: https://ai-engineer-roadmap.xyz/token-economics

#KVCache #LLMServing #MultiAgent #InferenceOptimization #vLLM

Over 60% of the content in your agent's skill files is doing nothing. SkillReducer studied 55,315 publicly available agent skills and found systemic waste: 26.4% carry no routing description at all, over 60% of body content is non-actionable, and reference files can inject tens of thousands of tokens per invocation. Every one of those tokens costs money and dilutes attention. The fix is not a bigger context window. It is compression plus progressive disclosure: separate the actionable core rules from supplementary content, and load the rest only on demand. Takeaways: • 48% description compression and 39% body compression, evaluated on 600 skills and the SkillsBench benchmark. • Functional quality did not merely hold. It improved by 2.8%. • The gains transfer across five models from four families, with a mean retention of 0.965. Call it the less-is-more effect: removing non-essential content reduces distraction in the context window. Your agent is not underperforming because it lacks instructions. It is underperforming because it has too many, and it has to read all of them before it does anything. More: https://ai-engineer-roadmap.xyz/token-economics

#PromptEngineering #AIAgents #ContextEngineering #TokenEconomics #CodingAgents

A stateless agent re-reads its entire history every single iteration. That is O(n) tokens per step and O(n^2) across the run. You are paying quadratically for a linear task. "Remember, Don't Re-read: Stateful ReAct Agents for Token-Efficient Autonomous Experimentation" makes the fix concrete. The authors rebuilt the autoresearch loop as a stateful ReAct agent in LangGraph, where typed persistent state carries experimental history across iterations through a tool-calling interface instead of being reconstructed from scratch each turn. The result on hyperparameter tuning: 90% fewer tokens. On code performance optimization, where each observation carries full source and benchmark output: 52% fewer tokens. Optimization quality stayed comparable on both. What matters here: → The saving is structural, not a prompt trick. The stateful agent works in a fixed-size conversation window at O(1) cost per iteration. → The longer the horizon, the bigger the gap. Quadratic growth is patient and it always wins. → If your agent's context is a transcript, you have a state management bug wearing a token bill as a disguise. More research on where agent tokens actually go: https://ai-engineer-roadmap.xyz/token-economics

#AIAgents #ReAct #LangGraph #TokenCost #ContextEngineering #AgentEconomics

Your coding agent spends 1000x more tokens than your chat window, and it has no idea it is doing it. "How Do AI Agents Spend Your Money?" is the first systematic study of token consumption in agentic coding tasks. The authors traced frontier LLMs across SWE-bench Verified and found agentic tasks consume 1000x more tokens than code reasoning and code chat, with input tokens, not output tokens, driving the bill. Worse, the spend is stochastic: two runs of the same task can differ by up to 30x in total tokens. And the models cannot see it coming. Asked to predict their own token cost before execution, frontier models managed correlations of only up to 0.39 with reality, and systematically underestimated. Three things to take from this: → Budget for the input side. Re-sent history, not generated text, is what you are paying for. → Do not let an agent self-report its budget. It underestimates, structurally. → More tokens is not more accuracy. The paper finds accuracy peaks at intermediate cost and saturates above it. The economics of agents are not the economics of chat. Full breakdown of the token-economics research here: https://ai-engineer-roadmap.xyz/token-economics

#AIAgents #TokenCost #LLMOps #SWEbench #AgentEconomics #InferenceCost

Throwing away most of an agent's context made it more accurate, not less. Reliability and cost moved in the same direction, which is not supposed to happen. "Less Context, Better Agents" ran GPT-5 on a 50-task enterprise expense-itemization benchmark over Model Context Protocol tools. Full conversation history reached 71.0% complete itemization and burned 1,480,996 tokens. Pruning the context to the last 5 tool call/response pairs and adding automated summarization reached 91.6% complete itemization using 553,374 tokens. Fewer tokens. Higher completion. Same model. The mechanism is attention dilution: verbose tool responses from enterprise systems cause context overflow and stale-state errors, and the agent starts acting on facts that are no longer true. Keeping the recent window plus a compact summary keeps it grounded. Takeaways: → Full-history retention is not the safe default. It is a failure mode with a receipt. → Recency plus summary beats append-everything on both axes for long-horizon tool use. → Stale state is a correctness bug before it is a cost bug. The rest of the agentic token-economics literature: https://ai-engineer-roadmap.xyz/token-economics

#AIAgents #ContextEngineering #TokenCost #MCP #ToolUse #LLMOps

The expensive part of agentic software engineering is not writing the code. Tokenomics traced 30 software development tasks through the ChatDev framework running a GPT-5 reasoning model, mapped every internal phase onto a lifecycle stage — Design, Coding, Code Completion, Code Review, Testing, Documentation — and then simply counted tokens. The iterative Code Review stage alone accounts for 59.4% of all token consumption. And input tokens, not output tokens, make up 53.9% of what you are billed for. Takeaways: • The primary cost of agentic engineering sits in automated refinement and verification, not in initial code generation. • When input is 53.9% of consumption, context hygiene beats output truncation every time. • Review loops need a stopping rule. Iteration is a budget line, not a free virtue. Most teams optimize the generation step because it is the step they can see in the demo. The bill is being run up afterwards, in the loop where agents read the code back to each other and ask for one more pass. Predict your expenses before you scale the workflow: https://ai-engineer-roadmap.xyz/token-economics

#SoftwareEngineering #AIAgents #LLMOps #TokenEconomics #ChatDev

Multi-agent orchestration has a broadcast problem that computer architects solved for shared-memory multiprocessors long ago. Token Coherence makes the mapping precise. When every agent rebroadcasts full state each round, synchronization cost explodes in agents, steps and artifact size at once. The paper argues this is a structural residue of full-state rebroadcast, not an inherent property of coordination, and that MESI-protocol invalidation transfers to artifact synchronization with minimal modification. The resulting Artifact Coherence System replaces rebroadcast with lazy invalidation, with a TLA+ verified protocol enforcing single-writer safety, monotonic versioning and bounded staleness. Takeaways: • Simulated token savings reach 95.0% when few agents actually read the shared artifact. • Even when most of them do read it, roughly 81% savings persist. The predicted collapse threshold never arrives. • It integrates with LangGraph, CrewAI and AutoGen through thin adapter layers, so this is a protocol change, not a rewrite. Agents do not need to be told everything that happened. They need to be told what changed, and only when they are about to care. More on what tokens actually cost: https://ai-engineer-roadmap.xyz/token-economics

#MultiAgent #DistributedSystems #LangGraph #AIAgents #TokenEconomics

Show an agent every tool you own and it succeeds 32.1% of the time. ToolMenuBench varies tool-menu size, distractor type, state-dependent task structure and risk exposure, then measures what the visible menu does to a multi-step agent across seven model backends. Under all-tools exposure, task success is 32.1%. With causal minimal tool filtering, it is 85.7%. And it is cheaper. Average token usage falls by roughly 98%. Takeaways: • A tool menu is not documentation. It is prompt content, re-paid on every single step of every single run. • Filtering does more than save money: wrong-tool calls, premature actions and risky-tool exposure all drop relative to unfiltered menus, lexical filtering and state-aware filtering. • Reliability and cost turn out to be the same knob here, which almost never happens and should be exploited when it does. If your agent is flaky, look hard at the menu before you reach for a bigger model. The question is not whether it can call a tool correctly. It is which tools should be visible, and when. More: https://ai-engineer-roadmap.xyz/token-economics

#ToolUse #AIAgents #FunctionCalling #AgentReliability #TokenEconomics

A pipeline with 14 agents issues 14 LLM calls per run. It does not have to. Agent Capsules treats multi-agent execution as an optimization problem with an empirical quality constraint. Merging agents into fewer calls saves tokens, but naive merging silently degrades quality through tool loss and prompt compression. So the runtime scores composition opportunities, then gates every mode switch on rolling-mean output quality, escalating back toward per-agent dispatch the moment the floor is threatened. Against a hand-crafted LangGraph implementation of a 14-agent competitive intelligence pipeline, it uses 51% fewer fine-mode input tokens, at higher quality rather than lower. Takeaways: • Against a DSPy due diligence pipeline it uses 68% fewer tokens than MIPROv2, with quality up by 0.052. • A controlled negative result earns its place here: injecting more context into a merged call worsens compression rather than relieving it. • Savings arrive before merging even fires, through automatic policy resolution, cache-aligned prompts and topology-aware context injection. Naive call merging is a quality regression waiting to ship. Gated call merging is an optimization. The difference is whether anything is watching. More: https://ai-engineer-roadmap.xyz/token-economics

#MultiAgent #DSPy #LangGraph #LLMOps #InferenceCost

Multi-agent debate can lose 94.5% of its token cost and barely notice. That number is not a compression trick. It is a measurement of how much of the conversation was never load-bearing. Multi-agent debate works: add agents, add debate rounds, reasoning improves. The problem is that every agent reads every other agent's output, so token cost scales with the square of the conversation while accuracy creeps up a few points. Scalability dies there. "S2-MAD: Breaking the Token Barrier to Enhance Multi-Agent Debate Efficiency" attacks it with sparsification: suppress the ineffective information exchanges and the unproductive discussions rather than letting every agent broadcast to every other. Across multiple datasets and models, the approach cuts token cost by up to 94.5% versus standard MAD, with performance degradation held below 2.0%. What this should change: → Debate topology is a cost decision, not just an accuracy decision. All-to-all is the expensive default nobody chose deliberately. → Most inter-agent messages carry no decision-relevant information. Measure before you scale the agent count. → If a technique survives losing most of its tokens, most of those tokens were never doing work. More on agentic token economics: https://ai-engineer-roadmap.xyz/token-economics

#MultiAgent #AIAgents #TokenCost #LLMReasoning #AgentEconomics #InferenceCost

Most of what a coding agent re-sends on every turn is waste, and you are paying for all of it. AgentDiet analyzed real agent trajectories and found three kinds of dead weight everywhere: useless content, redundant content, and expired content — facts that were true many steps ago and are now actively misleading. Multi-turn agents keep resending it, and input tokens are the dominant cost of an ever-growing trajectory. So prune at inference time. AgentDiet removes the waste during agent execution, implemented on a top-performing coding agent and evaluated on two LLMs and two benchmarks. Takeaways: • Input tokens drop by 39.9% to 59.7%. • Total computational cost drops by 21.1% to 35.9%, while maintaining the same agent performance. • No retraining and no new model. This is a runtime change to what you choose to resend. Long-horizon agents need the ability to forget far more than they need a longer context window. Efficiency has been largely overlooked in agent products, which is another way of saying it is unclaimed margin. More: https://ai-engineer-roadmap.xyz/token-economics

#CodingAgents #LLMOps #ContextEngineering #InferenceCost #TokenEconomics

Same benchmark results. Same multi-agent quality. $5.6 of tokens instead of $43.7. That is the headline finding of "Cut the Crap: An Economical Communication Pipeline for LLM-based Multi-Agent Systems". The authors were the first to formally define communication redundancy in LLM multi-agent pipelines: the inter-agent messages that get passed around, paid for, and never influence an outcome. Their method, AgentPrune, does one-shot pruning on the spatial-temporal message-passing graph, cutting redundant and even malicious messages before they are ever sent. Across six benchmarks it matches state-of-the-art communication topologies at $5.6 cost against their $43.7, and drops into existing multi-agent frameworks with a 28.1% to 72.8% token reduction. The bonus result is the interesting one: pruning the graph also defended against two types of agent-based adversarial attack, with a 3.5% to 10.8% performance boost under attack. Worth internalizing: → Redundant agent chatter is not free verbosity. It is an attack surface you are paying to keep open. → Communication topology, not model choice, is often the dominant cost term in a multi-agent system. → Prune the graph before you upgrade the model. More like this: https://ai-engineer-roadmap.xyz/token-economics

#MultiAgent #AIAgents #TokenCost #AgentEconomics #LLMOps #PromptEngineering

A self-hosted legal model just beat five frontier LLMs at contract extraction, for a fraction of the price. A Few Good Clauses pits Olava Extract, a domain-trained legal Mixture of Experts model, against five frontier models on structured contract extraction. Olava takes the strongest aggregate performance in the study: a macro F1 of 0.812 and a micro F1 of 0.842, while reducing inference cost by 78% to 97% compared with the frontier models tested. It also posts the highest precision, producing fewer hallucinated and unsupported extractions. In legal review that is not a nicety, it is the entire job, because every bad extraction becomes downstream human work. Takeaways: • On a narrow, structured task, domain training beats scale. The frontier model was never solving your problem, it was solving everyone's. • A cost reduction of 78% to 97% changes what is even worth automating. • Self-hosting takes the vendor out of the data path, which for contracts is often the actual blocker. The uncomfortable implication: valuable enterprise AI does not have to stay tied to ever larger models, massive infrastructure expenditure and centrally hosted providers. More: https://ai-engineer-roadmap.xyz/token-economics

#SmallLanguageModels #LegalTech #InformationExtraction #SelfHosted #InferenceCost

Running an LLM to judge your LLM means paying a frontier bill twice on every request. Luna-2 replaces LLM-as-a-judge with a small language model that returns a single token. Each metric — toxicity, hallucination, tool selection quality — is a lightweight LoRA head on a shared SLM backbone, so hundreds of specialized metrics run concurrently on one GPU, deterministically, deployed next to the system they guard. Accuracy lands at par with or higher than frontier LLM evaluators. Inference cost drops by over 80x and latency by over 20x. Takeaways: • Guardrails are the highest-volume inference in an AI product. They fire on every request, which makes them the worst possible place to park your most expensive model. • In production, Luna-2 protects over 100M AI sessions and processes over 100B tokens per month, with eval cost savings of over $30M annually. • Single-token evaluation is deterministic, which removes the operational non-determinism that multi-token generation quietly introduces. The judge does not need to be smarter than the thing it judges. It needs to be right about one narrow question, cheaply, forever. More: https://ai-engineer-roadmap.xyz/token-economics

#Guardrails #LLMEvaluation #SmallLanguageModels #LoRA #AISafety

219 training examples were enough to stop paying a frontier API. This work fine-tunes LLaMA 3.1 8B with LoRA, touching 2.05% of parameters, on only 219 curated examples, then couples it with a deterministic rule-based postprocessing layer for multi-label compliance evaluation of conversational transcripts across 18 heterogeneous output fields. Blind evaluation on 53 unseen production transcripts: 100% JSON structural validity, 83.0% human-validated overall accuracy, and 100% accuracy on the most critical classification field. The cost line is the part to read twice. USD 0.013 per evaluation, against USD 0.025 to 0.055 for the proprietary alternatives — a 46% to 76% saving — with inference completing in about 2 seconds on a single A100. Takeaways: • The data wall is lower than the industry pretends. Hundreds of curated examples, not hundreds of thousands. • Neural plus symbolic beats neural alone for structured output. The rules layer is what gets you to 100% valid JSON. • Privacy arrives free with the architecture, because the transcripts never leave your hardware. Frontier accuracy on a narrow task is not a frontier-model problem. More: https://ai-engineer-roadmap.xyz/token-economics

#LoRA #FineTuning #SmallLanguageModels #StructuredOutput #DataPrivacy

A usable model for a low-resource language now costs under $50 to build. Kakugo takes a language name as its only input. A large teacher model generates synthetic prompts and translates instruction datasets, and that data trains a general-purpose small language model. The pipeline produced training data and SLMs for 54 low-resource languages, with evaluations across translation, classification and question answering consistently improving on the base models. Total generation and training cost: under $50 per language. Takeaways: • Distillation is not only a cost play for rich tasks. It is an access play for the languages frontier labs will never prioritize. • The teacher is a one-time expense. Once the student exists, the expensive model leaves your inference path entirely. • A community can own its model instead of renting a token endpoint that handles its language badly and charges more for the privilege. Under $50 is a weekend, not a budget cycle. Any team sitting on a narrow domain and complaining about API bills should read the method and ask why they are still paying per token to a model nobody trained on their problem. More: https://ai-engineer-roadmap.xyz/token-economics

#SmallLanguageModels #LowResourceNLP #KnowledgeDistillation #SyntheticData #MultilingualNLP

Using an LLM as a one-time teacher rather than an inference engine produced a 261x cost reduction. Multi-Model Synthetic Training turned 3.2 billion AIS vessel-tracking records into 21,543 synthetic question and answer pairs, generated jointly by GPT-4o and o3-mini to prevent overfitting and keep the reasoning accurate. Those pairs fine-tuned a Qwen2.5-7B student. The student reaches 75% accuracy on maritime tasks, while the expensive model appears exactly once — at training time — and never again. Takeaways: • A 261x reduction is not a tuning gain, it is a change of category. Teachers are a capital expense; inference engines are an operating expense you renew every request. • Domains without annotated data are not blocked. Synthetic generation from raw records substitutes for manual annotation where annotation is infeasible. • Multi-model generation matters. Two teachers produce data that a single teacher's quirks would have quietly poisoned. If you are calling a frontier model repeatedly on a narrow, repetitive task, you are renting per query what you could have bought once, and the rent never stops. More: https://ai-engineer-roadmap.xyz/token-economics

#KnowledgeDistillation #SyntheticData #SmallLanguageModels #FineTuning #MLOps

Small models are not a compromise. Increasingly they are the answer. A survey of about 160 papers maps the small language models in the 1 to 8 billion parameter range and reaches a blunt conclusion: smaller models can perform as well as, or even outperform, large ones. Not on everything. On plenty of the tasks that people actually deploy. The survey separates task-agnostic general-purpose SLMs from task-specific SLMs, catalogues the techniques used to build them while balancing performance, efficiency, scalability and cost, and defines effective size — the capability an SLM delivers relative to a much larger model. Takeaways: • Scale was always a proxy for capability, never the thing itself. Effective size is the honest metric. • The 1 to 8 billion band is where deployment economics live: one GPU, no queue, no per-token meter running while your users think. • Is massive scale the only path forward? The literature has already answered no, at length, with receipts. Pick the smallest model that clears your bar, then spend the savings on the evaluation that proves it cleared it. More: https://ai-engineer-roadmap.xyz/token-economics

#SmallLanguageModels #ModelSelection #EdgeAI #InferenceCost #MLResearch

LinkedIn serves millions of semantic search requests per second, and it does not use a large model to do it. Their report on scaling small language models for semantic job search is refreshingly unglamorous. Pruning reduced model size by up to 40% while maintaining accuracy. Context compression cut input context length by up to 10 times with minimal accuracy loss. Serving-infrastructure work on GPUs did the rest. Taken together, throughput in a real-world deployment rose by 10 times, while meeting the quality bar they set in advance. Takeaways: • Two levers, not one. The model got smaller and the input got shorter, and context compression is the underrated half of that pair. • Set the quality bar first, then optimize until you touch it. Prune to 40%, not to zero. • LLM quality on relevance ranking was never the blocker. Latency and throughput were, and those are engineering problems. Prohibitively expensive is not a property of the model. It is a description of the serving stack you have not optimized yet, and someone will optimize it eventually. More: https://ai-engineer-roadmap.xyz/token-economics

#SmallLanguageModels #SemanticSearch #Pruning #LLMServing #MLOps

Sometimes the honest benchmark says the small model loses. Publish it anyway. A cost-versus-accuracy study evaluated 15 commonly used small language models, ranging from 1 billion to 70 billion parameters, on detecting phishing websites from raw HTML alone. The finding is not a triumph: the SLMs underperform state-of-the-art proprietary LLMs on classification accuracy. And yet they remain a viable, scalable alternative, because the comparison was never accuracy in isolation. It is accuracy weighed against operational cost, infrastructure, and who ends up holding the data. Takeaways: • Local deployment gives an organisation control over data and operations. For security telemetry, that control can outweigh a few points of accuracy. • The relevant baseline is not only the frontier API. Traditional machine learning here demanded extensive feature engineering, continuous retraining and costly maintenance. • The trade-off is the finding. Anyone telling you small models always win is selling you something. Research that publishes its losses is exactly what makes the wins elsewhere credible. This is a benchmark you can plan a budget against. More: https://ai-engineer-roadmap.xyz/token-economics

#SmallLanguageModels #CyberSecurity #PhishingDetection #SelfHosted #CostAwareAI

A 125M model can be pushed to the MMLU performance of a 600M model using 200 times fewer tokens. Adapt-Pruner gets there by refusing to treat pruning and training as separate phases. Prune a small portion of neurons, around 5% at a time, train, repeat. Layer-wise adaptive pruning, applied incrementally rather than in one destructive pass — and adaptive pruning with further training yields models comparable to pre-training from scratch. On LLaMA-3.1-8B it outperforms LLM-Pruner, FLAP and SliceGPT by an average of 1% to 7% accuracy on commonsense benchmarks. It also discovers a new 1B model that surpasses LLaMA-3.2-1B on multiple benchmarks. Takeaways: • Pruning is normally a lossy compromise. Interleaved with training, it becomes a genuine alternative to pre-training. • Compute, not capability, is what most teams are actually short of. 200 times fewer tokens is a training budget you already have. • Small models are not only trained from scratch. They can be grown out of the large ones you already downloaded. More: https://ai-engineer-roadmap.xyz/token-economics

#Pruning #SmallLanguageModels #ModelCompression #EdgeAI #TrainingEfficiency

130X faster and 25X less expensive, on a task you are probably still paying a frontier model to do. Performance-Guided Knowledge Distillation distils an LLM into a smaller task-specific model, but the interesting part is the loop. The teacher and student run an active learning routine: the LLM keeps generating new training data, steered by hard-negative mining, the student's validation performance, and early-stopping protocols. That targets the real failure modes of industrial text classification — many classes, sparse annotations — and it outperforms BERT-base and other distillation methods on several multi-class datasets. Takeaways: • Cost and latency benchmarking put the distilled models at up to 130X faster and 25X less expensive than the LLM for inference on the same classification task. • Distillation is not a one-shot dump of teacher outputs. Generation should be steered by where the student is currently failing. • Classification is the most over-served workload in production AI. Almost nobody needs a frontier model to route a support ticket. The framework extends beyond classification, language generation included, which makes the loop the reusable idea here. More: https://ai-engineer-roadmap.xyz/token-economics

#KnowledgeDistillation #TextClassification #MLOps #ActiveLearning #InferenceCost

70 open-source small language models, benchmarked on the hardware they will actually run on. Small Language Models: Survey, Measurements, and Insights covers transformer-based, decoder-only models in the 100M to 5B parameter range, and then does what most surveys skip. It measures. Architectures, training datasets and training algorithms on one axis. Commonsense reasoning, mathematics, in-context learning and long context on another. Then real inference latency and memory footprint on device. That last part is the whole point. An SLM's cost is not a price per token. It is a latency number and a memory number on hardware you already own. Takeaways: • SLMs get far less academic attention than LLMs despite being the models sitting in everyone's pocket. The measurement gap is the research gap. • The 100M to 5B band is not one thing. Latency and memory vary enormously across it, and this paper hands you the curve. • When the model runs on the device, the marginal cost of a request collapses. That reframes the entire build-versus-buy conversation. More: https://ai-engineer-roadmap.xyz/token-economics

#SmallLanguageModels #EdgeAI #OnDeviceAI #Benchmarking #Latency

The cheapest inference is the inference that never leaves the phone. SlimLM is a family of small language models tuned for document assistance — summarization, question answering, suggestions — and benchmarked where it matters: on a Samsung Galaxy S24. The paper maps the trade-off surface between model size, from 125M to 7B parameters, context length, and inference time. The smallest variant performs efficiently on the S24, while larger variants trade latency for capability within mobile constraints. Pre-trained on SlimPajama-627B, fine-tuned on a purpose-built DocAssist dataset, and shipped with an Android application so the deployment claims are actually checkable. Takeaways: • On-device processing reduces server cost and enhances privacy simultaneously. Those two goals usually fight each other. • Size is one axis of three. Context length and inference time move with it, and the optimum is a point on that surface, not a model name on a pricing page. • Document assistance is narrow, high-volume and boring. That is precisely the profile that should never touch a frontier API. More: https://ai-engineer-roadmap.xyz/token-economics

#OnDeviceAI #SmallLanguageModels #EdgeAI #MobileML #PrivacyByDesign

A 0.8B model landed within striking distance of an 8B production system on the same extraction task. How Small Can You Go? (2026) is a deployment study of 24 model variants pulling merchant names out of noisy bank transaction strings. The incumbent was a LoRA-fine-tuned LLaMA 3.1-8B at 96.95% F1. Qwen 3.5 4B with JSON-only prompting reached 96.60% F1, within 0.35 points, at roughly half the parameters. The 0.8B Qwen 3.5 still reached 94.75% F1. That is the fine-tuning trade laid bare. You are not choosing between accuracy and cost. You are choosing how many F1 points you are willing to sell, and at what price per point. • Dropping the LoRA rank from 32 to 8 cost 0.20 points of F1. Rank is a budget dial, not a quality cliff. • Benchmark numbers survived contact with production: 14 fine-tuned sub-8B models served with an average F1 change of only 0.8 points. • Reasoning supervision was wasted here. Think and nothink training templates produced nearly identical results on structured extraction. The whole cost-versus-quality literature, distilled and linked, lives here: https://ai-engineer-roadmap.xyz/token-economics

#LoRA #FineTuning #InferenceCost #MLOps #StructuredExtraction

Training the adapter is the easy part. Serving a hundred of them is what kills the business case. InfiniLoRA (2026) starts from an uncomfortable observation: existing LoRA serving designs couple adapter execution to base-model inference. Emerging architectures like MoE inflate LoRA memory cost, so the coupled design scales badly and inflates tail latency exactly when your multi-tenant fleet is busiest. Disaggregating LoRA execution from the base model, with a shared LoRA server and SLO-driven provisioning, raised the serviceable request rate an average of 3.05 times under strict latency SLOs and lifted the share of adapters meeting their SLO requirement by 54.0%. Read that as an economics result, not a systems one. The same GPUs absorb far more traffic, which is what moves the amortised cost per request. • A fine-tune only pays back if many adapters share hardware. Serving density is the payback term. • Tail latency, not average latency, decides whether an adapter fleet holds its SLO. • Your base-model choice changes your serving bill, not just your eval scores. More on where the money actually goes: https://ai-engineer-roadmap.xyz/token-economics

#LoRA #LLMServing #MultiTenancy #GPUEfficiency #SLO

An encoder model beat every prompting strategy thrown at it, on fewer computational resources. Fine-Tune, Don't Prompt (2026) benchmarked zero-shot prompting, in-context learning and supervised fine-tuning for detecting stigmatizing or privileging language in clinical notes. Fine-tuning with lexically primed inputs consistently beat the prompting approaches. A fine-tuned GatorTron scored an F1 of 0.96 on the OB-GYN test set, outperforming larger generative models while requiring minimal prompt engineering. The catch is the part practitioners skip. External validation on MIMIC-IV showed limited cross-domain generalizability, an F1 below 0.70 and a 44% drop. Training on the broader corpus and testing back on OB-GYN gave F1 of 0.71, an 11% drop, at the cost of reduced precision. • A fine-tune buys accuracy and cheap inference, but it buys them for one domain. • The same word carries different valence across specialties. Generality was the thing being traded away. • Budget for re-tuning per domain, or the cost model you sold to finance is fiction. The full cost-quality reading list is here: https://ai-engineer-roadmap.xyz/token-economics

#FineTuning #ClinicalNLP #PromptEngineering #DomainAdaptation #ModelEvaluation

Matching a teacher agent at 2.5x lower cost, with no fine-tuning and no prompt engineering at all. Inference-Time Distillation (2025) refuses the usual fork. Prompt engineering is brittle and slows iteration. Fine-tuning demands multi-day training and commitment to a fixed design. Instead: run the expensive teacher on a small subset of tasks, keep the demonstrations, then deploy a cheap student that retrieves relevant teacher traces as in-context examples. When several student samples agree, ship the answer. When they diverge, fall back to the teacher. On ALFWorld it matched teacher accuracy at 2.5x lower cost, 0.059 down to 0.024 per episode. On AppWorld it cut cost 3.5x while recovering 79% of teacher accuracy. • The cost-accuracy frontier moves without touching a single weight. That is a real option most teams never price. • Agility is a cost term. Multi-day training runs are paid in engineer-weeks nobody puts on the invoice. • Cascade thresholds, demo count and teacher-database size are the knobs that decide the bill. Where the cheap wins actually hide: https://ai-engineer-roadmap.xyz/token-economics

#LLMAgents #Distillation #InContextLearning #CostOptimization #Cascades

Serverless LoRA inference duplicates 99% of its weights across functions. You are paying rent on the same backbone, over and over. ServerlessLoRA (2025) diagnoses why pay-as-you-go GPU serving works for a plain LLM but falls apart for adapters: massive parameter redundancy between functions, artifact loading latency stacked on top of model loading, and magnified resource contention when several LoRA models are served at once. The result is GPU wastage, inflated time-to-first-token, and a bill nobody forecast. Sharing the backbone securely across isolated LoRA functions, pre-loading adapter artifacts, and batching with contention awareness cut TTFT by up to 86% and monetary cost by up to 89% on industrial workloads. • The fine-tune is small. The thing you keep paying for is the base model underneath it. • Cold starts are a cost line, not a latency footnote. Pre-loading artifacts is the cheapest fix on the list. • Before adopting a serving platform, ask what it duplicates per function. The rest of the token-economics evidence base: https://ai-engineer-roadmap.xyz/token-economics

#Serverless #LoRA #GPUUtilization #InferenceCost #ColdStart

Few-shot prompting lost to supervised fine-tuning on clinical entity recognition, and the longer prompt lost to the shorter one. Supervised Fine-Tuning or In-Context Learning? (2025) compared BERT-style encoders, GPT-4o with few-shot in-context learning, and GPT-4o with supervised fine-tuning on the CADEC corpus across five entity types. Fine-tuning achieved the strongest overall performance, an F1 of about 87.1%, albeit with higher cost. RoBERTa-large and BioClinicalBERT offered only limited improvements over BERT Base, showing the ceiling of that family. The buried finding is the useful one: simple in-context learning outperformed a longer, instruction-heavy prompt. Stuffing the prompt made results worse and the bill bigger at the same time. • The prompt-versus-tune debate has a spreadsheet answer. Higher training cost, lower marginal cost per call, and a crossover volume you can compute. • Longer prompts are not stronger prompts. Test the short one before you pay for tokens forever. • Simplifying the label space lifted accuracy. Task design is a cost lever. Full breakdown and papers: https://ai-engineer-roadmap.xyz/token-economics

#FineTuning #InContextLearning #ClinicalNLP #NER #PromptEngineering

Touching 0.8% of a frozen 7B model's parameters was enough to beat specialized reward networks of 27B to 70B parameters. Efficient Online RFT with Plug-and-Play LLM Judges (2025) attacks the cost bottleneck of RLHF: reward-model training that often needs tens of billions of parameters and an offline preference-tuning phase. The replacement is almost insultingly small. A frozen, instruction-tuned 7B LLM, a one-line JSON rubric, and a rank-16 LoRA adapter. That judge scores 96.2% accuracy on RewardBench and lets a 7B actor reach 92% exact match on GSM-8K with online PPO, against a top 70B DPO baseline at 61.8%. The ablations split the credit honestly. In-context demonstrations delivered most of the zero-to-few-shot gain, and the LoRA closed the rest, especially on safety and adversarial segments. • Prompting and tuning are not rivals here. The rubric does the framing, the adapter does the finishing. • Small adapters make an offline pipeline stage disappear entirely. • Cheap judges also read as more interpretable, which auditors will eventually ask about. More papers on the cost of quality: https://ai-engineer-roadmap.xyz/token-economics

#RLHF #RewardModeling #LoRA #LLMJudge #PEFT

Across 310 LoRA fine-tunes, 4-bit adapted models beat their base models by 34 points and GPT-4 by 10 points on average. LoRA Land (2024) is the most brutally practical answer to the buy-versus-tune question. Quantized low-rank adapters were trained across 10 base models and 31 tasks, then measured not only on quality but on how they serve. The serving half is the part that pays: LoRAX, a multi-LoRA inference server, hosts 25 Mistral-7B fine-tunes on a single A100 with 80GB of memory, sharing base weights and loading adapters dynamically. That is the arithmetic teams keep skipping. A frontier API charges you every call, forever. A shelf of specialized adapters shares one GPU and one set of base weights. • Specialized models beat one general-purpose model on both quality and cost in this setup. • Task complexity heuristics can partly predict which fine-tunes will pay off before you spend the compute. • Adapter serving density is what turns a training win into a margin win. The wider evidence, organized by cost lever: https://ai-engineer-roadmap.xyz/token-economics

#LoRA #PEFT #Quantization #ModelServing #OpenSourceLLMs

Idle GPU memory was sitting there the whole time. Filling it with popular adapters cut P99 time-to-first-token by 80.7%. Chameleon (2024) targets the environment most fine-tuning strategies actually land in: one base model, many adapters, heterogeneous traffic. Existing serving systems ignore that heterogeneity, hammer the interconnect with repeated adapter loading, and let head-of-line blocking wreck the tail. Two ideas fix most of it. Cache popular adapters in otherwise idle GPU memory, at no extra memory cost, and schedule with a non-preemptive multi-queue that avoids both head-of-line blocking and starvation. Under high load against production traces, P99 TTFT fell 80.7%, P50 fell 48.1%, and throughput rose 1.5x versus state-of-the-art baselines. • The cost of a fine-tune is dominated by how it is served, not how it was trained. • Adapter load time is a hidden tax on every request that misses the cache. • Scheduling policy is a cost decision. Head-of-line blocking is money. See how the levers compare, side by side: https://ai-engineer-roadmap.xyz/token-economics

#LoRA #LLMServing #Caching #Scheduling #TailLatency

The prompt-versus-fine-tune argument was settled in 2022, and most teams still have not read the paper. Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning makes the cost argument explicit. In-context learning incurs substantial computational, memory and storage costs because it processes all of the training examples every single time a prediction is made. You pay for those examples on every call, forever. Parameter-efficient fine-tuning trains a small set of parameters once, and then the examples are gone from the prompt. The paper introduces a method that scales activations by learned vectors, and a recipe called T-Few that needs no task-specific tuning. T-Few outperformed the state of the art on the RAFT benchmark by 6% absolute, with dramatically lower computational cost than few-shot prompting. • Every token in your few-shot block is a recurring line item. Fine-tuning converts it into a fixed one. • Better accuracy and lower cost were achieved together, not traded. • Run the crossover arithmetic before you scale a long prompt to production volume. The full token-economics library: https://ai-engineer-roadmap.xyz/token-economics

#PEFT #FewShotLearning #InContextLearning #FineTuning #TokenEconomics

Long-context prompting was more correct than RAG. It also cost 26 times more input tokens per query. The Token Tax of Epistemic Accuracy (2026) frames grounding as an allocation decision made before generation. Retrieve a few relevant passages, or load the whole document collection into context. The study evaluated 972 answers across three machines, two small language models and three retrieval or in-context prompting approaches, on an expert-validated manufacturing safety benchmark. Long-context prompting achieved the highest correctness, 73.1% versus 65.4% for semantic RAG. It bought that with 26 times the per-query token cost. The authors call the gap a token tax on broader evidentiary access, and it is the cleanest statement of the trade anyone has published. • The right question is not which architecture wins. It is what a point of correctness is worth to you. • For resource-constrained organizations, the tax is decisive. For a regulated safety workflow, it might be a bargain. • Whichever you pick, price the per-query token cost before you ship, not after the invoice. Every paper behind this trade-off: https://ai-engineer-roadmap.xyz/token-economics

#RAG #LongContext #TokenEconomics #GroundedGeneration #LLMArchitecture

Sophisticated RAG memory systems hit 30-45% accuracy where naive full-context prompting hit 70-82%. The Convomem Benchmark (2025) built 75,336 question-answer pairs across user facts, assistant recall, abstention, preferences, temporal changes and implicit connections, then made a point most memory startups would rather you missed. Conversational memory starts from zero and grows. Below 150 interactions, the corpus is small enough that exhaustive search is simply feasible, and simple full-context approaches reached 70-82% accuracy on the hardest multi-message evidence cases while systems like Mem0 reached only 30-45%. The paper is honest about where that stops. Long context excels for the first 30 conversations, stays viable with manageable trade-offs up to 150, and beyond that costs and latencies turn prohibitive. • Retrieval is not free engineering. It is a system you have to build, evaluate and maintain. • There is a corpus size below which retrieval actively loses. Measure yours before you build. • The transition point is a design input, not an afterthought. The whole cost-versus-accuracy corpus: https://ai-engineer-roadmap.xyz/token-economics

#RAG #LongContext #ConversationalMemory #Benchmarking #AgentMemory

Storing everything is a choice, not a default. Storing only what matters saved 72.4% of memory for 1.0% of F1. BudgetMem (2025) puts a blunt question to RAG orthodoxy: why does the store keep every chunk? Retrieval systems index the whole corpus by reflex, then pay for it in memory and infrastructure. BudgetMem instead learns what to remember, scoring salience from entity density, TF-IDF, discourse markers and position bias, gating what gets stored, and retrieving with sparse BM25. Across 700 question-answer pairs, on long documents of 5K-10K tokens with Llama-3.2-3B-Instruct, it lost 1.0% F1 while saving 72.4% of the memory a baseline RAG store needed. The benefit grew with document length. • Context windows stretched to 100K-1M tokens, but that capacity is priced per call. Selective memory is priced once. • Salience scoring is cheap, classical and unglamorous. It is also where the savings are. • The right comparison is not RAG versus long context. It is how much evidence you store, and how much you resend. Full research map on token cost: https://ai-engineer-roadmap.xyz/token-economics

#RAG #LongContext #MemorySystems #BM25 #CostEfficiency

Retrieved documents compressed 16x, with 0-3% accuracy loss and no pretraining required. PISCO (2025) takes the third path in the RAG-versus-long-context fight. You do not have to choose between resending everything and retrieving less. You can retrieve the same evidence and send far fewer tokens of it. The method leans on sequence-level knowledge distillation from document-based questions. No pretraining, no annotated data, and a fine-tune of a 7-10B model in 48 hours on a single A100. The result is a 16x compression rate at 0-3% accuracy loss across diverse retrieval-augmented question-answering tasks, outperforming existing compression models by 8% in accuracy. • RAG's scalability problem is not retrieval, it is the input tokens the retrieved documents cost on every call. • Compression is the cheapest lever in the stack because it changes the recurring cost, not the fixed one. • A method that needs no pretraining and no annotation is one an ordinary team can actually adopt this quarter. Compare it against the other levers here: https://ai-engineer-roadmap.xyz/token-economics

#RAG #ContextCompression #KnowledgeDistillation #RetrievalAugmentation #InferenceCost

Graph RAG has a bill almost nobody quotes: the tokens burned constructing the graph in the first place. TERAG (2025) names it. Existing graph-based retrieval systems improve reasoning and factuality, then quietly overlook the LLM token cost of building the graph, which is exactly what stops teams adopting them at scale. TERAG builds an informative graph far more cheaply and brings back Personalized PageRank at retrieval time. The trade it lands is the interesting part. TERAG keeps at least 80% of the accuracy of widely used graph-based RAG methods while consuming only 3%-11% of the output tokens to construct the graph. • Indexing cost is a real line item. It scales with your corpus and gets re-paid every time you re-index. • Ask what an accuracy point costs before adopting an architecture that quotes only accuracy. • Low token footprint is what makes a technique deployable at scale, not what makes it publishable. The full set of token-cost papers, sorted by lever: https://ai-engineer-roadmap.xyz/token-economics

#GraphRAG #RetrievalAugmentation #TokenEfficiency #PageRank #LLMCost

Every additional cost of long-context serving traces back to one single source: the size of the KV cache. Challenges in Deploying Long-Context Transformers (2024) is the paper to read before anyone on your team says "just use a bigger context window". Using a 34B model at 50K context on A100 NVLink as the running example, it shows how the KV cache alone creates four distinct deployment problems. Prefilling long inputs eats compute time and GPU memory. Once prefilled, the resident cache sharply limits how many concurrent users you can serve. During decoding, repeatedly reading that cache inflates latency. And when it overflows, swapping it out causes context-switching stalls. That is not a pricing quirk. It is the physics of the architecture, and it is why 100K to 10M token contexts stay prohibitively expensive next to 4K. • Long context does not just cost input tokens. It costs concurrency, which is your throughput. • The stated goal is making 1M context as cheap as 4K. Until then, retrieval is a cost strategy. • Model your KV cache before you model your token spend. The deployment economics, paper by paper: https://ai-engineer-roadmap.xyz/token-economics

#LongContext #KVCache #LLMServing #GPUMemory #InferenceEconomics

A 93x reduction in compute time while improving accuracy by 43%. Not a trade-off. A free lunch, briefly. Superposition Prompting (2024) starts from two facts everyone knows and few connect. Inference cost scales quadratically with sequence length, so long RAG prompts are expensive. And irrelevant context in the prompt degrades output quality, the distraction phenomenon. Stuffing more documents into the window makes you pay more for a worse answer. The fix needs no fine-tuning. Process retrieved documents in parallel prompt paths, and discard the paths that prove irrelevant. On NaturalQuestions-Open with an instruction-tuned MPT-7B, this delivered a 93x reduction in compute time while improving accuracy by 43% over naive RAG. • More context is not more evidence. Past a point it is noise you are paying to compute. • Gains were largest when the retrieved context was large relative to what the model was trained on. That is most production RAG. • It applies to pre-trained transformers directly, so there is no training budget to justify. More levers, all of them measured: https://ai-engineer-roadmap.xyz/token-economics

#RAG #PromptOptimization #LongContext #InferenceEfficiency #QuestionAnswering

Long-range reasoning without a long-context model, at 4x shorter contexts and up to 54% lower cost. PRISM (2024) picks apart the assumed choice. Long-context models need large compute budgets. Parameter-efficient fine-tuning needs training data. RAG entails complex task-specific designs. And in-context approaches with short-context models are usually inefficient, trading context for processing more tokens over and over. PRISM's answer is structured schemas. A short-context LLM processes the input incrementally, writing concise structured outputs instead of prose, which keeps the running state small and lets the key-value cache do real work. It outperformed baselines on diverse tasks with 4x shorter contexts and cut costs by up to 54%. • Output structure is an input-cost lever. Concise state means fewer tokens resent on the next chunk. • Schemas generate from a task description, so the setup cost stays small. • It scales down to tiny contexts without raising cost or sacrificing quality, which is the opposite of how long-context scaling behaves. The economics of context, with citations: https://ai-engineer-roadmap.xyz/token-economics

#LongContext #ShortContextLLM #KVCache #StructuredOutput #TokenEfficiency

Move the retrieved documents from the front of the prompt to the back, and RAG inference runs up to 4 times faster. FlashBack (2024) is a lesson in how much money sits in a detail everyone treats as arbitrary. Standard RAG prepends retrieved content to the input. That single choice invalidates the key-value cache, because a changed prefix means everything after it must be recomputed on every request. The retrieval is not what is slow. The cache miss is. FlashBack appends retrieved documents at the end of the context instead, adding marking tokens to bound the appended block during low-rank fine-tuning. On a 7B Llama 2, inference ran up to 4 times faster than the prepending counterpart, with decent generation quality in perplexity. • Prompt layout is a cost decision. Put the stable parts first and the volatile parts last. • Unnecessary recomputation is the tax you pay for ignoring cache mechanics. • A faster inference path lowers the per-call cost directly, before you optimise a single token. Prompt layout is one lever of many: https://ai-engineer-roadmap.xyz/token-economics

#RAG #KVCache #PromptCaching #InferenceOptimization #LLMEngineering

First-come, first-served is a pricing decision that nobody remembers making. When short interactive queries queue behind long throughput-oriented batch requests, head-of-line blocking wrecks tail latency and leaves the hardware underused. EWSJF, an adaptive scheduler with hybrid partitioning for mixed-workload LLM inference, learns the workload's structure at runtime instead: it discovers performance-homogeneous request groups, routes each request into the right queue, and scores them on urgency and fairness. Implemented in vLLM, it raises end-to-end throughput by over 30% and cuts average time-to-first-token for short requests by up to 4x against FCFS. Patience is a property of the request, not of the cluster. Once the scheduler can tell which jobs can wait, the same GPUs comfortably serve both tiers. • Label requests interactive or batch at the API boundary; a scheduler cannot exploit patience it cannot see • Head-of-line blocking is a scheduling bug, not a capacity shortfall, and buying more GPUs only hides the bill • This layer sits upstream of the execution scheduler, so you can adopt it without replacing your engine Full paper map: https://ai-engineer-roadmap.xyz/token-economics

#LLMServing #vLLM #SchedulingPolicy #Throughput #TTFT #InferenceOptimization

Tensor parallelism is treated as a deployment flag you set once. Nitsum treats it as a control surface you turn all day. The same deployment now handles latency-critical interactive requests and relaxed background workloads under one fixed GPU budget, and the mix keeps shifting. Most systems adapt only queuing and batching while execution configuration stays frozen. Nitsum, serving tiered LLM requests with adaptive tensor parallelism, jointly optimises the TP level, the prefill/decode GPU split and request scheduling, using TP-aware weight reuse and fast KV migration so that re-sharding is cheap enough to do often. On real traces it improves SLO-compliant goodput by up to 5.3 times. Goodput is the honest metric here: requests that satisfy both their TTFT and their TPOT target, not raw tokens per second. • Define tiers explicitly; background work with a loose deadline is the capacity that pays for your interactive tier • Static parallelism configs are tuned for a workload mix that no longer exists by the afternoon • Measure goodput per GPU, not throughput per GPU Full paper map: https://ai-engineer-roadmap.xyz/token-economics

#TensorParallelism #SLOaware #Goodput #LLMServing #GPUUtilization

Dedicating separate machines to batch work and interactive work is the most expensive simplification in LLM serving. That is the standard practice: one fleet tuned for low latency, another for high throughput, both badly utilised. Efficient LLM Serving on Hybrid Real-time and Best-effort Requests proposes BROS, which collocates the two on the same model. It formulates hybrid real-time and best-effort scheduling as a dynamic priority problem and adds bidirectional KV cache management, so real-time requests can share KV memory with best-effort ones instead of being blocked by it. The result is not a trade-off. Real-time latency drops by up to 74.20% and fine-grained SLO attainment rises up to 36.38x, with negligible throughput reduction for the best-effort batch work. • Back-of-house document processing is the perfect filler load: it has no user waiting on it • KV memory, not FLOPs, is what usually forces the two workloads apart; fix the memory policy first • Separate fleets look safe on an architecture diagram and look wasteful on an invoice Full paper map: https://ai-engineer-roadmap.xyz/token-economics

#LLMServing #KVCache #SLOAttainment #BatchInference #ResourceUtilization

Serving systems keep optimising average performance for workloads that have no average. Streaming chat cares about per-token latency. A tool-triggering call cares about the deadline for the full response. An agentic pipeline cares about dependencies that only reveal themselves mid-run. JITServe, an SLO-aware serving system built for imprecise request information, schedules on rough estimates of response length and dependency, then sharpens those estimates as generation proceeds. Its grouped margin goodput algorithm allocates just enough serving bandwidth to hit each request's SLO just in time, and banks the residual capacity for everyone else. Across chat, deep research and agentic pipelines this improves service goodput by 1.4x-6.3x, or alternatively delivers 28.5%-83.2% resource savings at the same goodput. • Read that second number again: the same quality of service on roughly a third of the hardware • You do not need exact output-length prediction; conservative estimates that tighten over time are enough • Every request that is not deadline-sensitive is slack the scheduler can sell back to you Full paper map: https://ai-engineer-roadmap.xyz/token-economics

#SLOaware #Goodput #RequestScheduling #LLMInference #CapacityPlanning

Not every request is worth the same amount of money, yet almost every scheduler pretends otherwise. Business-critical calls and background chores land in the same queue and receive the same treatment. ProServe, a unified multi-priority request scheduling framework for LLM serving, formalises this as service gain maximisation: satisfying the latency requirement of a high-priority request contributes more gain than satisfying a low-priority one. Its SlideBatching layer adapts batch formation under load with a sliding boundary, while preempted blocks are asynchronously offloaded and pipelined back in. Above that, gain-oriented routing reserves capacity for future high-priority or long requests. Across four open-source datasets and one industrial dataset it improves system gain by up to 35% and SLO attainment by up to 52%. • Encode business value in the queue, not in a runbook nobody reads at midnight • Preemption is only viable if offloading and reloading blocks is overlapped with compute • Reserving headroom for expensive requests beats scrambling for it once they arrive Full paper map: https://ai-engineer-roadmap.xyz/token-economics

#RequestScheduling #SLOaware #MultiTenancy #LLMServing #Preemption

Your batch size was tuned on a Tuesday, for traffic that no longer exists. Continuous batching is the throughput workhorse of every modern serving stack, but the batch size itself is usually a static configuration. It cannot adapt to fluctuating request rates or heterogeneous workloads, and the result is repeated SLA violations and jumpy latency. Adaptive Request Scheduling for CodeLLM Serving with SLA Guarantees introduces SABER, which predicts per-request SLA feasibility and adjusts scheduling decisions in real time rather than trusting one number. SABER improves goodput by up to 26% over the best static configuration, and cuts latency variability by up to 45%, without manual tuning or service restarts. Note the phrase best static configuration. That is not a strawman baseline, that is the tuned value you would have shipped. • Predict feasibility per request instead of reacting to violations after they happen • Latency variability is what your users actually feel; averages hide it • Self-hosted, resource-constrained code assistants are exactly where adaptive scheduling pays back fastest Full paper map: https://ai-engineer-roadmap.xyz/token-economics

#ContinuousBatching #CodeLLM #SLA #SelfHosting #Goodput

Running a thousand agent workflows one at a time is like running a thousand SQL queries without a query planner. Serving engines optimise each LLM call in isolation, and multi-agent frameworks orchestrate without any system-level performance planning. Repeated prompts, overlapping contexts and fragmented CPU-GPU execution produce enormous redundancy. Halo, from Batch Query Processing and Optimization for Agentic Workflows, represents each workflow as a query plan DAG, then consolidates a batch of them into one graph that exposes shared computation. A cost model covering prefill and decode costs, cache reuse and GPU placement drives plan-level optimisation, with adaptive batching, KV-cache sharing and CPU-GPU pipelining underneath. Across six benchmarks Halo reaches up to 3.6x speedup for batch inference and 2.6x higher throughput under online serving, with no quality loss. • Offline agentic analytics is a batch workload; stop serving it through an interactive path • Shared prefixes across sibling workflows are free money once something can see them together • Borrow from databases: plan first, execute second Full paper map: https://ai-engineer-roadmap.xyz/token-economics

#AgenticWorkflows #QueryOptimization #BatchInference #KVCache #LLMServing

Offline batched inference is bottlenecked by moving the KV cache around, not by matrix multiplication. Offloading-based batched inference extends GPU memory with host memory and storage, which is how long-context jobs run on affordable hardware. The catch is I/O: KV cache size scales with both batch size and context length, and the interconnect drowns. HILOS, a cost-effective near-storage processing solution for offline inference of long-context LLMs, moves the memory-intensive attention operations to near-storage accelerators, then adds a cooperative host-side cache, delayed KV writeback to hide write latency, and a memory-efficient attention accelerator. Evaluated on a real system with 16 SmartSSDs, HILOS reaches up to 7.86x the throughput of state-of-the-art offloading frameworks while cutting energy consumption by up to 85%. • If the work has no user waiting, optimise for tokens per dollar, not tokens per second • Energy is a line item too; an 85% cut shows up directly in a self-hosted bill • Long-context offline jobs are where exotic hardware pays for itself first Full paper map: https://ai-engineer-roadmap.xyz/token-economics

#OfflineInference #KVCache #NearStorageProcessing #LongContext #EnergyEfficiency

The cheapest GPU in your cluster is the one already paid for and currently doing nothing. Distributed training leaves prominent idle windows, caused by resource allocation and by collective communication stalls. SpecInF, which exploits idle GPU resources in distributed DL training via speculative inference filling, collocates inference instances with each primary training instance on the same GPU, detects those training bubbles, and adaptively fills them with online or offline inference work. The payoff: up to 14 times more offline inference throughput than TGS, and 67% lower online inference p95 latency than MPS, while the collocated training throughput is still guaranteed. That is the batch-scheduling thesis in its purest form. Work that can wait does not need dedicated capacity; it needs a scheduler willing to slot it into someone else's gaps. • Audit idle time on your training fleet before you rent one more inference GPU • Bubble-filling only works when the filler workload is preemptible, so keep offline jobs interruptible • Collocation must guarantee the primary job, otherwise nobody will let you do it twice Full paper map: https://ai-engineer-roadmap.xyz/token-economics

#GPUUtilization #SpeculativeInference #Collocation #OfflineInference #ClusterEfficiency

Nobody knows how long a generation will run, so schedulers guess conservatively and leave throughput on the floor. Sequence-level scheduling serves requests first-come first-served with static batching, so short generations wait behind long ones, and small batch sizes are chosen just to dodge out-of-memory errors. Iteration-level scheduling, the continuous batching that modern engines use, is better but still caps parallel requests to stay safe on memory, and neither approach balances load across instances. Slice-Level Scheduling for High Throughput and Load Balanced LLM Serving splits the maximum generation length into slices and serves batches slice by slice, which gives a precise range for the serving time and memory of a batch. That predictability is the whole point: it improves throughput by up to 315.8% over sequence-level and iteration-level schedulers, and flattens load imbalance. • Unpredictable output length is the root cause of both OOM paranoia and low batch occupancy • Bound the work per step and you can finally size batches aggressively • Load balancing across replicas is a scheduling win, not an infra win Full paper map: https://ai-engineer-roadmap.xyz/token-economics

#ContinuousBatching #RequestScheduling #Throughput #LoadBalancing #LLMServing

A Mixtral 8x7B batch job does not need a flagship datacentre GPU. It needs a better schedule. Mixture-of-Experts models raise capacity without a proportional rise in inference cost. But their sheer size locks them away from anyone without high-end hardware, so teams rent scarce accelerators to run work nobody is waiting on. MoE-Lightning, a high-throughput MoE batch inference system, attacks that with CGOPipe, a CPU-GPU-I/O pipelining schedule with paged weights, plus a hierarchical roofline performance model that searches for genuinely high-throughput policies rather than merely plausible ones. It reaches up to 10.3x higher throughput than state-of-the-art offloading-enabled inference systems for Mixtral 8x7B on a single 16GB T4, and when GPU memory is the binding constraint it hits the throughput upper bound with 2-3x less CPU memory. The same approach extends to Mixtral 8x22B and DBRX across a handful of low-cost cards. • For offline batches, cheap old GPUs plus a smart pipeline often beat renting scarce new ones • Overlapping CPU, GPU and I/O is where offloaded throughput actually comes from • Model your roofline before you tune, or you will optimise the wrong stage Full paper map: https://ai-engineer-roadmap.xyz/token-economics

#MixtureOfExperts #Offloading #BatchInference #GPUMemory #Pipelining

PCIe bandwidth, not the GPU, is what makes long-context offline inference slow. Growing context lengths and batch sizes blow up the KV cache, and the usual escape is to push it into host memory or onto SSDs. That trades a memory wall for an I/O wall: every decoding step drags KV data back across a narrow link. InstInfer, an in-storage attention offloading system for cost-effective long-context LLM inference, instead moves decoding attention and the KV cache itself into computational storage drives, exploiting their high internal bandwidth rather than fighting PCIe. For a 13B model on an A6000, it improves long-sequence throughput by up to 11.1 times over SSD-based solutions such as FlexGen. That is not a tuning win, that is a different memory hierarchy. • Profile where offline throughput is actually lost; it is usually data movement, not compute • Batch and offline work tolerates deeper memory hierarchies that interactive serving cannot stomach • Edge devices and personal machines get the same benefit as a rack, because the bottleneck is identical • Cost-effective long context is a storage architecture question before it is a model question Full paper map: https://ai-engineer-roadmap.xyz/token-economics

#ComputationalStorage #KVCache #LongContext #OfflineInference #Throughput

Serving a large encoder online is a recurring bill that most RAG budgets never itemise. Big embedding models retrieve better, so teams reach for them, and then pay to encode every query and every document with them forever. Score-Only Distillation for Compact Dense Retrieval asks whether a compact student can learn the teacher's ranking behaviour from score vectors alone, with no access to teacher hidden states. It trains on rows of ground-truth positives and negative candidates using a row-centred score-vector objective, a memory-efficient all-pairs PairMSE loss. On a fixed eight-task panel the protocol recovers up to 50% of the base-to-teacher quality gap, and the distilled 0.6B student encodes queries 4.7 times faster and documents 9.7 times faster than sequential online teacher fusion. Document encoding is the one that matters at corpus scale. That is the number your re-index job is charged for. • The teacher is a training-time asset, not an inference-time dependency • Score vectors are enough supervision; you do not need white-box access • External transfer stayed mixed, so evaluate on your own corpus before switching Full paper map: https://ai-engineer-roadmap.xyz/token-economics

#DenseRetrieval #Distillation #EmbeddingModels #RAG #InferenceCost

The per-query cost of approximate nearest neighbour search is a real line item, and it varies by an order of magnitude depending on who hosts it. The specialised vector database was sold as a necessity. Cost-Effective, Low Latency Vector Search with Azure Cosmos DB argues the opposite: deeply integrate DiskANN into a cloud-native operational database and you keep availability, durability and scale while getting the search. One vector index per partition, stored in the existing index trees and kept in sync with the underlying data. It serves under 20ms query latency over an index spanning 10 million vectors, holds recall stable under updates, and offers roughly 43x and 12x lower query cost than Pinecone and Zilliz serverless enterprise products, while partitioning out to billions of vectors. • Price your ANN queries per million, the same way you price LLM tokens • A separate vector store is also a separate sync problem and a separate failure domain • If your vectors already live next to their metadata, hybrid queries stop being an integration project Full paper map: https://ai-engineer-roadmap.xyz/token-economics

#VectorSearch #DiskANN #VectorDatabase #ANN #RAGInfrastructure

Scaling a vector index across nodes usually buys you load imbalance and a communication tax. Approximate nearest neighbour search underpins recommendation, image retrieval and RAG, and billions of high-dimensional vectors do not fit on one machine. So the index gets partitioned, and traditional partition strategies distribute the workload badly: some nodes melt, others idle, and every query fans out. Harmony, a scalable distributed vector database for high-throughput ANN search, combines dimension-based and vector-based partitioning into one multi-granularity strategy, and adds early-stop pruning that exploits the monotonicity of distance computations under dimension-based partitioning to cut both compute and communication. On real-world datasets it achieves 4.63 times the throughput of leading distributed vector databases on four nodes, and 58% better performance than traditional distribution on skewed workloads. • Skewed query distributions are the norm; benchmark on them, not on uniform ones • Communication overhead, not distance math, is what caps distributed recall throughput • Prune early: most candidates can be discarded before the full vector is ever touched Full paper map: https://ai-engineer-roadmap.xyz/token-economics

#ANN #VectorDatabase #DistributedSystems #Sharding #Throughput

Every embedding model upgrade quietly re-bills you for your entire corpus. The standard playbook is brutal: re-encode every document, rebuild the ANN index, run a dual index while you cut over, and hope nothing drifts. Drift-Adapter, a practical approach to near zero-downtime embedding model upgrades in vector databases, proposes a lightweight learnable transformation that bridges the old and new embedding spaces instead. New queries are mapped back into the legacy space, so the existing index keeps serving and the full re-computation is deferred. Three parameterisations are evaluated: orthogonal Procrustes, low-rank affine, and a compact residual MLP, each trained on a small sample of paired old and new embeddings. On MTEB text corpora and a CLIP image upgrade it recovers 95-99% of full re-embedding recall, adds under 10 microseconds of query latency, and reduces recompute cost by over 100 times. • Treat the embedding model as a versioned dependency with a migration path • Re-indexing cost, not model quality, is what freezes teams on an old encoder • Defer the rebuild; do it when it is convenient, not when the model ships Full paper map: https://ai-engineer-roadmap.xyz/token-economics

#VectorDatabase #EmbeddingModels #RAGOps #ZeroDowntime #IndexMigration

Most RAG stacks still store embeddings in float32, which is a habit, not a decision. Optimization of embeddings storage for RAG systems using quantization and dimensionality reduction techniques ran both levers systematically on MTEB. On quantization it compared float16, int8, binary and the low-bit float8 format. On dimensionality reduction it compared PCA, kernel PCA, UMAP, random projections and autoencoders. The findings are unusually actionable. float8 quantization achieves a 4x storage reduction with under 0.3% performance degradation, clearly outperforming int8 at the same compression level and being simpler to implement. PCA emerges as the most effective dimensionality reduction technique of the five tested. And combining moderate PCA, retaining 50% of dimensions, with float8 reaches 8x total compression with less performance impact than int8 alone, which only gives 4x. Storing high-dimensional vectors at scale is a memory problem long before it is a quality problem, and this is the cheapest fix in the whole RAG stack. • Dimensionality is a direct storage multiplier; halve it and you halve the vector store • Stack quantization and reduction, because they compose better than either lever alone • Plot the performance-storage trade-off space, then pick the point your memory budget allows Full paper map: https://ai-engineer-roadmap.xyz/token-economics

#Quantization #VectorStorage #RAG #PCA #MTEB

Embedding dimensionality is a storage multiplier hiding in a config field. Modern language model embeddings run to hundreds or thousands of dimensions, and every one of those floats is stored, indexed and compared on every single query. That scales badly the moment the corpus gets massive. PCA-RAG: Principal Component Analysis for Efficient Retrieval-Augmented Generation tests the boring classical fix on a real financial text corpus, comparing similarity and distance metrics under full-dimensional and PCA-compressed embeddings. Reducing vectors from 3,072 to 110 dimensions gives up to 60 times faster retrieval and an index roughly 28.6 times smaller, with only moderate declines in correlation against human-annotated similarity scores. For a real-time news and trading platform, where speed, memory and accuracy are optimised jointly, that trade lands firmly on the side of shipping. • Fit PCA on your own corpus; the right target dimension is domain-specific, never universal • Index size and query latency fall together, so one change bills back against storage and compute • Classical dimensionality reduction is still the highest-leverage line of code in a RAG budget Full paper map: https://ai-engineer-roadmap.xyz/token-economics

#PCA #RAG #VectorSearch #DimensionalityReduction #IndexSize

Keep only the first quarter of your embedding dimensions and almost nothing happens. Prompt-based text embedding models produce task-specific vectors that routinely run to thousands of dimensions, and those dimensions are charged to you twice: once in storage, once in every similarity computation. Redundancy, Isotropy, and Intrinsic Dimensionality of Prompt-based Text Embeddings applies naive post-hoc truncation and measures the damage across classification, clustering, retrieval and semantic textual similarity. Keeping only the first 25% of dimensions causes very slight degradation. For classification and clustering, performance holds even when embeddings are cut to under 0.5% of the original dimensionality. The analysis explains why: those tasks show lower intrinsic dimensionality and less isotropy than retrieval and STS, which need more of the space. • Truncate before you buy storage; the cheapest compression is the one with no training step • Budget dimensions per task, not per model: clustering needs far fewer than retrieval • Measure intrinsic dimensionality on your own vectors to find the floor Full paper map: https://ai-engineer-roadmap.xyz/token-economics

#EmbeddingModels #IntrinsicDimensionality #VectorStorage #Clustering #Retrieval

High-dimensional embeddings are the reason your vector bill scales faster than your corpus. LLM-generated embeddings capture rich semantics and, in exchange, inflate computational complexity and storage until deployment hurts. Matryoshka representation learning promised a dial, but training it well is harder than it looks. SMEC, Sequential Matryoshka Embedding Compression, tackles three specific failures: gradient variance during training, handled by sequential Matryoshka representation learning; information loss during dimension pruning, handled by an adaptive dimension selection module; and weak coupling between high- and low-dimensional embeddings, handled by a selectable cross-batch memory module. On BEIR, SMEC compresses LLM2Vec embeddings to 256 dimensions and still improves on Matryoshka-Adaptor by 1.1 points and Search-Adaptor by 2.7 points, with results holding across image, text and multimodal datasets. • Pick the smallest dimension that clears your recall bar, then hold it as a hard budget • Truncation is free but lossy; a trained compressor buys back the quality • Storage, index build time and query latency all fall with the same dial Full paper map: https://ai-engineer-roadmap.xyz/token-economics

#MatryoshkaEmbeddings #EmbeddingCompression #BEIR #RetrievalQuality #VectorStorage

A 250M-parameter retriever that beats models up to 10 times larger is an infrastructure argument, not a leaderboard one. Retrieving from a large document corpus is the most common industrial use of AI right now, and visual document retrieval does it on page screenshots directly, skipping the parsing pipeline. The usual recipe is to repurpose a large vision-language decoder as an embedding model, because it is cost-efficient to build. ModernVBERT: Towards Smaller Visual Document Retrievers shows that this very repurposing is what bottlenecks retrieval quality. Revisiting the whole training pipeline, the authors isolate the factors that actually matter: attention masking, image resolution, modality alignment data regimes, and late-interaction centred contrastive objectives. The result is a compact vision-language encoder that outperforms recent models up to 10 times larger when fine-tuned on document retrieval, and runs inference on cheap CPU hardware, cutting indexing latency and cost together. • Encoder size sets your indexing bill, and every document in the corpus pays it once per re-index • CPU-servable retrievers take the GPU out of the ingestion path entirely • Generative-decoder-as-encoder is convenient, not optimal Full paper map: https://ai-engineer-roadmap.xyz/token-economics

#VisualDocumentRetrieval #RAG #CPUInference #LateInteraction #EmbeddingModels

You can shrink OpenAI and Google embeddings two- to twelve-fold without touching their quality, and you do not need access to the model to do it. High-dimensional embeddings usually perform better, which is why providers ship them, and why teams store them at full width and then complain about latency and cost. Matryoshka-Adaptor is a tuning framework that modifies the embeddings themselves, downstream of the provider, so it works even with black-box APIs. It runs in both unsupervised and supervised settings and slots into any LLM architecture. Evaluated across English, multilingual and multimodal corpora, it delivers substantial dimensionality reduction while holding performance across multiple BEIR datasets. The point is that vector width is a decision you own, even when the model is not yours. • Do not accept the provider's default dimension as a law of nature • Black-box APIs still leave the storage and query cost on your side of the line • Tune the adaptor on your own corpus; that is where the redundancy lives Full paper map: https://ai-engineer-roadmap.xyz/token-economics

#MatryoshkaEmbeddings #BEIR #EmbeddingAPIs #DimensionalityReduction #VectorStorage

Over 20 commercial vector database systems appeared within five years. They are all fighting the same two costs. Embedding-based retrieval has been studied for over ten years and similarity search for more than half a century, so the sudden explosion of systems is not an algorithmic breakthrough. It is LLM applications demanding vast unstructured stores with fast, scalable query processing. The Survey of Vector Database Management Systems names five obstacles, and two of them are pure economics: the large size of vectors, and the high cost of similarity comparison. The other three are vagueness of semantic similarity, the lack of natural partitioning for indexing, and the difficulty of hybrid queries over attributes and vectors. Read the mitigations as a cost menu: vector compression through quantization, partitioning by randomization or learning, plan selection, and hardware-accelerated execution. • Vector size and comparison cost are structural, so design against them from day one • Hybrid queries are where naive vector stores fall over in production • Native versus extended systems is a cost trade-off, not a religion Full paper map: https://ai-engineer-roadmap.xyz/token-economics

#VectorDatabase #ANN #Quantization #HybridQueries #DataInfrastructure

Two orthographies. Identical linguistic content. One of them runs 16.5x slower on the same hardware. The Script Tax measured what a tokenizer quietly does to a writing system. Comparing two orthographic variants across mBERT and XLM-R, the higher-fragmentation script showed a 3.4x increase in fertility — 6.73-6.85 tokens per word against 2.10-2.35 — and that fragmentation propagated straight into throughput: 0.23 versus 3.8 sentences per second, same machine, same weights. Fertility is not a linguistics curiosity. It is a billing multiplier. → Tokens per word is the exchange rate between your language and the model's price list. A 3.4x fertility gap is a 3.4x tax on every prompt you will ever send. → Measured in bits per character, to dodge the NLL paradox that subword fragmentation creates, the information cost rises too: +19.7% for mBERT, +47.1% for XLM-R. The text did not just get longer. It got harder. → A round-trip conversion check confirms this is orthography-conditioned processing, not mapping noise. The tokenizer is the cause. Script choice is infrastructure. Nobody put it in the budget. The tokenization chapter, with the papers behind it: https://ai-engineer-roadmap.xyz/token-economics

#Tokenization #MultilingualNLP #InferenceCost #LLMOps #TokenEconomics

The most token-efficient language in one recent study was not English. It was Sanskrit. A quantitative study across GPT, Gemini and SentencePiece tokenizers took 701 parallel verses of the Bhagavad Gita in Sanskrit, English and Hindi, then measured what each language actually costs to encode: token count, characters per token, tokens per character. Under an unbiased SentencePiece baseline, Sanskrit encodes about 2x more compactly than English or Hindi. Translate a Sanskrit commentary into either and the token count inflates roughly 20x. Two things follow, and neither is really about Sanskrit. → Token efficiency is a joint property of a language and a tokenizer, never of the language alone. Morphology that packs meaning into fewer characters packs it into fewer tokens, and the meter reads tokens. → Tokenizers are converging, slowly. GPT o200k base, used by GPT-4o, and the latest Gemini tokenizer cut the bias substantially against cl100k base, yet still fail to fully capture Sanskrit's compactness. The authors name the consequence plainly: a penalty bias against non-English users, which inflates their token counts and their invoices. Read the tokenization chapter: https://ai-engineer-roadmap.xyz/token-economics

#Tokenization #SentencePiece #MultilingualNLP #TokenEconomics #LLMOps

Your context window is not a fixed size. It is a fixed size in tokens, which means it silently shrinks depending on which language you speak. Separate Before You Compress makes that concrete. Standard BPE shatters Abugida scripts, breaking multi-codepoint grapheme clusters into meaningless sub-character units. The WWHO architecture and its SGPE algorithm split the job in two: the script's linguistic rules first, statistical compression second, with a guarantee that no valid syllable is ever broken across tokens. Sinhala lands at a token-to-word ratio of 1.274 and 4.83 characters per token — a 61.7 percent reduction against OpenAI's o200k base. Hindi lands at 1.181, a 27.0 percent reduction. → Fewer tokens is not only cheaper, it is roomier: the usable context window extends by up to 4.38 times for these languages. → On mixed Sinhala, Devanagari and English text, SGPE cuts tokens by 36.7 percent against o200k base, 39.6 percent against Llama 4 Scout and 60.2 percent against DeepSeek V3. → The gain comes from a linguistic prior, not from a bigger vocabulary. The Global South pays this token tax on every single call. It is fixable, and the fix lives in the tokenizer. The tokenization chapter: https://ai-engineer-roadmap.xyz/token-economics

#Tokenization #BPE #LowResourceNLP #ContextWindow #TokenEconomics

Speakers of N'Ko pay up to 8.9x more than English speakers for the same sentence. Not a worse model. The same model. "The African Language Tax" measured tokenization across 20 African languages, five language families and three scripts, on 11 frontier and open tokenizers using parallel corpora — so the language effect is isolated from the content. Every single one of the 20 languages carries a premium over English. Median 1.88x on GPT-5 / o200k_base. Up to 8.92x for N'Ko. Ethiopic and N'Ko scripts reach 7-9x. Billing is per token, so the penalty is charged three times over. • Cost: up to 8.9x inference cost for the same meaning. • Latency: an equivalent generation-latency multiplier — 7.4x for Amharic vs English on GPT-5. • Context: as little as 11% of English's effective context window. Your RAG budget silently shrinks by an order of magnitude. Tokenizer choice is a mitigation, not a fix. Gemma 4 cuts the mean premium from 3.31x (cl100k_base) to 2.38x — the best available, and still a tax. The penalty is encoded in the subword vocabulary, before a model is ever invoked. More cost-aware papers, broken down: https://ai-engineer-roadmap.xyz/token-economics

#FinOps #InferenceCost #Tokenization #LLMOps #MultilingualNLP

Double the tokens and you quadruple the training cost. That single line is why tokenization inequity compounds instead of merely annoying. The Token Tax evaluated 10 large language models on AfriMMLU — 9,000 multiple-choice items, 5 subjects, 16 African languages — and showed that fertility, tokens per word, reliably predicts accuracy. Higher fertility, lower accuracy. Across every model. Across every subject. That is the part people miss. The token tax is not just a bill. It is a quality penalty that arrives attached to the bill. → Compute and accuracy are coupled through the tokenizer. Morphologically complex, low-resource languages get inflated compute and depressed scores from the same root cause. → The economics are not linear. Translating token inflation into money, a doubling in tokens results in quadrupled training cost and time. → Reasoning models, DeepSeek and o1 among them, outperform non-reasoning peers across both high and low resource languages here, narrowing gaps seen in earlier generations. The prescription is blunt: morphologically aware tokenization, fair pricing, and multilingual benchmarks that surface the gap instead of averaging it away. Read the tokenization chapter: https://ai-engineer-roadmap.xyz/token-economics

#Tokenization #AfricanNLP #LowResourceNLP #TokenEconomics #MLOps

Across more than 200 languages, saying the same thing costs three to five times more. Not in effort. In tokens. Tokenization Disparities as Infrastructure Bias ran a standardized tiktoken evaluation over 200-plus languages, measuring tokens per sentence and relative tokenization cost against an English baseline. Latin-script languages come out consistently efficient. Non-Latin and morphologically complex languages show relative tokenization cost ratios often 3-5 times higher. Call it what the title calls it: infrastructure bias. Nobody designed this. It falls out of a subword vocabulary fitted mostly to English text, and then it hardens into the pricing of every product built on top of it. → The overhead is charged twice: once in money, once in context. Inflated token counts also mean reduced effective context utilization for the very same passage. → It is measurable, not anecdotal. Relative tokenization cost is a number you can compute against your own corpus this afternoon. → The fix is upstream — typologically aware vocabulary construction, adaptive vocabularies — not a bigger context window bolted on afterwards. Speakers of low-resource and non-Latin languages pay a surcharge nobody voted for and almost nobody measures. Read the tokenization chapter: https://ai-engineer-roadmap.xyz/token-economics

#Tokenization #MultilingualNLP #AIEquity #TokenEconomics #LLMOps

Optimizing a vocabulary for average token length instead of raw frequency yields 14 to 18 percent fewer tokens than BPE. Same text. Same model. Smaller bill. The Length-MAX tokenizer casts vocabulary construction as a length-weighted objective — a graph partitioning problem solved with a greedy approximation — rather than the usual frequency-driven merge. On FineWeb and across diverse domains it beats Byte Pair Encoding at every vocabulary size from 10K to 50K, and still wins by 13.0 percent at 64K. The interesting part is what a shorter sequence does downstream. → Training gets cheaper. GPT-2 models at 124M, 355M and 1.3B parameters reach a fixed validation loss in 18.5, 17.2 and 18.5 percent fewer steps. → Serving gets cheaper. Inference latency drops 13.7 percent at 124M, throughput gains 16 percent, and embedding plus KV-cache memory falls 18 percent. → Quality did not pay for it. LAMBADA perplexity improves 11.7 percent and HellaSwag accuracy rises 4.3 percent, with vocabulary coverage at 99.62 percent and an out-of-vocabulary rate of just 0.12 percent. The cheapest token in your system is the one the tokenizer never had to emit. Read the tokenization chapter: https://ai-engineer-roadmap.xyz/token-economics

#Tokenization #BPE #InferenceCost #KVCache #TokenEconomics

Twenty-two official languages. Twelve tokenizers. The winner was not the one you would guess. A comprehensive evaluation of the tokenizers used by 12 LLMs across all 22 official languages of India measured normalized sequence length — how many tokens each tokenizer burns on identical content. The SUTRA tokenizer outperformed every other model, excelling in 14 languages, and it beat several Indic-specific models at their own game. Which is the uncomfortable finding. Being built for a language family is not the same as tokenizing it efficiently. → Normalized sequence length is the metric to demand from a vendor. It turns "multilingual support" from a marketing claim into a number you can put in a spreadsheet. → Progress is real but uneven. GPT-4o handles Indian languages measurably better than GPT-4 did, while Project Indus, built for the region, underperforms in certain languages. → Tokenizer choice is a procurement decision. If you serve Indic users, benchmark the tokenizer before you benchmark the model. Targeted tokenization is not a nicety for multilingual products. It is the floor under everything they will ever spend. Read the tokenization chapter: https://ai-engineer-roadmap.xyz/token-economics

#Tokenization #IndicNLP #MultilingualNLP #TokenEconomics #LLMOps

Byte-level models dodge tokenizer bias entirely, then drown in sequence length. MrT5 fixes the second problem without giving back the first. Skip the subword vocabulary and you skip its unfairness: no shattered conjuncts, no brittleness to spelling errors, no wildly inconsistent compression rates across scripts. The catch has always been that raw byte streams are long, and long is expensive to train and to serve. MrT5 puts a learned delete gate inside the ByT5 encoder: after a fixed number of layers, it decides which tokens to drop and merges their information into the survivors, using context from whatever remains. → Sequence lengths fall by up to 75% while accuracy stays comparable to ByT5 on XNLI, TyDi QA and character-level tasks. → Trained multilingually, the model adapts to each language's orthography and learns its own per-language compression rate, rather than inheriting one baked in by an English-fitted vocabulary. → The savings land in inference runtime, which is where you actually pay them. Dynamic merging is what a tokenizer would be if it were allowed to read the sentence first. Read the tokenization chapter: https://ai-engineer-roadmap.xyz/token-economics

#Tokenization #ByteLevelModels #InferenceCost #MultilingualNLP #TokenEconomics

Speakers of many supported languages are overcharged by commercial LLM APIs and get worse answers for the money. That was demonstrated back in 2023, and the pricing model has not changed since. Do All Languages Cost the Same? audited OpenAI's API across 22 typologically diverse languages, comparing cost against utility on multilingual benchmarks. What counts as a token is training-data and model dependent, so the number of tokens needed to convey identical information swings enormously between languages. The meter does not care. The kicker is where those speakers live. → The overcharged languages tend to be spoken in regions where the APIs are less affordable to begin with. The tax is regressive. → Cost and utility move in the wrong directions together: more tokens spent, poorer results returned. You do not get what you pay for. You get less. → Per-token pricing is a policy choice, not a law of physics. The authors push vendors toward transparency and a more equitable pricing policy. Before your team argues about which model is smartest, find out what your users' language costs to say. Read the tokenization chapter: https://ai-engineer-roadmap.xyz/token-economics

#Tokenization #APIPricing #MultilingualNLP #AIEquity #TokenEconomics

Translate one sentence into another language and it can come back up to 15 times longer. Not longer in words. Longer on the invoice. Language Model Tokenizers Introduce Unfairness Between Languages locates the disparity at the tokenization stage, well before the model is even invoked. Nothing about weights, training data or benchmarks is involved yet. The text is simply chopped into more pieces, and every piece is billable. → Encoding-length differences reach up to 15 times for identical content, and they persist even in tokenizers deliberately trained for multilingual support. Good intent is not enough. → Character-level and byte-level models are not the escape hatch people assume: some language pairs still show over 4 times the difference in encoding length. → Three costs get charged at once — the price of accessing the commercial service, the processing time and latency, and how much content can be fed as context in the first place. The authors' conclusion is the right one: train future models with multilingually fair subword tokenizers. Fairness here is not a values statement. It is a line item in the pricing table. Read the tokenization chapter: https://ai-engineer-roadmap.xyz/token-economics

#Tokenization #MultilingualNLP #AIEquity #LLMOps #TokenEconomics

Tokenizer choice has long been treated as a footnote in LLM training. Someone finally trained 24 models to test it, and the footnote turned into a 68% cost overrun. Tokenizer Choice For LLM Training: Negligible or Crucial? trained 24 mono- and multilingual LLMs at a 2.6B parameter scale, ablating tokenizer algorithms and parameterizations — the ablation nobody runs, because running it is expensive. Three findings that should change how you spec a pretraining run. → Applying an English-centric tokenizer to a multilingual model causes severe downstream degradation plus additional training costs of up to 68%. An inefficient vocabulary means buying more compute to say the same thing. → Multilingual tokenizers trained on the five most frequent European languages need vocabulary size increases of a factor three compared to English. Coverage is never free. → Fertility and parity, the metrics everyone quotes, are not always predictive of downstream performance. Treating them as a proxy is a mistake. The tokenizer is the one decision you cannot revise after pretraining. Get it wrong and you pay the difference on every token, for the life of the model. Read the tokenization chapter: https://ai-engineer-roadmap.xyz/token-economics

#Tokenization #LLMTraining #MultilingualNLP #ComputeCost #TokenEconomics

Self-hosted inference on consumer GPUs costs $0.001-0.04 per million tokens in electricity. That is 40-200x cheaper than budget-tier cloud APIs. "Private LLM Inference on Consumer Blackwell GPUs" benchmarked four open-weight models (Qwen3-8B, Gemma3-12B, Gemma3-27B, GPT-OSS-20B) across 79 configurations — quantization formats, context lengths from 8k to 64k, and three real workloads: RAG, multi-LoRA agentic serving, and high-concurrency APIs. The hardware pays for itself in under four months at 30M tokens/day. • Bigger is not automatically better per dollar. The RTX 5090 delivers 3.5-4.6x higher throughput than the 5060 Ti and 21x lower latency for RAG — but the budget GPUs win on throughput-per-dollar for API workloads with sub-second latency. • Quantization is nearly free money. NVFP4 gives 1.6x throughput over BF16 with 41% energy reduction and only 2-4% quality loss. • The exception matters: latency-critical long-context RAG still needs the high-end card. If your workload is steady and your data is sensitive, the cloud API is a subscription to someone else's margin. Read the full cost breakdown: https://ai-engineer-roadmap.xyz/token-economics

#FinOps #InferenceCost #SelfHosting #Quantization #LLMOps

The same set of requests costs 179 times more energy in Pashto than in English. The Language-Energy Divide measures inference energy across languages and finds the disparity is structural. Energy consumption per output token varies by up to 8.3 times. Total energy for a fixed set of requests runs from 17.6 kJ for the cheapest language, English, to 3,147 kJ for the most expensive, Pashto. Two factors compound. Complex or rare scripts cost more per token, and low-resource languages generate more tokens to say the same thing. Your tokenizer is a pricing function, and it is not neutral. Takeaways: • There is a double penalty. Languages with the highest energy footprints also tend to achieve the lowest task accuracy, so you pay more and get less. • The divide persists across models, hardware and tasks. It is systemic inequity, not an artifact of one deployment. • Per-token pricing quietly exports this cost to whoever happens to speak the wrong language. The recommendation is blunt: treat energy as a first-class evaluation axis, put it in reporting checklists and model cards, and adopt deployment-side mitigations. More on what a token really costs: https://ai-engineer-roadmap.xyz/token-economics

#GreenAI #Tokenizers #MultilingualNLP #InferenceCost #SustainableAI

Frontier LLMs run 1.5 to 2.5 times slower than the boring method they were supposed to replace. SimulCost benchmarks the cost that token-economics papers keep ignoring: tool-use cost. Running a physics simulation is expensive, so an agent that guesses bad parameters burns compute far beyond its token bill, and metrics like pass@k become impractical under a real budget. The benchmark spans 2,947 single-round and 1,931 multi-round tuning tasks across 13 simulators drawn from fluid dynamics, solid mechanics and plasma physics. Frontier models reach only 46-65% success on a single-round initial guess, dropping to 35-55% under high accuracy requirements. Trial-and-error across rounds lifts that to 72-81%, but at 1.5-2.5x the cost of traditional scanning. Takeaways: • Uneconomical is a legitimate benchmark verdict. Sometimes the finding is that the agent should not ship yet. • If your agent calls expensive tools, tokens are the cheap part of the bill. • Cost-aware agent design starts by measuring the tool, not the model. Ask what your agent's actions cost, not only what its thoughts cost. More: https://ai-engineer-roadmap.xyz/token-economics

#AgenticAI #ScientificComputing #Benchmarking #CostAwareAI #LLMEvaluation

Three of ten code models babble, and babbling is a line item on your energy bill. Towards Green AI splits LLM inference into its two phases, prefill and decoding, then measures energy across six 6B-7B and four 3B-4B models on HumanEval for generation and LongBench for understanding. Two findings should change how you buy tokens. First, the phases are coupled. Increases in prefill cost amplify the energy cost per token during decoding, by 1.3% to 51.8% depending on the model. Your prompt does not only cost you at prefill. It makes every generated token more expensive. Second, some models simply pad. Suppressing that babbling behaviour saved 44% to 89% of generation energy without affecting generation accuracy. Takeaways: • Decoding dominates energy consumption, so bounding output length is the first move, not the last. • Prompt bloat is a multiplier, not an addend. That is the counterintuitive part. • Choose models by tokens-to-answer, not by leaderboard position alone. Energy and cost are the same story told in different units, and one of them is easier to ignore. More: https://ai-engineer-roadmap.xyz/token-economics

#GreenAI #CodeLLMs #InferenceOptimization #SustainableAI #DeveloperTools

Your inference cluster is a grid asset, and it is being billed as if it were a blob. From Tokens to Energy Flexibility asks what a data centre can actually offer the power grid when its load is LLM inference. Conventional energy management shifts workloads in time and space and treats inference demand as one aggregate load, which throws away everything specific about how LLMs are served. This work opens the blob. Each model-and-quantization configuration maps to a compact set of dispatchable parameters, so numeric precision becomes a lever the grid can pull. The framework schedules model instance switching, request routing and precision selection together, co-optimized across campuses against electricity and carbon signals. Result: total data-centre operating cost falls by 34.3% without curtailing served token volume. Not fewer tokens. Cheaper ones. Takeaways: • Quantization is normally sold as a latency trick. It is also an energy-flexibility instrument. • Serving cost is not fixed per token. It depends on when you serve, and at what precision. • Grid and carbon signals belong in the serving stack, not only in the sustainability report. More: https://ai-engineer-roadmap.xyz/token-economics

#Quantization #DataCenters #GreenAI #LLMServing #EnergyEfficiency

GPT-5 achieves less than 75% exact match rate on the hardest cost-optimal planning tasks. Your agent completes the task. It just doesn't care what it spent. CostBench evaluates something benchmarks usually ignore: not whether an agent finishes, but whether it finishes cheaply. Tasks in a travel-planning domain are solvable via multiple sequences of atomic and composite tools, each with a customizable cost — so there is always a cheap path and an expensive one. The results are uncomfortable. • Even in static settings, agents frequently fail to identify the cost-optimal solution. Leading proprietary models included. • Under dynamic conditions — CostBench injects four types of blocking events like tool failures and cost changes — performance drops by around 40% further. Agents cannot replan economically when the world moves. • Task completion and economic rationality are separate capabilities. Optimizing one has not delivered the other. Every agent framework demo shows a task getting done. Almost none show the token bill it took to get there. That gap is where FinOps for agents actually lives. A benchmark score with no denominator is not a result. See the cost-aware evaluation papers: https://ai-engineer-roadmap.xyz/token-economics

#FinOps #InferenceCost #AgentEvaluation #LLMOps #ToolUse

The most energy-intensive models exceed 29 Wh per long prompt. That is over 65 times the most efficient systems — for the same job. "How Hungry is AI?" benchmarked energy, water and carbon across 30 state-of-the-art models in commercial datacenters, combining public API performance data with company-specific environmental multipliers and inferred hardware configurations. The scary number is not the big one. It is the small one. A single short query costs 0.42 Wh. Harmless. Scale it to 700M queries/day and it aggregates to annual electricity comparable to 35,000 U.S. homes, evaporative freshwater equal to the annual drinking needs of 1.2M people, and carbon emissions requiring a Chicago-sized forest to offset. • A 65x spread between models means model selection is an environmental decision, not just a latency one. • Per-query footprints look negligible precisely because they are amortized across a population. Multiply before you dismiss. • The paper ranks models by performance relative to environmental cost using cross-efficiency DEA — a denominator, finally. The paradox is the point: as AI gets cheaper and faster, adoption drives disproportionate consumption. More on measuring the real bill: https://ai-engineer-roadmap.xyz/token-economics

#FinOps #InferenceCost #GreenAI #SustainableComputing #LLMOps

Your model leaderboard is probably wrong, and that makes it a pricing problem. Do Repetitions Matter? re-evaluates eight state-of-the-art models on the AI4Math Benchmark with three independent runs per setting, then asks how much of a single-run ranking survives contact with statistics. The answer: 10 of 12 slices, 83%, invert at least one pairwise rank relative to the three-run majority. That ranking is how you picked your model. And the model you pick determines every token you buy afterwards. Takeaways: • Averaging runs shrinks standard error only modestly, around 5% going from one run to three, but the ranking gains are large. Two runs remove about 83% of single-run inversions. • Treat evaluation as an experiment. Report uncertainty, and use at least 2 repetitions under stochastic decoding. • An evaluation too cheap to be reliable is the most expensive line item you own, because it misroutes everything downstream of it. Pay for the second run. It costs less than a year of serving traffic to the wrong model, and the guidance here stays feasible for small teams. More: https://ai-engineer-roadmap.xyz/token-economics

#LLMEvaluation #Benchmarking #MLOps #Statistics #ModelSelection