LlamaIndex — The Complete Guide

LlamaIndex is a system that builds a guide page by reading the actual source code. It never answers from memory. Instead, it first splits the code into small chunks. Each chunk gets a fingerprint, called an embedding. These embeddings go into a cabinet, or vector index. When you ask a question, the system pulls only the few chunks that match. Then it reads those chunks to produce an answer. This is retrieval augmented generation. The whole point is grounding: every claim traces back to real text, not a model’s guess.

The trade-off is faithfulness versus ease. Fine-tuning a model on old code would give stale answers. Stuffing entire files into the context would drown the model in irrelevant text. The chosen approach retrieves only relevant pieces, answers from them, then verifies every identifier. If any symbol does not appear in the real source, it is flagged and the system retries. This ensures the page stays correct without manual updates.

Every part of the memory principles page is generated this way. It uses a code aware splitter that keeps functions intact. That avoids fragmenting important names across different chunks. Then it uses a grounding check to reject any made up code. Only excerpts whose identifiers match the real source survive. This makes the guide self updating whenever the code changes.

So LlamaIndex is not a model remembering facts. It is a pipeline that reads the on disk source at generation time. It splits, embeds, indexes, retrieves, and verifies. The result is a page that is faithful to the current code. That is why the memory principles page is not hand typed but generated by this pipeline. Every sentence is grounded in the actual source, not asserted from memory.

The pipeline starts with raw source files. It chops each file into small overlapping chunks. For regular text, it uses a sentence splitter. That splitter cuts by word count. It overlaps the chunks so nothing gets lost.

Next, each chunk goes through a local fast embedding model. That model turns the words into a numeric fingerprint called an embedding. The pipeline saves those embeddings in an index. The index stores the embedding, the original text, and some details about where it came from.

Once the index is built, it becomes searchable. When someone asks a question, the system finds the most relevant chunks by comparing their embeddings. That lets it pull the right pieces and build an answer grounded in the real code.

The design balances faithfulness against ease of generation. Splitting into overlapping chunks ensures context isn’t cut off. Embedding with a local model is fast and works offline. Storing everything in an index makes retrieval instant.

This whole process happens fresh each time. The index is rebuilt from the current source every run. That keeps the answers up to date without manual effort.

So the raw materials get split, encoded, and filed. Then the pipeline can quickly pull the needed pieces. It’s like a librarian who prepares a card catalog before anyone asks a question.

When you ask a question, the system first looks for the most relevant chunks of code. It retrieves the top twelve chunks by measuring how similar each one is to your question. This is the first stage: broad recall. Then a second stage applies a filter. Any chunk with a low similarity score below zero point one five gets dropped. That removes obvious noise, like chunks about imports when you asked about retrieval. After that filter, the remaining chunks are reordered. The most relevant ones are moved to the very beginning and the very end of the list. This takes advantage of how large language models pay more attention to the first and last items in a long context. The trade-off is precision versus position. You cannot reorder before filtering, because a noisy chunk would get a prime spot. And you cannot filter after reorder, because the cut would discard chunks you had deliberately placed. The similarity floor is set intentionally low at zero point one five. That is well below the typical relevance band, so it strips clear noise without starving the query down to nothing. Then the reordering exploits what is called the lost in the middle phenomenon. The result is a small, high-signal context for the writer. The system retrieves widely, then precisely selects and positions the best evidence. This two-stage process keeps the answer faithful to the real source code. It avoids drowning the model in irrelevant text. It also avoids relying on memory, which could be stale. Instead, the evidence is always fresh from the actual codebase. The whole pipeline runs at generation time, so every fact traces back to real identifiers. That is why the guide pages stay correct without manual updates.

A chapter is written and checked through a careful pipeline that keeps every statement true to the real code. First, the system reads the actual source files. It splits each file into small chunks. Then it turns every chunk into a unique fingerprint, called an embedding. These fingerprints are stored in an index for fast searching. When a new question arrives, the system does not guess from memory. Instead, it searches the index to find the few chunks that best match the question. It retrieves only those chunks. Then a large language model uses them to produce an answer. The model never sees the full source, only the relevant pieces. This keeps the answer focused and accurate.

But accuracy is not enough. The system also runs a verification step. It checks every code excerpt against the real source. It looks at each name in the excerpt and makes sure it appears as a whole word in the actual code. If too many names are missing, the system rejects the excerpt. It then asks the model to try again, pointing out which symbols are ungrounded. It will retry up to three times. If violations remain after that, it prints a warning instead of shipping a false example. This safety net ensures no made-up code ever appears in the chapter.

Once the text is written, it must pass through an audio gate before it can be spoken. This gate enforces rules for listenability. The narration must have a minimum readability score of fifty. That means sentences should be fairly easy to understand when heard only once. Long sentences and dense numbers are penalized. The system caps sentence length. It also avoids code punctuation and symbols. The goal is to create a natural, conversational voice. The audio gate makes sure every chapter sounds good when read aloud.

The whole design balances two goals: staying faithful to the source and being easy to generate. A fine-tuned model would be fast but could not keep up with changing code. A long context prompt would drown the model in irrelevant text. This pipeline chooses a middle path. It retrieves only what it needs, verifies every detail, and polishes the prose for the ear. The result is a chapter that is both accurate and pleasant to listen to. The verification step uses a strict check. It does not accept partial matches. For example, a fake function name would not count even if the real source has a similar name. The check demands exact whole-word matches. This prevents false claims. All identifiers in the code excerpt must trace back to the live source files. The pipeline also splits code files by function boundaries, so each retrieved chunk is a complete unit. This makes the answers more reliable. The audio gate then applies a readability floor and sentence length caps. It also enforces a conversational tone. Listeners cannot re-read, so the narration must stay clear and natural. Every step works together to produce a chapter that is truthful to the code and easy to follow by ear.

Here is the imageability and dual coding principle. Concrete picturable material gets stored twice. Once as words and once as a mental image. That gives you two independent routes to recall the same item. The effect is large and among the most replicated in memory research. You can measure how picturable something is automatically. A system can score each piece of content. For a candidate cue you feed it to a text to image model. Then you score how good and how consistent the pictures it produces are. That checks the whole scene, not just individual words. This helps even for abstract ideas where good cues are harder to find. For abstract content that cannot be pictured you only get the verbal route. So the second route does not help there. The machine learning equivalent is multimodal joint embeddings. A model called Contrastive Language Image Pre training, or CLIP for short, encodes content through two parallel channels. One channel for images and one for text. They line up in a shared space. That stores a concept as a mutually reinforcing picture and word.

The principle is well established. When you hear a concrete word like apple you form both the sound and a picture. Abstract words like justice form only the sound. Two independent routes mean two chances to retrieve. The effect is one of the most replicated findings in all of memory science. How picturable something is can be scored automatically. That is what the text to image model does. It generates pictures from the cue and then you evaluate them. Good cues produce consistent high quality images. That checks the whole concept, not just the words. This method works for concrete and abstract content alike because you test the whole scene.

For abstract items you only have the verbal route. The second route does not come in. That is the boundary. The fusion of two routes is mirrored in a pipeline stage called hybrid retrieval. Hybrid retrieval fuses a dense embedding search with a keyword search. A chunk can be found two independent ways. That echoes the dual coding idea. Two independent paths to the same piece of information. It gives you a higher chance of finding it when you need it.

The bizarreness effect explains why strange things stick in memory. A dog riding a bicycle is remembered better than a dog chasing one. The reason has to do with how our brains process unusual information. At the moment of encoding, bizarre items get more elaborative processing. They stand out as distinctive. At retrieval, they face less interference from other items. This makes them easier to recall. But there is a crucial catch. The advantage shows up almost only when the bizarre item is rare among ordinary ones. Make everything bizarre and the benefit collapses. Bizarreness is relative, not absolute. Also the effect mainly helps free recall. It can hurt associative binding if the bizarre element feels disconnected from the rest. And it weakens under cued recall. So the boundary condition is clear: bizarreness must be rare in its context.

This principle rhymes with a search technique called keyword exact-match retrieval. That method weights rare terms most heavily. It mirrors how distinctiveness reduces interference and surfaces unusual cues.

The isolation effect is a powerful memory principle. It was discovered in nineteen thirty three. An item that differs from its neighbors in color, size, or category is recalled much better. Memory rewards difference. This is not just a clever trick. It is a general organizing principle for how we encode and retrieve information. The reason is that our brain treats difference as a signal. It says this item is special, pay attention. That attention helps lock the memory in. But the benefit depends on context. Distinctiveness is relative to the set of items around the target. If everything is unique, nothing stands out anymore. The effect disappears when the whole set is made of oddballs. Consider what happens when a language model generates many cues at once. It often drifts toward stock imagery. Every item becomes similar. The benefit of difference is lost. To keep the effect strong, you must ensure that each cue is different from its neighbors. A diversity penalty across the batch is a cheap way to enforce that.

Now the pipeline stage that mirrors this principle is cross-encoder reranking. That stage scores each candidate chunk against the question. Then it lifts the one that stands out from its neighbors in the ranking. It is a natural place to apply a set-aware diversity penalty. This directly enforces the isolation effect in the generation system.

Memory is shaped by how deeply you engage with information. Thinking about what a word means works far better than thinking about how it sounds. And thinking about how it sounds beats just looking at its shape. But depth alone is not enough. The elaboration must hang together in a way that makes the original idea recoverable from the cue. Rich but disconnected detail offers no help at all.

Research shows that self-explanation deepens understanding for conceptual material. Chi and colleagues demonstrated this effect. Pressley and colleagues found that asking why something is true improves memory for facts. These studies illustrate the value of deep semantic processing. When you process information at a meaningful level, memory becomes stronger and more durable. The surface features like sound or appearance do not create lasting recall.

There is a crucial boundary. Elaboration only helps when it makes the meaning recoverable from the cue. If you elaborate on the wrong thing, you gain nothing. The connection between the cue and the target must be clear and coherent. Vague or scattered associations do not work.

The machine learning world has a similar approach. Models like BERT are trained to recover a hidden word from its surrounding context. That task forces deep semantic encoding. You cannot solve it by looking at surface features alone. This is exactly the same reason deep processing builds more durable memories.

In practice, you can test whether a cue is good. Show a second model only the scene and ask it to guess the meaning. If it cannot, the cue has failed. A good cue makes the target predictable from the context.

The system's index works on this same principle. It stores meaning, not just the surface string. When you retrieve information, matching is based on semantic content, not exact word matches. That is why the index applies the mechanism of deep processing directly. Each indexed representation embeds meaning that can be derived from the stored cue.

Interactive imagery is the principle of forming one scene where two items interact. In the original keyword studies, the instruction was very specific. You picture the keyword and its meaning doing something together. Not a separate image for each. They must be bound into a single scene. Motion almost forces this. Things colliding, chasing, or balancing on each other all create interaction. Interaction matters even more than bizarreness. A piano smoking a cigar beats a bizarre but separate piano and cigar. The boundary is clear. The interaction must bind the items. A scene where they merely sit side by side is not enough.

In the machine learning analog, a relation network fuses each pair of entities through a shared network. It stores them as one joint representation. This is the same idea. In the generation pipeline, a language model picks the keyword and writes the verbal cue. Then a text-to-image model draws the scene. The interaction rule is enforced on the words before any picture is made. A simple check is whether the keyword and the meaning share a verb.

This is exactly the same principle as the model behind the engine. That model must hold the question and the retrieved evidence in one integrated scene to answer from them. It cannot keep them separate.

The generation effect is a learning principle. Material you produce yourself is remembered better than material you are simply handed. Even a trivial act works. For instance, filling in a missing word rather than reading the full phrase. The effect is moderate but robust. It extends to mnemonics you make yourself versus ones you are given. There is an important twist, though. A good cue from an expert source often beats a poor cue a beginner generates. Quality and ownership trade off. The advantage only holds if you actually produce the material correctly. If you fail or produce a poor cue, you lose that benefit. Provided mnemonics from a good source often beat self-generated ones from novices. That is because novices produce low-quality cues. This creates a trade-off between quality and ownership. The generation effect is grouped under encoding. Encoding is how information is first learned. The machine learning analog is self-generated training data. In the STaR method, a model is fine-tuned on the correct reasoning it produced itself. Its own output becomes its training signal. That is the generation effect turned into a bootstrap loop. In generation systems, the resolution is to supply a high-quality cue. Then have the learner do something generative with it. They can edit a word. They can pick between two variants. Or they can spend three seconds visualizing it. The system supplies the quality floor. The learner supplies the generation.

The pipeline stage from index to page applies this principle. It prompts users to generate or refine their own search terms before retrieving results. This leverages the memory benefit of self-production with minimal generative involvement.

Emotional arousal and humor can strengthen your memory, but only under the right conditions. The mechanism works through consolidation priority. Arousing content gets special treatment in the brain. The amygdala modulates how the hippocampus encodes that information. The remarkable part is that the advantage actually grows over a delay. Most memory advantages fade, but this one gets stronger with time. That matters because what a learning product cares about is retention days later, not recall in the moment. The growth over a delay is a striking feature. It means the effect is not just a short-term boost. It deepens with time, which is exactly what long-term retention needs.

The safe channels into arousal are humor and surprise. However, there is no evidence that sexual or gory content outperforms humorous or surprising content for delayed verbal-associative learning. The key boundary condition is that the humorous element must be directly about the thing you need to remember. If the joke is about something else, it steals attention away from the very link you want to strengthen. So arousal sharpens memory for the central detail while degrading memory for the periphery. It is a trade, not a free gain. The delay advantage only works when the arousal is tied to the target. If it is not, you lose the benefit.

This principle belongs to the encoding group of learning science. In the product, it is applied during a stage called relevance scoring and a similarity floor. The relevance scoring stage assigns a score to each candidate chunk. The similarity floor then drops any chunk with a score below a threshold. This removes low-quality noise without starving the query. It ensures only the most central content passes through. That keeps the humor tied to the link and optimizes for the delayed retention advantage. The stage mirrors the trade-off of emotional arousal: keep the central detail, drop the periphery. The threshold is set low enough to retain useful content but high enough to remove clear noise. Without this floor, narrow queries would feed the system low-similarity information. The result would be diluted and less faithful.

The spacing effect is one of the largest known effects in learning. Spreading practice across time beats cramming it all at once. This finding comes from research going back to Ebbinghaus in 1885. A meta analysis of over eight hundred comparisons confirmed it. The effect works at every age and for every kind of material. Here is why it works so well. When you space practice, each repetition catches the memory just as it starts to fade. That struggle to recall strengthens the memory more than an easy, immediate review. The right gap between sessions is not random. It scales with how long you need to remember. For a test in one year, the ideal gap is about ten to twenty percent of that year. Shorter gaps waste time because the memory is still fresh. Longer gaps cause the memory to drop so low that recall fails. The schedule matters more than the cue. A better mnemonic helps a little, but spacing versus massing dwarfs that improvement. The boundary condition is simple. If the gap is too short, you get massed practice, which is worse. If the gap is too long, the memory is gone, and no learning happens. That is why expanding gaps work so well. Start with a short gap, then widen each time. This approach, supported by work from Cepeda and others, gives the best durability.

This principle rhymes with a specific stage in the pipeline. That stage splits a source into separated units instead of one undivided block. Doing so applies distributed practice to the material itself. It ensures that each part gets its own spaced review, not a single massed exposure.

Retrieval practice strengthens memory in a way that rereading does not. The effect is also called the testing effect. It was established in a classic study. In that study, students who practiced retrieving a passage remembered about fifty percent more than students who restudied it. That difference held up a week later. Importantly, the students who restudied felt they had learned more at the time. So there is a gap between what feels productive and what is actually productive. That gap is an illusion the design must fight.

Pulling a memory out effortfully strengthens it. Simply rereading does not create the same benefit. The retrieval must be effortful but successful. It feels harder than rereading. That is why learners consistently misjudge it as less effective.

In the generation system, this principle rhymes with a process called asking the index. That process retrieves a specific fact from memory rather than rereading the entire source. The system generates questions automatically to test in the direction of real use.

Desirable difficulties is a principle from memory science. Robert Bjork established it. He grouped spacing, testing, interleaving, and generation under this idea. These conditions slow how fast you learn at first. But they actually deepen long-term retention. The reason is effortful processing. Effortful processing builds storage strength. Storage strength is distinct from retrieval strength. Retrieval strength is how easy it feels right now. It fades quickly. Storage strength is durable. It grows with each difficult retrieval. When you struggle to recall something, you build durable memory. But there is a sharp boundary. A difficulty is only desirable if the learner can actually overcome it. If it is too hard, you get failed retrievals, not learning. Frustration follows. This is the crucial caveat Bjork himself stresses. Beginners need easier challenges. Excessive difficulty causes failure. The principle applies to many techniques. Spacing out practice sessions makes retrieval harder. That difficulty builds storage strength. Testing yourself forces effortful recall. Interleaving different topics forces you to tell them apart. Generation means creating your own answers instead of reading them. All these slow your apparent learning. But they enhance long-term retention. The mechanism is effortful processing. That processing strengthens the memory trace. Storage strength becomes more permanent. Retrieval strength is temporary. Effort transfers knowledge from temporary to permanent. In machine learning, there is an analog called curriculum learning. You order training examples from easy to hard. A model then converges faster and generalizes better. Difficulty becomes a schedulable variable. It is desirable exactly when the model can keep up. In generation systems, difficulty is now a tunable knob. Multi-agent frameworks create items at a targeted difficulty. They hold each learner near the edge of what they can just manage. Scaffolding is rich early on. As items mature, scaffolding is stripped away. This keeps the challenge just right. Now for the pipeline stage. In a retrieval pipeline, the verification step adds a hard check. It does not accept the first easy match. Instead it demands effortful retrieval or reasoning. That strengthens memory storage. But the verification challenge must remain surmountable for the learner's current level.

Chunking is the principle that working memory holds about four meaningful units, not raw pieces of information. Miller and Cowan established this idea decades ago. The key mechanism is recoding. When you take many small bits and group them into one larger meaningful chunk, you multiply your effective capacity. The classic evidence comes from expert memory. One runner pushed his digit span from seven to around eighty by recoding strings of digits into running times he already knew well. That is a huge leap. But there is a boundary condition. Recoding only works when the chunks are meaningful to that learner. An expert's chunk looks like noise to a novice. The famous limit of four chunks is not a limit on total information. It is a limit on the number of independently retrievable units. So the same raw material can hold vastly more information if you know how to group it.

Now, how does this rhyme with the pipeline stage? The system turns whole documents into smaller chunks. It splits prose by sentence boundaries and code by function boundaries. This makes each chunk small enough for the retrieval system and the language model to hold in memory. Without this step, each file would stay a single monolithic piece. That would make fine-grained retrieval impossible and degrade the quality of explanations. The same principle applies: break large texts into manageable units to fit working memory constraints and enable efficient processing.

The trade off is between chunk size and meaning. Too small and you lose context. Too large and the chunk exceeds what working memory can handle. The splitters use configurable chunk sizes and overlaps to balance these needs. They rely on natural boundaries like sentence ends or code function boundaries to keep chunks coherent. That coherence is what makes each chunk a meaningful unit. It is the same kind of recoding that the runner used. The system recodes a stream of raw text into reusable chunks. Each chunk then carries more useful information than a random slice of text.

In short, chunking multiplies capacity by packaging raw information into meaningful groups. Miller and Cowan showed the limit is about four chunks, not four items. The runner's digit span example proves it. The pipeline mirrors this by splitting documents into chunks that are both retrievable and meaningful. The boundary condition remains: the chunk must mean something to the learner. For the system, that means splitting at semantic boundaries so the language model can treat each chunk as a coherent fact. Without that meaningful grouping, the system would drown in noise. With it, it works efficiently.

The method of loci is the oldest memory technique on record. It was traced to Simonides around five hundred before the common era. The technique parasitizes your spatial memory. Spatial memory is evolutionarily old and has very high capacity. It comes with ordering and cues built in. You just walk the route again. Each location automatically provides a unique cue for what you placed there. This technique remains one of the most powerful ways to remember ordered material. It is inherently a scene with a person doing something absurd at a place. Six weeks of training in ordinary novices durably improved recall. Their brain connectivity shifted toward the pattern seen in memory athletes. A generated memory palace in virtual reality places an object at each spot. This was tested against random pairing at one week recall. One caution is that the locations must be your own. They must be your apartment or your commute. Without familiar locations the built in cues are lost. The locations must stay in a fixed order. Building that palace from your own places is the real cost. To use it, imagine a familiar route. Then place each item at a landmark along that route. Later, mentally walk the route and see each item at its spot. The machine learning analog is addressable external memory. A differentiable neural computer writes items to memory locations. It keeps a record of the order they were written in. Then it can traverse them in sequence. This is the machine version of storing things along a route and walking it back. The computer can return to any location it wrote to before. That gives it a stable addressable place for each chunk. The method of loci relies on associating items with familiar places. Each place becomes a retrieval cue. When you think of the place, you remember the item. The order of places gives you the order of items. That is why it is so good for lists. The six week training study showed brain changes. The changes in connectivity were durable. The virtual reality test used a generated environment. Objects were placed at specific locations along a route. The test measured recall at one week. Random pairing meant objects were not tied to places. This demonstrates the method can be implemented in virtual reality. The caution about locations applies to the traditional method. The method of loci is also called the memory palace. It has been used for over two thousand years. It remains popular among memory champions today. The machine analog creates a similar structure in software.

Encoding specificity means a retrieval cue only works if it was part of the encoding. Tulving and Thomson in nineteen seventy-three, and Godden and Baddeley in nineteen seventy-five, demonstrated this principle. Their evidence shows that retrieval succeeds when the cue was encoded with the target memory. Retrieval fails otherwise, even for strong semantic associates. The critical trade-off is that a cue seems memorable but is useless if it was absent at encoding. A boundary condition is important here. If a system regenerates mnemonics between sessions or tests different cue versions, it severs the encoding‑retrieval match. The mnemonic then silently stops working. The pipeline stage that rhymes with this principle embeds the question and the chunk into the same space. A query only finds a chunk when its encoding overlaps how that chunk was encoded.

This is a practice session over six encoding principles. For each principle, you will hear a question that tests its boundary condition. Then you will hear the answer. Then you will learn which pipeline stage it rhymes with.

First principle is imageability and dual coding. Content that can be pictured gets stored twice. Once as words, once as a mental image. Two routes to recall. But what kind of content cannot benefit from dual coding? Abstract content that cannot be pictured. It only has the verbal route. That is the boundary. This principle rhymes with a nearest-neighbor vector space pipeline.

Second principle is the bizarreness effect. Strange cues are remembered better. But there is a catch. When does the bizarreness effect fail? When everything is bizarre. The advantage only shows when the bizarre item is rare among ordinary ones. Make everything bizarre and the benefit collapses. This principle rhymes with prioritized experience replay.

Third principle is desirable difficulties. Conditions that slow learning can help long-term retention. But difficulty is not always good. When is difficulty not desirable? When the learner cannot overcome it. For beginners, too much difficulty causes failed retrievals and frustration. This principle rhymes with a trust but verify stage.

Fourth principle is chunking. Working memory is limited to roughly four chunks. Chunking recodes raw information into meaningful units. What is the limit of working memory that chunking addresses? Four chunks. This principle rhymes with the documents and chunks pipeline stage.

Fifth principle is method of loci. It uses spatial memory to store and retrieve items by walking a mental route. But there is a boundary. What happens if the loci are not personally anchored? The built-in cues are lost. The loci must be user-anchored, like your own apartment or commute. This principle rhymes with a nearest-neighbor vector space pipeline stage.

Sixth principle is encoding specificity and transfer-appropriate processing. The retrieval context should match the encoding context. With practice, what should happen to the mnemonic scaffolding? It should drop away. The learner retrieves meaning directly. This principle rhymes with embeddings without an API pipeline stage.

That concludes the practice session. Each principle has a boundary that separates it from the others. And each has a pipeline stage that applies the same idea.

This round covers the practice principles that make learning feel harder while it works. They share a trap. Your sense of how well you are learning runs backwards. When something feels easy, you are probably not storing it well. When it feels hard, that effort is building lasting strength.

First question. What is desirable difficulties?
Pause.
The answer. Conditions that slow apparent learning actually improve long-term retention. But the difficulty must be something you can overcome. For beginners, excessive difficulty causes failed retrievals and frustration. The trap is real. The pipeline stage that rhymes with this principle is trust but verify. That verification step demands effortful processing.

Second question. What is retrieval practice?
Pause.
The answer. It is the act of pulling information from memory instead of reviewing it passively. The index forces active recall. This builds stronger memory storage. The pipeline stage that rhymes with this one asks the index for direct retrieval. It retrieves stored information from memory on demand.

These two principles share the backward sense of progress. When learning feels slow and hard, that is exactly when the memory is strengthening. The verification step and the retrieval step both put you through that effort. That is why they work.

Let us begin the drill. First principle: chunking. Question: What must be present for chunking to multiply your memory? The answer: meaningful units. Raw information alone does nothing. You must recode it into chunks. Each chunk is a single meaningful unit. Working memory holds only around four of them. But if those chunks are well packed, you carry much more. The boundary condition is simple: without meaningful recoding, chunking fails. The pipeline stage that rhymes with this is documents and chunks. It splits sources into small nodes. That is what the sentence splitter and code splitter do.

Second principle: method of loci. Question: What is the one requirement for this technique to work? The answer: a route you already know well. The method parasitizes spatial memory. That memory is old and high capacity. But the built in cues only work if the locations are user anchored. If you do not have a familiar route, the effect collapses. The pipeline stage that rhymes with this is nearest neighbor vector space. It uses spatial proximity in embedding space. That mimics the route based cues of a real walk.

Third principle: encoding specificity. Question: What must be present for encoding specificity to give you an advantage? The answer: a cue that was there at the moment of encoding. The principle says the retrieval context should match the original learning context. That is called transfer appropriate processing. The boundary condition is that the learner must eventually retrieve meaning directly. But the cue at encoding is the key. The pipeline stage that rhymes with this is embeddings without an api. It stores information in a vector index. That index is queried by the same embedding model that encoded the content. So the cue at retrieval matches the cue at storage.

Now a quick recap on the whole set. Chunking requires recoding into meaningful units. Method of loci requires a known route. Encoding specificity requires a matching cue from encoding time. Without those, the principle does not work. With them, memory improves. The pipeline uses these same ideas to keep its answers grounded. It splits chunks, searches by spatial proximity, and forces active recall. That is how it stays truthful to the source code.

Here is the guide from beginning to end. It starts with raw source files. The system splits each file into small overlapping chunks. This step uses the chunking principle. Chunking means working memory can hold about four meaningful units. Breaking a large file into small chunks multiplies your capacity. It makes each piece easy to process later.

Next, each chunk gets a fingerprint called an embedding. That creates a searchable index. This step connects to imageability and dual coding. The principle says concrete picturable material gets stored twice. Once as words and once as a mental image. Two routes give two chances to recall. The embedding acts like a mental image for the text.

Now retrieval happens. When a question arrives, the system searches the index. It finds the few chunks that best match. This step rhymes with the method of loci. The method uses spatial memory to recall items in order. Walking a route gives built in cues. The vector space mimics that route.

Then grounding takes place. The system verifies every code excerpt against the real source. It checks each identifier for an exact whole word match. This is the trust but verify stage. It relates to desirable difficulties. The principle holds that effortful processing strengthens long term retention. But the difficulty must be surmountable. The verification forces careful checking.

Finally, an audio gate polishes the text for the ear. It enforces a readability floor. It caps sentence length. It removes code punctuation and symbols. The goal is a natural conversational voice.

So what does the whole guide claim? A retrieval pipeline and a remembering mind face the same problem. They must get the right thing back at the right moment. That is why the same ideas keep appearing on both sides.

The simplest search setup uses a vector store index and a query engine. Under the hood, it converts your query into an embedding. That embedding is a list of numbers that captures the meaning of your words. The engine then finds the closest matching chunks in the stored index by measuring cosine similarity. This is fast and requires no extra steps. For many collections of text, it is genuinely enough.

But this simplicity hides a trade-off. Embeddings capture topic, not exact tokens. If you search for an error code like err underscore conn underscore reset, the embedding treats it as some error-ish thing. It finds chunks about errors, but not the specific string you need. Cosine similarity happily returns twenty neighbors that are about the right subject and none of them about the right string. So dense retrieval fails on rare, exact identifiers.

That is why this setup matters as a baseline. Every later improvement, like hybrid search or reranking, must beat the plain vector score. Measurements like hit rate and mean reciprocal rank are compared against this starting point. The entire justification for adding more steps is a measurable lift over this simple method. If the baseline already works well, you may not need anything else.

The vector store plus query engine is the bottom rung of the ladder. It is what you build on top of. Everything downstream inherits whatever this stage hands it. So getting this first step right is crucial. You can later add hybrid search, which combines dense and lexical retrieval. You can add reranking to reorder the results. But you always come back to this baseline.

A common mistake is to blame the index when retrieval fails. Often the index is fine, and the question itself is broken. Simple vector search can still fail if the question uses different vocabulary than the corpus. But for many corpora, especially those with clear topic structure, the simple setup works well. It is fast, cheap, and easy to set up.

When you evaluate your retrieval, you compare against this baseline. Build your evaluation set first. Measure retrieval in isolation. Then you can see if an extra hop actually improves hit rate and mean reciprocal rank. If it does not, you saved yourself the complexity. If it does, you know the upgrade is worth it.

So remember: the vector store index and query engine are your starting point. They give you a clear number to beat. They handle topic matching well. They struggle with exact tokens. That is their strength and their weakness. And that is why you need to know them before you reach for anything fancier.

Overlap in chunking costs you duplicate embeddings. This overlap is a hedge, not a feature. It exists so a fact that straddles a boundary appears intact in at least one chunk. But you pay for duplicate embeddings and duplicate hits. When you re-ingest the same corpus, you want to avoid that cost if nothing has changed. One solution is the hierarchical node parser. It produces every chunk size at once: two thousand forty eight, five hundred twelve, and one hundred twenty eight. That saves you from re-chunking and re-embedding the same text multiple times. The trade-off is the upfront decision on chunk sizes. But that decision is a one-time cost. The real answer to the small versus large tension is to stop choosing. The hierarchical node parser gives you both at once. So you do not need to re-ingest for different chunk sizes. You pay the planning upfront, but you never again pay to embed text you already have when it has not changed. The semantic splitter is another tool. It embeds every sentence at ingest time. That cost is real. If the text is unchanged, you would pay that cost again. So the bookkeeping is the price you pay upfront. You decide on chunk sizes and overlap once. Then you can skip re-embedding the unchanged text. That is the trade-off: a planning investment for a permanent gain in ingest cost.

The way you split your text into pieces is the most common reason retrieval fails. To handle a fact that falls across a break, the system uses overlapping chunks. The overlap is a hedge, not a feature. The shipped defaults are one thousand twenty four characters per chunk with a two hundred character overlap. That works fine for normal prose. But for short FAQ entries you should use chunks of two hundred fifty six to five hundred twelve characters instead.

For structured text, a parser that reads headings is much better. The Markdown Node Parser splits on headers and stamps the header path into metadata. The Code Splitter uses a grammar tool called tree sitter to break at function boundaries. That way a function is never cut in half. Free structure beats inferred structure every time.

There is also a semantic splitter that people often reach for. It groups adjacent sentences into windows. Then it embeds every window and cuts wherever the topic shift is large. But it costs a lot. At ingest you embed the entire corpus once at sentence granularity. That is on top of the embedding for each resulting chunk. The chunk boundaries are also non deterministic across embedding model versions. This tool earns its keep on long, unsegmented text like transcripts or legal prose with no headings. On a well headered corpus it buys you nothing that the Markdown Node Parser already gave you for free. Decide whether to use it with an eval on retrieval hit rate, not by vibes.

Now consider what a chunk boundary through the middle of an answer costs you. The fact that straddles the boundary appears intact in at least one chunk because of the overlap. But you pay for it with duplicate embeddings and duplicate hits in your top k search. The real answer to the small versus large chunk tension is to stop choosing. The hierarchical node parser produces every level at once. It wires each node to its parent. You get small, medium, and large chunks all together. That way you never lose the complete answer.

Everything downstream inherits whatever this chunking stage hands it. Grounded citations and structured extraction all depend on it. So fix your chunking first. It is usually the real culprit when retrieval fails, not the embedding model.

Filtering by metadata before similarity search is the only correct way to scope a search to a specific tenant or time range. You place a condition on the metadata first, then run the vector search only on the matching documents. That guarantees you stay inside the right partition. But the filter itself can fail silently. If you make it too narrow, you get zero hits. The system does not raise an error. It simply returns a placeholder that says "Empty Response." No exception, no warning. Callers and agents will treat that as content and move on. That is a quiet breakdown.

A more dangerous failure is a near miss. The filter is correct on the surface, say the right service field but the wrong year. Or the right filter but the wrong tenant partition. The retriever returns a full, plausible list of nodes from the wrong partition. The answer is confidently wrong and includes citations. So you must check the number of nodes you get back. If you get zero, fall back to a more relaxed filter set. Do not trust an empty result. And when you do get nodes, check that their metadata actually match the filter you meant to apply. Do not trust a non-empty list either.

Now, how does the model know which filter to use in the first place? That is where auto-retrieval comes in. The model infers the filter from the question itself. It does not guess. It uses query transformations to rewrite the question before it ever touches the retriever. One method is called Hy D E. It asks the large language model to hallucinate a fake answer first. That fake answer is stylistically similar to real documents. It is embedded, and that embedding lands much closer to the correct chunks. The model never sees the fake answer as truth. It only uses it to bridge the gap between a short question and a long document.

Another method is multi-step querying. Some questions have a hidden chain of dependencies. For example, "What did the author do after the company he founded was acquired?" That question needs a first step: find the company. Then a second step: find what happened after. The multi-step query engine decomposes the question into sub-questions. It looks at the query, the index summary, and the reasoning so far. Then it emits the next sub-query. Step by step, it infers the right scope and filter for each sub-question.

These query transformations handle vocabulary mismatch. The corpus may use different terms than the question. For instance, a user asks "Why did the deploy blow up last night?" The runbooks say "Rollout aborted: readiness probe exceeded initialDelaySeconds." Those two strings are far apart in embedding space. Hy D E closes that gap. The technique costs an extra large language model call during ingestion and retrieval. But it is worth it when the question and answer use different styles.

In short, filtering by metadata before similarity is the only correct way to scope a search. But you need to handle silent failures. And the model can infer the right filter from the question using query transformations. Together, they give you precision and recall without guessing.

Dense retrieval captures topic but not exact tokens. An embedding turns a specific error code like ERR underscore CONN underscore RESET into just some error-ish idea. Cosine similarity happily returns twenty nearby chunks that are all about the wrong subject. None of them contain the right string.

BM twenty-five has the opposite flaw. It nails rare tokens and exact characters. But it misses paraphrases every time. Hybrid search combines both. It is the recognition that lexical and semantic retrieval fail on different queries. The union of their candidates gives far better recall than either method alone.

Now you have two ranked lists with scores that mean nothing to each other. A cosine of point eight three and a BM twenty-five score of fourteen point two are incomparable. Reciprocal rank fusion throws the scores away and keeps only the ranks. Each document scores the sum of one over K plus its rank from each list. A document ranked second by both retrievers beats one ranked first by only one of them.

The fusion retriever’s similarity top K is the recall budget. A document the fusion stage drops can never be recovered. That sets a ceiling on the whole pipeline’s quality. The reranker’s top N is the precision budget. Every node past it is context you pay for and attention you dilute.

Widening the candidate pool is nearly free. An approximate nearest neighbor scan and an inverted index lookup are cheap. But the rerank step is linear in the number of candidates. That is where your latency lives. A local Mini LM cross encoder on forty candidates takes fifty to a hundred fifty milliseconds on a GPU. On a CPU it can hit half a second. A hosted reranker like Cohere adds a network round trip but no local compute.

A cross encoder reorders candidates the bi encoder only roughly ranked. The bi encoder is fast but coarse. The cross encoder is slower but far more precise. You only want to rerank a small set. So retrieve wide to guarantee the answer is in the pool. Then rerank narrow to keep only the most relevant documents. That is the discipline.

Sometimes the problem is not the index but the question itself. The question might use different words than the documents. Or it might bundle several questions into one. Query transformations fix this by rewriting the question before it ever touches the retriever. They cost extra model calls. But they earn that cost when the question is broken.

One technique handles vocabulary mismatch. Instead of embedding the user's question, you ask the language model to imagine a fake answer. That fake answer is then embedded and used to find real documents. The fake answer is factually useless. But it matches the style of real documents, so it lands closer to the correct chunks. This adds one extra language model call. It earns its cost when the question uses different words than the corpus.

Another technique handles sequential dependency. Some questions depend on previous answers. For example, you asked what the author did after the company he founded was acquired. You first need to know which company. A multi-step engine breaks this into sub-questions. It looks at the query, the index summary, and previous reasoning to ask the next sub-question. This adds multiple calls, one per step. It earns its cost when the question has a built-in dependency.

A third technique is sub-question decomposition. It also breaks the question into smaller pieces. Each sub-question gets answered and then combined. Again, extra calls.

When are these transformations just latency? If your question is already well formed, with vocabulary that matches the corpus and no hidden steps, they add delay without benefit. You waste time and tokens.

Even with good question rewriting, a common failure sneaks in. If your filters are too narrow, you get zero results. The system returns a literal string that says empty response. It never calls the language model. So you get a placeholder that looks like content but is not. The real danger is a near miss. You get a full, plausible node list from the wrong partition. That produces a confidently wrong answer with citations. So you must check that your filters actually match, and fall back to a relaxed set when nothing comes back.

Know the trade off. Extra model calls are worth it when the question is the broken part. But they are just added cost when the question is already clear. And no matter how you rewrite, brittle filters can undo everything.

The first genuinely agentic step is a router. A router decides which index or search engine should answer a given question. It works by having a large language model read the description of each available tool. That description must be precise. It must state what data the tool covers. It must also imply what it does not cover. Two tools with vague descriptions make the model flip a coin. And a wrong route is expensive. Consider two types of questions. A summarise the corpus question wants broad topic matches. A find one fact question wants an exact token match. Dense retrieval captures topic but misses rare terms. Lexical retrieval captures rare terms but ignores meaning. So they need different indexes. Hybrid search combines both for better recall. The router can choose to use that hybrid approach. But it must also decide how to merge scores from two incompatible lists. The easiest way is to keep only the ranks. This is called reciprocal rank fusion. A document ranked second by both retrievers beats one ranked first by only one. That gives balanced results. The cost of a wrong route is not just a wrong answer. It is a confidently wrong answer with citations. The model will fabricate a fluent response over irrelevant chunks. That is hard to detect. The router also adds latency. One shot retrieval is one embedding call and one generation. The router loop may call multiple tools before returning. Each call adds cost. So the tool descriptions are the key to reliability. They must clearly mark the boundary of each corpus. They must state the input contract. A question about error codes should not go to a financial report index. A question about 2021 revenue should not go to a technical documentation index. If the filter misses, nothing raises an exception. The system returns an empty response or a plausible but wrong answer. So you must assert that the returned nodes actually match the filter. The router is the first agentic rung because it chooses the path. It does not just answer. It decides which engine can answer best. That decision is only as good as the tool descriptions it reads. That is the trade off: more capability, but more fragility. With clear descriptions and a careful loop, the router can handle diverse questions without you writing specific code for each one. You trust the model to route correctly. But you must evaluate the trajectory, not just the final string. That is the only way to catch a wrong route before it reaches the user.

Instead of a single retrieve and answer, you can hand the query engine to an agent as a tool. The agent itself decides whether to retrieve, how often, and when to stop. This is a different approach from the one shot retrieve then answer.

The LLM picks a tool by reading its description. The tool is a QueryEngineTool that runs the query engine and returns the synthesized answer as an observation. That observation goes back into the message history. Then the LLM decides again. Termination is the model saying nothing further. There is no fixed depth, just a backstop iteration cap.

What does this unlock? The agent can re query with sharper phrasing after a thin first hit. It can compare across two indexes without you writing any comparison code. You can subscribe to ToolCallResult and AgentStream to watch the queries it actually issued. You will be surprised, and that surprise is the whole point of evaluating the trajectory and not just the final string.

But the cost and latency model inverts. One shot RAG is one embedding call plus one generation with a bounded p99. Here, each tool call costs similar. The loop may iterate many times. You pay for multiple retrievals and multiple generations. The production failure is that this baseline has no way to fail. It cannot say I do not know. It returns chunks and confabulates an answer. With no eval harness attached, you find out from a user weeks later.

There are three knobs worth knowing. Return direct on a tool short circuits synthesis. The query engine answer is returned verbatim instead of being paraphrased by the agent. That saves a generation and stops the LLM from quietly rewriting a number. A Context held across calls carries the message history. So a follow up like what about 2022 reuses the tools without replaying the conversation. And run with max iterations bounds the loop. It defaults to twenty. Overrunning it raises an error unless you set early stopping method to generate. That forces one last synthesis from whatever the agent has gathered.

So the trade off is real. You gain flexibility and the ability to refine queries. The agent can adapt and retrieve exactly what it needs. But you also take on a loop and higher cost. Each step costs time and money. The single shot approach is cheap and fast. The agent approach is powerful but expensive. Choose based on the task.

Retrieval gives you chunks of prose, not fields. That is a problem. Downstream systems need typed data, not paragraphs. They need facts with a known shape. So you have to pull that shape out of the chunks. A schema does that. It is like a blueprint. It tells the model exactly what fields to fill. And it constrains what the model can say. The schema is the prompt. It is the instruction. Without it, the model can wander. With it, the output is predictable. That is the trade-off. You trade flexibility for reliability. The trade-off is worth it. Because a system that always returns the same structure can be trusted. A system that returns paragraphs is hard to use.

Now the concrete detail. How do you get that shape? The retrieval stage hands off to structured extraction. Everything downstream inherits what that stage produces. So if the chunks have good metadata, extraction is easier. For example, hierarchical nodes give you parent-child links. Those links help you assemble a full record. But the real work is in the schema. The schema defines what is allowed. It says the output must have a certain type. That type could be a number or a category. The model cannot ignore it. The schema enforces the shape. It is not a suggestion. It is the rule.

You can test whether the extracted data is faithful to the chunks. Faithfulness and relevancy are separate. A high-faithfulness extraction might be wrong if the chunks are off topic. So you need both. The schema also helps with evaluation. You can check if the output matches the schema. That is a simple test. It catches errors early.

But there is a fail case. If the retrieval misses the right chunks, the extraction will be wrong. The schema cannot fix missing data. So you need good retrieval first. That is why chunking matters. The splitter shapes what the retriever returns. If you split badly, you lose context. Then the schema cannot help.

The schema also works like a tool description for an agent. The agent picks a tool by reading its description. That description carries the corpus boundary and the input contract. It tells the model what to expect and what to output. So the schema is not just a format. It is the instruction that shapes the answer.

In summary, pulling typed data from chunks is the goal. The schema is the tool. It constrains the model. It gives you fields instead of paragraphs. That is what downstream systems need. They need structure. They need predictable output. And the source material must support it. Get the retrieval right. Then let the schema do its job. That way you turn messy prose into clean, usable data.

The top rung is evaluation. It makes every other step safe to change. Without it, you can only tell an answer changed. You cannot tell if it improved.

Three judges sit on that rung. First is the faithfulness evaluator. This one checks if the response sticks to the retrieved contexts. It needs no labels. You can run it against live traffic. It is your hallucination detector.

Second is the relevancy evaluator. It asks if the response and the contexts actually address the query. This catches a retrieval bug. The retriever may pull plausible but off topic chunks. Relevancy sees that.

Third is the correctness evaluator. It compares the answer to a reference. It scores from one to five. The default threshold is four point zero. This one requires ground truth. Save it for your golden set.

Faithfulness and relevancy are separate. One number cannot tell you which knob to turn. Two can. Run them both with the batch evaluator. It processes queries concurrently.

And then there is tracing. Scores tell you that a run failed. Spans tell you why. The instrumentation layer captures durations. It shows you the LLM calls and retrievals. Without it, you only see the output.

Now, the gate. A plain baseline RAG has no way to fail. It always returns the top two chunks. It cannot say it does not know. The synthesizer then makes up a fluent answer over those chunks. You only find out something is wrong from a user screenshot. That might be three weeks after launch.

Without an eval harness, you cannot tell if a change is an improvement. You just see a different answer. The three evaluators become the gate. They say whether the answer is faithful, relevant, and correct. They force the system to say when it does not know. That is the top rung. It makes every rung below it safe to adjust.

A document and the nodes it is split into are the two core building blocks of the whole framework. A document is not a subclass of a text node. That is a common mistake from older guides. Ingestion does not turn documents into nodes. Instead, nodes get split into smaller nodes. They keep relationships to their parent nodes.

Each node carries five key pieces of information. First is the text itself. This is what gets embedded and what the language model reads. Second is a stable identifier. Stability matters because if you regenerate random ids each time, syncing becomes a duplicate mess. Third is metadata. This is a simple dictionary. It is the most underused field in the framework. Metadata makes filtering possible. Filtering is what makes similarity search work in a multi-tenant corpus. Fourth are relationships. These link to source, previous, next, parent, and child nodes. They let a postprocessor retrieve a sentence and then hand the language model the paragraph around it. Fifth are the excluded metadata keys. This pair is an escape hatch that nobody reads about until it causes trouble.

Here is the trade-off. By default, a node's metadata is prepended to its text when going into both the embedding model and the language model prompt. That is usually what you want. A chunk with metadata like title Q3 Pricing Policy embeds as a chunk about Q3 pricing. That gives a semantic lift.

But here is the self-inflicted bug. If you stash a file path, an ingestion timestamp, or a long JSON blob on the node, that text also gets embedded. It dilutes the vector with meaningless text. When someone reports that retrieval got worse after adding metadata, this is nearly always why.

The fix is not to drop the metadata. You should exclude it from the embedding while keeping it available for filtering. That is the classic fix.

So remember: the document and node are the foundation. Metadata rides along with each node. And if you let metadata leak into the embedded text, you cause a retrieval bug yourself. Always set excluded metadata keys to keep your embeddings clean. That is a small change that saves a lot of trouble later.

The index you build does not live in just one place. It lives across three separate stores. There is the vector store, which holds the embeddings. There is the document store, which keeps the original nodes. And there is the index store, which tracks the structure. The seam that controls where an index physically lives is a single object called the storage context.

You attach this storage context when you build the index. But here is the trap. Many real vector databases automatically store the text of each node. When you use one, the storage context does not write the nodes to the document store. The document store stays empty. To force the nodes into the document store, you need to set a special flag to true. Without that flag, your document store has nothing.

Now persistence becomes a problem. If you try to persist the index, only the vector store gets saved. The document store and the index store are missing. So persistence fails silently. Multi tenancy also depends on this same seam. Metadata on each node lets you filter by tenant. But if that metadata is not stored in the document store, you cannot recover the full node later. The metadata is also prepended to the node text for embedding. That can dilute the vector with irrelevant information. You have to exclude certain keys to avoid that.

So the seam that decides the physical location also decides whether your metadata works correctly. Every time you switch from a simple local vector store to a real database, you change the behavior of this seam. The high level classes hide all these details. But once you know the seam by name, you can control exactly where your index lives.

The fix is always the same. Set the flag to true. Or use an ingestion pipeline instead of the from documents method. Then your document store will have the nodes. Persistence will save everything. Multi tenancy will work because the full nodes with metadata are available. You can filter by tenant using that metadata.

The index store also gets written only inside that same condition. Without the flag, the index structure is lost. So the three problems are really one problem. The storage context is the seam. Learn that one seam, and you solve them all.

An index is not something you own, like a folder of files. It is a strategy for building something you can query. The strategy you choose changes the kind of question that gets a good answer.

For example, the system can build an answer by summarizing all the pieces into one response. That mode is called tree summarization. It works in parallel and does not favor the nodes it saw first. You pair it with a summary index, and it is meant for the question “summarize everything.” When your question really is a global summary, this mode gives you a coherent answer. But there are cheaper ways. One mode called simple summarization just truncates everything into one prompt and answers once. It is the cheapest option, but it silently drops evidence. That is fine for a demo, but dangerous in a product. If you ask a question that needs all the evidence, simple summarization will lose information.

Another mode, called accumulate, does not summarize at all. It answers the query against each node separately and returns the list. That is not a summarizer, it is an extractor. If the consumer of the answer is code instead of a human, this is often the mode you actually wanted. It gives you a separate answer per chunk, not a combined one. So the question shape here is a list of individual facts, not a single narrative.

Then there are the refine and compact modes. These build the answer step by step, processing one node after another. The answer is built sequentially. That creates a real problem. Evidence in the last chunk is asked to overturn an answer the model has already committed to. If your users say “it found the right document but gave the old answer,” and the source nodes show the correct chunk was retrieved, you should look at this mode before blaming retrieval. The sequential bias means later evidence can hardly override the earlier conclusion. So a question that requires weighing information from many pieces will get a bad answer if the first few chunks set the wrong direction.

Tree summarization does not have that bias. It does not structurally favor the nodes it saw first. That makes it the right strategy when your question demands an even-handed synthesis across all retrieved chunks. The mismatch between strategy and question shape shows up as bad answers. If you use a mode that favors early nodes for a question that needs later corrections, you get an old answer. If you use a cheap summarizer that drops evidence for a question that needs every detail, you lose important facts.

The key is that each index and response mode answers a fundamentally different kind of question. The vector store index often works with embedding similarity and retrieves chunks based on meaning. But the response mode you choose on top of that retrieval decides how those chunks are turned into an answer. Picking the wrong mode for your question gives you a wrong answer, even when the right chunks were found. So the mismatch is not a retrieval failure; it is a response mismatch. Understanding that the index is a strategy, not a static thing, helps you pick the right tool for the question you are really asking.

The smallest interface is a list of chunks. It sits between the retriever and the language model. The retriever returns this list after a query. It is a ranked list of nodes. You can do things to this list before the model reads it. That is the postprocessor seam. It is the cheapest place to improve quality. Why is it so cheap? Because it only works on about ten chunks. Not on your whole collection of documents. A postprocessor is a function. It takes a list of nodes and returns a new list. It can filter out weak chunks. For example, a similarity postprocessor drops low scoring chunks. It uses a score cutoff to remove them. Or you can reorder the list. A cross encoder reranker rescues all the chunks. This model reads the query and each chunk together. That gives a better ranking than just embedding alone. You retrieve twenty chunks first. Then you rescore them. You keep the best five. This is a high leverage addition to a basic pipeline. You can also rewrite chunks. A metadata postprocessor replaces part of a chunk. That makes small chunks more viable. The narrowness of this seam is its strength. It is an interception point. You can swap in different postprocessors easily. You do not change the retriever or the language model. This makes the whole stack composable. Each part is a small interface. You can replace one piece without breaking others. That is the beauty of a narrow seam. It keeps the architecture flexible and simple. You can think of it as a gate. Every node passes through this gate. The gate can remove bad nodes. It can also change the order. This control is very cheap. It costs almost nothing to run. Because you only process ten items. This is why it is the highest leverage point. Many improvements happen here. You do not need to change the retriever. You do not need to change the model. Just swap the postprocessor. That makes the system easy to tune. This seam is a small interface. It has a question in and a ranked list out. That list is what you work with. You can swap the retriever that produces it. You can add postprocessors to change it. You can choose different synthesizers to read it. All these swaps happen at the same seam. That is what makes the stack composable. Each part connects through this list. You can upgrade one part at a time. This prevents breaking everything else. It is the core design pattern here. The whole architecture depends on this narrow seam. It is the key to building powerful systems.

After the retriever returns its top nodes, but before the model reads them, there is a list. This list is called node postprocessors. This is the cheapest place to improve quality. A postprocessor can filter nodes. It can reorder them. It can even rewrite what the model sees.

One type is a similarity postprocessor. It drops nodes below a score cutoff. Another type is a reranking model. It over-retrieves many nodes first. Then it rescores all of them with a model that reads the query and the chunk together. It is more accurate than comparing separate embeddings. The reranking model keeps the best few nodes.

Another type replaces metadata to expand the context around a small chunk. That makes small chunks viable.

But order matters here. When the response is built in sequence, evidence in the last chunk must overturn an answer the model already committed to. This creates a bias. The mode called tree summarize avoids this bias. It does not favor the first nodes seen. So it is better for summarizing everything.

Remember, the order of nodes in the list decides which duplicate survives. If you merge lists yourself, the ordering you hand back is important. This stage gives you control. You can add a score cutoff. You can add a reranking model. You can filter by metadata. You can expand context. It is the place to put these adjustments.

This small list of postprocessors is powerful. It works on only about ten nodes. So it is fast. But it must be done right. Order sensitivity can make it easy to get wrong. With the correct mode, like tree summarize, you avoid the bias. You get a fairer answer.

When you retrieve several chunks of text, the system must collapse them into one single answer. There are different modes that do this, and the mode you use decides how much it costs.

The simplest mode is called simple summarise. It stuffs every retrieved chunk into one big prompt and answers once. This is the cheapest option, but it will silently drop evidence if the prompt gets too long. It is fine for a demo but dangerous in a real product.

Another mode is refine, and its close cousin compact. These modes answer the query against each chunk one at a time. The system reads the first chunk, produces an answer, then reads the next chunk and tries to refine or compact that answer. The result is built sequentially. This means each additional chunk adds another call, so the cost grows with every new chunk. There is a hidden problem: because the answer is built step by step, evidence in the last chunk has to overturn what the model already committed to. If your users find the right document but get the old answer, this mode might be the cause.

The third mode is tree summarise. It does not favour the nodes it saw first. It still makes one call per chunk of text, but importantly those calls can run in parallel. So while the number of calls still scales with the number of chunks, the total time does not have to increase in a straight line. This is the mode to use when you want to summarise everything and avoid any bias toward earlier chunks.

In short, simple summarise packs everything into one prompt and costs the least. Refine and compact process chunks one after another, so cost climbs with each chunk. Tree summarise also needs a call per chunk, but because those calls can happen at the same time, the cost in time is not sequential. Your choice among these three directly shapes how much work the system does and how well it handles all the evidence.

The framework offers several high-level ways to run a query. The first is the query engine. It is not a single thing. It is a factory that builds a retriever, a list of postprocessors, and a synthesizer. The retriever grabs nodes from the index. The postprocessors filter or rerank them. The synthesizer forms the final answer. Together they give you a complete response from your data.

Another high-level facade is the agent. An agent is a workflow. A workflow uses steps that send events to each other. An agent's steps call the language model and run the tool it picks. The loop is not a special structure. It is just an event going from one step back to an earlier step. That lets the agent decide and act repeatedly.

So how do you choose which one you need? If you just need to answer one query with retrieved context, use the query engine. If you need a loop with tool use and choices, use the agent. These two cover the most common needs.

There is a global container that holds your model, your embedding model, and your text splitter. It is called Settings. Many parts of the framework pull from it by default. That is why the quickstart works with just five lines. You do not have to specify everything yourself. But the convenience has a cost. If you change a setting, it can affect things already created. For example, changing the chunk size mutates the node parser in place. Other components might be holding that same parser. That leads to unexpected behavior across your codebase. The same global container also has other traps. When you attach a docstore, it might not write to it if the vector store stores text. That is another silent issue. The framework is built from small interfaces with swappable implementations. The high level classes are just pre wired compositions. The global container supplies the defaults. Once you know the parts, you can replace pieces. The global container makes it easy to start but hard to debug. It is the reason the five line demo works so smoothly. It is also the reason a larger codebase can drift apart silently. That is the trade off you get with convenience.

The agent abstractions in LlamaIndex run on an event-driven workflow system. Steps in a workflow consume one type of event and return another. The runtime then routes each event to the step that accepts it. There is no fixed chain or central graph. A step can emit an event that an earlier step consumes, which creates a loop naturally. You do not need a special loop construct. Fanning out means sending several events at once. Fanning in means a step waits for several events to arrive. This design gives you control because you can see exactly how events flow. You can also replace any piece of the pipeline. The high-level classes are just ready-made assemblies of smaller parts. When your data does not fit the tutorial example, you need to open those defaults. Knowing which seam to cut is most of the fix. For example, if retrieval pulls the wrong chunk, it is a metadata or storage problem. If answers drop half the evidence, it is a synthesizer mode problem. If costs grow with the number of chunks, it is also a synthesizer mode issue. And if you cannot observe the pipeline, it is a dispatcher problem. You should drop to the workflow directly when you need to customize how steps connect. The agents themselves are workflows whose steps simply call the language model and run a tool. They are not a separate system. The event loop is the same one that powers your custom workflows. So you gain full visibility into what each step does. The trade-off is that you lose the convenience of the ready-made engine. But when your application grows beyond a demo, that trade-off is worth it. You stop fighting the facade and start replacing pieces.

The runtime that moves events between every stage is a dispatcher. It does not have a fixed path. Instead, it routes each event to whichever step accepts that event. This design gives you loops without any special loop construct. A step can emit an event that another step consumes. If that event loops back to the first step, you have a natural cycle. That is how agents work inside the framework.

Every stage in this system is a small interface with a swappable implementation. The high-level classes are just pre-wired compositions. So the dispatcher is the seam where you can plug in observation. A pipeline you cannot watch is a pipeline you cannot debug. That is why the dispatcher matters. It emits an event for every step that runs. Each event carries its own data. As events flow from step to step, they form a tree of spans. Each span is a single request and response. You can trace the whole journey from start to finish.

The trade-off is that the dispatcher is invisible by default. It does its work silently. You get correct results but no visibility into how they arrived. If something goes wrong, you have no record of which event fired when. That makes debugging very hard. The fix is to hook into the dispatcher. Tracing integrations do exactly that. They attach to the runtime and record every event. Then you can inspect the span tree later. You can see which step took too long or which event got lost.

The framework used to have graph-shaped pipelines. Those pipelines could not express loops. Agents are loops, so those old designs did not work well. Workflows replaced them. Workflows are event-driven. Each step declares the event it consumes and the event it returns. The runtime does the rest. Fan-out means emitting several events at once. Fan-in means a step that waits for several events before proceeding. All of that is handled by the dispatcher.

So when you build a RAG pipeline, you are really building a workflow. The retriever step emits a Retrieved event. The synthesizer step receives it. If it needs to loop and re-query, it emits another Retrieved event. The runtime routes it back. That is the same event round-trip as any other loop. And because the dispatcher records each event, you can trace the entire sequence.

The key insight is that the dispatcher is the seam. Knowing its name is most of the fix. You can replace the default runtime with one that logs everything. Or you can attach a tracing library that listens to events. That is how every major tracing integration hooks in. They intercept the dispatcher and build the span tree. Without that, you are blind. With it, you can see every step, every event, and every decision.

Shipping the five-line quickstart query path to production is tempting.
It looks like a one-line change.
It requires no understanding of the system.
No evaluation set, no chunking decision, no reranker latency budget.
But this path can never say it does not know.

The silent cost is huge.
Almost every quality problem in the first year of a retrieval augmented generation system, called RAG, is actually a retrieval problem.
It only looks like a generation problem.
A bigger model cannot read a chunk you never retrieved.
If the answer was not in the context window, generation quality is irrelevant.

The smallest responsible path starts with two numbers.
Measure retrieval in isolation with hit rate.
If hit rate is zero point six, then forty percent of your queries are unanswerable.
No prompt or model can fix that.

Then split the failures with faithfulness and relevancy evaluation.
High faithfulness and low relevancy means the retriever pulled the wrong documents.
A bigger model makes that worse because it summarizes wrong evidence more convincingly.
Low faithfulness with high relevancy is the only case where the model is the problem.

You also must tune chunk size on hit rate, not on how fluent the answers read.
As you make chunks bigger, retrieval gets worse.
You cannot see the collapse because you are grading only the queries that still worked.

The defaults are a fine starting point for dense prose.
But the point is you answer with a retrieval metric, not by reading answers.
That metric is the ceiling on everything downstream.
At zero point six hit rate, no amount of prompt engineering gets you past sixty percent correct.

So resist the quickstart path.
Add a cutoff, add a reranker, and always keep an abstention set.
That set is the only thing that tests whether your pipeline can honestly say it does not know.

Every night, your ingestion script reads every single document in your corpus, splits them all into chunks, and embeds every chunk. It does this even if nothing changed. That is the largest avoidable expense in most production retrieval systems. And nobody notices because the job succeeds. The index looks correct. The only clue is a line on an invoice that keeps going up. The root cause is simple. The from-documents call that tutorial taught you has no memory. It cannot remember what it saw last run because you never gave it a place to remember. That place is called the docstore. It maps a document’s unique identifier to its hash. An ingestion pipeline with upserts uses that. It processes a document only when its identifier is new or its hash has changed. It also deletes old nodes before inserting new ones. And an ingestion cache sits behind that. It skips expensive steps like embedding for content you have already processed. Here is the trap that catches almost everyone who knows about the docstore. You attach a docstore to a storage context. Then you call from-documents. You think the docstore is being updated. It is not. The reason is that real vector databases like Qdrant or Pinecone store the text themselves. So the code that would write to the docstore is skipped. Your docstore stays empty. Your dedup is a no-op. It does nothing. It raises no error because the index queries perfectly. The vector store has the text, so everything seems fine. But the cost goes on forever. The fix is to stop trying to make from-documents remember things. Instead, use an ingestion pipeline from the start. It handles caching and upserts properly. The content hash caching means that if a document has not changed, your system skips re-embedding entirely. That is how you stop paying twice for the same corpus. Think of it this way: every night you are throwing away all the work from the previous night and starting from scratch. That is like paying for a full meal every day even when you only eat leftovers. The ingestion pipeline is the refrigerator. It keeps the work you already did, so you only cook what is new. The cost survives because it is invisible. The only symptom is a number on an invoice that someone else reads. Now you know where to look. And the fix is just a few lines of code.

You tune the chunk size until the answers read better. But that optimizes the wrong objective. It measures fluency, a noisy proxy for quality. As you increase the chunk size, the retrieved text becomes more fluent. But the recall underneath collapses. You cannot see the collapse because you only grade the queries that still worked. The ones that fail are invisible.

The problem is a fundamental trade-off. Generation wants big chunks. More surrounding prose means fewer severed sentences. The answer is less likely to be cut in half at a boundary. Retrieval wants small chunks. A large chunk that covers several topics embeds as the average of those topics. That average vector is not close to any specific question. So as you turn the dial up, the answers you read get smoother. Meanwhile the retrieval hit rate drops silently.

The default settings are a chunk size of one thousand twenty four and an overlap of two hundred. Those are a fine starting point for dense prose. But the real question is not which number is right. It is how you find the right number. And you cannot answer that by reading answers. You answer it with a retrieval metric, measured in isolation, with no large language model in the loop.

That metric is hit rate. Hit rate is the ceiling on everything downstream. If your hit rate is sixty percent, then forty percent of your queries are unanswerable. No prompt and no model can get you past sixty percent correct. Every dollar you spend on a bigger model is spent on the wrong half of the pipeline. A larger model cannot read a chunk you never retrieved. If the answer was not in the context window, generation quality is irrelevant. You have paid a premium for a more articulate way of not knowing.

So stop tuning chunk size by how the answers read. Measure retrieval in isolation. Use a retriever evaluator and look at hit rate. That number tells you the true ceiling. Only after you have optimized retrieval should you worry about generation. Otherwise you are optimizing the wrong variable through a noisy proxy. The answers get more fluent while the recall underneath collapses. And you will never see the collapse because you are only looking at the queries that still work. Measure what matters first.

You think that adding a reranker improves your search, so applying a score cutoff on top must make it even better. But the reranker does not give you the kind of score you expect.

Here is the concrete detail. A regular vector store returns a cosine similarity score. That score is bounded, usually between zero and one. A reranker like a cross-encoder returns a completely different number. It produces an unbounded logit. That logit can be negative. A score of negative one point two is perfectly possible for the most relevant document.

Now imagine you set a cutoff at zero point five. Every node that falls below that value gets thrown away. But the reranker’s negative logit does not mean the node is bad. It means the reranker’s scoring scale is different. So your cutoff rejects the very documents the reranker ranked most confidently. You end up with an empty context. Or the model answers from its own training data because nothing was retrieved. That is the silent recall you lose.

The trade off is painful. You implemented a reranker to improve quality. Then you added a threshold that silently removes the best results. You never see the error because the system just returns less. The only visible symptom might be an empty response or a model that sounds confident but wrong.

The correct approach is not to filter by score value at all. Score values are not comparable across different producers. A cosine similarity, a BM25 relevance score, and a cross-encoder logit all land in the same field but with wildly different scales. If you ever write code that checks if the score is greater than some number, you have hard coded a threshold to one model’s distribution. That threshold will not survive the next model upgrade.

Instead, filter by rank. After the reranker, simply take the top N results. That respects the reranker’s ordering without assuming anything about the scores themselves. Rank is stable. Score is not.

There is another subtle failure. A postprocessor that expects a similarity score will drop nodes that have no score at all. Some retrievers do not populate scores. If you put a similarity postprocessor after them with a cutoff of zero point zero, thinking it is harmless, every node gets silently dropped. That turns a working pipeline into one that always returns nothing.

So remember: a reranker improves the ranking. But its scores are not calibrated the way you assume. Do not apply a value cutoff after it. You will throw away your best retrieval work. Measure retrieval in isolation. Keep the reranker’s output ordered by rank. Let the top N through. That is the only safe filter. Everything else is a trap that wastes your model budget and hides the real problem.

Putting tenant restrictions in the system prompt is a tempting fix. It requires no understanding of the retrieval system. It is a one line change. It needs no evaluation set. No chunking decision. No filter schema. No reranking budget. But it is wrong for a reason that should be on the wall. A bigger model cannot read a chunk you never retrieved. If the answer was not in the context window, generation quality is irrelevant. You have paid a premium for a more articulate way of not knowing.

Now apply that to tenancy. The system prompt reads only use documents belonging to this tenant. Never reveal information from other tenants. It works in testing. It works in the demo. But it is not access control. It is a request. The request is addressed to a stochastic text generator about text you already placed inside its context window. What happened before that prompt was ever read? The retriever ran an approximate nearest neighbor search across the whole collection. It pulled the top chunks. Some of those chunks belong to another tenant. They were serialized into the prompt. The data has already left the store. The only thing standing between another customer's contract and your user is a single sentence. That sentence is competing for attention with everything else in the context. In a forty thousand token context, it can easily be ignored. Any prompt injection in a retrieved document becomes a peer of your instruction, not a subject of it.

So relying on the prompt is a correctness and privacy failure waiting to happen. The prompt cannot remove what the retriever already placed in the context. It is not a style choice. It is a fundamental misunderstanding of how the pipeline works. The cost is not visible until production. But when it fails, it fails silently. The system prompt is not a gate. It is a suggestion. And suggestions are weak when the context is full of contradictory information. A retrieved document from another tenant might contradict the system prompt. The model has to choose. It might follow the strong evidence from the document instead of the abstract instruction. That is the failure mode. It is silent. The user sees the other tenant's data. The system never raises an error. The only way to catch it is to inspect the output. But most teams do not do that. They trust the prompt. But the prompt is not a filter. The retriever already decided what the model will see. The prompt cannot undo what the retrieval already placed in the window. Instead, filter by metadata before retrieval. That way, the wrong data never enters the context. That is the real fix. It requires understanding the retrieval step. But it is the only way to protect tenant data.

Your system has a hidden path. It can fabricate answers. When a retriever finds nothing, the node count is zero. If nothing checks that number, an empty context goes to the generator. The generator then answers from its own training. It sounds confident but the content is made up. This is a hallucination nobody sees. The guard that fixes this is straightforward. You assert on the number of nodes. If it is zero, you fall back. Then you synthesize an honest response. That response says I do not know. The abstention set tests this exact ability. It is the only thing that proves the pipeline can refuse. Without it, the system has no way to say no. All answers look correct but many are built on missing information. The golden set often misses unanswerable questions. So the system passes tests but fails in production. The fix is simple. Measure retrieval alone. Use hit rate to find true accuracy. If retrieval finds nothing, the model cannot answer anything real. The guard saves the day by stopping the empty path. It turns a fabrication into an honest no.

A common mistake is to switch to a larger language model when the answers are wrong. But a bigger model cannot read a chunk you never retrieved. Most quality problems in the first year are actually retrieval problems wearing generation problems clothes. To diagnose, measure retrieval by itself with a hit rate. If the hit rate is zero point six, then four out of ten queries are unanswerable no matter what model you use. Every dollar spent on a larger model then goes to the wrong half of the pipeline. Then split the generation failures using two separate checks. One checks faithfulness to the retrieved context. The other checks relevancy of that context. Faithfulness means the model sticks to the text it was given. Relevancy means that text actually answers the question. These two are independent. High faithfulness with low relevancy shows a retrieval bug. The model summarized the wrong documents. A bigger model makes that worse because it summarizes wrong evidence more convincingly. Low faithfulness with high relevancy is the only case where the model itself is the problem. That indicates the model ignored the context. Chunk size also plays a role. Generation wants big chunks for more surrounding prose. Retrieval wants small chunks so the embedding is not an average of many topics. Tuning chunk size by reading the outputs hides the collapse in retrieval. The answers get more fluent while the recall underneath falls. You cannot see that collapse because you grade the queries that still worked. The right way is to answer that question with a retrieval metric in isolation. No language model in the loop. Measure hit rate, then tune on that number. That is the only way to distinguish a retrieval failure from a generation failure.

Another common trap is using a golden set built from the same generator. Every question was written from the node that answers it. The retriever finds it trivially because it is a vocabulary matched paraphrase. The set contains no unanswerable questions. So the hit rate looks very high, like zero point nine four. But in production the system confabulates over irrelevant chunks. The evaluation certifies the exact failure users report. This is the most expensive mistake because it requires no understanding of the system. It is a one line change. But it is wrong because a bigger model cannot read a chunk never retrieved. If the answer was not in the context window, generation quality is irrelevant. You have paid for a more articulate way of not knowing.

To fix this, you must measure retrieval in isolation using a hit rate metric. Hit rate is the ceiling on everything downstream. If it is low, no prompt and no model can fix it. Then and only then use faithfulness and relevancy checks. Those checks need no labels and can run on live traffic. Faithfulness and relevancy are orthogonal. Holding them apart is the whole diagnostic. High faithfulness plus low relevancy means the retriever pulled on topic but wrong documents. A bigger model makes that problem worse. Low faithfulness with high relevancy is the only genuine model problem. The chunk boundary also matters. A chunk is both the unit of retrieval and the unit of context. These two want opposite things. Generation wants large chunks to avoid severed sentences. Retrieval wants small chunks so the vector is specific. As you increase chunk size, answers get more fluent but recall collapses underneath. You cannot see the collapse because you grade the output of queries that still worked. So tune chunk size on a retrieval metric, not on how the prose reads. That is the core of the misdiagnosis. The most common error is to blame the generator when the real fault is retrieval. A bigger model cannot read a chunk never retrieved. You settle this with two numbers, not with an argument. Hit rate first. Then faithfulness versus relevancy. That tells you where the problem really lies.

One quiet mistake hits many retrieval systems. It is the global settings container. Two processes run with two different sets of settings. One uses OpenAI embeddings. Another uses a model called B G E. The vectors from these models are incompatible. The index was built with one model. When a query comes in using the other model, the retrieval cannot find the right chunks. The system does not crash. No error appears. The retrieval simply returns the wrong documents. The generator then produces an answer from those wrong documents. The answer looks fluent. But it is based on irrelevant information. The user gets a confident wrong answer. The operator sees a quality problem. They might blame the generator. They might upgrade to a bigger model. But that will not help. A bigger model cannot read a chunk that was never retrieved. The real problem is the embedding mismatch. It is a retrieval problem disguised as a generation problem. Almost every quality problem in the first year of a retrieval system is a retrieval problem wearing a generation problem's clothes. This mistake is one of ten that never raise an exception. The cost shows up only in metrics that nobody checks. You cannot argue your way out of this. The numbers tell the truth. Measure the retrieval hit rate in isolation. Without any generator. If the hit rate is zero point six, then forty percent of your queries are unanswerable. No prompt and no model can get you past sixty percent correct. The embedding swap makes the hit rate drop. The retrieval evaluator would reveal it. But without that measurement, the problem hides. The evaluation would show that the generator is faithful to the retrieved documents. But those documents are irrelevant. That is a clear sign of a retrieval problem. The solution is simple. You must pin the embedding model explicitly. In your code, set the embed model directly. Do not rely on a global container. At boot time, check that the vector dimension matches what you expect. This adds a few lines. Those lines are worth it. They prevent a whole class of silent failures. They make the system predictable. They ensure that every call uses the same model. If you ever need to change the embedding model, you must rebuild the index. That is a deliberate act. It is not a hidden side effect. The global settings container makes changes invisible. Explicit wiring makes them visible. That visibility is the whole point. You need to know what your system is actually doing. The extra lines are not overhead. They are insurance. They save you from chasing generation problems that are really retrieval problems. They save you from paying for a bigger model that cannot fix a missing chunk. They save you from the cost that nobody is looking at.

Reaching for an agent loop on every question is a common mistake. It turns retrieval into something unpredictable. A query engine is simpler and much faster. For most questions, a single call to a query engine works. That is roughly eighty percent of cases. So the honest test is straightforward. Ask yourself if the question really needs a decision at all. If not, skip the agent loop entirely. This saves latency and cost. It also keeps your system deterministic.

An agent adds complexity and randomness. Retrieval stops being a pure function. You introduce extra steps and more variables. The result is harder to debug and slower to run. A bigger model does not fix this. If the answer was never retrieved, no model can save it. You pay a premium for a more articulate way of not knowing. That is why measuring retrieval in isolation matters. Use a retriever evaluator without any large language model. Check the hit rate. A hit rate of zero point six means forty percent of queries are unanswerable. No agent or model can fix that. The ceiling on your system is that number.

The honest test is simple. Before you build an agent, test whether a query engine can handle the question. Try a direct retrieval and synthesis. If it works, you are done. If it fails, then consider an agent. But do not default to an agent for every question. That route adds latency and cost for no gain. It also makes the system nondeterministic. The same question can get different answers. That is a problem for reliability.

Look at the two numbers that matter. Faithfulness and relevancy. High faithfulness with low relevancy means the retriever pulled the wrong documents. A bigger model makes that worse. It summarizes wrong evidence more convincingly. Low faithfulness with high relevancy means the model ignored the context. That is the rare case where the model is the problem. For most issues, the retrieval is the culprit. So fix retrieval first.

The agent loop is a heavy tool. Use it only when the question truly needs a decision. The easy eighty percent of questions do not. They just need a fast, accurate answer from a query engine. That engine is simple. It retrieves and synthesizes. No extra steps, no branching, no tool calls. It is deterministic and cheap.

When you do use an agent, be careful. Make sure your retrieval is solid. Measure it in isolation. Do not rely on the agent to fix bad retrieval. It cannot. The agent will only amplify the problem. It will make the wrong answer sound more fluent. That is dangerous.

So the rule is: start with a query engine. Test it. If it works, stop. Only if it clearly fails should you reach for an agent. This saves time and money. It also avoids the hidden cost of nondeterminism. Your users get consistent answers. Your system stays simple and easy to debug.

The honest test is not about how smart the agent is. It is about whether the question needs a decision at all. Most questions are straightforward. They have a single correct answer. A query engine finds that answer fast. An agent only adds unnecessary steps. That is the core insight. Route the easy questions to a query engine. Keep agents for the hard ones. Your latency will drop. Your cost will shrink. And your answers will be more reliable.

When you build an evaluation set using the same model you are testing, the result looks good but is actually worthless. The questions come directly from the chunks that answer them. They are paraphrases that match the vocabulary of those chunks. So the retriever finds them easily. You see a hit rate of zero point nine four. The test passes with a green light. But in production, your system confabulates over irrelevant chunks. Users report exactly that failure. The evaluation set has zero unanswerable questions. The generator cannot produce one. So it never tests whether your pipeline can say I do not know. This mistake is very common because it requires no understanding of the system. It is a one line change. No chunking decision. No rerank budget. It is wrong for a simple reason. A bigger model cannot read a chunk you never retrieved. If the answer was not in the context window, generation quality does not matter. You have paid for a more articulate way of not knowing.

Almost every quality problem in the first year of a rag system is actually a retrieval problem. It wears a generation problems clothes. You settle this with two numbers. Measure retrieval in isolation. No large language model in the loop. Use retriever evaluator and hit rate. If hit rate is zero point six, then forty percent of your queries are unanswerable no matter what generates over them. Every dollar on a big model is spent on the wrong half. Then split generation failures with faithfulness and relevancy evaluators. Faithful but irrelevant means the model summarized the wrong documents. That is a retrieval bug. Relevant but unfaithful means the model ignored the context. That is a prompt or model bug. These two are orthogonal. Holding them apart is the whole diagnostic.

High faithfulness with low relevancy is a retriever that pulled on topic but wrong documents. A bigger model makes that worse. It summarizes the wrong evidence more convincingly. Low faithfulness with high relevancy is the only case where the model is genuinely the problem. A defensible golden set requires an abstention set. That is the only thing that ever tests whether your pipeline can say I do not know. You must hand edit the golden set. Add unanswerable questions. Do not use the unedited output of the generator. That output contains no unanswerable questions and guarantees a false green pass. Hand edit it. Include questions that cannot be answered from the retrieved chunks. Then your evaluation will tell you the truth.