LlamaIndex Architecture — Audio Guide

A document and the nodes it is split into are the two core building blocks of the whole framework. A document is not a subclass of a text node. That is a common mistake from older guides. Ingestion does not turn documents into nodes. Instead, nodes get split into smaller nodes. They keep relationships to their parent nodes.

Each node carries five key pieces of information. First is the text itself. This is what gets embedded and what the language model reads. Second is a stable identifier. Stability matters because if you regenerate random ids each time, syncing becomes a duplicate mess. Third is metadata. This is a simple dictionary. It is the most underused field in the framework. Metadata makes filtering possible. Filtering is what makes similarity search work in a multi-tenant corpus. Fourth are relationships. These link to source, previous, next, parent, and child nodes. They let a postprocessor retrieve a sentence and then hand the language model the paragraph around it. Fifth are the excluded metadata keys. This pair is an escape hatch that nobody reads about until it causes trouble.

Here is the trade-off. By default, a node's metadata is prepended to its text when going into both the embedding model and the language model prompt. That is usually what you want. A chunk with metadata like title Q3 Pricing Policy embeds as a chunk about Q3 pricing. That gives a semantic lift.

But here is the self-inflicted bug. If you stash a file path, an ingestion timestamp, or a long JSON blob on the node, that text also gets embedded. It dilutes the vector with meaningless text. When someone reports that retrieval got worse after adding metadata, this is nearly always why.

The fix is not to drop the metadata. You should exclude it from the embedding while keeping it available for filtering. That is the classic fix.

So remember: the document and node are the foundation. Metadata rides along with each node. And if you let metadata leak into the embedded text, you cause a retrieval bug yourself. Always set excluded metadata keys to keep your embeddings clean. That is a small change that saves a lot of trouble later.

The index you build does not live in just one place. It lives across three separate stores. There is the vector store, which holds the embeddings. There is the document store, which keeps the original nodes. And there is the index store, which tracks the structure. The seam that controls where an index physically lives is a single object called the storage context.

You attach this storage context when you build the index. But here is the trap. Many real vector databases automatically store the text of each node. When you use one, the storage context does not write the nodes to the document store. The document store stays empty. To force the nodes into the document store, you need to set a special flag to true. Without that flag, your document store has nothing.

Now persistence becomes a problem. If you try to persist the index, only the vector store gets saved. The document store and the index store are missing. So persistence fails silently. Multi tenancy also depends on this same seam. Metadata on each node lets you filter by tenant. But if that metadata is not stored in the document store, you cannot recover the full node later. The metadata is also prepended to the node text for embedding. That can dilute the vector with irrelevant information. You have to exclude certain keys to avoid that.

So the seam that decides the physical location also decides whether your metadata works correctly. Every time you switch from a simple local vector store to a real database, you change the behavior of this seam. The high level classes hide all these details. But once you know the seam by name, you can control exactly where your index lives.

The fix is always the same. Set the flag to true. Or use an ingestion pipeline instead of the from documents method. Then your document store will have the nodes. Persistence will save everything. Multi tenancy will work because the full nodes with metadata are available. You can filter by tenant using that metadata.

The index store also gets written only inside that same condition. Without the flag, the index structure is lost. So the three problems are really one problem. The storage context is the seam. Learn that one seam, and you solve them all.

An index is not something you own, like a folder of files. It is a strategy for building something you can query. The strategy you choose changes the kind of question that gets a good answer.

For example, the system can build an answer by summarizing all the pieces into one response. That mode is called tree summarization. It works in parallel and does not favor the nodes it saw first. You pair it with a summary index, and it is meant for the question “summarize everything.” When your question really is a global summary, this mode gives you a coherent answer. But there are cheaper ways. One mode called simple summarization just truncates everything into one prompt and answers once. It is the cheapest option, but it silently drops evidence. That is fine for a demo, but dangerous in a product. If you ask a question that needs all the evidence, simple summarization will lose information.

Another mode, called accumulate, does not summarize at all. It answers the query against each node separately and returns the list. That is not a summarizer, it is an extractor. If the consumer of the answer is code instead of a human, this is often the mode you actually wanted. It gives you a separate answer per chunk, not a combined one. So the question shape here is a list of individual facts, not a single narrative.

Then there are the refine and compact modes. These build the answer step by step, processing one node after another. The answer is built sequentially. That creates a real problem. Evidence in the last chunk is asked to overturn an answer the model has already committed to. If your users say “it found the right document but gave the old answer,” and the source nodes show the correct chunk was retrieved, you should look at this mode before blaming retrieval. The sequential bias means later evidence can hardly override the earlier conclusion. So a question that requires weighing information from many pieces will get a bad answer if the first few chunks set the wrong direction.

Tree summarization does not have that bias. It does not structurally favor the nodes it saw first. That makes it the right strategy when your question demands an even-handed synthesis across all retrieved chunks. The mismatch between strategy and question shape shows up as bad answers. If you use a mode that favors early nodes for a question that needs later corrections, you get an old answer. If you use a cheap summarizer that drops evidence for a question that needs every detail, you lose important facts.

The key is that each index and response mode answers a fundamentally different kind of question. The vector store index often works with embedding similarity and retrieves chunks based on meaning. But the response mode you choose on top of that retrieval decides how those chunks are turned into an answer. Picking the wrong mode for your question gives you a wrong answer, even when the right chunks were found. So the mismatch is not a retrieval failure; it is a response mismatch. Understanding that the index is a strategy, not a static thing, helps you pick the right tool for the question you are really asking.

The smallest interface is a list of chunks. It sits between the retriever and the language model. The retriever returns this list after a query. It is a ranked list of nodes. You can do things to this list before the model reads it. That is the postprocessor seam. It is the cheapest place to improve quality. Why is it so cheap? Because it only works on about ten chunks. Not on your whole collection of documents. A postprocessor is a function. It takes a list of nodes and returns a new list. It can filter out weak chunks. For example, a similarity postprocessor drops low scoring chunks. It uses a score cutoff to remove them. Or you can reorder the list. A cross encoder reranker rescues all the chunks. This model reads the query and each chunk together. That gives a better ranking than just embedding alone. You retrieve twenty chunks first. Then you rescore them. You keep the best five. This is a high leverage addition to a basic pipeline. You can also rewrite chunks. A metadata postprocessor replaces part of a chunk. That makes small chunks more viable. The narrowness of this seam is its strength. It is an interception point. You can swap in different postprocessors easily. You do not change the retriever or the language model. This makes the whole stack composable. Each part is a small interface. You can replace one piece without breaking others. That is the beauty of a narrow seam. It keeps the architecture flexible and simple. You can think of it as a gate. Every node passes through this gate. The gate can remove bad nodes. It can also change the order. This control is very cheap. It costs almost nothing to run. Because you only process ten items. This is why it is the highest leverage point. Many improvements happen here. You do not need to change the retriever. You do not need to change the model. Just swap the postprocessor. That makes the system easy to tune. This seam is a small interface. It has a question in and a ranked list out. That list is what you work with. You can swap the retriever that produces it. You can add postprocessors to change it. You can choose different synthesizers to read it. All these swaps happen at the same seam. That is what makes the stack composable. Each part connects through this list. You can upgrade one part at a time. This prevents breaking everything else. It is the core design pattern here. The whole architecture depends on this narrow seam. It is the key to building powerful systems.

After the retriever returns its top nodes, but before the model reads them, there is a list. This list is called node postprocessors. This is the cheapest place to improve quality. A postprocessor can filter nodes. It can reorder them. It can even rewrite what the model sees.

One type is a similarity postprocessor. It drops nodes below a score cutoff. Another type is a reranking model. It over-retrieves many nodes first. Then it rescores all of them with a model that reads the query and the chunk together. It is more accurate than comparing separate embeddings. The reranking model keeps the best few nodes.

Another type replaces metadata to expand the context around a small chunk. That makes small chunks viable.

But order matters here. When the response is built in sequence, evidence in the last chunk must overturn an answer the model already committed to. This creates a bias. The mode called tree summarize avoids this bias. It does not favor the first nodes seen. So it is better for summarizing everything.

Remember, the order of nodes in the list decides which duplicate survives. If you merge lists yourself, the ordering you hand back is important. This stage gives you control. You can add a score cutoff. You can add a reranking model. You can filter by metadata. You can expand context. It is the place to put these adjustments.

This small list of postprocessors is powerful. It works on only about ten nodes. So it is fast. But it must be done right. Order sensitivity can make it easy to get wrong. With the correct mode, like tree summarize, you avoid the bias. You get a fairer answer.

When you retrieve several chunks of text, the system must collapse them into one single answer. There are different modes that do this, and the mode you use decides how much it costs.

The simplest mode is called simple summarise. It stuffs every retrieved chunk into one big prompt and answers once. This is the cheapest option, but it will silently drop evidence if the prompt gets too long. It is fine for a demo but dangerous in a real product.

Another mode is refine, and its close cousin compact. These modes answer the query against each chunk one at a time. The system reads the first chunk, produces an answer, then reads the next chunk and tries to refine or compact that answer. The result is built sequentially. This means each additional chunk adds another call, so the cost grows with every new chunk. There is a hidden problem: because the answer is built step by step, evidence in the last chunk has to overturn what the model already committed to. If your users find the right document but get the old answer, this mode might be the cause.

The third mode is tree summarise. It does not favour the nodes it saw first. It still makes one call per chunk of text, but importantly those calls can run in parallel. So while the number of calls still scales with the number of chunks, the total time does not have to increase in a straight line. This is the mode to use when you want to summarise everything and avoid any bias toward earlier chunks.

In short, simple summarise packs everything into one prompt and costs the least. Refine and compact process chunks one after another, so cost climbs with each chunk. Tree summarise also needs a call per chunk, but because those calls can happen at the same time, the cost in time is not sequential. Your choice among these three directly shapes how much work the system does and how well it handles all the evidence.

The framework offers several high-level ways to run a query. The first is the query engine. It is not a single thing. It is a factory that builds a retriever, a list of postprocessors, and a synthesizer. The retriever grabs nodes from the index. The postprocessors filter or rerank them. The synthesizer forms the final answer. Together they give you a complete response from your data.

Another high-level facade is the agent. An agent is a workflow. A workflow uses steps that send events to each other. An agent's steps call the language model and run the tool it picks. The loop is not a special structure. It is just an event going from one step back to an earlier step. That lets the agent decide and act repeatedly.

So how do you choose which one you need? If you just need to answer one query with retrieved context, use the query engine. If you need a loop with tool use and choices, use the agent. These two cover the most common needs.

There is a global container that holds your model, your embedding model, and your text splitter. It is called Settings. Many parts of the framework pull from it by default. That is why the quickstart works with just five lines. You do not have to specify everything yourself. But the convenience has a cost. If you change a setting, it can affect things already created. For example, changing the chunk size mutates the node parser in place. Other components might be holding that same parser. That leads to unexpected behavior across your codebase. The same global container also has other traps. When you attach a docstore, it might not write to it if the vector store stores text. That is another silent issue. The framework is built from small interfaces with swappable implementations. The high level classes are just pre wired compositions. The global container supplies the defaults. Once you know the parts, you can replace pieces. The global container makes it easy to start but hard to debug. It is the reason the five line demo works so smoothly. It is also the reason a larger codebase can drift apart silently. That is the trade off you get with convenience.

The agent abstractions in LlamaIndex run on an event-driven workflow system. Steps in a workflow consume one type of event and return another. The runtime then routes each event to the step that accepts it. There is no fixed chain or central graph. A step can emit an event that an earlier step consumes, which creates a loop naturally. You do not need a special loop construct. Fanning out means sending several events at once. Fanning in means a step waits for several events to arrive. This design gives you control because you can see exactly how events flow. You can also replace any piece of the pipeline. The high-level classes are just ready-made assemblies of smaller parts. When your data does not fit the tutorial example, you need to open those defaults. Knowing which seam to cut is most of the fix. For example, if retrieval pulls the wrong chunk, it is a metadata or storage problem. If answers drop half the evidence, it is a synthesizer mode problem. If costs grow with the number of chunks, it is also a synthesizer mode issue. And if you cannot observe the pipeline, it is a dispatcher problem. You should drop to the workflow directly when you need to customize how steps connect. The agents themselves are workflows whose steps simply call the language model and run a tool. They are not a separate system. The event loop is the same one that powers your custom workflows. So you gain full visibility into what each step does. The trade-off is that you lose the convenience of the ready-made engine. But when your application grows beyond a demo, that trade-off is worth it. You stop fighting the facade and start replacing pieces.

The runtime that moves events between every stage is a dispatcher. It does not have a fixed path. Instead, it routes each event to whichever step accepts that event. This design gives you loops without any special loop construct. A step can emit an event that another step consumes. If that event loops back to the first step, you have a natural cycle. That is how agents work inside the framework.

Every stage in this system is a small interface with a swappable implementation. The high-level classes are just pre-wired compositions. So the dispatcher is the seam where you can plug in observation. A pipeline you cannot watch is a pipeline you cannot debug. That is why the dispatcher matters. It emits an event for every step that runs. Each event carries its own data. As events flow from step to step, they form a tree of spans. Each span is a single request and response. You can trace the whole journey from start to finish.

The trade-off is that the dispatcher is invisible by default. It does its work silently. You get correct results but no visibility into how they arrived. If something goes wrong, you have no record of which event fired when. That makes debugging very hard. The fix is to hook into the dispatcher. Tracing integrations do exactly that. They attach to the runtime and record every event. Then you can inspect the span tree later. You can see which step took too long or which event got lost.

The framework used to have graph-shaped pipelines. Those pipelines could not express loops. Agents are loops, so those old designs did not work well. Workflows replaced them. Workflows are event-driven. Each step declares the event it consumes and the event it returns. The runtime does the rest. Fan-out means emitting several events at once. Fan-in means a step that waits for several events before proceeding. All of that is handled by the dispatcher.

So when you build a RAG pipeline, you are really building a workflow. The retriever step emits a Retrieved event. The synthesizer step receives it. If it needs to loop and re-query, it emits another Retrieved event. The runtime routes it back. That is the same event round-trip as any other loop. And because the dispatcher records each event, you can trace the entire sequence.

The key insight is that the dispatcher is the seam. Knowing its name is most of the fix. You can replace the default runtime with one that logs everything. Or you can attach a tracing library that listens to events. That is how every major tracing integration hooks in. They intercept the dispatcher and build the span tree. Without that, you are blind. With it, you can see every step, every event, and every decision.