LlamaIndex Architecture

The LlamaIndex hub gives you the two-phase contract — ingestion builds an index offline, a query reads it online — and the patterns page gives you the ladder of things you build on top of it. This page is the layer underneath both: the object graph, and the control flow that walks it.

It is worth learning as its own thing, because almost every real LlamaIndex problem is a seam problem. The framework's five-line quickstart is a stack of defaults that have been collapsed into one call, and the moment your corpus stops looking like the tutorial's, you need to know which default to prise back open. Retrieval pulling the wrong tenant is a StorageContext/metadata problem. Answers that quietly drop half the retrieved evidence are a response-synthesizer problem. Costs that scale with the number of chunks are a synthesizer mode problem. A pipeline you can't observe is a dispatcher problem. Knowing the seam by name is most of the fix.

The one structural idea to hold on to: every stage in LlamaIndex is a small interface with a swappable implementation, and the high-level classes are just pre-wired compositions of them. index.as_query_engine() is not a thing — it's a factory that builds a retriever, a list of postprocessors, and a synthesizer, and hands you the assembly. Once you can name the parts, you stop fighting the facade and start replacing pieces of it.

The whole graph, end to end — ingestion on the left, query on the right, the storage layer as the seam they meet at:

Read that diagram as two pipelines that share a middle. Everything from Readers to StorageContext runs offline, once per corpus change, and is allowed to be slow and expensive. Everything from Retriever to Response runs online, once per question, in front of a waiting user. Work you can push left of the storage layer is work the user never waits for — that single trade explains most of the design decisions below.

Documents and Nodes: the only data model

There is exactly one data structure in LlamaIndex, and it is the Node. A Document is not a separate thing — it is a BaseNode like every other, a node that happens to hold a whole file. (Be precise about the family tree, because half the blog posts get it wrong: Document subclasses Node, and TextNode is its sibling — both descend from BaseNode. Document is emphatically not a subclass of TextNode, whatever an older tutorial told you.) Ingestion is therefore not "documents become nodes" so much as nodes get split into smaller nodes and given relationships to their parents. Once you see that, SimpleDirectoryReader → SentenceSplitter → VectorStoreIndex stops looking like three conversions and starts looking like one recursive shape.

A node carries five things that matter, and four of them are the ones people forget:

• text — the span itself. This is what gets embedded and what the LLM eventually reads.
• id_ — a stable identifier. Stable matters: incremental re-ingestion (the dedup/upsert pattern) keys on it, so a reader that regenerates random ids on every run quietly turns "sync" into "duplicate everything".
• metadata — an arbitrary dict, and the single most under-used field in the framework. Metadata is what makes filtering possible, and filtering is what makes similarity search survive contact with a multi-tenant corpus.
• relationships — links to SOURCE, PREVIOUS, NEXT, PARENT, CHILD nodes. This is the graph that lets a postprocessor retrieve a sentence and then hand the LLM the paragraph around it.
• excluded_embed_metadata_keys / excluded_llm_metadata_keys — the escape hatch nobody reads about until it bites.

That last pair deserves the paragraph it never gets. By default, a node's metadata is prepended to its text on the way into both the embedding model and the LLM prompt. That is usually what you want — a chunk that carries {"title": "Q3 Pricing Policy"} embeds as a chunk that is about Q3 pricing, which is exactly the semantic lift metadata is for. But it means that a file_path, an ingestion timestamp, or a 400-character JSON blob you stashed on the node is also being embedded, diluting the vector with text that has nothing to do with meaning. When someone reports "retrieval got worse after we added metadata", this is nearly always why. The fix is not to drop the metadata — it's to exclude it from the embedding while keeping it available for filtering.

from llama_index.core.schema import TextNode

node = TextNode(
    text="Enterprise tier moves from seat-based to usage-based billing on Oct 1.",
    metadata={
        "title": "Q3 Pricing Policy",     # semantic — worth embedding
        "tenant_id": "acme",              # a filter key, NOT meaning
        "ingested_at": "2025-09-30T12:00:00Z",
        "file_path": "/mnt/corpus/pricing/q3.pdf",
    },
    # Keep these out of the vector: they're routing keys, not semantics.
    excluded_embed_metadata_keys=["tenant_id", "ingested_at", "file_path"],
    # And out of the prompt: the LLM doesn't need the file path to answer.
    excluded_llm_metadata_keys=["tenant_id", "ingested_at", "file_path"],
)


print(node.get_content(metadata_mode="embed"))   # what the embedding model reads
print(node.get_content(metadata_mode="llm"))     # what lands in the prompt
print(node.get_content(metadata_mode="none"))    # the raw span

get_content(metadata_mode=...) is the highest-value thirty seconds you can spend in this framework. It shows you the actual string being embedded, which is very often not the string you thought you were embedding.

Fails when: the metadata you never wrote is the metadata that poisons the vector. SimpleDirectoryReader's default_file_metadata_func stamps file_path, file_name, file_type, file_size, creation_date, last_modified_date and last_accessed_date onto every document it loads, and it puts none of them on excluded_embed_metadata_keys — so a corpus you thought was pure prose is embedding an absolute path and a byte count into every single chunk. The extractors do it too: BaseExtractor.aprocess_nodes takes excluded_embed_metadata_keys as an argument that defaults to None, so a TitleExtractor or SummaryExtractor dropped into an IngestionPipeline writes its new keys into node.metadata and excludes them from nothing. And the exclusion lists are ingest-time state — they are consumed once, when get_content(metadata_mode="embed") is called on the way into the embedding model, and the resulting vector is what lives in the store forever. Adding a key to excluded_embed_metadata_keys after the fact changes the next embedding, not the one you already paid for. The symptom is a corpus whose recall is mysteriously flat across unrelated queries, and the trap is that the node you inspect in a REPL is a node you constructed by hand — not the one the reader built, which is the one in the store.

The StorageContext: where an index actually lives

An index is not a file. It is a view over three stores, bundled into a StorageContext, and knowing which store holds what is the difference between "persistence works" and "persistence appears to work until you restart".

• The docstore holds the nodes themselves — text, metadata, relationships. It is what makes dedup and incremental upsert possible: to know whether a node changed, you need last run's copy of it.
• The index store holds index metadata — the bookkeeping that says which node ids belong to which index, and for the non-vector index types, the structure itself.
• The vector store holds the embeddings, and it is the only one of the three that people usually think about. The default is SimpleVectorStore: a Python dict, in process memory, doing brute-force cosine similarity over every vector on every query. It is genuinely fine up to a few tens of thousands of chunks and a genuinely bad idea after that.

The default StorageContext keeps all three in memory and writes nothing to disk unless you ask, which produces the single most common beginner bug in LlamaIndex: the index works perfectly, the process exits, and the index is gone. storage_context.persist() writes the three stores as JSON to a directory; load_index_from_storage reads them back.

from llama_index.core import (
    VectorStoreIndex, SimpleDirectoryReader, StorageContext, load_index_from_storage,
)

docs = SimpleDirectoryReader("data").load_data()
index = VectorStoreIndex.from_documents(docs)
index.storage_context.persist(persist_dir="./storage")   # or it dies with the process


storage_context = StorageContext.from_defaults(persist_dir="./storage")
index = load_index_from_storage(storage_context)

Once you move to a real vector database, the shape of that round-trip changes, and this is the part that surprises people. A production vector store already persists the vectors and (usually) the node text and metadata as payload. So you don't reload the index from a JSON directory — you point at the live collection and build the index from the store, with no re-embedding and no local files at all:

import qdrant_client
from llama_index.core import VectorStoreIndex, StorageContext
from llama_index.vector_stores.qdrant import QdrantVectorStore

client = qdrant_client.QdrantClient(url="http://localhost:6333")
vector_store = QdrantVectorStore(client=client, collection_name="corpus")

# Ingestion: pass the store in, and from_documents() writes through to it.
storage_context = StorageContext.from_defaults(vector_store=vector_store)
index = VectorStoreIndex.from_documents(docs, storage_context=storage_context)

# Query time, any process, no persist_dir: the collection IS the index.
# Note what this call does NOT take: it pops any storage_context= you hand it
# (docstore and all), and raises ValueError unless vector_store.stores_text.
index = VectorStoreIndex.from_vector_store(vector_store)

The docstore is the store people skip, and skipping it has a cost: without an attached docstore, the ingestion pipeline has no memory of what it saw last run, so upsert-style dedup degrades into "insert everything again". If re-ingesting the same corpus is something you'll do more than once — and it always is — you want one.

But wanting one is not the same as having one, and this is the trap that catches almost everyone: attaching a docstore to the StorageContext does not populate it. VectorStoreIndex._add_nodes_to_index gates the docstore write behind if not self._vector_store.stores_text or self._store_nodes_override: — and Qdrant, Pinecone, Weaviate and pgvector all set stores_text=True, while store_nodes_override defaults to False. So with a real vector database, from_documents(docs, storage_context=sc) writes the nodes to the vector store and nowhere else. Your docstore stays empty, index.ref_doc_info returns {}, and the dedup you thought you wired is a no-op that raises nothing, because the index queries perfectly — the vector store has the text. If you want nodes in the docstore, pass store_nodes_override=True, or stop using from_documents and drive an IngestionPipeline(docstore=...) instead.

Fails when: you attach a docstore to the StorageContext, hand it to VectorStoreIndex.from_documents(), and it never gets written to. Read VectorStoreIndex._add_nodes_to_index and the branch is explicit: if not self._vector_store.stores_text or self._store_nodes_override: — the docstore and index_struct writes live inside that condition. Qdrant, Pinecone, Weaviate and pgvector all set stores_text=True, and store_nodes_override defaults to False, so with a real vector database the nodes go into the store and nowhere else. Your docstore stays empty, index.ref_doc_info returns {}, storage_context.persist() writes a docstore.json containing nothing, and the dedup you thought you wired is a no-op. Nothing raises — the index queries perfectly, because the vector store has the text. And VectorStoreIndex.from_vector_store() makes it worse on the read side: it calls kwargs.pop("storage_context", None) and builds a fresh default context, so a docstore you pass it is discarded silently. If you want nodes in the docstore, you either set store_nodes_override=True or you stop using from_documents and drive an IngestionPipeline with docstore= — attaching the store is not enough.

Indexes: a build strategy, not a data structure

The word "index" is doing unhelpful work here. In LlamaIndex, an index is better read as a strategy for turning a node set into a retrievable structure, plus the retriever that knows how to walk it. The four you'll actually meet:

• VectorStoreIndex — embed every node, retrieve by cosine similarity, top-k. The default, and the right default: it answers "find me the facts about X" at any corpus size, and it is the one every other rung of the patterns ladder is bolted onto.
• SummaryIndex (the old ListIndex) — keep the nodes in a flat, ordered list, and by default retrieve all of them. That sounds absurd until you realise what it's for: "summarize this whole document" is not a top-k question. No similarity threshold will give you a faithful summary of a corpus, because the answer isn't in the two nearest chunks — it's in all of them. A SummaryIndex over a tree_summarize synthesizer is the correct machine for that job. Be warned that "all of them" means it: the default retriever_mode is SummaryIndexRetriever, which returns every node in the index, and as_query_engine(similarity_top_k=5) accepts that argument and ignores it — similarity_top_k only exists on the retriever_mode="embedding" path, where it then defaults to 1.
• PropertyGraphIndex — extract entities and relations into a graph, then retrieve by traversal as well as by similarity. Reach for it when the answer requires hops ("which customers are affected by the outage in the region this vendor serves?") that no single chunk contains.
• DocumentSummaryIndex — index a generated summary per document, retrieve documents by summary, then return their nodes. A cheap, surprisingly strong answer to the "similarity keeps matching the wrong document" problem.

The reason to know the whole list is routing: once a corpus has to serve structurally different question types, the right move usually isn't a cleverer retriever — it's two indexes and a router that picks. Different index types, same nodes, same storage context.

Fails when: you point a SummaryIndex at a real corpus and try to make it behave. summary_index.as_query_engine(similarity_top_k=5) accepts that kwarg without complaint and then ignores it, because retriever_mode defaults to "default", which is SummaryIndexRetriever, which reads index_struct.nodes and returns every node in the index, scored None. Twenty thousand chunks go into a compact synthesizer, which packs them to the context window and refines sequentially across however many prompts that takes — so a call you budgeted at one LLM round-trip becomes hundreds, in series, and your first sign of it is a timeout or an invoice. The obvious escape is retriever_mode="embedding", and it has its own knife: SummaryIndexEmbeddingRetriever defaults to similarity_top_k=1. You go from reading the whole corpus to reading exactly one chunk, and both of those look like "the summary index is broken" rather than "you never chose a k."

Retrievers: the narrowest useful interface

BaseRetriever is the smallest and most important interface in the framework, and it is almost insultingly simple:

from llama_index.core.retrievers import BaseRetriever
from llama_index.core.schema import NodeWithScore, QueryBundle

class MyRetriever(BaseRetriever):
    def _retrieve(self, query_bundle: QueryBundle) -> list[NodeWithScore]:
        ...   # that's the whole contract: a query in, scored nodes out

A query in, a list of scored nodes out. That's it. Every retrieval idea in the ecosystem — hybrid search, fusion across multiple indexes, auto-generated metadata filters, recursive retrieval into sub-documents — is somebody's _retrieve. And because a query engine takes a retriever as a constructor argument, you can drop a custom one into the standard pipeline without forking anything downstream of it.

NodeWithScore is the other half of the contract, and its score field carries a wart worth knowing: the number is not comparable across retrievers. A cosine similarity from a vector store, a BM25 relevance score, and a cross-encoder rerank score all land in the same field with wildly different scales and distributions. This is exactly why naive hybrid search — "just merge the two lists by score" — produces worse results than either retriever alone, and why fusion retrievers use reciprocal rank fusion, which throws the scores away and merges on rank instead. If you ever find yourself writing if node.score > 0.7, know that you've hard-coded a threshold to one specific embedding model's score distribution, and it will not survive the model upgrade.

Fails when: you implement _retrieve and stop. _aretrieve is not abstract — BaseRetriever ships a default body that reads, in full, return self._retrieve(query_bundle), with a # TODO: make this abstract above it. So your custom retriever has a working async path from day one, and that async path is a synchronous blocking call sitting on the event loop. Everything that fans out concurrently — QueryFusionRetriever(use_async=True), SubQuestionQueryEngine, a @step in a workflow, any aquery — will await your coroutine, get no yield point, and serialize. It is not a bug you can see: the results are correct, nothing warns, and the only symptom is that use_async=True bought you nothing and p95 is the sum of your sub-retrievals rather than the max. Override _aretrieve properly, and while you're in there remember that retrieve() runs your output through _handle_recursive_retrieval, which resolves any IndexNode through object_map and then dedupes by node_id, keeping the first occurrence — so if you merge lists yourself, the ordering you hand back is the ordering that decides which duplicate survives.

Node postprocessors: the interception seam

Between "the retriever returned k nodes" and "the LLM reads them" there is a list, and you get to put things in it. That list is node_postprocessors, and it is the cheapest place in the entire architecture to buy quality, because everything in it operates on ~10 nodes rather than on your whole corpus.

A postprocessor is a function from a node list to a node list. Which means it can filter, reorder, or rewrite what the LLM sees:

• SimilarityPostprocessor — drop nodes below a score cutoff. The blunt one, with the score-scale caveat above.
• SentenceTransformerRerank / LLMRerank / CohereRerank — over-retrieve (say similarity_top_k=20), then rescore all 20 with a model that actually reads the query and the chunk together, and keep the best 5. A bi-encoder embedding compares two vectors that were computed in ignorance of each other; a cross-encoder reads the pair. That asymmetry is why reranking is the single highest-leverage addition to a naive RAG pipeline, and it belongs here, in the postprocessor list.
• MetadataReplacementPostProcessor — the trick that makes small chunks viable. Embed sentences (precise retrieval), but at synthesis time swap each retrieved sentence for the window of sentences around it (sufficient context). You get the precision of small chunks and the context of large ones, and it costs one postprocessor.
• LongContextReorder — LLMs attend worse to the middle of a long context ("lost in the middle"), so move the highest-scoring nodes to the ends of the list. A pure reordering, no model call, occasionally free accuracy.

from llama_index.core.postprocessor import (
    SimilarityPostprocessor, SentenceTransformerRerank, LongContextReorder,
)

query_engine = index.as_query_engine(
    similarity_top_k=20,                     # over-retrieve: cheap, it's just cosine
    node_postprocessors=[
        SimilarityPostprocessor(similarity_cutoff=0.5),
        SentenceTransformerRerank(model="cross-encoder/ms-marco-MiniLM-L-6-v2", top_n=5),
        LongContextReorder(),                # best evidence at the edges of the prompt
    ],
)

Order matters, and the order above is the one that makes sense: filter cheaply, then rerank what survived, then arrange what won. Not for the reason you'd guess, though — sparing the expensive step a few nodes you were about to discard is the trivial half of it. The sharp half is that a reranker overwrites the very score it sorts on, which turns a threshold placed after it into a context-shredder.

Fails when: you forget that score is a single mutable field that every postprocessor in the list writes to. SentenceTransformerRerank._postprocess_nodes does node.score = score — unconditionally, destroying the retrieval score unless you pass keep_retrieval_score=True, which stashes the old value in node.metadata["retrieval_score"] and is False by default. What replaces it is a cross-encoder logit, which is unbounded and routinely negative. So invert the order in that snippet — put SimilarityPostprocessor(similarity_cutoff=0.5) after the reranker, which reads like a perfectly sensible "keep the good ones" — and you delete your entire context, because -2.4 < 0.5. The same field bites from the other side: SimilarityPostprocessor treats score is None as a fail (if similarity is None: should_use_node = False), so any retriever that doesn't populate scores — SummaryIndexRetriever, most hand-rolled ones — has every node silently dropped by a postprocessor whose cutoff you set to 0.0 thinking it was a no-op. And MetadataReplacementPostProcessor fails upward: it calls node.metadata.get(self.target_metadata_key, node.get_content(MetadataMode.NONE)), so if the key isn't there — you swapped the node parser, or the window metadata didn't survive the store round-trip — it quietly substitutes the node's own text and you get precise little sentences with none of the context you built the whole pipeline to fetch.

Response synthesizers: how N chunks become one answer

You retrieved five nodes. They do not fit in one prompt, or they do but you'd rather they didn't. The synthesizer is the strategy for collapsing N chunks into one answer, and choosing the wrong mode is where LlamaIndex bills get strange and answers get lossy.

• refine — one LLM call per node, each call handed the previous answer and asked to improve it. Thorough, sequential, and O(n) calls: five nodes is five round-trips.
• compact (the default) — the same refine loop, but first pack as many nodes into each prompt as the context window allows. Usually collapses to a single call. This is the default for a good reason.
• tree_summarize — recursively summarize pairs/groups of nodes up a tree until one answer remains. O(n) calls but parallelizable and, crucially, the only mode that doesn't structurally favour the nodes it saw first. This is the mode for "summarize everything", and it's what you pair with a SummaryIndex.
• simple_summarize — truncate everything into one prompt and answer once. Cheapest, and it will silently drop evidence. Fine for a demo, dangerous in a product.
• accumulate — answer the query against each node separately and return the list. Not a summarizer at all — an extractor. When your consumer is code rather than a human, this is often the mode you actually wanted.

The failure mode worth naming: with refine and compact, the answer is built sequentially, so evidence in the last chunk is being asked to overturn an answer the model already committed to. If your users report "it found the right document but gave the old answer", and source_nodes shows the correct chunk was retrieved, look here before you blame retrieval. tree_summarize doesn't have that bias.

from llama_index.core import get_response_synthesizer
from llama_index.core.query_engine import RetrieverQueryEngine


query_engine = RetrieverQueryEngine(
    retriever=index.as_retriever(similarity_top_k=10),
    # Always name the model and the top_n: the class default model is an STS
    # *similarity* model (cross-encoder/stsb-distilroberta-base), not a relevance
    # reranker — and top_n defaults to 2, so it keeps two nodes and drops the rest.
    node_postprocessors=[
        SentenceTransformerRerank(model="cross-encoder/ms-marco-MiniLM-L-6-v2", top_n=4),
    ],
    response_synthesizer=get_response_synthesizer(response_mode="tree_summarize"),
)

response = query_engine.query("What changed in the Q3 pricing policy?")
print(response)                # the answer
print(response.source_nodes)   # the evidence it actually saw — log this in prod, always

That snippet is the whole point of this page. as_query_engine() is a convenience that builds exactly this; the moment any of the three arguments needs to be something else, you write the assembly out longhand and keep everything else.

Fails when: you assume the LLM reads your nodes. It doesn't — it reads whatever PromptHelper.repack produces, and repack is three lines: combined_str = "\n\n".join([c.strip() for c in text_chunks if c.strip()]), then a TokenTextSplitter sized to the remaining context window, then split_text(combined_str). Your node boundaries are concatenated away and re-cut on a token budget. compact does this, and so does tree_summarize. Two consequences you will meet in production. The chunk the model actually sees can be half of node 3 glued to the front of node 4 — so a "citation" the model produces is attributed to a span that never existed as a node, and any faithfulness judge you point at (response, source_nodes) is comparing the answer against text the model was not shown in that shape. And repack is lossless by splitting, not by truncating, which means the moment your retrieved nodes exceed the window, compact does not warn and does not drop — it silently becomes an N-call Refine chain, with all of the last-chunk-overturns-the-answer bias described above, at N times the cost. The one mode that genuinely truncates is simple_summarize, which calls prompt_helper.truncate and throws the overflow away without telling anyone. source_nodes will show you all five nodes either way; it is a record of what was retrieved, never a record of what was read.

Query engines, chat engines, agents

Three top-level abstractions, and they form a ladder of increasing autonomy — which is to say, increasing capability and increasing non-determinism. Pick the least autonomous one that answers the question.

• Query engine — one shot. Retrieve, synthesize, return. Stateless, predictable, cheap, and easy to evaluate because the same input gives you the same retrieval. Most "chat with your docs" products should be this plus a rewritten query.
• Chat engine — retrieval plus conversation memory, and one genuinely important extra step: condensing the history and the new message into a standalone query before retrieving. "What about the enterprise tier?" is un-retrievable on its own; against three turns of history it means something specific. That condensation step is the whole value, and it's why bolting a chat UI onto a bare query engine feels broken. Note what it is not, though: as_chat_engine() defaults to ChatMode.BEST, which resolves to a CondensePlusContextChatEngine built from self.as_retriever(**kwargs) — the retriever, not the query engine you configured. Your postprocessors and your synthesizer mode do not come with you; the chat engine brings its own.
• Agent — the LLM decides whether and what to retrieve, possibly several times, possibly against several tools. A query engine wrapped as a QueryEngineTool is just one more tool it can call. This is agentic RAG, and it buys you multi-hop questions and multi-corpus routing at the cost of a loop you can no longer predict the shape of. See LlamaAgents & Workflows.

The evaluation consequence is the part teams learn late: a query engine is testable with a fixed set of question/answer pairs because retrieval is a pure function of the question. An agent is not — the same question can take a different path on a different day. Do not step up this ladder for free; step up it when the question genuinely needs it, and bring the eval gate with you.

Fails when: you climb this ladder by accident, because as_chat_engine() has a default and it is not "query engine plus memory". chat_mode defaults to ChatMode.BEST, and BaseIndex.as_chat_engine resolves that to CondensePlusContextChatEngine.from_defaults(retriever=self.as_retriever(**kwargs), ...) — note the retriever, not the query engine you configured. Every node_postprocessors list, every synthesizer mode, every response_mode you set on the query-engine path is gone, replaced by the chat engine's own internal CompactAndRefine. Then _condense_question fires on every turn where len(chat_history) > 0, unconditionally, with no notion of whether the message needs retrieval at all — so "thanks, that's useful" gets rewritten by an LLM into a fully-formed standalone question about the previous topic and retrieved against, and turn six is answering a question nobody asked. Worst of all, the condensed string is the thing that was actually embedded, and it appears nowhere on the response object — it is written to logger.info and to stdout under verbose=True, and that is the entire audit trail. You will debug retrieval against a query you cannot see. (ChatMode.REACT and ChatMode.OPENAI are marked deprecated and unsupported in ChatMode's own docstrings; the agent rung is FunctionAgent, reached deliberately, not through a chat-engine kwarg.)

Settings: the global injection container

Settings is a process-global singleton that every component reaches into for its defaults: the LLM, the embedding model, the node parser, the chunk size, the callback manager, the tokenizer. It replaced the old ServiceContext, and it is a genuine ergonomic win — you configure the models once, and forty components stop asking you for them.

It is also, being a global, the source of the two most confusing bugs in the framework:

"Why is this calling OpenAI? I never configured OpenAI." Because Settings.llm and Settings.embed_model default to OpenAI, and any component you didn't explicitly hand a model to has quietly resolved one from the global. The fix is to set the globals before you construct anything.

"Why is my chunk size being ignored?" Because Settings.chunk_size tunes the default node parser, while Settings.transformations replaces the entire transformation list. Set both, and the list wins — your chunk size is being applied to a node parser that is no longer in the pipeline.

from llama_index.core import Settings
from llama_index.llms.anthropic import Anthropic
from llama_index.embeddings.huggingface import HuggingFaceEmbedding


Settings.llm = Anthropic(model="claude-sonnet-5")
Settings.embed_model = HuggingFaceEmbedding(model_name="BAAI/bge-small-en-v1.5")
Settings.chunk_size = 512

# Local override beats the global — pass the dependency explicitly when it matters.
engine = index.as_query_engine(llm=Anthropic(model="claude-opus-4-8"))

The rule that keeps this sane: globals for the boring defaults, explicit arguments for anything a reviewer would want to see. A cheap model for the fifty extraction calls in ingestion and an expensive one for the final synthesis is a decision that belongs in the code that makes it, not in a global someone has to go hunting for.

Fails when: the ingest process and the query process disagree about Settings.embed_model, and nothing anywhere notices. VectorStoreIndex.from_vector_store(vector_store) takes an optional embed_model that defaults to None, which resolves to Settings.embed_model, which resolves — via resolve_embed_model("default") — to OpenAIEmbedding() if you never set it. The collection carries no record of which model wrote it. So an ingest job that set Settings.embed_model = HuggingFaceEmbedding(...) and a query service that forgot to, or that pinned a different OpenAI model, will happily talk to the same store: the query vector has the right dimensionality, the ANN index accepts it, cosine comes back in the 0.6–0.8 band that looks exactly like normal retrieval, and the nearest neighbours are noise. No exception, no warning, no dimension mismatch to save you — just a system that has quietly become a random-chunk generator with citations. Set the embedding model explicitly on the index at both ends, and hard-fail on startup if the two don't match. (The same global has a smaller trap: the chunk_size setter mutates Settings.node_parser in place, so it retroactively reconfigures a parser object other components are already holding — and raises ValueError outright if the configured parser has no chunk_size attribute.)

Workflows: the runtime underneath the agents

The pipeline abstractions LlamaIndex used to ship (query pipelines, DAG-shaped chains) had the flaw every DAG has: a DAG cannot express a loop, and agents are loops. Workflows are the replacement, and they're event-driven rather than graph-shaped.

You write @step methods. Each step declares the event type it consumes and returns the event it emits. There is no central graph — the runtime routes events to whichever step accepts them, which means a step that emits an event consumed by an earlier step is a loop, expressed without any special looping construct. Fan-out is emitting several events; fan-in is a step that waits for several.

from llama_index.core.workflow import Workflow, StartEvent, StopEvent, Event, Context, step

class Retrieved(Event):
    nodes: list

class RAGWorkflow(Workflow):
    @step
    async def retrieve(self, ctx: Context, ev: StartEvent) -> Retrieved:
        retriever = ev.index.as_retriever(similarity_top_k=5)
        return Retrieved(nodes=await retriever.aretrieve(ev.query))

    @step
    async def synthesize(self, ctx: Context, ev: Retrieved) -> StopEvent:
        # ...judge the nodes; emit a Retrieved again to loop and re-query...
        return StopEvent(result="...")

result = await RAGWorkflow(timeout=60).run(index=index, query="What changed in Q3?")

The architectural payoff is that the agents are not a separate system. A LlamaIndex agent is a workflow whose steps happen to be "call the LLM" and "run the tool it picked", and its loop is the same event round-trip as the one above. That's what makes human-in-the-loop and durable execution tractable: the runtime already has a notion of "an event arrived and a step ran", so "the event arrives from a human, three hours later" is a variation on a theme rather than a rewrite. This is the seam that LlamaAgents & Workflows picks up, and the same shape you'll recognise from LangGraph.

Fails when: the fan-in count and the fan-out count drift apart, and the workflow simply stops. ctx.collect_events(ev, [RetrievedEvent] * 3) returns None until all three have arrived — which is why the collecting step must be typed -> StopEvent | None and must return None on the miss — and the 3 is a literal you hard-coded, with nothing tying it to the number of events the fan-out step actually emitted. Emit four and one sits in the buffer forever; emit two and the join never fires. Static validation does not catch it: the type graph is perfectly well-formed, every event type has a consumer, and ctx.send_event is a runtime call that the graph checker cannot see through anyway. So there is no error. There is a workflow that sits there, doing nothing, until Workflow.__init__'s timeout — which defaults to 45.0 seconds, not None — expires and raises WorkflowTimeoutError. From the outside this is indistinguishable from a slow LLM or a hung network call, and the only thing that tells you otherwise is the WorkflowTimedOutEvent published to the stream, whose active_steps list will be conspicuously empty. The same event loop hands you the second one for free: steps run concurrently, so two steps doing await ctx.store.get("x") / set("x", ...) around an LLM call will lose writes unless you wrap the read-modify-write in async with ctx.store.edit_state().

Instrumentation: the dispatcher and the span tree

Every component in the framework emits events and opens spans through a central dispatcher, arranged in a tree that mirrors the call stack: a query span contains a retrieve span contains an embedding span, and each carries its own payload. This is the mechanism the observability integrations (Arize/Phoenix, LangSmith, Langfuse, OpenTelemetry exporters) hook into — they are subscribers, not wrappers.

The reason to care is a specific, recurring production question: "why did it retrieve that?" You cannot answer it from the response object. The answer lives in the span tree — the query that was actually embedded after condensation and transformation, the filters that were actually applied, the scores before and after reranking, the chunks that lost. A one-line integration gets you the whole trace, and it is the difference between debugging RAG and guessing at it.

The rule of thumb that survives contact with production: log response.source_nodes from day one, and wire a real tracer before your first quality complaint, not after. Nearly every "the model is dumb" bug turns out to be visible in the trace as a chunk that was never retrieved — and, as the patterns page puts it, a bigger model cannot read a chunk you never gave it.

Fails when: your handler raises and the dispatcher eats it. Read Dispatcher.event() and the loop is literally for h in c.event_handlers: try: h.handle(event, **kwargs) except BaseException: pass — span_enter, span_exit and span_drop do the same. Every exception your observability layer throws is swallowed, silently, by design, so the framework can never be broken by a bad subscriber. Which is correct, and which means a BaseEventHandler that dies on a KeyError against a payload field that moved, or a span exporter whose collector is refusing connections, produces exactly the same output as no handler at all: no traces, no errors, no log line, an application that looks perfectly healthy. You will re-check your add_event_handler call four times before you think to put a bare print inside handle(). Two smaller edges from the same code: handlers run inline and synchronously on the request path, so a handler that does network I/O is latency you added to every query; and propagate defaults to True, so a handler attached to both a module dispatcher and the root sees each event twice — which is how token counters end up reporting exactly double.

Where to go next. The patterns ladder is these seams put to work, rung by rung — each pattern is, in the end, a specific choice about which of the layers above to replace. And the hub is the two-phase model this whole graph is an elaboration of.