LlamaIndex Patterns

Every one of these ten patterns lives inside the two-phase model from the LlamaIndex hub: ingestion builds an index offline, a query reads it online. What changes from pattern to pattern is how much machinery you put on each side — and the honest way to read this page is as a ladder. You start at the bottom with a VectorStoreIndex and a query engine, which is genuinely enough for a surprising number of corpora. Then it breaks, and each rung above is a specific answer to a specific way it broke.

That framing matters because of the most common failure in production RAG: the answers are wrong, and the team reaches for a bigger model. A bigger model cannot read a chunk you never retrieved. Almost every quality problem in the first year of a RAG system is a retrieval problem wearing a generation problem's clothes — the chunk boundary cut the answer in half, similarity search pulled last year's tenant, the exact error code was invisible to the embedding. Climb the ladder before you climb the price list.

The rungs get progressively more agentic — routing, then query engines as tools, which is where this page hands off to llama-agents — and the top rung is the eval gate. Put that in place and every rung below becomes safe to change, because you can finally tell better from merely different.

The Baseline: VectorStoreIndex + Query Engine

Every LlamaIndex tutorial opens with the same five lines, and they are worth taking seriously — not because you will ship them, but because the baseline is the control group: it is the number that every reranker, every metadata filter, and every hybrid retriever you add later has to beat. If you can't say what as_query_engine().query() scores on your corpus, you have no way to know whether the pipeline you spent two weeks on is an improvement or an expensive lateral move.

Read VectorStoreIndex.from_documents as a pipeline that has been collapsed into one call. SimpleDirectoryReader walks a folder and hands back Document objects. from_documents runs them through the ingestion transformations — by default Settings.transformations, which is just [Settings.node_parser], which is a bare SentenceSplitter(): 1024-token chunks with 200 tokens of overlap (SentenceSplitter's own SENTENCE_CHUNK_OVERLAP default — not the DEFAULT_CHUNK_OVERLAP = 20 constant you'll see quoted in half the blog posts). It embeds each resulting TextNode with whatever Settings.embed_model resolves to (OpenAI, so it fails loudly without an API key), and stuffs the vectors into a SimpleVectorStore: a Python dict in process memory doing brute-force cosine similarity. Nothing is persisted unless you call index.storage_context.persist().

as_query_engine() collapses a second pipeline. It builds a VectorIndexRetriever with similarity_top_k=2 — yes, two chunks — and a response synthesizer in compact mode, which packs those chunks into as few LLM calls as the context window allows and refines an answer across them. So the whole of "RAG" here is: embed the question, cosine-rank the corpus, take the top 2 chunks, and ask the LLM to answer over them. There is no filtering, no reranking, and no notion of whether the retrieved chunks are actually relevant — only that they were the closest two vectors, which is true even when the corpus contains nothing relevant at all.

from llama_index.core import SimpleDirectoryReader, VectorStoreIndex, Settings
from llama_index.core.node_parser import SentenceSplitter


Settings.chunk_size = 1024
Settings.chunk_overlap = 200        # SentenceSplitter's real default, not 20
# ...or replace the transformation list outright. Setting `transformations`
# replaces the whole pipeline, so it wins over the two lines above.
Settings.transformations = [SentenceSplitter(chunk_size=1024, chunk_overlap=200)]

documents = SimpleDirectoryReader("data").load_data()
index = VectorStoreIndex.from_documents(documents)   # embed -> SimpleVectorStore (in-memory)

query_engine = index.as_query_engine()               # similarity_top_k=2, response_mode="compact"
response = query_engine.query("What changed in the Q3 pricing policy?")

print(response)                 # the synthesized answer
print(response.source_nodes)    # the 2 chunks it saw — always log these
index.storage_context.persist(persist_dir="./storage")   # otherwise it dies with the process

Two habits make the baseline useful rather than merely fast. First, always read response.source_nodes alongside the answer: it is the only honest signal about whether the retriever did its job, and most "the LLM hallucinated" bug reports turn out to be "the retriever returned the wrong two chunks." Second, treat as_query_engine() as sugar over VectorIndexRetriever + get_response_synthesizer + RetrieverQueryEngine — the moment you need a node_postprocessors reranker or metadata filters, you drop to that composition, and knowing it is the same object graph keeps the migration boring. See /rag for the retrieval strategies layered on top, /structured-outputs for turning answers into typed payloads, and /llama-agents for wrapping the query engine as a tool an agent can call.

The production failure is not that vector search is bad; it's that this baseline has no way to fail. It cannot say "I don't know," because top-2 always returns two chunks. It cannot honor "only 2025 contracts," because the chunks carry no queryable metadata. It cannot adapt top_k to a question that needs eight chunks or one. And with no eval harness attached, you find out about all of this from a user, in a screenshot, three weeks after launch.

Fails when: the answer spans more than two chunks, or the corpus doesn't contain it at all — similarity_top_k=2 cheerfully returns the two nearest-but-irrelevant chunks and the synthesizer confabulates a fluent answer over them. There is no score cutoff in the default path (SimilarityPostprocessor is opt-in via node_postprocessors), no reranker, and no metadata filter anywhere in the chain to stop it.

Ingestion Pipeline: Caching, Dedup, Incremental Sync

Retrieval is a two-phase contract. The online half — retrieve, rerank, synthesize — is what /rag is about. The offline half is the one that quietly bankrupts you: every time a cron job re-reads the same 40k-document corpus and re-embeds it end to end, you pay full price for a result you already have. Re-embedding an unchanged corpus is the single largest source of avoidable spend in a production RAG system, and it is entirely a bookkeeping problem, not a modelling one.

IngestionPipeline from llama_index.core.ingestion is that bookkeeping. It is an ordered list of TransformComponents — a splitter, maybe a TitleExtractor, an embedding model — applied to Documents to produce BaseNodes, plus two side stores that make the whole thing idempotent.

The two hashes

There are two independent hash checks, and confusing them is where people get burned.

The cache (IngestionCache) is keyed per transformation step, and — this is the part everyone gets wrong — per batch, not per node. get_transformation_hash concatenates the content of every node handed to that step and hashes it together with the transformation's serialized config. So the key is all-or-nothing: one new document in the batch changes the key, and the whole step re-runs on everything. Change the chunk size and the key changes too, so the cache correctly misses. What makes it useful is not the cache alone but the cache behind the docstore, which narrows the batch to only the documents that actually changed before the key is ever computed. Cache hits then cover the expensive steps — TitleExtractor-style LLM calls, embeddings — for content you've already processed.

The docstore is keyed per document, on doc_id → doc.hash. This is what docstore_strategy governs. DocstoreStrategy.UPSERTS (the default) processes a document only when its ID is new or its hash has moved, and when the hash has moved it calls docstore.delete_ref_doc and vector_store.delete(ref_doc_id) before inserting the new nodes. That deletion step is the whole point: without it you don't get dedup, you get accumulation — the edited doc's stale chunks stay in the index forever, competing with the fresh ones at retrieval time.

The trap: upsert semantics require an attached vector_store. With a docstore but no vector store, run() warns and silently falls back to DUPLICATES_ONLY for that run — there is nowhere to issue the delete, so it doesn't. DocstoreStrategy.UPSERTS_AND_DELETE goes further and evicts documents absent from the current run, which is what you want when the source of truth is a folder or a Drive listing rather than an append-only feed; it has the same vector-store dependency. DUPLICATES_ONLY dedups on node.hash alone, ignoring doc_id — an edited document is simply a new hash, so it gets appended and nothing is cleaned up. Use it only for immutable corpora.

import qdrant_client
from llama_index.core.ingestion import (
    IngestionPipeline, IngestionCache, DocstoreStrategy,
)
from llama_index.core.node_parser import SentenceSplitter
from llama_index.core.storage.docstore import SimpleDocumentStore
from llama_index.embeddings.openai import OpenAIEmbedding
from llama_index.vector_stores.qdrant import QdrantVectorStore

client = qdrant_client.QdrantClient(url="http://localhost:6333")
vector_store = QdrantVectorStore(client=client, collection_name="corpus")

pipeline = IngestionPipeline(
    transformations=[SentenceSplitter(chunk_size=512), OpenAIEmbedding()],
    docstore=SimpleDocumentStore(),
    vector_store=vector_store,           # required for UPSERTS to actually delete
    docstore_strategy=DocstoreStrategy.UPSERTS,
    cache=IngestionCache(),              # SimpleCache: in-memory until you persist
)
nodes = pipeline.run(documents=docs, num_workers=4, show_progress=True)
pipeline.persist("./pipeline_storage")   # cache + docstore; .load() next run

persist()/load() is the toy version — and so is the default IngestionCache(), which is a SimpleCache living in process memory until you write it to disk. Fine for a single box, useless the moment two workers ingest concurrently. In production, back both stores with something shared:

from llama_index.storage.docstore.redis import RedisDocumentStore
from llama_index.storage.kvstore.redis import RedisKVStore as RedisCache

pipeline = IngestionPipeline(
    transformations=[SentenceSplitter(chunk_size=512), OpenAIEmbedding()],
    docstore=RedisDocumentStore.from_host_and_port("127.0.0.1", 6379, namespace="ingest"),
    vector_store=vector_store,
    docstore_strategy=DocstoreStrategy.UPSERTS,
    cache=IngestionCache(
        cache=RedisCache.from_host_and_port(host="127.0.0.1", port=6379),
        collection="ingest",
    ),
)

(IngestionCache.cache is typed BaseKVStore, so any KV store — Redis, Mongo, Firestore — drops straight in.) Then a second worker on the same corpus finds every hash already present and does nothing, which is exactly the behaviour you want from a job that runs every fifteen minutes.

Wiring it to the read path

When you pass a vector_store, run() inserts the nodes for you and returns them; you don't build an index from documents at all. The query side attaches to the same store:

VectorStoreIndex.from_vector_store(vector_store) — and from there it's a retriever, a query engine, or a tool handed to an agent (/llama-agents, /llamaindex). Ingestion and query are two processes sharing one store, not one script.

Two operational notes. Your doc_id must be stable and derived from the source — a URL, a primary key, a file path — not autogenerated per run, or every sync looks like a corpus of brand-new documents and the docstore buys you nothing. And treat the transformation list as a versioned artifact: it belongs in the same review as your prompts and your /evals, because changing a chunk size silently invalidates every cache entry and triggers a full re-embed you should have budgeted for. Metadata extractors that emit typed fields pair well with /structured-outputs when you need those fields to survive as filterable columns rather than prose blobs.

Fails when: the source system rewrites doc_ids (or you regenerate them per run). The docstore can no longer match the old ref_doc_id, so it never issues the delete — every document looks new, the splitter mints fresh node IDs, and the vector store ends up holding two copies of every chunk. Retrieval starts returning near-duplicate context and your top-k budget is spent on redundancy. (The embedding bill is the second-order damage: the transformation hash is content-based, so a bit-identical corpus may still hit cache — but any real change shifts the batch hash and you pay for the whole re-embed anyway. The duplicates you pay for every query, forever.)

Chunking & Node Parsing

Chunking is the only decision in a RAG pipeline that you make before you know what the questions will be. Every downstream knob — embedding model, top-k, reranker — operates on units you already committed to at ingest time. A chunk is simultaneously the unit of retrieval and the unit of context, and those two want different sizes: retrieval wants chunks small enough that the embedding isn't a smear of five unrelated topics, generation wants chunks large enough that the answer isn't cut in half. Most of the node-parser zoo in llama_index.core.node_parser exists to decouple those two.

The default, and the one you should not move away from without evidence, is SentenceSplitter. It packs text up to chunk_size tokens while preferring to break on sentence and paragraph boundaries, with chunk_overlap tokens of tail repeated into the next chunk. TokenTextSplitter is the same idea without the sentence awareness — it splits on a separator (" ", falling back to backup_separators) and will happily bisect a sentence. Use it only for text with no prose structure. The overlap is a hedge, not a feature: it exists so a fact straddling a boundary appears intact in at least one chunk, and it costs you duplicate embeddings and duplicate hits in top-k. The shipped defaults are 1024/200, which is a reasonable starting point for dense prose; drop to 256–512 when your corpus is FAQ-shaped and each atomic fact is short.

For structured source text, prefer a parser that reads the structure rather than one that guesses at it. MarkdownNodeParser splits on headers and stamps the header path into node metadata; CodeSplitter uses tree-sitter to split at AST boundaries so a function is never severed mid-body (it needs a language and the tree-sitter language pack installed). Free structure beats inferred structure every time.

SemanticSplitterNodeParser is the one people over-reach for. It needs an embed_model, groups adjacent sentences into windows (buffer_size, default 1), embeds every window in one batched get_text_embedding_batch call, walks the document computing cosine distance between consecutive windows, and cuts wherever the distance exceeds the breakpoint_percentile_threshold (default 95). The result is variable-length chunks that respect topic shifts. The cost is real: at ingest you embed the entire corpus once at sentence granularity, on top of the embedding per resulting chunk, and the chunk boundaries are non-deterministic across embedding model versions. It earns its keep on long, unsegmented, topic-drifting text — transcripts, scraped pages, legal prose with no headings. On a well-headered corpus it buys you nothing that MarkdownNodeParser didn't already give you for free. Decide this with an eval on retrieval hit-rate, not by vibes.

The real answer to the small-vs-large tension is to stop choosing. HierarchicalNodeParser.from_defaults(chunk_sizes=[2048, 512, 128]) produces every level at once, wiring each node to its parent. You index only get_leaf_nodes(nodes) in the vector store, put the full node list in a docstore, and hand both to AutoMergingRetriever: it retrieves precise 128-token leaves, then — when enough siblings under one parent are hit — swaps them for the parent, so the LLM sees the coherent 512-token block rather than three shards of it. SentenceWindowNodeParser is the cheap cousin of the same trick: embed one sentence, stash its neighbours in metadata, and expand at query time with MetadataReplacementPostProcessor.

from llama_index.core import VectorStoreIndex, StorageContext
from llama_index.core.node_parser import HierarchicalNodeParser, get_leaf_nodes
from llama_index.core.storage.docstore import SimpleDocumentStore
from llama_index.core.retrievers import AutoMergingRetriever

parser = HierarchicalNodeParser.from_defaults(chunk_sizes=[2048, 512, 128])
nodes = parser.get_nodes_from_documents(documents)   # all levels, parent-linked

docstore = SimpleDocumentStore()
docstore.add_documents(nodes)                        # every level lives here
storage = StorageContext.from_defaults(docstore=docstore)

index = VectorStoreIndex(get_leaf_nodes(nodes), storage_context=storage)
retriever = AutoMergingRetriever(
    index.as_retriever(similarity_top_k=12), storage, verbose=True
)

All of this rides on node relationships, which is the part people skip. Every NodeParser sets include_prev_next_rel=True by default, so each node carries NodeRelationship.PREVIOUS / NEXT pointers to its neighbours and SOURCE back to the originating Document; the hierarchical parser adds PARENT / CHILD. These live in node.relationships as RelatedNodeInfo, and they are what makes post-retrieval expansion possible at all. The relationships themselves usually survive a round-trip through a vector store — most integrations serialize the whole node into a _node_content metadata blob — but the targets don't: you only indexed the leaves, so a PARENT id points at a node the vector store has never seen. AutoMergingRetriever resolves it with storage_context.docstore.get_node(...), which is why the docstore is load-bearing rather than optional. Note also that prev/next links are only wired between nodes that share a SOURCE, so relationships never leak across document boundaries.

Parsers are just TransformComponents, so they compose the same way everywhere: pass transformations=[parser] to VectorStoreIndex.from_documents, set Settings.node_parser globally, or drop them into an IngestionPipeline where the doc-hash cache means re-running ingest on an unchanged corpus is free. If you're extracting fields per chunk rather than retrieving them, see structured outputs; if chunks are being fetched by a tool-calling agent, LlamaIndex agents will happily retrieve at the wrong granularity forever without telling you.

Fails when: you tune chunk_size against generation quality instead of retrieval quality — large chunks make answers read fine while the embedding of a 2048-token chunk averages four topics into a vector that matches no specific query, so recall silently collapses and you blame the retriever.

Metadata Filtering & Auto-Retrieval

Embeddings only know what the text says, not what the row is. A chunk from last year's postmortem and this year's are near-identical in vector space; so are the same policy document for two different tenants. When your corpus is partitioned along an axis the embedding cannot see — tenant_id, year, service, doc_type, ACL group — similarity search will happily return the wrong partition with a great score. A metadata filter is a correctness constraint, not a relevance hint: it changes the candidate set before the ANN search runs, where raising top_k only changes how much of the wrong set you look at.

That distinction is the whole pattern. Retrieving top_k=50 instead of top_k=5 to "make sure the right doc is in there" costs you latency, tokens, and — because rerankers and LLMs are both distractible — accuracy. Pushing MetadataFilters down into the vector store costs you nothing: Qdrant, Pinecone, Weaviate, and pgvector all translate the filter into a native pre-filter, so you get five results out of the right thousand rows instead of five out of the wrong million. Anything that must be true of an answer — tenancy, ACL, freshness window — belongs in the filter, never in the prompt.

The construction is MetadataFilters(filters=[...], condition=...), where each MetadataFilter is a key / value / operator triple. FilterOperator covers EQ, NE, GT/GTE/LT/LTE, IN/NIN, CONTAINS, ANY/ALL, TEXT_MATCH, TEXT_MATCH_INSENSITIVE, and IS_EMPTY; FilterCondition gives you AND, OR, NOT, and MetadataFilters nests inside itself for compound predicates. Hand it to index.as_retriever(filters=...) or as_query_engine(filters=...) and you're done. (ExactMatchFilter still imports, but it is a backwards-compat alias of MetadataFilter — write the triple.) The catch: the operators a given store actually supports vary, and metadata keys must exist on the nodes at ingest time — see /rag for why you attach that metadata during chunking rather than bolting it on later.

Hard filters assume the caller knows the filter. Often the question knows it: "what broke in checkout in 2024?" contains a service and a year. VectorIndexAutoRetriever closes that gap by giving the LLM a VectorStoreInfo — a content_info string plus a list of MetadataInfo(name, type, description) — and asking it to emit a VectorStoreQuerySpec: a rewritten query string, the filters it inferred, and an optional top_k. It is structured output applied to the retrieval layer, and the field descriptions are the prompt. Describe the enum ("one of checkout, auth, billing"), not the column.

from llama_index.core.retrievers import VectorIndexAutoRetriever
from llama_index.core.vector_stores import (
    FilterOperator, MetadataFilter, MetadataFilters, MetadataInfo, VectorStoreInfo,
)

info = VectorStoreInfo(
    content_info="engineering incident postmortems",
    metadata_info=[
        MetadataInfo(name="service", type="str", description="one of: checkout, auth, billing"),
        MetadataInfo(name="year", type="int", description="year the incident occurred"),
    ],
)
retriever = VectorIndexAutoRetriever(
    index,
    vector_store_info=info,
    extra_filters=MetadataFilters(  # AND-ed on; condition must not be OR
        filters=[MetadataFilter(key="tenant_id", value=tid, operator=FilterOperator.EQ)]
    ),
    similarity_top_k=5,
    empty_query_top_k=10,  # used when the LLM infers filters but a blank query string
    max_top_k=10,          # hard ceiling: clamps both similarity_top_k and the LLM's top_k
    verbose=True,          # logs the inferred spec
)
nodes = retriever.retrieve("what broke in checkout in 2024?")  # list[NodeWithScore]

Two constructor arguments carry most of the production weight. extra_filters is AND-ed onto whatever the LLM infers — that is where tenancy and ACLs go, because you must never let a model decide whether to scope a query to the caller's tenant. (It must not be an OR; the retriever raises ValueError("extra_filters cannot be OR condition") on one.) And empty_query_top_k — 10 by default, None to fall back to similarity_top_k — handles the case where the LLM correctly infers filters but leaves the query string blank, which is what a pure-lookup question like "list every 2024 auth incident" should produce: without it, you're doing similarity search against nothing. Watch max_top_k too: it defaults to 10 and silently clamps both the LLM-chosen and your own similarity_top_k, so raising one without the other does nothing. Combine them and the auto-retriever becomes a safe front-end — the LLM narrows, your code constrains.

Auto-retrieval is a call the model can get wrong, so treat it as one. Log the inferred spec (verbose=True, or wrap _generate_retrieval_spec), and in evals score filter precision and recall separately from answer quality: a query that returns zero nodes because the LLM hallucinated service="cart" looks identical, downstream, to a genuinely empty corpus. In an agent, this is usually the retriever behind a single tool rather than a tool the agent parameterises itself — the agent asks in natural language, the auto-retriever does the translating, and the index stays the only thing that knows your schema.

Fails when: the filter misses and nothing raises. Two distinct shapes, and only one of them is the loud one. Over-narrow to zero hits — a hallucinated service="cart", a mis-cased "Checkout", year=2025 on a 2024 corpus, an AND chain no node satisfies — and the response synthesizer short-circuits on len(nodes) == 0 and hands back the literal string "Empty Response" without ever calling the LLM: not a wrong answer, but a placeholder that raises no exception and that callers, agents, and eval harnesses will cheerfully treat as content. The genuinely dangerous shape is the near miss — right service, wrong year; correct filter, missing tenant — which returns a full, plausible node list from the wrong partition and produces a confidently wrong answer with citations. So: assert on len(nodes) and fall back to a relaxed filter set rather than trusting an empty retrieve, and assert that the returned nodes' metadata actually match the filter you meant to apply rather than trusting a non-empty one.

Hybrid Search + Reranking

Dense retrieval fails in a specific, predictable way: embeddings capture topic, not tokens. A query for ERR_CONN_RESET or invoice INV-2024-8871 embeds as "some error-ish thing" or "some invoice-ish thing", and cosine similarity happily returns twenty neighbours that are all about the right subject and none of them about the right string. BM25 has the mirror flaw — it nails the rare token and misses the paraphrase. Hybrid search is not a hedge between two mediocre retrievers; it is the recognition that lexical and semantic matching fail on disjoint query classes, so the union of their candidates has far better recall than either alone.

The union is the easy half. The hard half is that you now have two ranked lists with incomparable scores — a cosine of 0.83 and a BM25 score of 14.2 mean nothing to each other, and normalising them is a tuning rabbit hole. Reciprocal rank fusion sidesteps the whole problem by throwing the scores away and keeping only the ranks: each document scores sum(1 / (k + rank_i)) across the lists it appears in. A document ranked #2 by both retrievers beats one ranked #1 by only one of them. It has no hyperparameters worth tuning and it cannot be broken by a retriever with a badly-scaled score distribution. In LlamaIndex this is QueryFusionRetriever with mode=FUSION_MODES.RECIPROCAL_RANK.

But RRF is still a cheap positional heuristic — it never looks at the query and the document together. That is what the reranker is for. A cross-encoder like SentenceTransformerRerank runs query and passage through one transformer forward pass jointly, with full cross-attention between them, and emits a relevance score. This is far more accurate than any bi-encoder retrieval and completely unusable as a retriever, because it is O(corpus) — you cannot precompute anything. Hence the shape of the whole pattern: retrieve wide with things that are cheap and approximate, rerank narrow with the thing that is expensive and correct. Fuse 40 candidates, score all 40 with the cross-encoder, keep the best 5 for the context window.

from llama_index.core import VectorStoreIndex
from llama_index.core.retrievers import QueryFusionRetriever
from llama_index.core.retrievers.fusion_retriever import FUSION_MODES
from llama_index.core.query_engine import RetrieverQueryEngine
from llama_index.core.postprocessor import SentenceTransformerRerank
from llama_index.retrievers.bm25 import BM25Retriever

index = VectorStoreIndex(nodes)
fusion = QueryFusionRetriever(
    [index.as_retriever(similarity_top_k=20),
     BM25Retriever.from_defaults(nodes=nodes, similarity_top_k=20)],
    mode=FUSION_MODES.RECIPROCAL_RANK,
    num_queries=1,        # >1 asks an LLM to paraphrase the query first
    similarity_top_k=40,  # wide: this is the reranker's candidate pool
    use_async=True,
)
rerank = SentenceTransformerRerank(model="cross-encoder/ms-marco-MiniLM-L-6-v2", top_n=5)
engine = RetrieverQueryEngine.from_args(fusion, node_postprocessors=[rerank])

Two numbers govern everything here, and they pull against each other. similarity_top_k on the fusion retriever is the recall budget — a document the fusion stage drops can never be recovered, so this is a ceiling on the whole pipeline's quality. top_n on the reranker is the precision budget — every node past it is context you pay for and attention you dilute. Widening the pool is nearly free on the retrieval side (an ANN scan and an inverted-index lookup) but linear on the rerank side, which is where your latency lives.

That latency is worth being concrete about. A local MiniLM cross-encoder on 40 candidates costs roughly 50–150ms on GPU and can hit half a second on CPU; a hosted reranker like CohereRerank adds a network round-trip but no local compute. LLMRerank is the expensive option — it asks an LLM to score batches of documents, which means real tokens and seconds, not milliseconds, and it is best kept for offline evaluation or genuinely low-QPS paths rather than an interactive endpoint. Note also that num_queries > 1 silently inserts an LLM call before retrieval to paraphrase the query; it helps on vague questions and it doubles your p50, so make that trade knowingly rather than by leaving the default on. Reranking is a node_postprocessor, so it composes the same way in an agentic retrieval tool as in a plain query engine — see /llama-agents for the workflow framing and /llamaindex for the surrounding index and ingestion machinery.

Do not adopt this on faith. The entire justification for the extra hop is a measurable lift in hit-rate and MRR over the plain vector baseline on your queries, and reranker quality is famously domain-sensitive — a model tuned on MS MARCO web passages may do nothing at all for your internal API docs. Build the eval set first (/evals), then measure retrieval in isolation before you measure the generated answer, so you can tell a retrieval regression from a prompting one. Everything downstream — grounded citations, structured extraction — inherits whatever this stage hands it, which is the general point made in /rag.

Fails when: the reranker's top_n is tuned as if it were a retrieval knob. Shrink the fusion pool to 10 to make the cross-encoder cheap and you have thrown away the recall that hybrid search bought you — the reranker can only reorder what it is given, so it now sorts a candidate set that no longer contains the answer, and returns a confidently-ranked wrong document instead of an obviously-empty result.

Query Transformations: HyDE, Multi-Step, Sub-Question

Most retrieval failures get blamed on the index. You re-chunk, you swap the embedding model, you add a reranker — and recall barely moves. Sometimes the index was fine and the question was the broken part: it was phrased in vocabulary the corpus never uses, or it silently bundled three lookups into one sentence. Query transformations attack that half of the problem. They rewrite the question before it ever touches the retriever, and LlamaIndex ships three of them that fail in usefully different ways.

HyDEQueryTransform handles vocabulary mismatch. The user asks "why did the deploy blow up last night?"; your runbooks say "rollout aborted: readiness probe exceeded initialDelaySeconds." Those two strings are not close in embedding space, because a question and an answer are different genres of text — questions are short and interrogative, documents are long and declarative. HyDE closes the gap by asking the LLM to hallucinate an answer first, then embedding that fake document instead of the question. The fabricated passage is factually worthless but stylistically correct, and it lands much nearer the real chunks. You wrap an existing engine with TransformQueryEngine; include_original=True (the default) keeps the raw query in the embedding mix as a hedge.

MultiStepQueryEngine handles sequential dependency — questions where step two cannot be written until step one is answered. "What did the author do after the company he founded was acquired?" requires knowing the company first. StepDecomposeQueryTransform looks at the query, the index_summary, and the reasoning so far, then emits the next sub-query. It is a loop, bounded by num_steps (default 3) and short-circuited by early_stopping, and each turn costs an LLM call plus a retrieval. Note that index_summary defaults to the literal string "None" — if you don't pass a real description of what the index contains, you are telling the planner it has nothing to work with, and the sub-queries degrade accordingly.

SubQuestionQueryEngine handles parallel composition — questions that fan out over several sources at once. You hand it QueryEngineTools, each with a ToolMetadata description, and it plans a batch of sub-questions routed to specific tools, runs them concurrently under use_async=True, and synthesizes over the collected answers. The tool descriptions are the whole ballgame here; a vague one gets every sub-question routed to the wrong index. This is the same routing-by-description contract you meet again in agents, and the same reason structured outputs matter — the sub-question plan is a typed object the LLM has to fill in correctly.

from llama_index.core.indices.query.query_transform import HyDEQueryTransform
from llama_index.core.query_engine import TransformQueryEngine, SubQuestionQueryEngine
from llama_index.core.tools import QueryEngineTool, ToolMetadata

base = index.as_query_engine()

# 1. Vocabulary mismatch: embed a hypothetical answer, not the question.
hyde = TransformQueryEngine(base, HyDEQueryTransform(include_original=True))

# 2. Compositional: fan out across described sources, then synthesize.
tools = [
    QueryEngineTool(
        query_engine=hyde,
        metadata=ToolMetadata(name="incidents", description="Postmortems, 2023-2025"),
    ),
]
engine = SubQuestionQueryEngine.from_defaults(query_engine_tools=tools, use_async=True)
print(engine.query("Did deploy failures rise after we moved off Kubernetes?"))

The engineering judgment is knowing that all three are latency amplifiers, not accuracy features. HyDE adds one generation to the critical path. Sub-question adds a planning call plus N retrievals plus a synthesis call. Multi-step adds up to num_steps rounds of both — three sequential LLM-plus-retrieval turns before you even synthesize, with the tail entirely at the planner's discretion. On a well-tuned RAG pipeline serving simple lookup questions, every one of them makes p95 worse and answer quality identical. So don't reach for them on a hunch: put the failing queries in an eval set first, and only adopt the transform whose failure mode you actually have. If your bad queries are single-fact lookups phrased oddly, HyDE. If they contain "and", "compare", or "across", sub-question. If they contain "after" or "then", multi-step. If they're just wrong because the chunks are wrong, none of this helps — go fix the index.

Fails when: the query is ambiguous out of context and HyDE hallucinates the wrong domain — ask about "Bel" and the LLM writes a paragraph on the Semitic deity, which then retrieves confidently and completely irrelevant chunks. Rewriting a question you misunderstood just makes retrieval more precise about the wrong thing.

Routing: Picking the Right Index

Top-k vector search is the wrong tool for "summarize this contract." It will hand you five chunks out of two hundred and the model will confidently summarize 2.5% of the document. Meanwhile a summary index — which reads every node and tree-summarizes upward — is a ruinous way to answer "what is the termination notice period." These are not competing implementations of one thing; they are answers to structurally different questions. Routing is how you keep both and pick per query.

A RouterQueryEngine is a BaseSelector plus a list of QueryEngineTools. Each tool is a query engine wrapped in a ToolMetadata (name, description). At query time the selector sees only the descriptions — never the documents, never the index type — and returns a SelectorResult carrying the chosen index and a reason. The router then delegates the original query, unmodified, to that engine. The selector is a text classifier whose entire feature set is the strings you wrote in description — which is the whole pattern in one sentence, and also its whole failure surface.

from llama_index.core import SummaryIndex, VectorStoreIndex
from llama_index.core.query_engine import RouterQueryEngine
from llama_index.core.selectors import PydanticSingleSelector
from llama_index.core.tools import QueryEngineTool

summary_tool = QueryEngineTool.from_defaults(
    query_engine=SummaryIndex(nodes).as_query_engine(response_mode="tree_summarize"),
    name="summary",
    description="Whole-document questions: themes, tone, obligations overall. Reads every node.",
)
vector_tool = QueryEngineTool.from_defaults(
    query_engine=VectorStoreIndex(nodes).as_query_engine(similarity_top_k=5),
    name="facts",
    description="Point lookups: a named clause, a date, a number, a quote from one passage.",
)
router = RouterQueryEngine(
    selector=PydanticSingleSelector.from_defaults(), query_engine_tools=[summary_tool, vector_tool]
)
print(router.query("What is the notice period?").metadata["selector_result"])

Which selector

LLMSingleSelector prompts the model to emit JSON ({"choice": 1, "reason": ...}) and parses it. PydanticSingleSelector gets the same answer through a function-calling program that fills a SingleSelection model, so the schema is enforced by the provider rather than by a parser — the same argument made on structured outputs, applied to a two-token decision. Prefer the Pydantic one when your LLM does tool calls; fall back to the LLM one when it doesn't. (RouterQueryEngine.from_defaults(...) does exactly that fallback for you.) The *MultiSelector variants return several indices and tree-summarize the responses together — right when a question genuinely spans sources ("compare the 2023 and 2024 filings"), wasteful when it doesn't. EmbeddingSingleSelector skips the LLM entirely and cosine-matches the query against the descriptions: fast and free, but it cannot reason about intent, only surface similarity, so it routes "don't summarize this, just find the date" straight into the summary tool.

Price it honestly. A single-selector call is one short LLM round trip over len(tools) descriptions — cheap, but on the critical path of every query and before any retrieval starts. That's a real latency floor, and it's why routing earns its keep at two-to-five coarse, genuinely distinct indices, not at fifty. Past that, you don't want a classifier over descriptions; you want retrieval over tools (ObjectIndex / an agent that searches its own toolset — see llama-agents) or an agentic loop that can try, fail, and re-route.

The description is the contract

Vague descriptions are where this pattern rots, and it rots silently. "Useful for questions about the 10-K" and "Useful for information from the 10-K" are, to a selector, the same string. It will pick one — it always picks one — and half your traffic gets the wrong index while returning fluent, plausible, subtly incomplete answers. Write descriptions as routing rules with negative space: say what the tool is for, what shape of question it answers, and implicitly what it isn't for. Then treat routing accuracy as its own metric. Because the router stamps response.metadata["selector_result"] (with .ind, .reason) onto every response, you can label a hundred real queries with their correct index and score the selector in isolation — a confusion matrix, not a vibes check — well before you start scoring answer quality in your eval harness. The two failures look identical from the outside and have completely different fixes.

Routing composes downward, not just sideways: the "index" behind a tool can be a whole RAG pipeline with its own reranker, a SQL engine, or another router. And routing is the cheap ancestor of agentic tool use — the same QueryEngineTool you register here is the one an agent picks from — so it's worth reaching for first, and only graduating to an agent when one query provably needs more than one hop.

Fails when: two tool descriptions overlap in meaning. The selector still returns exactly one index with a confident reason, the wrong engine answers, and the response reads perfectly — so the bug shows up as a slow quality drift in eval scores rather than as an error.

Agentic RAG: The Query Engine as a Tool

Classic RAG is a straight line: embed the question, retrieve k chunks, stuff them into a prompt, answer. The retrieval step is unconditional and it happens exactly once. That works right up until the question needs two lookups, or a lookup whose phrasing the user never supplied — "did our margin improve faster than Uber's?" is not one search, it's two searches and a subtraction.

Agentic RAG dissolves the pipeline by demoting retrieval to a tool. You build the index and query engine exactly as before, then wrap it with QueryEngineTool.from_defaults(query_engine=..., name=..., description=...) and hand it to a FunctionAgent (or ReActAgent) from llama_index.core.agent.workflow. The retrieval step stops being a stage in your pipeline and becomes one option in the model's action space — the agent decides whether to search at all, what to search for, which corpus to search, and whether the result was good enough to answer with.

from llama_index.core import VectorStoreIndex
from llama_index.core.tools import QueryEngineTool
from llama_index.core.agent.workflow import FunctionAgent, AgentStream, ToolCallResult
from llama_index.core.workflow import Context
from llama_index.llms.openai import OpenAI

tools = [
    QueryEngineTool.from_defaults(
        query_engine=lyft_index.as_query_engine(similarity_top_k=3),
        name="lyft_10k",
        description="Lyft's 2021 10-K. Input: a detailed plain-text question.",
    ),
    QueryEngineTool.from_defaults(
        query_engine=uber_index.as_query_engine(similarity_top_k=3),
        name="uber_10k",
        description="Uber's 2021 10-K. Input: a detailed plain-text question.",
    ),
]

agent = FunctionAgent(tools=tools, llm=OpenAI(model="gpt-4.1"), timeout=120)

ctx = Context(agent)  # hold this; it is the conversation
handler = agent.run(
    user_msg="Compare Lyft and Uber revenue growth in 2021.",
    ctx=ctx,
    max_iterations=10,
)

async for ev in handler.stream_events():
    if isinstance(ev, ToolCallResult):
        print(ev.tool_name, ev.tool_kwargs)
    elif isinstance(ev, AgentStream):
        print(ev.delta, end="", flush=True)

response = await handler

The description is not documentation — it is the routing logic. The LLM picks a tool by reading it, so the description carries the corpus boundary ("Lyft's 2021 10-K"), the input contract ("a detailed plain-text question", not keywords), and implicitly the negative space. Two tools with vague descriptions produce an agent that flips a coin. This is the same discipline as any tool schema; see structured outputs for why the schema is the prompt.

The loop itself is a workflow (the same machinery behind llama-agents): the LLM emits tool calls, QueryEngineTool runs the query engine and returns the synthesized answer as an observation, the observation goes back into the message history, and the LLM decides again. Termination is the model saying nothing further — no k, no fixed depth, just a backstop iteration cap. Everything the agent has seen is in context, which is what lets it re-query with a sharper phrasing after a thin first hit, or compare across two indexes without you ever writing the comparison code. Subscribe to ToolCallResult and AgentStream on the handler returned by agent.run(), as above, if you want to watch the queries it actually issued; you will be surprised, and that surprise is the whole point of evaluating the trajectory and not just the final string.

Three knobs are worth knowing. return_direct=True on a tool short-circuits synthesis — the query engine's answer is returned verbatim instead of being paraphrased by the agent, which saves a generation and stops the LLM from quietly rewriting a number. A Context held across calls and passed to run(ctx=...) carries the message history, so a follow-up ("what about 2022?") reuses the tools without replaying the conversation — construct it once and keep it, don't build a fresh one per turn. And run(max_iterations=...) bounds the loop: it defaults to 20, and overrunning it raises rather than returning a half-answer, unless you set early_stopping_method="generate" to force one last synthesis from whatever the agent has.

Fails when: the cost and latency model inverts and nobody notices. One-shot RAG is one embedding call plus one generation with bounded p99. An agent with two query-engine tools routinely makes four to six LLM calls per question — a routing call, one generation inside each query engine's response synthesizer, and a final synthesis over observations that already contain the retrieved chunks. The live context grows linearly with each observation, but you re-send the whole history every turn, so cumulative tokens grow quadratically in the number of tool calls, and a "simple" question that could have been answered by a single retrieval now costs 10x and takes 20 seconds. The loop is bounded, but the default bound is generous: an agent that keeps getting mediocre retrievals will happily burn all 20 iterations, or trip your timeout first and raise instead of answering. Lower max_iterations to something you can actually pay for, cache aggressively, and route the easy 80% of questions to plain RAG before they ever reach the agent — see LlamaIndex for where the query engine ends and the agent begins.

Structured Extraction Over Retrieved Context

Most RAG pipelines end by handing prose to a human. This pattern ends by handing a typed record to code — a row to upsert, a payload to POST, a branch condition for an agent. The retrieval half is unchanged; what changes is that the synthesis step now has a schema, and a schema is something you can fail. The Pydantic model is not a formatting convenience, it is the contract boundary between the probabilistic half of your system and the deterministic half — everything upstream of it may hallucinate, everything downstream of it may assume.

LlamaIndex gives you the same capability at three altitudes, and the choice is mostly about how much of the call you want to own. llm.as_structured_llm(output_cls=MyModel) returns a StructuredLLM that behaves like any other LLM — you can pass it as llm= to index.as_query_engine(...), so the whole retrieve-and-synthesize path stays intact and only the payload changes: you still get a Response back, with the validated model instance on response.response. llm.structured_predict(MyModel, prompt, **prompt_args) is the one-liner when you want to drive the prompt yourself: it takes the output class, a PromptTemplate or ChatPromptTemplate, and the template variables (plus an optional llm_kwargs dict). Under the hood it resolves a program via get_program_for_llm, which dispatches to a FunctionCallingProgram when llm.metadata.is_function_calling_model is true and falls back to LLMTextCompletionProgram with a PydanticOutputParser when it isn't — which is the whole reason function-calling models are more reliable here: the schema is enforced by the decoder rather than requested in the prompt. Reach for PydanticOutputParser directly only when you need to customise how the raw text is coerced.

from pydantic import BaseModel, Field
from llama_index.core import VectorStoreIndex
from llama_index.core.prompts import PromptTemplate
from llama_index.llms.openai import OpenAI

class Incident(BaseModel):
    service: str
    severity: int = Field(ge=1, le=5)
    root_cause: str | None = None  # default matters: no default = still required

llm = OpenAI(model="gpt-4o")
index = VectorStoreIndex.from_documents(documents)
query = "What broke during the checkout outage?"

nodes = index.as_retriever(similarity_top_k=6).retrieve(query)
context = "\n\n".join(n.get_content() for n in nodes)

incident = llm.structured_predict(
    Incident,
    PromptTemplate("Context:\n{context}\n\nExtract the incident.\n"),
    context=context,
)

Note what the schema is doing there: severity: int = Field(ge=1, le=5) and root_cause: str | None = None are not documentation, they are the retry trigger. A validation error is the cheapest signal you will ever get that retrieval underdelivered, and validate-then-retry is the actual reliability lever in this pattern, not the schema itself. Catch ValidationError, feed the error text back as an additional turn, and retry once or twice with a cap; if it still fails, widen similarity_top_k or fail loudly rather than persisting a half-filled record. And make optional fields genuinely optional — in Pydantic v2, str | None without a default is still a required field that merely tolerates null, so give it = None if you want the model to be able to say "not in the context" instead of inventing a plausible value. An over-strict schema converts an honest abstention into a hallucination.

The other half of the pattern runs at ingestion, not at query time. The extractors in llama_index.core.extractors — TitleExtractor, QuestionsAnsweredExtractor, SummaryExtractor, KeywordExtractor, and the schema-driven PydanticProgramExtractor (which wraps a program such as FunctionCallingProgram) — are transformations you chain in an IngestionPipeline after a SentenceSplitter, and they write into each node's metadata. That metadata is then available for pre-retrieval filtering and, unless you exclude it via excluded_embed_metadata_keys, gets embedded alongside the chunk text — so it improves recall and gives the extraction step cleaner material to work from. The cost is real: one LLM call per node per extractor, paid on every re-ingest. Run them over the corpus you actually query, not the one you hope to have.

For the wider taxonomy of constrained decoding, JSON mode, and tool-calling as an extraction substrate, see structured outputs; for the LlamaIndex surface these methods hang off, the same structured_predict exists on every LLM subclass, with astructured_predict for the async path and stream_structured_predict if you want partial objects. And because the output is typed, this is one of the few RAG stages you can evaluate without a judge model: field-level exact match against a labelled set, plus a validation-failure rate you can alert on.

Fails when: the schema is stricter than the retrieved context supports — required fields with no grounding force the model to fabricate, and because the fabrication is well-typed, it passes validation and lands in your database looking exactly like a fact.

Evaluation & Observability: The Gate

The other nine patterns are all changes: swap the embedding model, add a reranker, restructure the prompt, hand a step to an agent. Without a gate, every one of those changes is a coin flip that you resolve in production. An eval suite is not a quality initiative — it is the mechanism that makes the rest of the system safe to touch. It converts "the answers feel worse since Tuesday" into a number that moves, and it is the only reason you can refactor a RAG pipeline or a workflow without holding your breath.

LlamaIndex splits evaluation along the same seam as the pipeline itself: retrieval and generation. Confusing the two is the most common mistake, and it wastes weeks — you tune the prompt for a month when the right chunk was never in the context window.

Retrieval: did the right chunk make it into the window?

RetrieverEvaluator scores the retriever in isolation, with no LLM in the loop. It compares the node IDs your retriever returns against the IDs you expected, and reports hit rate (was any expected node retrieved at all?) and MRR (how high up was it?). Hit rate is the ceiling on everything downstream: if it's 0.6, no amount of prompt engineering gets you past 60% correct answers. MRR is what a reranker actually improves.

You need labelled query→node pairs. generate_qa_embedding_pairs bootstraps them synthetically from your own corpus — an LLM reads each node and writes questions it can answer — producing an EmbeddingQAFinetuneDataset (queries, corpus, relevant_docs) that you check in and treat as a fixture. Note that it lives in the llama-index-finetuning package, not in core; the older generate_question_context_pairs / RetrieverEvaluator.aevaluate_dataset pairing that still shows up in blog posts and stale example notebooks was removed from llama_index.core along with the llama_dataset module. Today you drive the evaluator over the dataset yourself, which is three lines and a good deal clearer about what is being measured.

# pip install llama-index-finetuning
from llama_index.core.evaluation import RetrieverEvaluator
from llama_index.finetuning import (
    EmbeddingQAFinetuneDataset, generate_qa_embedding_pairs,
)

# generate once, then check the JSON in and load it in CI
# qa_dataset = generate_qa_embedding_pairs(nodes=nodes, llm=llm, num_questions_per_chunk=2)
# qa_dataset.save_json("eval/qa_dataset.json")
qa_dataset = EmbeddingQAFinetuneDataset.from_json("eval/qa_dataset.json")

evaluator = RetrieverEvaluator.from_metric_names(
    ["hit_rate", "mrr"], retriever=index.as_retriever(similarity_top_k=5)
)

results = [                                        # list[RetrievalEvalResult]
    await evaluator.aevaluate(query=query, expected_ids=expected_ids)
    for query, expected_ids in qa_dataset.query_docid_pairs
]

# each result carries .metric_vals_dict -> {"hit_rate": 1.0, "mrr": 0.5}
hit_rate = sum(r.metric_vals_dict["hit_rate"] for r in results) / len(results)
assert hit_rate >= 0.85, f"retrieval regressed: {hit_rate:.2f}"

That last line is the whole pattern. It is an assertion, in CI, on a checked-in dataset.

Generation: three evaluators, three distinct failure modes

All the LLM-judge evaluators live in llama_index.core.evaluation and share one interface — evaluate(query=..., response=..., contexts=...), or evaluate_response(query=..., response=response) if you'd rather hand them a Response object straight from a query engine. Each returns an EvaluationResult with .passing, .score, and .feedback.

• FaithfulnessEvaluator asks: is the answer supported by the retrieved contexts? It accepts a query and stores it on the result, but never uses it — the judge only sees the response and the contexts. This is your hallucination detector, and it needs no labels at all — which is why it's the one evaluator you can run against live production traffic.
• RelevancyEvaluator asks: do the response and the contexts actually address the query? It sees the query and fails when the retriever pulled plausible-looking but off-topic chunks.
• CorrectnessEvaluator asks: does the answer match a reference answer? It scores 1–5 against score_threshold=4.0 by default, and it's the only one of the three that requires ground truth (evaluate(..., reference=...)). Reserve it for your golden set.

Faithfulness and relevancy are orthogonal, and holding that distinction is the point. A high-faithfulness, low-relevancy answer is a model dutifully summarising the wrong documents — a retrieval bug. A high-relevancy, low-faithfulness answer is a model that ignored the context and answered from its weights — a prompt or model bug. One number can't tell you which knob to turn; two can. BatchEvalRunner({"faithfulness": f, "relevancy": r}, workers=8) runs both across a query set concurrently via aevaluate_queries(query_engine, queries=[...]), because sequential LLM-judge calls over a few hundred questions is a coffee break you don't need.

Observability: the instrumentation layer

Scores tell you that a run failed; spans tell you why. Modern LlamaIndex exposes llama_index.core.instrumentation — a dispatcher hierarchy modelled on Python's logging, and the layer that is superseding the older CallbackManager (which still exists and still works, but is where new integrations no longer go). dispatcher = instrument.get_dispatcher(__name__) gets you a dispatcher; subclass BaseEventHandler from llama_index.core.instrumentation.event_handlers (implement handle()) to observe discrete events like LLM calls and retrievals, or BaseSpanHandler from llama_index.core.instrumentation.span_handlers to observe the enclosing durations. Because dispatchers propagate to parents, one root handler captures the whole tree, and that's exactly where an OpenTelemetry exporter or an LLM-judge panel belongs.

The payoff is that the same failing eval case, the trace that produced it, and the retrieved node IDs are one object. Instrument first, then evaluate — otherwise a failing score is just a mystery with a number attached.

Wiring it into the loop

Treat the eval set the way you treat a test fixture: version it, grow it from real failures, and make the CI job assert on aggregate thresholds rather than individual runs (judges are stochastic; a single flipped verdict is noise, a five-point drop in mean faithfulness is not). Pair this with structured outputs wherever you can — a Pydantic-validated field is a free, deterministic, zero-token eval, and every assertion you can move from an LLM judge to a schema check is one you no longer have to pay for or debug. The judge is for what the schema cannot express.

Once the gate exists, the other patterns in this LlamaIndex guide stop being opinions and start being experiments you can settle.

Fails when: your golden set was written by the same model you're grading — which is precisely what generate_qa_embedding_pairs does if you never edit its output — or drawn from queries that already worked, so it encodes the system's existing blind spots as ground truth, every threshold passes, and the eval gate certifies the exact failure your users are hitting.