LlamaIndex Patterns — Audio Guide

The simplest search setup uses a vector store index and a query engine. Under the hood, it converts your query into an embedding. That embedding is a list of numbers that captures the meaning of your words. The engine then finds the closest matching chunks in the stored index by measuring cosine similarity. This is fast and requires no extra steps. For many collections of text, it is genuinely enough.

But this simplicity hides a trade-off. Embeddings capture topic, not exact tokens. If you search for an error code like err underscore conn underscore reset, the embedding treats it as some error-ish thing. It finds chunks about errors, but not the specific string you need. Cosine similarity happily returns twenty neighbors that are about the right subject and none of them about the right string. So dense retrieval fails on rare, exact identifiers.

That is why this setup matters as a baseline. Every later improvement, like hybrid search or reranking, must beat the plain vector score. Measurements like hit rate and mean reciprocal rank are compared against this starting point. The entire justification for adding more steps is a measurable lift over this simple method. If the baseline already works well, you may not need anything else.

The vector store plus query engine is the bottom rung of the ladder. It is what you build on top of. Everything downstream inherits whatever this stage hands it. So getting this first step right is crucial. You can later add hybrid search, which combines dense and lexical retrieval. You can add reranking to reorder the results. But you always come back to this baseline.

A common mistake is to blame the index when retrieval fails. Often the index is fine, and the question itself is broken. Simple vector search can still fail if the question uses different vocabulary than the corpus. But for many corpora, especially those with clear topic structure, the simple setup works well. It is fast, cheap, and easy to set up.

When you evaluate your retrieval, you compare against this baseline. Build your evaluation set first. Measure retrieval in isolation. Then you can see if an extra hop actually improves hit rate and mean reciprocal rank. If it does not, you saved yourself the complexity. If it does, you know the upgrade is worth it.

So remember: the vector store index and query engine are your starting point. They give you a clear number to beat. They handle topic matching well. They struggle with exact tokens. That is their strength and their weakness. And that is why you need to know them before you reach for anything fancier.

Overlap in chunking costs you duplicate embeddings. This overlap is a hedge, not a feature. It exists so a fact that straddles a boundary appears intact in at least one chunk. But you pay for duplicate embeddings and duplicate hits. When you re-ingest the same corpus, you want to avoid that cost if nothing has changed. One solution is the hierarchical node parser. It produces every chunk size at once: two thousand forty eight, five hundred twelve, and one hundred twenty eight. That saves you from re-chunking and re-embedding the same text multiple times. The trade-off is the upfront decision on chunk sizes. But that decision is a one-time cost. The real answer to the small versus large tension is to stop choosing. The hierarchical node parser gives you both at once. So you do not need to re-ingest for different chunk sizes. You pay the planning upfront, but you never again pay to embed text you already have when it has not changed. The semantic splitter is another tool. It embeds every sentence at ingest time. That cost is real. If the text is unchanged, you would pay that cost again. So the bookkeeping is the price you pay upfront. You decide on chunk sizes and overlap once. Then you can skip re-embedding the unchanged text. That is the trade-off: a planning investment for a permanent gain in ingest cost.

The way you split your text into pieces is the most common reason retrieval fails. To handle a fact that falls across a break, the system uses overlapping chunks. The overlap is a hedge, not a feature. The shipped defaults are one thousand twenty four characters per chunk with a two hundred character overlap. That works fine for normal prose. But for short FAQ entries you should use chunks of two hundred fifty six to five hundred twelve characters instead.

For structured text, a parser that reads headings is much better. The Markdown Node Parser splits on headers and stamps the header path into metadata. The Code Splitter uses a grammar tool called tree sitter to break at function boundaries. That way a function is never cut in half. Free structure beats inferred structure every time.

There is also a semantic splitter that people often reach for. It groups adjacent sentences into windows. Then it embeds every window and cuts wherever the topic shift is large. But it costs a lot. At ingest you embed the entire corpus once at sentence granularity. That is on top of the embedding for each resulting chunk. The chunk boundaries are also non deterministic across embedding model versions. This tool earns its keep on long, unsegmented text like transcripts or legal prose with no headings. On a well headered corpus it buys you nothing that the Markdown Node Parser already gave you for free. Decide whether to use it with an eval on retrieval hit rate, not by vibes.

Now consider what a chunk boundary through the middle of an answer costs you. The fact that straddles the boundary appears intact in at least one chunk because of the overlap. But you pay for it with duplicate embeddings and duplicate hits in your top k search. The real answer to the small versus large chunk tension is to stop choosing. The hierarchical node parser produces every level at once. It wires each node to its parent. You get small, medium, and large chunks all together. That way you never lose the complete answer.

Everything downstream inherits whatever this chunking stage hands it. Grounded citations and structured extraction all depend on it. So fix your chunking first. It is usually the real culprit when retrieval fails, not the embedding model.

Filtering by metadata before similarity search is the only correct way to scope a search to a specific tenant or time range. You place a condition on the metadata first, then run the vector search only on the matching documents. That guarantees you stay inside the right partition. But the filter itself can fail silently. If you make it too narrow, you get zero hits. The system does not raise an error. It simply returns a placeholder that says "Empty Response." No exception, no warning. Callers and agents will treat that as content and move on. That is a quiet breakdown.

A more dangerous failure is a near miss. The filter is correct on the surface, say the right service field but the wrong year. Or the right filter but the wrong tenant partition. The retriever returns a full, plausible list of nodes from the wrong partition. The answer is confidently wrong and includes citations. So you must check the number of nodes you get back. If you get zero, fall back to a more relaxed filter set. Do not trust an empty result. And when you do get nodes, check that their metadata actually match the filter you meant to apply. Do not trust a non-empty list either.

Now, how does the model know which filter to use in the first place? That is where auto-retrieval comes in. The model infers the filter from the question itself. It does not guess. It uses query transformations to rewrite the question before it ever touches the retriever. One method is called Hy D E. It asks the large language model to hallucinate a fake answer first. That fake answer is stylistically similar to real documents. It is embedded, and that embedding lands much closer to the correct chunks. The model never sees the fake answer as truth. It only uses it to bridge the gap between a short question and a long document.

Another method is multi-step querying. Some questions have a hidden chain of dependencies. For example, "What did the author do after the company he founded was acquired?" That question needs a first step: find the company. Then a second step: find what happened after. The multi-step query engine decomposes the question into sub-questions. It looks at the query, the index summary, and the reasoning so far. Then it emits the next sub-query. Step by step, it infers the right scope and filter for each sub-question.

These query transformations handle vocabulary mismatch. The corpus may use different terms than the question. For instance, a user asks "Why did the deploy blow up last night?" The runbooks say "Rollout aborted: readiness probe exceeded initialDelaySeconds." Those two strings are far apart in embedding space. Hy D E closes that gap. The technique costs an extra large language model call during ingestion and retrieval. But it is worth it when the question and answer use different styles.

In short, filtering by metadata before similarity is the only correct way to scope a search. But you need to handle silent failures. And the model can infer the right filter from the question using query transformations. Together, they give you precision and recall without guessing.

Dense retrieval captures topic but not exact tokens. An embedding turns a specific error code like ERR underscore CONN underscore RESET into just some error-ish idea. Cosine similarity happily returns twenty nearby chunks that are all about the wrong subject. None of them contain the right string.

BM twenty-five has the opposite flaw. It nails rare tokens and exact characters. But it misses paraphrases every time. Hybrid search combines both. It is the recognition that lexical and semantic retrieval fail on different queries. The union of their candidates gives far better recall than either method alone.

Now you have two ranked lists with scores that mean nothing to each other. A cosine of point eight three and a BM twenty-five score of fourteen point two are incomparable. Reciprocal rank fusion throws the scores away and keeps only the ranks. Each document scores the sum of one over K plus its rank from each list. A document ranked second by both retrievers beats one ranked first by only one of them.

The fusion retriever’s similarity top K is the recall budget. A document the fusion stage drops can never be recovered. That sets a ceiling on the whole pipeline’s quality. The reranker’s top N is the precision budget. Every node past it is context you pay for and attention you dilute.

Widening the candidate pool is nearly free. An approximate nearest neighbor scan and an inverted index lookup are cheap. But the rerank step is linear in the number of candidates. That is where your latency lives. A local Mini LM cross encoder on forty candidates takes fifty to a hundred fifty milliseconds on a GPU. On a CPU it can hit half a second. A hosted reranker like Cohere adds a network round trip but no local compute.

A cross encoder reorders candidates the bi encoder only roughly ranked. The bi encoder is fast but coarse. The cross encoder is slower but far more precise. You only want to rerank a small set. So retrieve wide to guarantee the answer is in the pool. Then rerank narrow to keep only the most relevant documents. That is the discipline.

Sometimes the problem is not the index but the question itself. The question might use different words than the documents. Or it might bundle several questions into one. Query transformations fix this by rewriting the question before it ever touches the retriever. They cost extra model calls. But they earn that cost when the question is broken.

One technique handles vocabulary mismatch. Instead of embedding the user's question, you ask the language model to imagine a fake answer. That fake answer is then embedded and used to find real documents. The fake answer is factually useless. But it matches the style of real documents, so it lands closer to the correct chunks. This adds one extra language model call. It earns its cost when the question uses different words than the corpus.

Another technique handles sequential dependency. Some questions depend on previous answers. For example, you asked what the author did after the company he founded was acquired. You first need to know which company. A multi-step engine breaks this into sub-questions. It looks at the query, the index summary, and previous reasoning to ask the next sub-question. This adds multiple calls, one per step. It earns its cost when the question has a built-in dependency.

A third technique is sub-question decomposition. It also breaks the question into smaller pieces. Each sub-question gets answered and then combined. Again, extra calls.

When are these transformations just latency? If your question is already well formed, with vocabulary that matches the corpus and no hidden steps, they add delay without benefit. You waste time and tokens.

Even with good question rewriting, a common failure sneaks in. If your filters are too narrow, you get zero results. The system returns a literal string that says empty response. It never calls the language model. So you get a placeholder that looks like content but is not. The real danger is a near miss. You get a full, plausible node list from the wrong partition. That produces a confidently wrong answer with citations. So you must check that your filters actually match, and fall back to a relaxed set when nothing comes back.

Know the trade off. Extra model calls are worth it when the question is the broken part. But they are just added cost when the question is already clear. And no matter how you rewrite, brittle filters can undo everything.

The first genuinely agentic step is a router. A router decides which index or search engine should answer a given question. It works by having a large language model read the description of each available tool. That description must be precise. It must state what data the tool covers. It must also imply what it does not cover. Two tools with vague descriptions make the model flip a coin. And a wrong route is expensive. Consider two types of questions. A summarise the corpus question wants broad topic matches. A find one fact question wants an exact token match. Dense retrieval captures topic but misses rare terms. Lexical retrieval captures rare terms but ignores meaning. So they need different indexes. Hybrid search combines both for better recall. The router can choose to use that hybrid approach. But it must also decide how to merge scores from two incompatible lists. The easiest way is to keep only the ranks. This is called reciprocal rank fusion. A document ranked second by both retrievers beats one ranked first by only one. That gives balanced results. The cost of a wrong route is not just a wrong answer. It is a confidently wrong answer with citations. The model will fabricate a fluent response over irrelevant chunks. That is hard to detect. The router also adds latency. One shot retrieval is one embedding call and one generation. The router loop may call multiple tools before returning. Each call adds cost. So the tool descriptions are the key to reliability. They must clearly mark the boundary of each corpus. They must state the input contract. A question about error codes should not go to a financial report index. A question about 2021 revenue should not go to a technical documentation index. If the filter misses, nothing raises an exception. The system returns an empty response or a plausible but wrong answer. So you must assert that the returned nodes actually match the filter. The router is the first agentic rung because it chooses the path. It does not just answer. It decides which engine can answer best. That decision is only as good as the tool descriptions it reads. That is the trade off: more capability, but more fragility. With clear descriptions and a careful loop, the router can handle diverse questions without you writing specific code for each one. You trust the model to route correctly. But you must evaluate the trajectory, not just the final string. That is the only way to catch a wrong route before it reaches the user.

Instead of a single retrieve and answer, you can hand the query engine to an agent as a tool. The agent itself decides whether to retrieve, how often, and when to stop. This is a different approach from the one shot retrieve then answer.

The LLM picks a tool by reading its description. The tool is a QueryEngineTool that runs the query engine and returns the synthesized answer as an observation. That observation goes back into the message history. Then the LLM decides again. Termination is the model saying nothing further. There is no fixed depth, just a backstop iteration cap.

What does this unlock? The agent can re query with sharper phrasing after a thin first hit. It can compare across two indexes without you writing any comparison code. You can subscribe to ToolCallResult and AgentStream to watch the queries it actually issued. You will be surprised, and that surprise is the whole point of evaluating the trajectory and not just the final string.

But the cost and latency model inverts. One shot RAG is one embedding call plus one generation with a bounded p99. Here, each tool call costs similar. The loop may iterate many times. You pay for multiple retrievals and multiple generations. The production failure is that this baseline has no way to fail. It cannot say I do not know. It returns chunks and confabulates an answer. With no eval harness attached, you find out from a user weeks later.

There are three knobs worth knowing. Return direct on a tool short circuits synthesis. The query engine answer is returned verbatim instead of being paraphrased by the agent. That saves a generation and stops the LLM from quietly rewriting a number. A Context held across calls carries the message history. So a follow up like what about 2022 reuses the tools without replaying the conversation. And run with max iterations bounds the loop. It defaults to twenty. Overrunning it raises an error unless you set early stopping method to generate. That forces one last synthesis from whatever the agent has gathered.

So the trade off is real. You gain flexibility and the ability to refine queries. The agent can adapt and retrieve exactly what it needs. But you also take on a loop and higher cost. Each step costs time and money. The single shot approach is cheap and fast. The agent approach is powerful but expensive. Choose based on the task.

Retrieval gives you chunks of prose, not fields. That is a problem. Downstream systems need typed data, not paragraphs. They need facts with a known shape. So you have to pull that shape out of the chunks. A schema does that. It is like a blueprint. It tells the model exactly what fields to fill. And it constrains what the model can say. The schema is the prompt. It is the instruction. Without it, the model can wander. With it, the output is predictable. That is the trade-off. You trade flexibility for reliability. The trade-off is worth it. Because a system that always returns the same structure can be trusted. A system that returns paragraphs is hard to use.

Now the concrete detail. How do you get that shape? The retrieval stage hands off to structured extraction. Everything downstream inherits what that stage produces. So if the chunks have good metadata, extraction is easier. For example, hierarchical nodes give you parent-child links. Those links help you assemble a full record. But the real work is in the schema. The schema defines what is allowed. It says the output must have a certain type. That type could be a number or a category. The model cannot ignore it. The schema enforces the shape. It is not a suggestion. It is the rule.

You can test whether the extracted data is faithful to the chunks. Faithfulness and relevancy are separate. A high-faithfulness extraction might be wrong if the chunks are off topic. So you need both. The schema also helps with evaluation. You can check if the output matches the schema. That is a simple test. It catches errors early.

But there is a fail case. If the retrieval misses the right chunks, the extraction will be wrong. The schema cannot fix missing data. So you need good retrieval first. That is why chunking matters. The splitter shapes what the retriever returns. If you split badly, you lose context. Then the schema cannot help.

The schema also works like a tool description for an agent. The agent picks a tool by reading its description. That description carries the corpus boundary and the input contract. It tells the model what to expect and what to output. So the schema is not just a format. It is the instruction that shapes the answer.

In summary, pulling typed data from chunks is the goal. The schema is the tool. It constrains the model. It gives you fields instead of paragraphs. That is what downstream systems need. They need structure. They need predictable output. And the source material must support it. Get the retrieval right. Then let the schema do its job. That way you turn messy prose into clean, usable data.

The top rung is evaluation. It makes every other step safe to change. Without it, you can only tell an answer changed. You cannot tell if it improved.

Three judges sit on that rung. First is the faithfulness evaluator. This one checks if the response sticks to the retrieved contexts. It needs no labels. You can run it against live traffic. It is your hallucination detector.

Second is the relevancy evaluator. It asks if the response and the contexts actually address the query. This catches a retrieval bug. The retriever may pull plausible but off topic chunks. Relevancy sees that.

Third is the correctness evaluator. It compares the answer to a reference. It scores from one to five. The default threshold is four point zero. This one requires ground truth. Save it for your golden set.

Faithfulness and relevancy are separate. One number cannot tell you which knob to turn. Two can. Run them both with the batch evaluator. It processes queries concurrently.

And then there is tracing. Scores tell you that a run failed. Spans tell you why. The instrumentation layer captures durations. It shows you the LLM calls and retrievals. Without it, you only see the output.

Now, the gate. A plain baseline RAG has no way to fail. It always returns the top two chunks. It cannot say it does not know. The synthesizer then makes up a fluent answer over those chunks. You only find out something is wrong from a user screenshot. That might be three weeks after launch.

Without an eval harness, you cannot tell if a change is an improvement. You just see a different answer. The three evaluators become the gate. They say whether the answer is faithful, relevant, and correct. They force the system to say when it does not know. That is the top rung. It makes every rung below it safe to adjust.