LlamaIndex Anti-Patterns — Audio Guide

Shipping the five-line quickstart query path to production is tempting.
It looks like a one-line change.
It requires no understanding of the system.
No evaluation set, no chunking decision, no reranker latency budget.
But this path can never say it does not know.

The silent cost is huge.
Almost every quality problem in the first year of a retrieval augmented generation system, called RAG, is actually a retrieval problem.
It only looks like a generation problem.
A bigger model cannot read a chunk you never retrieved.
If the answer was not in the context window, generation quality is irrelevant.

The smallest responsible path starts with two numbers.
Measure retrieval in isolation with hit rate.
If hit rate is zero point six, then forty percent of your queries are unanswerable.
No prompt or model can fix that.

Then split the failures with faithfulness and relevancy evaluation.
High faithfulness and low relevancy means the retriever pulled the wrong documents.
A bigger model makes that worse because it summarizes wrong evidence more convincingly.
Low faithfulness with high relevancy is the only case where the model is the problem.

You also must tune chunk size on hit rate, not on how fluent the answers read.
As you make chunks bigger, retrieval gets worse.
You cannot see the collapse because you are grading only the queries that still worked.

The defaults are a fine starting point for dense prose.
But the point is you answer with a retrieval metric, not by reading answers.
That metric is the ceiling on everything downstream.
At zero point six hit rate, no amount of prompt engineering gets you past sixty percent correct.

So resist the quickstart path.
Add a cutoff, add a reranker, and always keep an abstention set.
That set is the only thing that tests whether your pipeline can honestly say it does not know.

Every night, your ingestion script reads every single document in your corpus, splits them all into chunks, and embeds every chunk. It does this even if nothing changed. That is the largest avoidable expense in most production retrieval systems. And nobody notices because the job succeeds. The index looks correct. The only clue is a line on an invoice that keeps going up. The root cause is simple. The from-documents call that tutorial taught you has no memory. It cannot remember what it saw last run because you never gave it a place to remember. That place is called the docstore. It maps a document’s unique identifier to its hash. An ingestion pipeline with upserts uses that. It processes a document only when its identifier is new or its hash has changed. It also deletes old nodes before inserting new ones. And an ingestion cache sits behind that. It skips expensive steps like embedding for content you have already processed. Here is the trap that catches almost everyone who knows about the docstore. You attach a docstore to a storage context. Then you call from-documents. You think the docstore is being updated. It is not. The reason is that real vector databases like Qdrant or Pinecone store the text themselves. So the code that would write to the docstore is skipped. Your docstore stays empty. Your dedup is a no-op. It does nothing. It raises no error because the index queries perfectly. The vector store has the text, so everything seems fine. But the cost goes on forever. The fix is to stop trying to make from-documents remember things. Instead, use an ingestion pipeline from the start. It handles caching and upserts properly. The content hash caching means that if a document has not changed, your system skips re-embedding entirely. That is how you stop paying twice for the same corpus. Think of it this way: every night you are throwing away all the work from the previous night and starting from scratch. That is like paying for a full meal every day even when you only eat leftovers. The ingestion pipeline is the refrigerator. It keeps the work you already did, so you only cook what is new. The cost survives because it is invisible. The only symptom is a number on an invoice that someone else reads. Now you know where to look. And the fix is just a few lines of code.

You tune the chunk size until the answers read better. But that optimizes the wrong objective. It measures fluency, a noisy proxy for quality. As you increase the chunk size, the retrieved text becomes more fluent. But the recall underneath collapses. You cannot see the collapse because you only grade the queries that still worked. The ones that fail are invisible.

The problem is a fundamental trade-off. Generation wants big chunks. More surrounding prose means fewer severed sentences. The answer is less likely to be cut in half at a boundary. Retrieval wants small chunks. A large chunk that covers several topics embeds as the average of those topics. That average vector is not close to any specific question. So as you turn the dial up, the answers you read get smoother. Meanwhile the retrieval hit rate drops silently.

The default settings are a chunk size of one thousand twenty four and an overlap of two hundred. Those are a fine starting point for dense prose. But the real question is not which number is right. It is how you find the right number. And you cannot answer that by reading answers. You answer it with a retrieval metric, measured in isolation, with no large language model in the loop.

That metric is hit rate. Hit rate is the ceiling on everything downstream. If your hit rate is sixty percent, then forty percent of your queries are unanswerable. No prompt and no model can get you past sixty percent correct. Every dollar you spend on a bigger model is spent on the wrong half of the pipeline. A larger model cannot read a chunk you never retrieved. If the answer was not in the context window, generation quality is irrelevant. You have paid a premium for a more articulate way of not knowing.

So stop tuning chunk size by how the answers read. Measure retrieval in isolation. Use a retriever evaluator and look at hit rate. That number tells you the true ceiling. Only after you have optimized retrieval should you worry about generation. Otherwise you are optimizing the wrong variable through a noisy proxy. The answers get more fluent while the recall underneath collapses. And you will never see the collapse because you are only looking at the queries that still work. Measure what matters first.

You think that adding a reranker improves your search, so applying a score cutoff on top must make it even better. But the reranker does not give you the kind of score you expect.

Here is the concrete detail. A regular vector store returns a cosine similarity score. That score is bounded, usually between zero and one. A reranker like a cross-encoder returns a completely different number. It produces an unbounded logit. That logit can be negative. A score of negative one point two is perfectly possible for the most relevant document.

Now imagine you set a cutoff at zero point five. Every node that falls below that value gets thrown away. But the reranker’s negative logit does not mean the node is bad. It means the reranker’s scoring scale is different. So your cutoff rejects the very documents the reranker ranked most confidently. You end up with an empty context. Or the model answers from its own training data because nothing was retrieved. That is the silent recall you lose.

The trade off is painful. You implemented a reranker to improve quality. Then you added a threshold that silently removes the best results. You never see the error because the system just returns less. The only visible symptom might be an empty response or a model that sounds confident but wrong.

The correct approach is not to filter by score value at all. Score values are not comparable across different producers. A cosine similarity, a BM25 relevance score, and a cross-encoder logit all land in the same field but with wildly different scales. If you ever write code that checks if the score is greater than some number, you have hard coded a threshold to one model’s distribution. That threshold will not survive the next model upgrade.

Instead, filter by rank. After the reranker, simply take the top N results. That respects the reranker’s ordering without assuming anything about the scores themselves. Rank is stable. Score is not.

There is another subtle failure. A postprocessor that expects a similarity score will drop nodes that have no score at all. Some retrievers do not populate scores. If you put a similarity postprocessor after them with a cutoff of zero point zero, thinking it is harmless, every node gets silently dropped. That turns a working pipeline into one that always returns nothing.

So remember: a reranker improves the ranking. But its scores are not calibrated the way you assume. Do not apply a value cutoff after it. You will throw away your best retrieval work. Measure retrieval in isolation. Keep the reranker’s output ordered by rank. Let the top N through. That is the only safe filter. Everything else is a trap that wastes your model budget and hides the real problem.

Putting tenant restrictions in the system prompt is a tempting fix. It requires no understanding of the retrieval system. It is a one line change. It needs no evaluation set. No chunking decision. No filter schema. No reranking budget. But it is wrong for a reason that should be on the wall. A bigger model cannot read a chunk you never retrieved. If the answer was not in the context window, generation quality is irrelevant. You have paid a premium for a more articulate way of not knowing.

Now apply that to tenancy. The system prompt reads only use documents belonging to this tenant. Never reveal information from other tenants. It works in testing. It works in the demo. But it is not access control. It is a request. The request is addressed to a stochastic text generator about text you already placed inside its context window. What happened before that prompt was ever read? The retriever ran an approximate nearest neighbor search across the whole collection. It pulled the top chunks. Some of those chunks belong to another tenant. They were serialized into the prompt. The data has already left the store. The only thing standing between another customer's contract and your user is a single sentence. That sentence is competing for attention with everything else in the context. In a forty thousand token context, it can easily be ignored. Any prompt injection in a retrieved document becomes a peer of your instruction, not a subject of it.

So relying on the prompt is a correctness and privacy failure waiting to happen. The prompt cannot remove what the retriever already placed in the context. It is not a style choice. It is a fundamental misunderstanding of how the pipeline works. The cost is not visible until production. But when it fails, it fails silently. The system prompt is not a gate. It is a suggestion. And suggestions are weak when the context is full of contradictory information. A retrieved document from another tenant might contradict the system prompt. The model has to choose. It might follow the strong evidence from the document instead of the abstract instruction. That is the failure mode. It is silent. The user sees the other tenant's data. The system never raises an error. The only way to catch it is to inspect the output. But most teams do not do that. They trust the prompt. But the prompt is not a filter. The retriever already decided what the model will see. The prompt cannot undo what the retrieval already placed in the window. Instead, filter by metadata before retrieval. That way, the wrong data never enters the context. That is the real fix. It requires understanding the retrieval step. But it is the only way to protect tenant data.

Your system has a hidden path. It can fabricate answers. When a retriever finds nothing, the node count is zero. If nothing checks that number, an empty context goes to the generator. The generator then answers from its own training. It sounds confident but the content is made up. This is a hallucination nobody sees. The guard that fixes this is straightforward. You assert on the number of nodes. If it is zero, you fall back. Then you synthesize an honest response. That response says I do not know. The abstention set tests this exact ability. It is the only thing that proves the pipeline can refuse. Without it, the system has no way to say no. All answers look correct but many are built on missing information. The golden set often misses unanswerable questions. So the system passes tests but fails in production. The fix is simple. Measure retrieval alone. Use hit rate to find true accuracy. If retrieval finds nothing, the model cannot answer anything real. The guard saves the day by stopping the empty path. It turns a fabrication into an honest no.

A common mistake is to switch to a larger language model when the answers are wrong. But a bigger model cannot read a chunk you never retrieved. Most quality problems in the first year are actually retrieval problems wearing generation problems clothes. To diagnose, measure retrieval by itself with a hit rate. If the hit rate is zero point six, then four out of ten queries are unanswerable no matter what model you use. Every dollar spent on a larger model then goes to the wrong half of the pipeline. Then split the generation failures using two separate checks. One checks faithfulness to the retrieved context. The other checks relevancy of that context. Faithfulness means the model sticks to the text it was given. Relevancy means that text actually answers the question. These two are independent. High faithfulness with low relevancy shows a retrieval bug. The model summarized the wrong documents. A bigger model makes that worse because it summarizes wrong evidence more convincingly. Low faithfulness with high relevancy is the only case where the model itself is the problem. That indicates the model ignored the context. Chunk size also plays a role. Generation wants big chunks for more surrounding prose. Retrieval wants small chunks so the embedding is not an average of many topics. Tuning chunk size by reading the outputs hides the collapse in retrieval. The answers get more fluent while the recall underneath falls. You cannot see that collapse because you grade the queries that still worked. The right way is to answer that question with a retrieval metric in isolation. No language model in the loop. Measure hit rate, then tune on that number. That is the only way to distinguish a retrieval failure from a generation failure.

Another common trap is using a golden set built from the same generator. Every question was written from the node that answers it. The retriever finds it trivially because it is a vocabulary matched paraphrase. The set contains no unanswerable questions. So the hit rate looks very high, like zero point nine four. But in production the system confabulates over irrelevant chunks. The evaluation certifies the exact failure users report. This is the most expensive mistake because it requires no understanding of the system. It is a one line change. But it is wrong because a bigger model cannot read a chunk never retrieved. If the answer was not in the context window, generation quality is irrelevant. You have paid for a more articulate way of not knowing.

To fix this, you must measure retrieval in isolation using a hit rate metric. Hit rate is the ceiling on everything downstream. If it is low, no prompt and no model can fix it. Then and only then use faithfulness and relevancy checks. Those checks need no labels and can run on live traffic. Faithfulness and relevancy are orthogonal. Holding them apart is the whole diagnostic. High faithfulness plus low relevancy means the retriever pulled on topic but wrong documents. A bigger model makes that problem worse. Low faithfulness with high relevancy is the only genuine model problem. The chunk boundary also matters. A chunk is both the unit of retrieval and the unit of context. These two want opposite things. Generation wants large chunks to avoid severed sentences. Retrieval wants small chunks so the vector is specific. As you increase chunk size, answers get more fluent but recall collapses underneath. You cannot see the collapse because you grade the output of queries that still worked. So tune chunk size on a retrieval metric, not on how the prose reads. That is the core of the misdiagnosis. The most common error is to blame the generator when the real fault is retrieval. A bigger model cannot read a chunk never retrieved. You settle this with two numbers, not with an argument. Hit rate first. Then faithfulness versus relevancy. That tells you where the problem really lies.

One quiet mistake hits many retrieval systems. It is the global settings container. Two processes run with two different sets of settings. One uses OpenAI embeddings. Another uses a model called B G E. The vectors from these models are incompatible. The index was built with one model. When a query comes in using the other model, the retrieval cannot find the right chunks. The system does not crash. No error appears. The retrieval simply returns the wrong documents. The generator then produces an answer from those wrong documents. The answer looks fluent. But it is based on irrelevant information. The user gets a confident wrong answer. The operator sees a quality problem. They might blame the generator. They might upgrade to a bigger model. But that will not help. A bigger model cannot read a chunk that was never retrieved. The real problem is the embedding mismatch. It is a retrieval problem disguised as a generation problem. Almost every quality problem in the first year of a retrieval system is a retrieval problem wearing a generation problem's clothes. This mistake is one of ten that never raise an exception. The cost shows up only in metrics that nobody checks. You cannot argue your way out of this. The numbers tell the truth. Measure the retrieval hit rate in isolation. Without any generator. If the hit rate is zero point six, then forty percent of your queries are unanswerable. No prompt and no model can get you past sixty percent correct. The embedding swap makes the hit rate drop. The retrieval evaluator would reveal it. But without that measurement, the problem hides. The evaluation would show that the generator is faithful to the retrieved documents. But those documents are irrelevant. That is a clear sign of a retrieval problem. The solution is simple. You must pin the embedding model explicitly. In your code, set the embed model directly. Do not rely on a global container. At boot time, check that the vector dimension matches what you expect. This adds a few lines. Those lines are worth it. They prevent a whole class of silent failures. They make the system predictable. They ensure that every call uses the same model. If you ever need to change the embedding model, you must rebuild the index. That is a deliberate act. It is not a hidden side effect. The global settings container makes changes invisible. Explicit wiring makes them visible. That visibility is the whole point. You need to know what your system is actually doing. The extra lines are not overhead. They are insurance. They save you from chasing generation problems that are really retrieval problems. They save you from paying for a bigger model that cannot fix a missing chunk. They save you from the cost that nobody is looking at.

Reaching for an agent loop on every question is a common mistake. It turns retrieval into something unpredictable. A query engine is simpler and much faster. For most questions, a single call to a query engine works. That is roughly eighty percent of cases. So the honest test is straightforward. Ask yourself if the question really needs a decision at all. If not, skip the agent loop entirely. This saves latency and cost. It also keeps your system deterministic.

An agent adds complexity and randomness. Retrieval stops being a pure function. You introduce extra steps and more variables. The result is harder to debug and slower to run. A bigger model does not fix this. If the answer was never retrieved, no model can save it. You pay a premium for a more articulate way of not knowing. That is why measuring retrieval in isolation matters. Use a retriever evaluator without any large language model. Check the hit rate. A hit rate of zero point six means forty percent of queries are unanswerable. No agent or model can fix that. The ceiling on your system is that number.

The honest test is simple. Before you build an agent, test whether a query engine can handle the question. Try a direct retrieval and synthesis. If it works, you are done. If it fails, then consider an agent. But do not default to an agent for every question. That route adds latency and cost for no gain. It also makes the system nondeterministic. The same question can get different answers. That is a problem for reliability.

Look at the two numbers that matter. Faithfulness and relevancy. High faithfulness with low relevancy means the retriever pulled the wrong documents. A bigger model makes that worse. It summarizes wrong evidence more convincingly. Low faithfulness with high relevancy means the model ignored the context. That is the rare case where the model is the problem. For most issues, the retrieval is the culprit. So fix retrieval first.

The agent loop is a heavy tool. Use it only when the question truly needs a decision. The easy eighty percent of questions do not. They just need a fast, accurate answer from a query engine. That engine is simple. It retrieves and synthesizes. No extra steps, no branching, no tool calls. It is deterministic and cheap.

When you do use an agent, be careful. Make sure your retrieval is solid. Measure it in isolation. Do not rely on the agent to fix bad retrieval. It cannot. The agent will only amplify the problem. It will make the wrong answer sound more fluent. That is dangerous.

So the rule is: start with a query engine. Test it. If it works, stop. Only if it clearly fails should you reach for an agent. This saves time and money. It also avoids the hidden cost of nondeterminism. Your users get consistent answers. Your system stays simple and easy to debug.

The honest test is not about how smart the agent is. It is about whether the question needs a decision at all. Most questions are straightforward. They have a single correct answer. A query engine finds that answer fast. An agent only adds unnecessary steps. That is the core insight. Route the easy questions to a query engine. Keep agents for the hard ones. Your latency will drop. Your cost will shrink. And your answers will be more reliable.

When you build an evaluation set using the same model you are testing, the result looks good but is actually worthless. The questions come directly from the chunks that answer them. They are paraphrases that match the vocabulary of those chunks. So the retriever finds them easily. You see a hit rate of zero point nine four. The test passes with a green light. But in production, your system confabulates over irrelevant chunks. Users report exactly that failure. The evaluation set has zero unanswerable questions. The generator cannot produce one. So it never tests whether your pipeline can say I do not know. This mistake is very common because it requires no understanding of the system. It is a one line change. No chunking decision. No rerank budget. It is wrong for a simple reason. A bigger model cannot read a chunk you never retrieved. If the answer was not in the context window, generation quality does not matter. You have paid for a more articulate way of not knowing.

Almost every quality problem in the first year of a rag system is actually a retrieval problem. It wears a generation problems clothes. You settle this with two numbers. Measure retrieval in isolation. No large language model in the loop. Use retriever evaluator and hit rate. If hit rate is zero point six, then forty percent of your queries are unanswerable no matter what generates over them. Every dollar on a big model is spent on the wrong half. Then split generation failures with faithfulness and relevancy evaluators. Faithful but irrelevant means the model summarized the wrong documents. That is a retrieval bug. Relevant but unfaithful means the model ignored the context. That is a prompt or model bug. These two are orthogonal. Holding them apart is the whole diagnostic.

High faithfulness with low relevancy is a retriever that pulled on topic but wrong documents. A bigger model makes that worse. It summarizes the wrong evidence more convincingly. Low faithfulness with high relevancy is the only case where the model is genuinely the problem. A defensible golden set requires an abstention set. That is the only thing that ever tests whether your pipeline can say I do not know. You must hand edit the golden set. Add unanswerable questions. Do not use the unedited output of the generator. That output contains no unanswerable questions and guarantees a false green pass. Hand edit it. Include questions that cannot be answered from the retrieved chunks. Then your evaluation will tell you the truth.