LlamaIndex Anti-Patterns

The patterns page is a ladder and the architecture page is the object graph underneath it. This page is the other thing entirely: the ten ways teams actually get LlamaIndex wrong. Not gaps in knowledge — gaps in judgment. Every one of these is a decision that looked correct at the time, shipped without complaint, and then charged you later.

That is what separates an anti-pattern from mere ignorance. Nobody wires a bad pipeline on purpose. They wire a seductive one: the quickstart works, so it ships. The reranker improves scores, so a threshold on top must improve them more. The answers are wrong, so a better model must make them right. Every mistake below is locally reasonable and globally wrong, and every one of them fails silently — LlamaIndex's defaults are tuned for a notebook that runs, not a service that is correct, so the framework will let you do all ten and raise nothing.

Read this as the debugging index for a RAG system that is already in production and already disappointing you. Each section names the mistake, says why you made it, shows the mechanism of the damage, and gives you the fix.

The inverse map — down the left, what you reached for, in the order the pipeline runs; on the arrow, what it silently charges you; on the right, what you should have reached for instead:

The dotted spine is the order to read them in, not a chain of cause and effect — each of the ten is independently shippable and independently silent. What they share is the arrow: nothing on the left ever raises, so the cost is only ever visible as a number nobody is looking at.

The Quickstart Is Not a Product

Five lines. A directory, an index, a query engine, an answer. It is the best onboarding experience in the ecosystem and that is precisely the problem — it works, so it ships, and it ships carrying every default the tutorial author chose to keep the snippet short.

Unpack what you actually deployed. as_query_engine() builds a VectorIndexRetriever with similarity_top_k=DEFAULT_SIMILARITY_TOP_K, and that constant is literally 2 in llama_index/core/constants.py. There is no SimilarityPostprocessor — score cutoffs are opt-in, passed by hand through node_postprocessors. There is no reranker. There is no metadata filter. So the whole of your retrieval layer is: embed the question, cosine-rank the corpus, take the two nearest vectors, synthesize.

The consequence is not "sometimes the answers are mediocre." The consequence is structural: this system has no representation of failure. Top-2 always returns two chunks. It returns two chunks when the corpus contains the answer, and it returns two chunks when the corpus has never heard of the subject — the same two nearest-but-irrelevant neighbours, handed to an LLM that will write a fluent paragraph over them because that is what you asked it to do. It cannot abstain, because nothing in the chain is capable of producing zero nodes.

The fix is to give the pipeline a floor and a ceiling. A cutoff so it can return nothing, a reranker so the nodes that survive were actually read against the query, and a top_k wide enough that the reranker has something to choose from.

from llama_index.core.postprocessor import SentenceTransformerRerank, SimilarityPostprocessor

engine = index.as_query_engine(
    similarity_top_k=20,                 # wide: the reranker's candidate pool
    node_postprocessors=[
        SimilarityPostprocessor(similarity_cutoff=0.5),   # BEFORE the rerank — see below
        SentenceTransformerRerank(
            model="cross-encoder/ms-marco-MiniLM-L-6-v2",  # name it: the class default is an STS model
            top_n=5,                                       # name it: the class default is 2
        ),
    ],
)
resp = engine.query("What changed in the Q3 pricing policy?")
if not resp.source_nodes:            # the pipeline can now say "I don't know"
    return "No supporting documents found."

Note the two named arguments in the reranker. SentenceTransformerRerank.__init__ defaults to top_n=2 and model="cross-encoder/stsb-distilroberta-base" — a semantic textual similarity model, not a relevance reranker. Leave both at their defaults and you have added a hop that keeps two nodes and ranks them with the wrong objective. Defaults in this framework are for notebooks. Name everything on the query path.

Fails when: the corpus doesn't contain the answer at all. similarity_top_k=2 returns the two nearest vectors regardless — there is no threshold in the default chain, because SimilarityPostprocessor is opt-in via node_postprocessors and nothing else in as_query_engine() looks at node.score — so the synthesizer receives two irrelevant chunks and confabulates over them. The operator sees a confident, well-cited, entirely fictional answer, and there is no error, no low-confidence signal, and no log line anywhere that distinguishes it from a good one.

Paying Twice for the Same Corpus

The ingest script works. It runs nightly, it takes forty minutes, and nobody has looked at it since the week it was written. It reads every document, splits every document, and embeds every chunk — every night, whether or not a single byte changed. This is the largest avoidable line item in most production RAG budgets, and it survives because it is invisible: the job succeeds, the index is correct, and the only symptom is a number on an invoice that someone else reads.

The seduction here is that VectorStoreIndex.from_documents(docs) is the call the tutorial gave you, and it has no memory. It does not know what it saw last run because you never gave it anywhere to remember. That is what the docstore is for — doc_id → doc.hash — and it is what IngestionPipeline exists to drive. docstore_strategy defaults to DocstoreStrategy.UPSERTS, which processes a document only when its ID is new or its hash has moved, and deletes the old nodes from both the docstore and the vector store before inserting the new ones. IngestionCache sits behind that, keyed per transformation step, so the expensive steps (LLM extractors, embeddings) are skipped for content you have already processed.

Now the trap, and it is the one that catches almost everyone who does know about the docstore. Attaching a docstore to a StorageContext does not populate it. VectorStoreIndex._add_nodes_to_index gates the docstore write behind if not self._vector_store.stores_text or self._store_nodes_override: — and Qdrant, Pinecone, Weaviate and pgvector all set stores_text=True, while store_nodes_override defaults to False. So with a real vector database, from_documents(docs, storage_context=sc) writes the nodes to the vector store and nowhere else. Your docstore is empty — and the probe you'd reach for won't even show you an empty dict: against a text-storing store, index.ref_doc_info raises NotImplementedError ("Vector store integrations that store text in the vector store are not supported by ref_doc_info yet."); the {} return only exists when stores_text=False. The dedup you carefully wired is a silent no-op, because the index queries perfectly — the vector store has the text.

Stop trying to make from_documents remember things. Drive the pipeline directly:

from llama_index.core.ingestion import IngestionPipeline, IngestionCache, DocstoreStrategy
from llama_index.core.node_parser import SentenceSplitter
from llama_index.core.storage.docstore import SimpleDocumentStore

pipeline = IngestionPipeline(
    transformations=[SentenceSplitter(chunk_size=512), embed_model],
    docstore=SimpleDocumentStore(),      # here it IS written to
    vector_store=vector_store,           # required, or UPSERTS silently degrades
    docstore_strategy=DocstoreStrategy.UPSERTS,
    cache=IngestionCache(),
)
nodes = pipeline.run(documents=docs, num_workers=4)
pipeline.persist("./pipeline_storage")   # cache + docstore; .load() next run

And make doc_id stable and source-derived — a URL, a primary key, a file path. A reader that mints a fresh UUID per run turns every sync into a corpus of brand-new documents, and the docstore buys you exactly nothing.

Fails when: you attach the docstore to a StorageContext and hand it to from_documents() instead of to an IngestionPipeline. The write is gated on not self._vector_store.stores_text or self._store_nodes_override, both of which are false against any real vector DB, so the docstore stays empty (index.ref_doc_info raises NotImplementedError against a stores_text=True store if you think to check it). The pipeline itself raises nothing — the index queries fine — and the operator's only symptom is that the re-embed bill never drops and the same document accumulates a fresh copy of its chunks on every run.

Tuning Chunk Size Against Generation Quality

You read a blog post about chunk size. You tried 512, then 1024, then 2048. At 2048 the answers read better — longer, more coherent, fewer half-sentences — so you shipped 2048. This is one of the most common tuning loops in RAG and it optimizes the wrong variable, because a chunk is simultaneously the unit of retrieval and the unit of context, and those two want opposite things.

Generation wants big chunks: more surrounding prose, fewer severed sentences, an answer that isn't cut in half at a boundary. Retrieval wants small ones: a 2048-token chunk that covers four topics embeds as the average of four topics, and that average vector is not close to any specific question. So as you turn the knob up, the answers you're reading get more fluent while the recall underneath them collapses — and you cannot see the collapse, because you are grading the output of the queries that still worked.

SentenceSplitter defaults are chunk_size=DEFAULT_CHUNK_SIZE (1024) and chunk_overlap=SENTENCE_CHUNK_OVERLAP (200) — not the 20 that half the blog posts quote. They are a fine starting point for dense prose. The point is not which number is right; the point is that you cannot answer that question by reading answers. You answer it with a retrieval metric, in isolation, with no LLM in the loop.

RetrieverEvaluator is that measurement and it takes three lines. Its metric registry in llama_index/core/evaluation/retrieval/metrics.py gives you hit_rate, mrr, precision, recall, ap, ndcg. Hit rate is the ceiling on everything downstream: at 0.6, no prompt and no model gets you past 60% correct.

from llama_index.core.evaluation import RetrieverEvaluator

for chunk_size in (256, 512, 1024, 2048):
    nodes = SentenceSplitter(chunk_size=chunk_size, chunk_overlap=chunk_size // 5).get_nodes_from_documents(docs)
    idx = VectorStoreIndex(nodes)
    ev = RetrieverEvaluator.from_metric_names(
        ["hit_rate", "mrr"], retriever=idx.as_retriever(similarity_top_k=5)
    )
    results = [
        await ev.aevaluate(query=q, expected_ids=ids)
        for q, ids in qa_dataset.query_docid_pairs
    ]
    hit = sum(r.metric_vals_dict["hit_rate"] for r in results) / len(results)
    print(chunk_size, round(hit, 3))     # THIS is the number you tune against

If your chunks have structure, stop guessing at it entirely: MarkdownNodeParser splits on headers, CodeSplitter on AST boundaries. Free structure beats inferred structure, and it beats a knob.

Fails when: you turn chunk_size up until the prose reads well. A 2048-token chunk averages several topics into one vector that matches no specific query, so hit_rate silently drops while answer fluency — the only thing you were looking at — improves. The operator sees confident, well-written answers that are increasingly about the wrong document, and blames the retriever, or the embedding model, or the LLM. All three are innocent.

Thresholding After the Reranker

This is the single most destructive one-line mistake in LlamaIndex, and it is destructive precisely because it is such good reasoning. You added a cross-encoder reranker. It sorts nodes by real relevance. So the obvious next move is to keep only the relevant ones — put a SimilarityPostprocessor(similarity_cutoff=0.5) at the end of the list and drop everything below the bar. Sensible. Defensible in review. And it deletes your entire context.

Here is the mechanism. node_postprocessors is an ordered list, and the stages are asymmetric about the score field: a reranker writes it, a filter only reads it. Read SentenceTransformerRerank._postprocess_nodes and the write is explicit: node.score = score — unconditionally, destroying the retrieval score unless you pass keep_retrieval_score=True, which stashes the old value in node.metadata["retrieval_score"] and defaults to False. SimilarityPostprocessor never writes anything back; it just reads whatever is in node.score and drops the node if it falls under the cutoff. And what replaces it is not a cosine similarity. It is a cross-encoder logit: unbounded, uncalibrated, and routinely negative. A genuinely excellent match can score -1.2. So SimilarityPostprocessor(similarity_cutoff=0.5) running after the reranker compares -1.2 < 0.5, drops the node, drops all of them, hands the synthesizer an empty list, and — see the next section — gets back the literal string "Empty Response" with no exception raised anywhere.

Order the list so each stage reads a score it understands:

from llama_index.core.postprocessor import SimilarityPostprocessor, SentenceTransformerRerank

engine = index.as_query_engine(
    similarity_top_k=20,
    node_postprocessors=[
        SimilarityPostprocessor(similarity_cutoff=0.5),   # 1. cosine scale: cutoff is meaningful
        SentenceTransformerRerank(                        # 2. overwrites score with a logit
            model="cross-encoder/ms-marco-MiniLM-L-6-v2",
            top_n=5,
            keep_retrieval_score=True,   # old score -> node.metadata["retrieval_score"]
        ),
        # 3. NOTHING that reads node.score numerically goes here. top_n is your filter now.
    ],
)

The general law: NodeWithScore.score is not comparable across producers. A cosine from a vector store, a BM25 relevance score, and a cross-encoder logit all land in the same float field with wildly different scales. If you ever write if node.score > 0.7, you have hard-coded a threshold to one specific model's score distribution and it will not survive the next model upgrade. Filter by rank (top_n) after a reranker, never by value.

The same field bites from the other direction, too: SimilarityPostprocessor treats a missing score as a failure — if similarity is None: should_use_node = False — so any retriever that doesn't populate scores (SummaryIndexRetriever, most hand-rolled ones) has every node silently dropped by a postprocessor whose cutoff you set to 0.0 thinking it was a harmless no-op.

Fails when: a SimilarityPostprocessor sits after a reranker in node_postprocessors. SentenceTransformerRerank._postprocess_nodes executes node.score = score, replacing the bounded cosine with an unbounded cross-encoder logit that is frequently negative, so the cutoff you set at 0.5 now rejects every node including the perfect ones. The operator sees "Empty Response" — or a model answering from its weights over an empty context — on exactly the queries the reranker just ranked most confidently.

Tenancy in the Prompt

The system prompt reads: "Only use documents belonging to tenant acme. Never reveal information from other tenants." It works in testing. It works in the demo. It is not access control — it is a request, addressed to a stochastic text generator, about text you have already placed inside its context window.

Understand what happened before that prompt was ever read. The retriever ran an ANN search across the whole collection, pulled the top-k chunks, and those chunks — some of them another tenant's — were serialized into the prompt. The data has already left the store. The only thing standing between another customer's contract and your user's screen is a sentence, and that sentence is competing for attention with everything else in a 40k-token context. Any prompt injection in any retrieved document is a peer of your instruction, not a subject of it.

Filters are not a relevance hint. A MetadataFilters is a correctness constraint that changes the candidate set before the ANN search runs. VectorStoreQuery in llama_index/core/vector_stores/types.py carries a filters: Optional[MetadataFilters] field, and VectorIndexRetriever passes whatever you handed its filters= argument straight into it — which the store then translates into its native pre-filter. Qdrant, Pinecone, Weaviate and pgvector all do this in the engine. You get five results out of the right thousand rows, not five out of the wrong million.

from llama_index.core.vector_stores import FilterOperator, MetadataFilter, MetadataFilters

filters = MetadataFilters(filters=[
    MetadataFilter(key="tenant_id", value=request.tenant_id, operator=FilterOperator.EQ),
])
engine = index.as_query_engine(filters=filters, similarity_top_k=10)


retriever = VectorIndexAutoRetriever(
    index,
    vector_store_info=info,
    extra_filters=filters,   # AND-ed onto whatever the model infers; must not be OR
)

Two rules follow and neither is negotiable. Anything that must be true of an answer — tenancy, ACL group, freshness window, legal jurisdiction — goes in the pre-filter and never in the prompt. And where an LLM is allowed to infer filters, the security-bearing ones go in extra_filters, which is AND-ed on top, because you must never let a model decide whether to scope a query to the caller's tenant. Then assert it: after retrieval, check that every returned node's metadata["tenant_id"] actually equals the caller's. A filter that silently didn't apply and a filter that correctly returned nothing look identical from the outside.

Fails when: the tenant scope lives in the prompt instead of in MetadataFilters. The ANN search has already pulled another tenant's chunks into the context window by the time the model reads your instruction, so a single prompt injection in any indexed document — or just an ordinary long-context attention lapse — leaks them. The operator sees nothing: no error, no anomaly, a perfectly fluent answer. The customer sees another customer's contract.

Trusting an Empty Retrieve

Nowhere in your code do you check len(nodes). Why would you? Every other data-access layer you have ever used raises on a miss, or returns something falsy you would trip over immediately. So you wrote answer = engine.query(q).response and moved on.

LlamaIndex does neither. Read BaseSynthesizer.synthesize in llama_index/core/response_synthesizers/base.py: the first thing it does is if len(nodes) == 0:, and in that branch it constructs Response("Empty Response") and returns it — without ever calling the LLM, and without raising anything. That is a plain Response object. It has a .response attribute. It is a string. It is truthy. It will be JSON-serialized into your API payload, stored in your chat history, passed back to an agent as a tool observation, and scored by your eval harness as if it were content.

The reason this is an anti-pattern rather than a mere gotcha is how many upstream mistakes funnel into it. An over-narrow metadata filter — a hallucinated service="cart", a mis-cased "Checkout", year=2025 against a 2024 corpus, an AND chain no node satisfies. A SimilarityPostprocessor placed after a reranker, per the section above. A cutoff tuned against a different embedding model. Every one of these produces zero nodes, and every one of them arrives at the caller as the same cheerful two-word string.

So make the empty case a case:

nodes = retriever.retrieve(query)

if not nodes:
    # Do NOT synthesize. Decide.
    nodes = fallback_retriever.retrieve(query)      # relax the filters, widen top_k
    if not nodes:
        return "I don't have documents covering that."

resp = engine.synthesize(query, nodes=nodes)


assert all(n.node.metadata.get("tenant_id") == tenant for n in resp.source_nodes)

Decompose the query engine when you need this — retrieve, inspect, then synthesize — rather than letting query() collapse the whole thing into one call whose failure mode is a string. And note the genuinely dangerous sibling of the empty retrieve: the near miss. Right service, wrong year. Correct filter, missing tenant. That returns a full, plausible node list from the wrong partition and produces a confidently wrong answer with citations. Neither trusting an empty retrieve nor trusting a non-empty one is safe; assert on both.

Fails when: retrieval returns zero nodes and you never check. BaseSynthesizer.synthesize short-circuits on len(nodes) == 0 and returns Response("Empty Response") — a literal two-word string, no LLM call, no exception — which your API serializes, your agent consumes as a tool observation, and your eval harness scores as a real answer. The operator sees a 200, a normal-looking response body, and green dashboards, while every filtered query in production is returning a placeholder.

Reaching for a Bigger Model

The answers are wrong. The meeting concludes that the model isn't smart enough. Somebody upgrades the LLM, the bill triples, quality moves by roughly nothing, and the meeting reconvenes.

This is the most expensive mistake on this page and it is the most understandable one, because it is the only lever that requires no understanding of the system. It is a one-line diff. It requires no eval set, no chunking decision, no filter schema, no rerank latency budget. And it is wrong for a reason that should be a poster on the wall: a bigger model cannot read a chunk you never retrieved. If the answer was not in the context window, generation quality is irrelevant — you have paid a premium for a more articulate way of not knowing.

Almost every quality problem in the first year of a RAG system is a retrieval problem wearing a generation problem's clothes. The chunk boundary cut the answer in half. Similarity search pulled last year's tenant. The exact error code was invisible to the embedding because embeddings capture topic, not tokens — ERR_CONN_RESET embeds as "some error-ish thing," and cosine returns twenty neighbours about the right subject and none about the right string.

You settle this with two numbers, not an argument. Measure retrieval in isolation — no LLM in the loop — with RetrieverEvaluator and hit_rate. If hit rate is 0.6, then 40% of your queries are unanswerable no matter what generates over them, and every dollar spent on the model is spent on the wrong half of the pipeline. Then, and only then, split the generation failures with the LLM judges in llama_index.core.evaluation:

from llama_index.core.evaluation import FaithfulnessEvaluator, RelevancyEvaluator


# relevant but unfaithful -> the model ignored the context   -> prompt/model bug
faith = FaithfulnessEvaluator(llm=judge)     # needs no labels: runs on live traffic
rel = RelevancyEvaluator(llm=judge)

Those two are orthogonal, and holding them apart is the whole diagnostic. High faithfulness plus low relevancy is a retriever that pulled plausible, on-topic, wrong documents — a bigger model makes that worse, because it summarizes the wrong evidence more convincingly. Low faithfulness with high relevancy is the only case where the model is genuinely the problem. Climb the patterns ladder — chunking, filters, hybrid, rerank — before you climb the price list.

Fails when: you upgrade the LLM without ever computing hit_rate. Retrieval hit rate is a hard ceiling on answer correctness — at 0.6, no model reaches 61% — so the upgrade buys fluency and costs money while the actual defect (a chunk that never entered the context window) is untouched. The operator sees the bill move and the eval scores not move, and reaches for an even bigger model.

Letting the Globals Drift

Settings is a process-global singleton and a genuine ergonomic win — configure the models once and forty components stop asking you for them. It is also a global, in a system whose two halves run in two different processes, and that is a trap with no floor.

Your ingest job is a batch script. Your query service is a web server. They share a vector store and nothing else. The ingest job sets Settings.embed_model = HuggingFaceEmbedding(model_name="BAAI/bge-small-en-v1.5") because someone wanted to stop paying OpenAI for embeddings. The query service was written six weeks earlier, in a different file, and never sets it at all.

Now trace what happens. VectorStoreIndex.from_vector_store(vector_store) takes an embed_model: Optional[EmbedType] = None. None resolves to Settings.embed_model, whose getter is if self._embed_model is None: self._embed_model = resolve_embed_model("default") — and resolve_embed_model("default") constructs an OpenAIEmbedding(). So the query service embeds every incoming question with OpenAI, against a collection written by BGE.

And here is why this is the quietest failure in the entire framework: the collection carries no record of which model wrote it. There is no dimension mismatch to save you if the two models happen to agree on width — and if they don't, you get a loud error, which is the lucky outcome. When the widths match, the ANN index accepts the query vector, cosine comes back in the 0.6–0.8 band that looks exactly like healthy retrieval, and the nearest neighbours are noise. No exception. No warning. Just a system that has quietly become a random-chunk generator with citations.

Stop treating the embedding model as ambient configuration. Pin it in one place both processes import, pass it explicitly, and hard-fail on startup:

EMBED_MODEL_ID = "BAAI/bge-small-en-v1.5"
EMBED_DIM = 384

def embed_model():
    return HuggingFaceEmbedding(model_name=EMBED_MODEL_ID)

# query service, at startup — explicit, not via Settings:
index = VectorStoreIndex.from_vector_store(vector_store, embed_model=embed_model())

# and assert the store agrees, before you serve a single request:
info = client.get_collection("corpus")
assert info.config.params.vectors.size == EMBED_DIM, "embedding model drift"

Write the model ID into the collection's payload or name at ingest time (corpus_bge_small_v1) and check it at boot. The rule that keeps Settings sane in general: globals for the boring defaults, explicit arguments for anything a reviewer would want to see. An embedding model is the second kind.

Fails when: ingest and query processes disagree about Settings.embed_model. from_vector_store() defaults embed_model=None, which falls through to resolve_embed_model("default") and silently constructs OpenAIEmbedding() — and the collection records nothing about which model wrote it, so if the dimensions happen to match, queries return plausible-looking cosine scores over semantically meaningless neighbours. The operator sees normal latency, normal scores, and answers that are subtly, inexplicably about the wrong documents.

An Agent Where a Query Engine Would Do

Agentic RAG is the most fun thing in the framework, and that is exactly the problem. Wrapping your query engine in a QueryEngineTool and handing it to a FunctionAgent from llama_index.core.agent.workflow takes ten minutes, demos beautifully, and handles the multi-hop question your PM asked in the meeting. So it becomes the default path — every question, easy or hard, goes through the loop.

What you gave up is not obvious until you try to improve the system. A query engine's retrieval is a pure function of the question: the same query embeds to the same vector, hits the same ANN index, and returns the same nodes. That determinism is what makes RetrieverEvaluator possible — you can hand it (query, expected_ids) pairs and get a hit_rate because there is a stable set of retrieved ids to compare against. An agent has no such thing. The same question takes a different path on a different day: it may search once, or three times with reworded queries, or reason its way to an answer without searching at all. There are no fixed expected_ids because there is no fixed retrieval. The moment you make retrieval agentic, you make it unevaluable by the only cheap, deterministic, LLM-free metric you had.

The cost model inverts too, and nobody notices. One-shot RAG is one embedding plus one generation with a bounded p99. An agent with two query-engine tools routinely makes four to six LLM calls per question — a routing call, one generation inside each query engine's synthesizer, and a final synthesis over observations that already contain the retrieved chunks. Context grows with each observation and you re-send the whole history every turn, so cumulative tokens grow quadratically in tool calls. A question that a single retrieval would have answered now costs 10x and takes twenty seconds.

Climb this ladder deliberately. Route the easy 80% to a plain query engine and let only the questions that provably need multiple hops reach the agent:

from llama_index.core.query_engine import RouterQueryEngine
from llama_index.core.selectors import PydanticSingleSelector
from llama_index.core.tools import QueryEngineTool

router = RouterQueryEngine(
    selector=PydanticSingleSelector.from_defaults(),
    query_engine_tools=[
        QueryEngineTool.from_defaults(
            query_engine=index.as_query_engine(similarity_top_k=5),
            name="lookup",
            description="Single-fact lookups: one clause, one date, one number, one passage.",
        ),
        QueryEngineTool.from_defaults(
            query_engine=agent_engine,
            name="multi_hop",
            description="Questions spanning several corpora, or requiring a comparison or a follow-up query.",
        ),
    ],
)

And if you do need the agent, evaluate the trajectory, not just the final string — subscribe to ToolCallResult on the handler and log every query it actually issued. You will be surprised, and the surprise is the point.

Fails when: you make every query agentic and then try to run RetrieverEvaluator over it. Agent retrieval is not a pure function of the question — the same input searches once today and three times tomorrow — so there are no stable expected_ids to score against and hit_rate stops being computable at all. The operator loses the one deterministic, LLM-free quality metric they had, and every subsequent retrieval change becomes a coin flip resolved in production.

Grading Your RAG With a Golden Set the Model Wrote

You did the responsible thing. You built an eval set. You ran generate_qa_embedding_pairs(nodes=nodes, llm=llm) (from llama-index-finetuning, or DatasetGenerator in llama_index.core.evaluation.dataset_generation), got back an EmbeddingQAFinetuneDataset of a few hundred question/node pairs, checked the JSON in, wired the assertion into CI, and shipped. Every threshold passes. It has passed every day since.

It will keep passing while your users suffer, and the reason is worth being precise about, because "the model graded itself" is only half of it.

A generator that reads chunk N and writes a question answerable from chunk N produces questions that are paraphrases of chunk N in the embedding model's own vocabulary. The retriever's job is then to find chunk N given a lightly-rewritten copy of chunk N — which is the easiest possible retrieval task, and one that dense vector search is structurally guaranteed to be good at. Your hit rate is 0.94, and it is 0.94 because the questions were derived from the answers.

Real questions aren't like that. Real users ask in vocabulary the corpus never uses ("why did the deploy blow up?" against a runbook that says "rollout aborted: readiness probe exceeded initialDelaySeconds"). They ask questions that span three chunks. They ask about the exact error code, the invoice number, the thing the embedding cannot see. They ask questions your corpus cannot answer — and a synthetic set contains none of those, because every question was generated from a node that answers it, so your suite has zero examples of the case where the right behaviour is to say "I don't know." You have built a fixture that certifies the exact failures your users are hitting.

So use the generator for what it is — a scaffold, not a golden set:

# 3. Then add the two classes the generator structurally cannot produce:
#      - real logged production queries, labelled by hand
#      - unanswerable questions, with expected_ids = []  (the abstention set)
qa = EmbeddingQAFinetuneDataset.from_json("eval/qa_dataset.json")   # checked in, hand-curated

hit = mean(r.metric_vals_dict["hit_rate"] for r in results)
assert hit >= 0.85, f"retrieval regressed: {hit:.2f}"
assert abstain_rate >= 0.9, "the system answered questions it should have refused"

Grow the set from real failures: every bug report becomes a case, forever. And keep the abstention set — it is the only thing that ever tests whether the pipeline you built in the first section can actually say "I don't know."

Fails when: the golden set is the unedited output of generate_qa_embedding_pairs. Every question was written from the node that answers it, so it is a vocabulary-matched paraphrase of the chunk and the retriever finds it trivially — and the set contains zero unanswerable questions, because the generator cannot produce one. The operator sees a hit rate of 0.94 and a green CI gate while production is confabulating over irrelevant chunks, and the eval suite certifies the exact failure the users are reporting.