Reranking — Audio Guide

The embedding model scores the question and each passage independently. That makes vector search extremely fast. But because it never sees the question and the passage together, it is comparing two summaries rather than judging a true match. As a result, the top of the ranking can be imprecise. The chunk that truly answers a question can get lost among near duplicates that appear relevant.

A cross-encoder reads the query and the candidate passage jointly as one input. It scores true relevance, not separate summaries. That joint attention is materially more precise. But it is only affordable over a small candidate set. So the system retrieves wide with the fast bi-encoder first, then reranks narrow with the cross-encoder. Fusion and reranking compose rather than compete. One widens the pool, the other orders it.

The design is a deliberate trade. Speed for the first pass, precision for the second. The bi-encoder handles the entire corpus quickly. The cross-encoder refines the top results. This two‑stage approach prevents any single component from becoming a bottleneck.

The cross-encoder runs on the open neural network exchange runtime through the same library the platform already uses. No deep learning framework, no graphics card, no separate service. Cross‑encoding twenty candidates costs CPU milliseconds. That is negligible next to language model synthesis.

But the cross-encoder is not always available. If its weights cannot load, the builder returns nothing and the caller keeps the plain unreranked top results. Every stage can be switched off by an environment flag. Grounding degrades in quality but never fails outright. That rule governs the whole retrieval path.

The bi-encoder’s weakness is inherent. It never sees the question and passage together, so it relies on cosine similarity of separate embeddings. That works well for paraphrase but misses exact lexical matches. Keyword search with BM25 captures those, so the system fuses dense and sparse results to widen recall. Then the cross-encoder reorders for precision.

The result is a retrieval stack that balances speed, recall, and precision. The bi-encoder provides the speed. The fusion provides the breadth. The cross-encoder provides the final ordering. None of these stages is a hard dependency. They compose gracefully. The user gets the best available results under any circumstances.

That is how the platform ensures that the chunk that truly answers a question rises to the top, despite the bi-encoder’s inherent limitation. The trade‑off is explicit and deliberate. Speed at the first stage, precision at the last.

The reranker model does not score the question and each passage separately. Instead, it reads them together as a single input. That is the key change. It scores true relevance directly. This is far more precise because it judges the actual match, not just two summaries. The bi-encoder model scores each piece by itself. It compares a summary of the question to a summary of the passage. That is fast but imprecise. The reranker avoids that problem. It looks at the full pair.

This precision comes at a cost. The reranker must process every pair individually. That is expensive for a large set. So it is only used on a short shortlist. The retrieval system first gathers a wide pool of candidates. It uses cheap methods for that step. Then the reranker reorders those few candidates. This keeps the overall system fast while fixing precision where it matters. The reranker never replaces the first-stage retriever. It only refines it.

The design is deliberate. The reranker runs on small, fast models that require no graphics card. It works on CPU in milliseconds. That is important because the whole stack ships in a lightweight image. Reranking is switched off in production because of memory limits. It is only used in development and pipeline profiles. This is a documented trade-off.

The reranker also includes quiet extras that improve its output. The raw scores are squashed onto a zero-to-one scale. This gives downstream consumers a stable number. Near-duplicate chunks are removed, keeping the best copy. This prevents the shortlist from filling up with redundant text. Query expansion widens recall on the question side. It splits compound names and expands acronyms. This helps match passages that spell things differently. But the expansion has a trap. If too many variants are added, the real question gets diluted. The fix caps the number of variants and repeats the original question to keep it dominant.

None of this is a hard dependency. If the reranker weights cannot load, the system simply keeps the unreranked results. If the keyword retriever fails, the dense retriever is used alone. The rule is that quality degrades but never fails. That is the design principle.

In the serving chain, the reranker comes after a guard. Retrieved text is fenced against prompt injection before the reranker sees it. That matters because the reranker is a model reading attacker-controlled input. It decides what the synthesizer sees. So it must be protected.

The reranker is measured, not felt. Changes are gated on retrieval metrics like mean reciprocal rank and hit rate. These are computed over fixed sets of questions and the correct answers. The reranker is deterministic, so it can be scored reliably. The argument is never that it feels better. It is that the numbers are better. That is the standard.

Every retrieval here follows a two-stage shape. The first pass is deliberately wide. It aims for recall, not precision. The system fetches twenty candidates from each search leg. One leg uses dense embeddings. The other leg uses keyword search. A technique called reciprocal rank fusion merges those two lists into one broad candidate set.

Then comes the second stage. A cross-encoder reorders that shortlist. It scores true relevance by reading the question and each passage together as a single pair. That narrow pass fixes precision where it matters.

The two stages compose rather than compete. One widens the pool. The other orders it. They solve different problems. If you remove the fusion step, the reranker has nothing good to find. It gets a shallow or lopsided candidate set. If you remove the reranker, the best chunk might sit at rank eleven. Nothing reads it there. The system loses the best answer.

The design is deliberate. First you gather broadly, then you rank precisely. Neither stage can replace the other. Fusion improves recall. Reranking improves ordering. Taking away either one degrades a different thing. Drop fusion and recall narrows. Drop the reranker and precision suffers.

The cross-encoder is small and efficient. It runs on the same library the embeddings already use. There is no new package, no separate service. It costs only milliseconds for twenty candidates. That is fast enough to sit between the first stage and the final synthesis.

The system also handles failures gracefully. If the reranker cannot load, the caller keeps the plain top results. If the keyword retriever cannot build, the caller gets dense retrieval only. Nothing hard fails. Quality degrades, but the system still works.

So the shape is simple. Retrieve wide. Then rerank narrow. The two stages are partners. Losing one breaks the balance. You need both to find the best supporting fact for your answer. That is why every retrieval path here follows that pattern. It is not an accident. It is the only way to get both recall and precision without making the whole thing too slow.

The cross encoder emits raw relevance scores that are unbounded. They have no fixed limit. They also drift depending on which model you use. That makes them hard to compare. So the platform transforms these raw numbers. It passes them through a logistic function. This squashes them onto a scale from zero to one. Crucially, this does not change the ordering. The best chunk still ranks highest. The second best stays second. The relative ranking is completely untouched. What changes is the scale. Instead of a wild number that shifts around, you get a consistent score between zero and one. That stable number is what every downstream consumer reads. Downstream consumers include the synthesizer, the faithfulness checks, and the evaluation harness. They all need a score that means the same thing every time. A raw logit from one run might be eight. From another run it might be three, even for the same relevance. That drift makes comparisons meaningless. Squashing to zero to one eliminates that problem. The same passage will get the same score each time. That is because the logistic function is deterministic. And because the cross encoder is also deterministic with no credentials needed, this score is stable across repeated calls.

The trade off is that you lose nothing in precision. The ordering remains identical. All you gain is a consistent scale. That consistency is critical for evaluation. Without it, the evaluation harness cannot trust its own numbers. The system would have to treat each score as relative only within that single query. That is much less useful. So the logistic squash is a small but vital step. It fixes a real problem with raw unbounded scores. Every downstream consumer gets a stable, comparable number. That is far more useful than a raw score that drifts with the model. The platform does this without any extra cost or complexity. It is part of the built in postprocessor. The whole retrieval stack runs in a lightweight processor only image. The logistic squash adds no new dependencies. It is just a simple mathematical transformation. But without it, the entire evaluation pipeline would be unreliable. The synthesis engine would not know if a score of two is good or bad. The faithfulness checks would have no baseline. The evaluation harness could not compare results across different runs. So the squash is essential, not just a nice extra. It is essential for producing trustworthy numbers.

Wide recall often surfaces near identical chunks. They are the same paragraph or code block but re split at a chunk boundary. This happens because documents are split into smaller pieces. A paragraph may be split at different boundary points. So two chunks can hold the same content. Wide recall pulls in both. Keeping several of them uses up the scarce kept set budget on redundancy instead of new coverage. That is exactly what starves an answer of a supporting fact. The solution is to drop near duplicates before the set is cut down to size. Only the highest scoring copy stays. The copy with the highest reranker score wins. The cross encoder gives each chunk a relevance score between zero and one. That score determines which copy of a near duplicate survives. The highest score stays. This way the reranker spends its limited space on diverse useful chunks. Without this step, the reranker might keep two copies of the same information. That leaves a gap that could have held a different useful chunk. This refinement is one of three quiet extras around the core reranker. The others are score scaling and query expansion. But near duplicate removal directly addresses the redundancy problem. It happens after the cross encoder re scores the shortlist. Then the set is truncated to the top results. This ensures the final set contains a range of supporting facts. The answer gets better coverage. The whole process is called retrieve wide then rerank narrow. Fusion widens the pool, and the reranker orders it. They compose rather than compete. One widens the pool, the other orders it. Removing either one degrades a different thing. Drop fusion, and the reranker has nothing good to find. Drop the reranker, and the best chunk sits at rank eleven where nothing reads it. So near duplicate removal is a key piece. It stops the reranker from wasting its budget on redundancy. That keeps the system efficient and effective. Each chunk in the kept set is a chance to support a different part of the answer. If two chunks say the same thing, that chance is wasted. The answer loses a potential supporting fact. By dropping duplicates first, the reranker maximises the diversity of information. This is a deliberate design choice. It is built into the postprocessor. The near duplicate detection uses the reranker scores to pick the best copy. Then the set is truncated. This order matters. If truncation happened first, you might keep duplicates and miss a unique chunk. So the removal comes before the cut. That is the correct sequence. The result is a final set that is both precise and broad. It has the best scoring chunks without redundancy. That is how the platform delivers complete answers from a limited budget.

Semantic similarity alone misses chunks whose relevance is lexical. A dense embedding model is a bi-encoder. It scores the question and each passage independently. It never looks at them together. So it compares two summaries, not true relevance. Exact matches like an identifier or an error code can be lost.

Keyword search catches those exact tokens directly. It matches the letter, not the meaning. But keyword and dense rankings use different score scales. They are incomparable. Fusion merges them by rank position instead of by score. This is called reciprocal rank fusion. The highest rank from each list gets combined. No normalizing needed.

A wider candidate pool is the result. The cross‑encoder reranker then re‑scores that wider set. It reads the question and candidate together for true relevance. So fusion and reranking compose. One widens recall, the other orders precision. The order is deliberate: retrieve wide, then rerank narrow.

Every stage degrades gracefully. If keyword search cannot build, the dense retriever is used alone. If the reranker fails, the plain top results are kept. Grounding never hard fails. The whole stack runs in a single lightweight processor‑only image.

Keyword search increases recall for exact tokens. Dense search covers paraphrases. Combined by rank position, they enrich the candidate set without fixing a score scale. The reranker then narrows that set precisely. It is a trade‑off that works well for code and technical prose.

The cross‑encoder is deterministic and needs no credentials. It can be measured with mean reciprocal rank and hit rate. That makes improvements quantifiable, not just felt. The feature is switched on by an environment flag. It defaults to off in production to save memory. In development, it refines the top results.

This design avoids hard dependencies. If the reranker weights cannot load, the caller keeps the plain results. If the corpus is too small for keyword search, the dense retriever stands alone. The whole retrieval path follows the same rule: grounding degrades in quality, never fails outright.

Query expansion widens recall on the question side with no model call at all. Compound identifiers are split into their spaced forms. Domain acronyms are expanded to their phrases and back again. This means a question that names something one way will still match a passage that spells it the other way. The process is purely lexical. It is deterministic. It costs nothing, no model required.

But there is a trap. When you supply several query strings together, they get averaged into an unweighted mean of their embeddings. Those extra strings replace the original question instead of joining it. With eight variants, the real question falls to roughly one ninth of that average. The retrieval point then drifts toward the lexical variants. Recall does not widen. It re-centers. That is strictly worse than not expanding at all.

The fix is simple. Cap how many variants reach the aggregate. Repeat the original question several times inside the aggregate so it keeps the majority share of the mean. The variants still pull toward passages that spell a name differently. But the question stays dominant.

Keyword search and the cross-encoder are unaffected either way. Both read the original question text directly. So you get the benefit of wider recall without the drift. The expansion list must stay small and high-precision. Too many variants, and the original question gets lost. Keep the list tight, repeat the question, and the retrieval stays accurate.

Query expansion has a hidden problem. When you supply several query strings together, they get averaged into one single point. The extra strings replace the original question instead of joining it. That means too many variants dilute the real question.

Say you have eight variants. The original question then makes up only about one ninth of that average. The retrieval point drifts toward the lexical variants. Recall does not widen. It actually re-centres. That is strictly worse than not expanding at all.

The fix is two-sided. First, cap how many variants reach the aggregate. Second, repeat the original question several times inside the aggregate so it keeps the majority share of the mean. The variants still pull toward passages that spell a name differently, but the question stays dominant.

Keyword search and the cross encoder read the original question text directly, so they are unaffected either way. The cross encoder is a model that reads the question and a candidate passage together and scores true relevance. It is precise but only affordable over a small candidate set. That is why you retrieve wide first, then rerank narrow.

The failure mode is called the centroid trap. It is easy to overlook. The naive version of query expansion is actively harmful. You think you are widening recall, but you are just moving the centre. The original question gets buried under its own variants.

The two-sided fix works because it keeps the original majority. The variants still help but they do not overpower. Keyword search and the cross encoder stay reliable because they see the original text. The embedding model sees the averaged point, so only that part needs the fix.

That is the trade-off. Query expansion is free and deterministic. It uses no model call. But if you do not handle the centroid trap, you make things worse. The narrow fix of capping and weighting turns a harmful feature into a helpful one.

The order of the postprocessors is deliberate. Retrieved text is fenced against prompt injection first. Nothing downstream reads it until that fence is in place. Only after that is the text reordered by the reranker. A reranker that reads unfenced hostile text becomes part of the attack surface. It is a model reading attacker controlled input. That model decides what the synthesizer ends up seeing. So the fence must come before the reordering. This protects the whole chain. The guard runs first, then the rerank. This is not an accident. It is a planned trade off. The reranker is powerful. It orders the best chunks. But it can be tricked. Hostile text can hide inside a chunk. If the guard has not yet run, the reranker reads that hostile text. Then it makes a decision. That decision influences what the synthesizer receives. The attack can spread. So the guard must always run first. It fences the text. It strips out prompts. Then the reranker sees only safe text. Then it orders the chunks. The synthesizer gets a clean input. This order is fixed in the serving path. It is not optional. The platform ensures this order is followed. No chunk goes to the reranker unfenced. This keeps the system secure. It also keeps the output reliable. The reranker is a model. Models can be exploited. So the fence is the first line of defense. Then the reranker does its job. Then the synthesizer builds the answer. This three step chain is designed carefully. The guard is not just a nice extra. It is essential. A reranker without a guard is a weak point. Attackers could feed it malicious text. That text could sway the reranker. Then the synthesizer would get wrong material. The answer could be harmful. So the guard is mandatory. It is not switched off. Every retrieved chunk passes through it. Then the reranker can safely reorder. The platform documentation states this clearly. The postprocessor order is deliberate. Guard first, then rerank. This is the rule. It protects the user. It protects the system. The reranker does not need to see hostile text. It only sees clean text. That is the safe way. The trade off is simple. The guard costs a little time. But it prevents a big risk. So the design is correct. The order is never reversed. That is how the platform stays secure. The reranker still works well. It still improves precision. But it works on safe text. That is the key point. The fence makes the reranker safe. Without the fence, the reranker is dangerous. The platform knows this. So it puts the guard first. Then it reranks. Then it synthesizes. That is the deliberate order. It is the right order. And it is not changed.

None of this is a hard dependency. That design is intentional. If the cross encoder weights cannot load, maybe because the system is offline or has a cold cache, the builder simply returns nothing. The caller then keeps the plain top results without any reranking. No error stops the retrieval. If the keyword retriever cannot be built, say over an empty or tiny corpus, the caller gets back the dense retriever unchanged. That means recall still works. It just uses the simpler method.

Every stage in the retrieval path can be switched off by an environment flag. The feature is off by default on the production host. That host serves only dense retrieval. It never pays for a model it cannot hold. The small memory budget of five hundred and twelve mebibytes is the reason. So the cross encoder belongs to the full development profile instead. That trade is documented, not a bug.

Grounding degrades in quality but never breaks outright. That is the rule the whole retrieval path follows. It is graceful degradation. The highest scoring results stay. The reranker is not a hard dependency. It is optional. The keyword retriever is also optional. If either cannot run, the system quietly falls back to what does work. No failure message. No crash.

The fusion step also degrades gracefully. It merges dense and sparse candidates. But if the keyword part cannot be built, the caller just gets the dense retriever. The design makes sure every stage can be removed. You can switch off any part by setting an environment flag. That means you control the trade between quality and resource use.

The production host chooses to keep it simple. Reranking is off. Dense retrieval alone is enough for that environment. The development environment has more room. It can load the cross encoder. But even there, if the weights fail to load, the system just skips reranking. The user never sees an error. They just get the plain top results.

That is the concrete detail. The trade off is between quality and reliability. You get better ordering when the cross encoder works. But you never depend on it. If it is missing, you still get a good enough answer. The same for the keyword retriever. It adds recall for exact token hits. But if it cannot be built, the dense retriever still gives good results.

Every stage has an off switch. That makes the entire retrieval path resilient. It never fails outright. It only degrades in quality. That is the design principle. It is measured and deliberate. The cross encoder is deterministic. It can be scored precisely. But it is never required. The system always finds a way to return something useful.

Reranker changes are gated on pinned retrieval metrics. Those metrics are mean reciprocal rank and hit rate. They are computed over fixed sets of questions, each paired with the source that should answer it. That makes every change measurable, not just felt.

"The new reranker feels better" is never the argument. Feelings are not a score. A reranker that feels better today might feel worse tomorrow. Metrics do not change with mood. They give a stable, repeatable number. That number tells you if the reranker actually finds the right chunk more often.

The cross encoder can be scored this way because it is deterministic. It gives the same output every time you run the same query on the same chunk. It also needs no credentials. You can hand it an injected scorer and it works. That makes it easy to test and compare.

A language model based reranker cannot be scored the same way. It sits at the engine level. It costs a call per query. And it gives a different answer on a rerun. Non determinism means the metric can change between runs. You cannot pin a score to a single run. So you cannot gate changes on a fixed metric the same way.

The cross encoder runs in process. It uses the same small model that the embeddings already rely on. No new package, no graphics card, no separate service. Cross encoding twenty candidates costs CPU milliseconds. That is negligible next to language model synthesis. And results are cached for repeat queries.

The whole retrieval stack ships in one lightweight processor only image. That is not an accident. It is the constraint the design was solved under. Reranking is switched off in production because the model exceeds the memory budget. It belongs to the full development and pipeline profile.

In the serving path the postprocessor order is deliberate. Guard first, then rerank. Retrieved text is fenced against prompt injection before anything downstream consumes it. Only then is it reordered. A reranker that reads unfenced hostile text is part of the attack surface too. It is a model reading attacker controlled input, and it decides what the synthesizer sees.

The quiet extras around the reranker solve distinct failures. Raw cross encoder logits are squashed through a logistic function onto a zero to one scale. That gives every downstream consumer a stable, comparable number. Near duplicates are dropped, highest scoring copy winning. Query expansion widens recall on the question side with no model call at all. Compound identifiers are split into spaced forms. Domain acronyms are expanded to their phrases and back again.

None of this is a hard dependency. If the cross encoder weights cannot load, the builder returns nothing. The caller simply keeps the plain top results. If the keyword retriever cannot be built, the caller gets the dense retriever unchanged. Every stage can be switched off by an environment flag. Grounding degrades in quality, never fails outright. That is the rule the whole retrieval path is written to.