FastEmbed cross-encoder

Why a cross-encoder at all

The embedding model is a bi-encoder: it scores query and passage independently, which is what makes vector search fast — and what makes its top-of-ranking imprecise, because the model never sees the query and the passage together. A cross-encoder attends over the pair jointly. That's exactly what final ordering needs, and it's only affordable on a small candidate set.

Hence the shape of every retrieval here: retrieve wide → rerank narrow. RRF fusion merges dense and BM25 candidates deliberately wide (top-20 per leg, recall-oriented); the cross-encoder re-scores that shortlist and fixes precision where it matters. Fusion composes with the reranker — one widens, the other orders. (Both belong to the full dev/pipeline profile: the production host serves dense-only with reranking off, because the cross-encoder's lazy ONNX load alone exceeds its 512Mi memory budget — the trade is documented in the RAG serving deep dive.)

The implementation constraint that became a feature

The reranker is a custom LlamaIndex BaseNodePostprocessor wrapping a small English cross-encoder that runs on ONNX via FastEmbed — the same dependency the embeddings already pull in. No torch, no GPU inference service, no new package. Cross-encoding 20 candidates is CPU milliseconds — negligible next to LLM synthesis, and cached anyway for repeat queries.

The whole retrieval stack — embeddings, vector store, fusion, reranking — therefore ships in one lightweight, CPU-only image. That's not an accident; it's the constraint the design was solved under.

Placement in the chain

In the serving path the postprocessor order is deliberate: guard → rerank — retrieved text is fenced against prompt injection before anything downstream consumes it, then reranked. A reranker that reads unfenced hostile text is part of the attack surface too.

Measured, not felt

Reranker changes are gated on pinned retrieval metrics — MRR and hit rate over fixed query→expected-source sets (the eval page) — so "the new reranker feels better" is never the argument. Alongside the model there's also deterministic, LLM-free query expansion: lexical variants widen recall with zero model calls.

Numbers

• Cross-encodes a top-20-per-leg fused shortlist in CPU milliseconds — negligible next to synthesis, and cached for repeats.
• Zero new dependencies: the ONNX cross-encoder rides the same FastEmbed package the embeddings already pull in.
• Off in production: its lazy ONNX load alone busts the 512Mi serving budget — the trade documented in the RAG serving deep dive.
• Gated on MRR + hit rate over pinned query→source sets, never on feel.