Back to the stack guide
Embeddings

FastEmbed bge-small

Index builds embed in-process on ONNX — no torch, no GPU — with a content-keyed cache that makes re-ingestion nearly free; the memory-constrained prod host embeds queries remotely with the same model.

In-process, ONNX, CPU

Every index build on this platform embeds with FastEmbed running bge-small on ONNX Runtime, inside the pipeline process. There is no embedding microservice, no torch dependency, no GPU. For a lesson-sized corpus this beats the alternatives on every axis that matters here: no HTTP overhead, no batching infrastructure, no cold starts, and the whole retrieval stack ships in one lightweight image.

The one exception is the public demo box, and it is a concession, not the design: the hosted service runs on a 512Mi free instance where the ONNX runtime alone pushes it to ~616Mi. There, EMBED_REMOTE=1 embeds queries through the Cloudflare AI Gateway's Workers AI route (@cf/baai/bge-small-en-v1.5 — the same model, served remotely) to fit. Run the stack the way it is meant to run — locally — and embeddings are in-process again, with the LLM called directly and no Cloudflare model anywhere in the path. Builds always embed locally, and the two runtimes are never mixed within one collection. The memory numbers are in the RAG serving deep dive.

The model instance is process-cached — one load per lane, shared between the retriever build and everything else. Lexical-only paths (BM25 without vectors) deliberately skip building the vector index at all, avoiding even the model load they don't need.

Why bge-small

384 dimensions, fast on CPU, strong English retrieval quality — the sweet spot for a technical corpus of this size. The known trade-off of a small bi-encoder (precision at the top of the ranking) is recovered downstream by a cross-encoder reranker that runs on the same FastEmbed dependency: retrieval stays recall-oriented, ordering gets fixed where joint attention over the query–passage pair is affordable.

Content-keyed caching

Embeddings are cached through LlamaIndex's IngestionPipeline + IngestionCache in the RAG pipeline, keyed on content hash. Unchanged nodes are never re-embedded — a full corpus re-sync dropped from 392 seconds to 4 once the cache landed. Editing one lesson invalidates exactly that lesson's nodes.

Model upgrades fit the same mechanism: swapping the embedding model changes the pipeline signature, which invalidates cleanly, and the cache re-fills incrementally instead of forcing a big-bang re-index.

Zero-egress mode

Because embeddings are local, the entire stack can run with no network at all: FastEmbed embeddings + embedded Qdrant retrieval + a local llama.cpp reasoner for generation. VECTOR_STORE=memory goes one step further for hermetic tests — same embedding path, throwaway store. Deterministic CI and a $0 dev loop fall out of the same design decision.

Numbers

  • 384-dimension vectors — bge-small's sweet spot of CPU speed and English retrieval quality.
  • Content-keyed cache: full corpus re-sync 392s → 4s.
  • The ONNX runtime alone pushes the 512Mi prod instance to ~616Mi — the measured fact behind EMBED_REMOTE=1.
  • One model, two runtimes: bge-small-en-v1.5 in-process and via Workers AI, never mixed within a collection.