In-process embeddings (FastEmbed)

Worked Example: In-Process FastEmbed Embedding with Process Cache

This is a fully solved example. Study it carefully to understand how the production code in kg/llm.py creates an in-process FastEmbed embedding model (bge-small, 384-dim) with a process-wide cache. The same pattern is used across all memory lanes to avoid re-loading ONNX weights.

Why this pattern?

• FastEmbedEmbedding loads the ONNX model into memory when instantiated. If several lanes (ingest, query, rerank) each call make_embed(), they'd load the same weights multiple times, wasting seconds and RAM.
• The solution caches the embedding object in a module-level dictionary _EMBED_CACHE keyed by model name. This ensures only one instance is created per model per process.
• Environment variables EMBED_MODEL, EMBED_THREADS, and EMBED_CACHE_DIR allow tuning without changing code. EMBED_MODEL defaults to "BAAI/bge-small-en-v1.5" (384 dims).
• If the cache already holds a model for the current model_name, it returns it directly – no new embedding instantiation.

Your learning task:

Read the complete solution below. The starter code provides the skeleton with imports and function signature. The solution code fills in the logic. Verify you understand:

• How _EMBED_CACHE is used to cache instances.
• How environment variables are read (os.environ.get with defaults).
• How FastEmbedEmbedding is constructed with optional threads and cache_dir parameters.
• Why this approach avoids repeated ONNX model loading.

After studying, you should be able to explain each step and the rationale.