The Stack

One substrate, many lanes

Everything retrievable on this platform — lessons, case-study source code, ground sources, research papers — lives in one Qdrant store. Locally that store is embedded (a directory at data/qdrant, no server process); in production the same client code targets Qdrant Cloud via QDRANT_URL — a free-tier cluster holding ai_engineer_roadmap (22,210 lesson + code nodes), which the serving host mounts strictly load-only (QDRANT_LOAD_ONLY=1: a controlled build job owns writes; the service can never wipe or rebuild it). That cloud collection is the only copy of the prod retrieval index, so a daily cron snapshots it to R2 — the ops story is in the RAG serving deep dive. Locally, the main collection kg-corpus holds ~8,000 nodes; namespace collections (kg-ns-*) carry lane-specific corpora such as research papers and learning-science sources.

The RAG service mounts the store read-only and serves per-namespace views of the shared corpus: the glossary, the tutor, select-to-explain, and the research lanes all query the same substrate through their own filter. There is no second index to keep in sync — even the BM25 leg hydrates its corpus back out of Qdrant at boot via store_nodes().

Hybrid retrieval with RRF

In the full (dev/pipeline) profile, queries run two legs — production serving runs the dense leg only, because hydrating 22k nodes for BM25 does not fit its 512Mi instance:

• dense — FastEmbed bge-small vectors, cosine similarity — catches paraphrase and conceptual matches;
• sparse (BM25) — exact identifiers, API names, error strings — the things a technical corpus is full of and embeddings blur.

The legs are merged by Reciprocal Rank Fusion (QueryFusionRetriever in reciprocal_rerank mode). RRF fuses ranks, not scores — BM25 and cosine scores aren't calibrated against each other, so score interpolation needs per-corpus weight tuning while rank fusion needs none. Fusion deliberately retrieves wide (top-20 per leg) to maximize recall; a cross-encoder reranker then narrows for precision.

The single-process rule

Embedded Qdrant takes a file lock: one process at a time. That constraint shaped the workflow instead of fighting it — batch ingestion lanes run while the service is down, then the service boots and mounts the result read-only. If the corpus collection is missing, boot fails fast with the fix in the error message ("run: make corpus-ingest") rather than silently rebuilding an index at startup.

Why Qdrant

One store had to serve two very different lives: an embedded directory in dev (no server process, nothing to install) and a managed cluster in production — with the same client code targeting both. Qdrant is one of the few vector stores where that's a first-class mode switch, and it carries dense and sparse (BM25) legs plus payload-filtered namespace views in the same collection, so hybrid retrieval never means a second system to operate. The embedding and reranking legs stay CPU-only; the store had to match that lightweight profile.

A cache is also a collection

Near-duplicate "explain this" queries on the same page hit a semantic cache — its own Qdrant collection where the query embedding is the key. A hit above the cosine threshold reuses the stored answer; a payload index scopes lookups to the page. The vector store turns out to be a perfectly good cache backend: same infra, same client, one more collection.

Numbers

• ~8,000 nodes in the main local corpus, 8 collections total; 22,210 lesson
  • code nodes in the production cloud collection.
• Content-keyed embedding cache cuts a full corpus re-sync from 392s to 4s.
• Repeat explanations answer from cache in ~0.05s vs ~1.9s cold.
• Fusion retrieves top-20 per leg (dense + BM25) before the reranker narrows.

In-process, ONNX, CPU

Every index build on this platform embeds with FastEmbed running bge-small on ONNX Runtime, inside the pipeline process. There is no embedding microservice, no torch dependency, no GPU. For a lesson-sized corpus this beats the alternatives on every axis that matters here: no HTTP overhead, no batching infrastructure, no cold starts, and the whole retrieval stack ships in one lightweight image.

The one exception is the public demo box, and it is a concession, not the design: the hosted service runs on a 512Mi free instance where the ONNX runtime alone pushes it to ~616Mi. There, EMBED_REMOTE=1 embeds queries through the Cloudflare AI Gateway's Workers AI route (@cf/baai/bge-small-en-v1.5 — the same model, served remotely) to fit. Run the stack the way it is meant to run — locally — and embeddings are in-process again, with the LLM called directly and no Cloudflare model anywhere in the path. Builds always embed locally, and the two runtimes are never mixed within one collection. The memory numbers are in the RAG serving deep dive.

The model instance is process-cached — one load per lane, shared between the retriever build and everything else. Lexical-only paths (BM25 without vectors) deliberately skip building the vector index at all, avoiding even the model load they don't need.

Why bge-small

384 dimensions, fast on CPU, strong English retrieval quality — the sweet spot for a technical corpus of this size. The known trade-off of a small bi-encoder (precision at the top of the ranking) is recovered downstream by a cross-encoder reranker that runs on the same FastEmbed dependency: retrieval stays recall-oriented, ordering gets fixed where joint attention over the query–passage pair is affordable.

Content-keyed caching

Embeddings are cached through LlamaIndex's IngestionPipeline + IngestionCache in the RAG pipeline, keyed on content hash. Unchanged nodes are never re-embedded — a full corpus re-sync dropped from 392 seconds to 4 once the cache landed. Editing one lesson invalidates exactly that lesson's nodes.

Model upgrades fit the same mechanism: swapping the embedding model changes the pipeline signature, which invalidates cleanly, and the cache re-fills incrementally instead of forcing a big-bang re-index.

Zero-egress mode

Because embeddings are local, the entire stack can run with no network at all: FastEmbed embeddings + embedded Qdrant retrieval + a local llama.cpp reasoner for generation. VECTOR_STORE=memory goes one step further for hermetic tests — same embedding path, throwaway store. Deterministic CI and a $0 dev loop fall out of the same design decision.

Numbers

• 384-dimension vectors — bge-small's sweet spot of CPU speed and English retrieval quality.
• Content-keyed cache: full corpus re-sync 392s → 4s.
• The ONNX runtime alone pushes the 512Mi prod instance to ~616Mi — the measured fact behind EMBED_REMOTE=1.
• One model, two runtimes: bge-small-en-v1.5 in-process and via Workers AI, never mixed within a collection.

The eval stack

Nothing generated ships unexamined. The lanes, cheapest first:

1. Deterministic gates — citation presence, structure, readability, length budgets. Free, instant, zero variance. They run before any LLM judge; a judge is the expensive last line, not the first.
2. LlamaIndex response evals — faithfulness, relevancy, correctness on RAG answers; retrieval metrics (MRR, hit rate) over pinned query→expected-source sets, so retriever changes are measured, not felt.
3. A DeepEval lane — runs in its own venv (make eval-deepeval) because deepeval's pydantic pin conflicts with the service image; eval dependencies never leak into production.
4. Eight GEval-style metrics on three axes (audio experience, system design, content) judge every generated audio chapter. Chapters that fail don't ship.

Rule 1: a verdict without evidence is not a verdict

An audit of judge output found 54% of "supported" claims couldn't actually be cited back to a source — the judge said yes without receipts. The fix is structural, not prompt-level: any asserted verdict must persist the evidence it used. A "supported" with no stored citation is treated as unverified, full stop. Judges are pinned, too — scores are only comparable within a judge+prompt version, so the judge-stack identity is folded into the cache key.

Rule 2: automated repair is worse than no repair

The grounding watchdog could auto-"refute" claims it thought were wrong. In practice it produced roughly two bogus refutations for every real one — an auto-repair loop that vandalizes good content faster than it fixes bad content. Refutations now require a human in the loop; the automated lane runs with repair off.

Testing the non-deterministic

The same philosophy handles agent and RAG testing generally: deterministic checks wherever a property can be deterministic (structure, citations, tool-call shapes, scripted-LLM integration tests), LLM-as-judge only for qualities that genuinely need judgment — with evidence persisted and refutation reviewed. Related pages: the serving layer ships a scripted LLM for zero-model integration tests; the agents page covers deterministic trajectory testing; the full evaluation architecture — dataset contracts, trajectory evaluators, the coverage gate — is the Evaluation & Feedback guide.

Numbers

• Eight GEval-style metrics on three axes judge every generated audio chapter; the judge bar is ≥0.80.
• The audit that forced Rule 1: 54% of "supported" verdicts couldn't be cited back to a source.
• The audit that forced Rule 2: ~2 bogus refutations per real one from automated repair.
• Eval dependencies live in their own venv — zero eval packages in the production image.

The pipeline is idempotent end to end

Ingestion runs through LlamaIndex's IngestionPipeline with an IngestionCache keyed on content hash: unchanged nodes are never re-embedded (re-sync: 392s → 4s). Nodes carry stable, content-derived IDs, so upserts into Qdrant replace rather than duplicate, and every downstream cache — embeddings, judge verdicts, explanations — keys cleanly across runs.

The chain is: content hash → stable ID → cached embedding → upsert. Edit one lesson and exactly that lesson's nodes invalidate; nothing else moves.

External documents (PDFs, papers) enter through LlamaParse, gated on an API key and optional by design — the pipeline degrades gracefully without it.

The flagship lane: select-to-explain

Highlight any text on a lesson page and the platform explains it from its own corpus: the selection goes to the RAG service, retrieval runs over the shared store (hybrid in the full profile; dense-only on the memory-capped prod host — see the RAG serving deep dive), and the answer comes back grounded in the retrieved excerpts, with citations.

Grounding is strict: the engine answers only from what it retrieved, and "the excerpts don't cover it" is a legitimate, expected answer. A confident answer with no support isn't style — it's a gate failure.

Two caches on the answer path

• a persistent explanation cache — repeat queries answer in ~0.05s instead of ~1.9s;
• a semantic cache in Qdrant — near-duplicate queries on the same page (cosine above threshold) reuse the stored answer.

Between them, most real traffic never touches an LLM.

Retrieval shape

512/64 chunking with a retriever/synthesizer split, so concepts embed clean; one shared corpus with per-namespace views at query time — glossary, tutor cards, and research lanes each see their slice of the same substrate. In the full profile, recall comes from wide hybrid fusion, precision from the cross-encoder reranker, and quality from response evals measuring faithfulness, relevancy, and correctness against pinned retrieval metrics.

Why LlamaIndex

The pipeline leans on primitives that would be expensive to rebuild honestly: IngestionPipeline/IngestionCache for content-keyed idempotence, LlamaParse for messy external documents, StorageContext persistence that snapshots to R2 through fsspec, and the retriever/postprocessor seams the fencing and reranking stages plug into. The framework is load-bearing at the seams, and replaceable inside them.

Numbers

• Content-keyed re-sync: 392s → 4s; unchanged nodes never re-embed.
• Repeat explanations: ~0.05s from cache vs ~1.9s cold — most real traffic never touches an LLM.
• Chunking: 512/64 with stable, content-derived node IDs.
• Fusion width in the full profile: top-20 per leg before reranking.

Durable by construction

Research workflows here (multi-step runs over the corpus — company benchmarks, cross-corpus briefs) are LlamaIndex Workflow subclasses, and their execution is durable: after every non-terminal step the workflow context is serialized (ctx.to_dict(JsonSerializer())) into a run store. A crash, deploy, or restart loses nothing — the run rebuilds from its last step and continues.

Steps are the natural checkpoint boundary: events in, events out, state in Context. Serialize at the seams and durability is a property of the design rather than a bolted-on retry layer.

Human-in-the-loop that costs nothing while it waits

When a workflow needs a person, it emits InputRequiredEvent and parks — the run sits in a waiting_human state consuming zero compute. A HumanResponseEvent resumes it exactly where it paused (ctx.wait_for_event). The interrupt happens before the sensitive action, not after: HITL as a guardrail on autonomy, not an afterthought. A demo workflow exercises the full pause → approve → resume cycle end to end in the runs console.

ReAct with real tools — and the lab

A ReActAgent runs over the platform's actual toolbox (the paper-triage tools, wrapped as LlamaIndex FunctionTools). Around it, the agents lab implements the major orchestration patterns from scratch — ReAct, LATS, ReWOO, Reflexion, LLM-Compiler, plan-and-execute, tree-of-thoughts, self-refine, debate, supervisor — each one runnable, not a diagram.

Testing agents deterministically

Live-LLM agents hide their trajectory, which makes them untestable. The knowledge-graph lane runs a scripted ReAct loop — same thought/action/observation structure, deterministic trajectory — so agent behavior is assertable in CI. Non-determinism is confined to the outermost layer and judged there (evals); everything inside is exact.

The same ideas appear in the LangGraph curriculum here — durable checkpointers, HITL interrupts, multi-agent supervision — grounded in a real 60+ subgraph production case study rather than toy examples; the Agents & Workflows guide walks that system end to end.

Numbers

• Context serialized after every non-terminal step — a crash or deploy loses nothing.
• A parked waiting_human run consumes zero compute until its HumanResponseEvent arrives.
• Ten orchestration patterns implemented runnable-from-scratch in the agents lab.
• The companion LangGraph case study spans a 60+ subgraph production system.

Why a graph next to a vector store

Some questions embeddings can't answer no matter how good the vectors are: what are the prerequisites of X, transitively? what does this lesson unlock? which concepts bridge these two topics? Those are traversals, not similarity lookups. This platform keeps both: Qdrant for "similar to this", Neo4j for "connected to this".

Ingestion: extraction is generation, so it's gated

Lessons and parsed docs flow through LlamaIndex's PropertyGraphIndex with LLM-driven entity/relation extraction into Neo4j AuraDB. Extraction prompts pin the allowed node and edge types — concepts, lessons, techniques, papers; prerequisite/covers/cites edges — so the model can't invent schema.

And because extraction is LLM output like any other, it passes the same deterministic + judge gates before landing. In a learning-path graph a hallucinated edge is worse than a missing one: it sends a learner down a wrong path confidently.

Query: two retrievers, fused

• TextToCypher — natural language to Cypher for structural questions ("what depends on X?"). Powerful, and treated with respect: generated Cypher is still generated code, so it runs schema-constrained and read-only.
• VectorContext — embedding similarity over graph nodes for fuzzy, conceptual questions that don't map to a clean traversal.

Fusing the two at the answer level is the pragmatic version of GraphRAG: vector retrieval for context, graph retrieval for structure, without a monolithic pipeline in between.

Why Neo4j

The graph store had to satisfy the same discipline as the rest of the stack: a first-class LlamaIndex PropertyGraphIndex backend (so ingestion and retrieval reuse the pipeline's primitives instead of a bespoke loader), a declarative query language a model can be constrained in — generated Cypher runs schema-pinned and read-only, the same posture as the SQL guards — and a managed deployment (AuraDB) so the graph adds zero operational surface next to Qdrant.

Where it surfaces

The graph powers the platform's interactive graph view — concept nodes, prerequisite edges, lesson links — and feeds recommendation lanes that need to know what a learner is ready for, not just what's textually similar to what they last read. The full extraction-to-query walkthrough is the Knowledge Graph guide.

Numbers

• Extraction schema pinned to four node types (concepts, lessons, techniques, papers) and three edge families (prerequisite / covers / cites) — the model can't invent schema.
• Two retrievers fused at the answer level: TextToCypher for structure, VectorContext for similarity.
• Generated Cypher runs with zero write access — read-only, schema-constrained, like any other generated code here.

Memory as another retrieval leg

The chat tutor attaches a Mem0-backed memory per user (via llama_index.memory.mem0): conversations are distilled into persistent facts, and each turn retrieves only the relevant ones — capped by a search limit — instead of replaying transcripts into the prompt. Architecturally it's just another retrieval leg next to the corpus: user-scoped facts, fetched per turn, subject to the same grounding discipline.

Why Mem0

What Mem0 adds over a message buffer is the part that's genuinely hard to roll yourself: consolidation — dedupe, update-vs-add decisions, forgetting. The store is the easy 20%; the memory lifecycle is the product. And because it ships as a LlamaIndex memory integration, it drops into the existing chat engine as a component, not a parallel system — the same reason the rest of the pipeline leans on framework seams. (For what short-term/working memory looks like on the other side of that boundary, see the short-term memory deep dive.)

Local-first: the engine runs in-process

Memory is a mode, not a cloud dependency. MEM0_MODE=local (the localhost default) runs the open-source mem0 engine inside the RAG service itself: fact extraction and consolidation on the same DeepSeek credentials the service already holds, embeddings from the in-process FastEmbed model it already loaded, storage in a service-owned collection inside the shared embedded Qdrant, and a SQLite history log — no Mem0 account, no memory API key, no telemetry egress. cloud keeps the hosted Mem0 Platform for the deployed worker; the wire contract is identical either way. Beyond the chat lane, the same engine seasons the memorize tutor, the grounded /query path, and raw completions: remembered learner facts join the prompt as tone and difficulty guidance — explicitly fenced off from being treated as evidence, so the grounding gates keep their authority.

Never a hard dependency

Memory off, or the engine fails at init? The chat engine falls back to a memoryless default buffer — logged, not raised. Chat never breaks because memory is down, and a mid-turn consolidation failure degrades the same way (the answer still ships; only the remembering is skipped). That fallback also keeps the platform honest about coupling: memory is an enhancement, not a load-bearing wall.

Isolation is the design requirement

Memory is keyed by user identity from the constructor down — not filtered after the fact. Cross-user leakage is the one unrecoverable failure mode of a memory system, so isolation isn't a query parameter that could be forgotten; it's the shape of the API.

The case-study system pushes this further: long-term memory on Mem0 Cloud with per-contact isolation and source-trust tiers — a fact a contact stated in their own email carries different weight than one an agent inferred. Memory without provenance becomes confident hearsay.

What gets remembered

Learner preferences and progress context on the chat and tutor lanes (a practice card only enters memory once it passes the mnemonic gate); verified contact facts with provenance on the case-study lane. The rule of thumb: nothing enters memory that the user didn't say or that wasn't verified — memory is curated state, not a transcript dump.

The invariants

• Per-turn retrieval is capped by a search limit — relevant facts, never replayed transcripts.
• Memory failure degrades, never breaks: no key or an outage at init falls back to a memoryless buffer, logged not raised.
• Isolation is keyed by user identity from the constructor down — not a filter that could be forgotten.
• Every remembered fact carries provenance; inferred and stated facts are never the same trust tier.

One gateway, cross-cutting concerns at the edge

All hosted-LLM traffic passes through a Cloudflare AI Gateway endpoint. Logging, response caching, rate limits, and per-lane cost visibility live there — enforced at the edge once — instead of being sprinkled through application code. The RAG service is gateway-only by design: one base URL and one token in settings; swapping providers is configuration.

Since the 2026-07 serving rebuild the same gateway carries two model families on one credential: DeepSeek chat for synthesis, and Workers AI (workers-ai/@cf/baai/bge-small-en-v1.5 via the compat /embeddings route) for production query embeddings (EMBED_REMOTE=1 — the free-tier memory profile in the RAG serving deep dive). One gotcha the compat endpoint enforces: model ids must be provider-prefixed (deepseek/deepseek-chat) or it answers 400.

Three tiers, one API shape

Batch pipelines route across a fallback chain:

1. direct DeepSeek API — bulk generation at commodity cost;
2. local llama.cpp — an OpenAI-compatible reasoner proxy on localhost;
3. Cloudflare AI Gateway — the managed path.

Every tier speaks the same OpenAI-compatible API, so the chain is a list of base URLs, not three integrations. Extraction and query paths use plain text completion; agent paths get native function calling. When a tier is down, the next one answers — the circuit-breaker pattern applied to model providers.

Zero-egress mode

Flip one flag (LLM_LOCAL=1) and generation points at the local proxy — combined with in-process FastEmbed embeddings and embedded Qdrant, the dev and pipeline stack runs with no network egress at all. Deterministic dev, no API spend, and nothing sensitive leaves the machine. (Production serving is the deliberate exception: its queries embed remotely and its vectors live in Qdrant Cloud — a memory trade, not a design reversal.)

Cost tiering in practice

The routing rule is economic: batch lanes prefer the local tier; bulk generation uses DeepSeek (an order of magnitude cheaper than frontier models); frontier-quality calls are reserved for the judge and eval passes where the quality delta actually pays for itself. The gateway's per-request accounting is what makes that rule enforceable rather than aspirational — usage is a dashboard, not an end-of-month surprise.

Numbers

• Three tiers, one API shape — the fallback chain is a list of base URLs, not three integrations.
• Two model families on one credential since the 2026-07 rebuild: DeepSeek chat + Workers AI embeddings.
• Bulk generation on DeepSeek runs ~an order of magnitude cheaper than frontier models.
• Compat-endpoint gotcha: an unprefixed model id answers 400 — always deepseek/deepseek-chat.

Two runtimes, one trace shape

LLM work happens in two places here — the Next.js/Workers app (TypeScript) and the pipeline lanes (Python). Both trace to Langfuse, and the Python uploader's REST payload is parity-pinned to the TypeScript client's shape: same trace structure from either runtime, so a dashboard doesn't care which side of the stack produced a span.

Offline-first: capture is not shipping

Pipelines don't take a hard dependency on an observability SaaS being up. Every lane writes trace events as JSONL locally, always — even with no Langfuse credentials configured. Tracing can never block or break a run. A separate uploader ships batches to Langfuse's ingestion API later, and it's idempotent: a per-directory ledger tracks what shipped, so re-running the uploader never duplicates events.

That decoupling buys durability (traces survive network failures), replay (reprocess old runs into a fresh project), and honest structured logging — append-only, schema'd, machine-readable — as a side effect.

What a trace contains

Per-call: model, tokens, latency, cost. Per-lane: spans for each stage — retrieve → rerank → synthesize → judge — so a quality regression is attributable to a stage, not to "the pipeline". Judge scores land next to the traces they evaluate, which turns "which prompt version regressed faithfulness?" into a query instead of an investigation — the closing of the loop the Evaluation & Feedback guide builds toward.

The wider net

The case-study system layers LangSmith over OpenTelemetry on a single trace across Next.js, Python, and Rust — with eval failures routed to annotation queues, so failures become labeled data rather than log lines. Cost telemetry feeds per-workflow daily budgets with a global kill switch: observability closing the loop into control, not just visibility.

The invariants

• Two runtimes, one parity-pinned trace shape — a dashboard never cares which side of the stack produced a span.
• Capture is JSONL-locally-always, even with no credentials configured — tracing can never block or break a run.
• The uploader is idempotent by ledger: re-running it duplicates zero events.
• Every lane run is spanned per stage (retrieve → rerank → synthesize → judge), so regressions attribute to a stage.

Why a cross-encoder at all

The embedding model is a bi-encoder: it scores query and passage independently, which is what makes vector search fast — and what makes its top-of-ranking imprecise, because the model never sees the query and the passage together. A cross-encoder attends over the pair jointly. That's exactly what final ordering needs, and it's only affordable on a small candidate set.

Hence the shape of every retrieval here: retrieve wide → rerank narrow. RRF fusion merges dense and BM25 candidates deliberately wide (top-20 per leg, recall-oriented); the cross-encoder re-scores that shortlist and fixes precision where it matters. Fusion composes with the reranker — one widens, the other orders. (Both belong to the full dev/pipeline profile: the production host serves dense-only with reranking off, because the cross-encoder's lazy ONNX load alone exceeds its 512Mi memory budget — the trade is documented in the RAG serving deep dive.)

The implementation constraint that became a feature

The reranker is a custom LlamaIndex BaseNodePostprocessor wrapping a small English cross-encoder that runs on ONNX via FastEmbed — the same dependency the embeddings already pull in. No torch, no GPU inference service, no new package. Cross-encoding 20 candidates is CPU milliseconds — negligible next to LLM synthesis, and cached anyway for repeat queries.

The whole retrieval stack — embeddings, vector store, fusion, reranking — therefore ships in one lightweight, CPU-only image. That's not an accident; it's the constraint the design was solved under.

Placement in the chain

In the serving path the postprocessor order is deliberate: guard → rerank — retrieved text is fenced against prompt injection before anything downstream consumes it, then reranked. A reranker that reads unfenced hostile text is part of the attack surface too.

Measured, not felt

Reranker changes are gated on pinned retrieval metrics — MRR and hit rate over fixed query→expected-source sets (the eval page) — so "the new reranker feels better" is never the argument. Alongside the model there's also deterministic, LLM-free query expansion: lexical variants widen recall with zero model calls.

Numbers

• Cross-encodes a top-20-per-leg fused shortlist in CPU milliseconds — negligible next to synthesis, and cached for repeats.
• Zero new dependencies: the ONNX cross-encoder rides the same FastEmbed package the embeddings already pull in.
• Off in production: its lazy ONNX load alone busts the 512Mi serving budget — the trade documented in the RAG serving deep dive.
• Gated on MRR + hit rate over pinned query→source sets, never on feel.

The boundary

The Next.js app on Cloudflare Workers owns the edge — auth, caching, static content. Everything that needs the Python AI stack lives in one FastAPI service: querying, the glossary, tutor cards, chat, memorization scoring, research workflows — about fifteen endpoints behind a typed JSON/SSE API.

Streaming: SSE, not WebSockets

Token-by-token delivery (tutor cards, research briefs) uses server-sent events. The traffic is strictly server→client, which is exactly what SSE gives you over plain HTTP — built-in reconnect semantics, no connection-state machinery, no upgrade dance. First tokens hit the UI while synthesis is still running; a WebSocket here would be paying for bidirectionality nobody uses.

Boot discipline

• Settings resolve once at startup; /health reports liveness plus the wired backends (vector store, embedding backend, caches).
• Heavy imports are deferred — importing the routes module never forces corpora or Qdrant to load.
• The shared store mounts read-only; ingestion happens in separate lanes.
• If the corpus is missing, boot fails fast with the fix in the message ("run: make corpus-ingest") instead of half-starting or silently rebuilding.

The production box is the design constraint

In production this service runs on a 512Mi free-tier Render instance, and that number dictates the serving profile: query embeddings go remote (EMBED_REMOTE=1 through the AI Gateway — boot drops from 496Mi to 242Mi), retrieval is dense-only, and the cross-encoder reranker stays off. A Cloudflare cron pings /health every 10 minutes so the free instance never cold-starts mid-request, deploys fire only through a private deploy hook, and a daily cron snapshots the Qdrant collection to R2 because the cloud cluster is the index's only copy. The complete profile, measured memory numbers, and the failure catalog live in the RAG serving deep dive.

Dependency lanes

The base image stays minimal because optional capabilities install into their own venvs on demand: the eval lane (whose pydantic pin conflicts with the service), the NeMo guardrails lane, and the R2 snapshot lane each carry their own requirements file. Dead weight never rides along to production.

Deterministic integration tests

A scripted LLM stands in for the real model in tests: the full route stack — including streaming — exercises end to end with zero model calls and exact expected outputs. Non-determinism is confined to production model calls and judged there; the service's own behavior is asserted exactly. A smoke script drives the real service the same way after deploys.

Numbers

• ~15 endpoints behind one typed JSON/SSE API.
• The production budget is 512Mi; remote query embedding drops boot memory from 496Mi to 242Mi.
• A cron pings /health every 10 minutes so the free instance never cold-starts mid-request.
• A daily cron snapshots the production Qdrant collection to R2 — it's the index's only copy.

The threat model RAG actually has

A RAG system's most under-appreciated input channel is its own corpus: anything retrieved — a parsed PDF, a scraped doc, pasted content — flows into the prompt. If a document says "ignore your instructions and…", retrieval will happily deliver it. So this platform treats retrieved text as attacker-controlled input, the way a web app treats query strings.

Fencing: first in the chain, never destructive

The core defense is injection fencing, a custom LlamaIndex node postprocessor that runs first — before the reranker, before synthesis: guard → rerank. Nothing downstream ever consumes unfenced retrieved text.

Detection is a set of compiled regexes, one per attack family (instruction override, role hijack, exfiltration patterns, …) — deterministic, in-process, microseconds, zero model calls. Flagged content is wrapped, not deleted: fenced blocks preserve the original substring and mark it as untrusted data, so the LLM can still cite a document that happens to contain hostile text without obeying it. A destructive filter would corrupt the corpus's honesty; a fence keeps the content and removes its authority.

On the way out, output screening applies the same family checks to what the model produced — the second half of the contract.

NeMo rails, in their own lane

NeMo Guardrails layers policy rails on the conversational paths. Like every optional capability here it lives in its own dependency lane (requirements-guard.txt), never in the base service image, and its verdicts reduce to a small status the routes can act on. The integration carries an injection seam for tests, so rail behavior is assertable in CI without a live model.

Design position

Deterministic defenses first — they're free, fast, and auditable; model-based policy second; and every layer testable offline. The same philosophy as the eval stack: don't ask an LLM to do a regex's job, and don't trust any layer you can't test deterministically. The general theory — guardrail taxonomies, layered defenses, HITL as a rail — is the Agent Guardrails deep dive.

The invariants

• Fencing runs first in the postprocessor chain — nothing downstream ever reads unfenced retrieved text.
• Detection is one compiled regex family per attack pattern — in-process, microseconds, zero model calls.
• Flagged content is wrapped, never deleted — the corpus keeps its honesty, hostile text loses its authority.
• The same family checks screen output on the way out — the second half of the contract.

Index snapshots: StorageContext over fsspec

LlamaIndex persistence has a quietly great property: StorageContext writes through fsspec, so "persist to disk" and "persist to a bucket" are the same code path with a different filesystem handle. This platform's snapshot lane hands it an s3fs filesystem pointed at R2: the index snapshots to object storage and restores from it — same API, no custom serialization.

Like every optional capability in the service, the lane is opt-in (R2_STORE=1, its own requirements file): s3fs drags in aiobotocore, which is dead weight the production runtime doesn't carry unless the lane is actually in use.

Why R2

The platform already runs on Cloudflare (Workers, D1, Queues) — but the deciding feature is zero egress fees. The audio library is served straight from a public R2 bucket: 40+ minute generated audio guides are exactly the kind of asset where egress pricing on other clouds quietly becomes the bill.

What lives there

• Index snapshots — the StorageContext lane above; a fresh environment restores retrieval state without re-ingesting.
• Audio artifacts — every chapter the TTS pipeline produces, published under a public domain after passing its quality gates.
• Backups — a scheduled cron lane snapshots platform data to the bucket.

The pattern

Compute is disposable here — Workers, a Render service, batch lanes that run and exit. R2 is the durable layer they all agree on: artifacts are written once, content-addressed where it matters, and any environment can be reconstructed from the bucket plus the repo. Object storage as the source of recovery, not just a file dump.

Numbers

• Zero egress fees — the deciding feature for serving 40+ minute audio guides from a public bucket.
• Index snapshot/restore is one code path — StorageContext over fsspec, disk and bucket interchangeable.
• A daily cron backs up the production Qdrant collection — the retrieval index's only copy — via the serving ops lane.
• The s3fs lane is opt-in (R2_STORE=1, own requirements file): aiobotocore never rides to production unused.