FastAPI with SSE streaming

The boundary

The Next.js app on Cloudflare Workers owns the edge — auth, caching, static content. Everything that needs the Python AI stack lives in one FastAPI service: querying, the glossary, tutor cards, chat, memorization scoring, research workflows — about fifteen endpoints behind a typed JSON/SSE API.

Streaming: SSE, not WebSockets

Token-by-token delivery (tutor cards, research briefs) uses server-sent events. The traffic is strictly server→client, which is exactly what SSE gives you over plain HTTP — built-in reconnect semantics, no connection-state machinery, no upgrade dance. First tokens hit the UI while synthesis is still running; a WebSocket here would be paying for bidirectionality nobody uses.

Boot discipline

• Settings resolve once at startup; /health reports liveness plus the wired backends (vector store, embedding backend, caches).
• Heavy imports are deferred — importing the routes module never forces corpora or Qdrant to load.
• The shared store mounts read-only; ingestion happens in separate lanes.
• If the corpus is missing, boot fails fast with the fix in the message ("run: make corpus-ingest") instead of half-starting or silently rebuilding.

The production box is the design constraint

In production this service runs on a 512Mi free-tier Render instance, and that number dictates the serving profile: query embeddings go remote (EMBED_REMOTE=1 through the AI Gateway — boot drops from 496Mi to 242Mi), retrieval is dense-only, and the cross-encoder reranker stays off. A Cloudflare cron pings /health every 10 minutes so the free instance never cold-starts mid-request, deploys fire only through a private deploy hook, and a daily cron snapshots the Qdrant collection to R2 because the cloud cluster is the index's only copy. The complete profile, measured memory numbers, and the failure catalog live in the RAG serving deep dive.

Dependency lanes

The base image stays minimal because optional capabilities install into their own venvs on demand: the eval lane (whose pydantic pin conflicts with the service), the NeMo guardrails lane, and the R2 snapshot lane each carry their own requirements file. Dead weight never rides along to production.

Deterministic integration tests

A scripted LLM stands in for the real model in tests: the full route stack — including streaming — exercises end to end with zero model calls and exact expected outputs. Non-determinism is confined to production model calls and judged there; the service's own behavior is asserted exactly. A smoke script drives the real service the same way after deploys.

Numbers

• ~15 endpoints behind one typed JSON/SSE API.
• The production budget is 512Mi; remote query embedding drops boot memory from 496Mi to 242Mi.
• A cron pings /health every 10 minutes so the free instance never cold-starts mid-request.
• A daily cron snapshots the production Qdrant collection to R2 — it's the index's only copy.