Back to the stack guide
LLM gateway

Cloudflare AI Gateway

Every hosted-LLM call goes through one gateway — and a three-tier fallback chain (direct API → local llama.cpp → gateway) makes providers a config value, not a code path.

One gateway, cross-cutting concerns at the edge

All hosted-LLM traffic passes through a Cloudflare AI Gateway endpoint. Logging, response caching, rate limits, and per-lane cost visibility live there — enforced at the edge once — instead of being sprinkled through application code. The RAG service is gateway-only by design: one base URL and one token in settings; swapping providers is configuration.

Since the 2026-07 serving rebuild the same gateway carries two model families on one credential: DeepSeek chat for synthesis, and Workers AI (workers-ai/@cf/baai/bge-small-en-v1.5 via the compat /embeddings route) for production query embeddings (EMBED_REMOTE=1 — the free-tier memory profile in the RAG serving deep dive). One gotcha the compat endpoint enforces: model ids must be provider-prefixed (deepseek/deepseek-chat) or it answers 400.

Three tiers, one API shape

Batch pipelines route across a fallback chain:

  1. direct DeepSeek API — bulk generation at commodity cost;
  2. local llama.cpp — an OpenAI-compatible reasoner proxy on localhost;
  3. Cloudflare AI Gateway — the managed path.

Every tier speaks the same OpenAI-compatible API, so the chain is a list of base URLs, not three integrations. Extraction and query paths use plain text completion; agent paths get native function calling. When a tier is down, the next one answers — the circuit-breaker pattern applied to model providers.

Zero-egress mode

Flip one flag (LLM_LOCAL=1) and generation points at the local proxy — combined with in-process FastEmbed embeddings and embedded Qdrant, the dev and pipeline stack runs with no network egress at all. Deterministic dev, no API spend, and nothing sensitive leaves the machine. (Production serving is the deliberate exception: its queries embed remotely and its vectors live in Qdrant Cloud — a memory trade, not a design reversal.)

Cost tiering in practice

The routing rule is economic: batch lanes prefer the local tier; bulk generation uses DeepSeek (an order of magnitude cheaper than frontier models); frontier-quality calls are reserved for the judge and eval passes where the quality delta actually pays for itself. The gateway's per-request accounting is what makes that rule enforceable rather than aspirational — usage is a dashboard, not an end-of-month surprise.

Numbers

  • Three tiers, one API shape — the fallback chain is a list of base URLs, not three integrations.
  • Two model families on one credential since the 2026-07 rebuild: DeepSeek chat + Workers AI embeddings.
  • Bulk generation on DeepSeek runs ~an order of magnitude cheaper than frontier models.
  • Compat-endpoint gotcha: an unprefixed model id answers 400 — always deepseek/deepseek-chat.