Deep Dives

Pointer. The canonical copy of this reference now lives in the infra-spec lane at specs/infra/deep-dives/observability.md (root ai-apps repo), which owns changes to the observability code. This copy stays for the audio guide's teaching/render use (mission.md link + roadmap Phase 8.3). Keep them in sync if the implementation changes — the infra-lane copy is the source of truth.

Companion to the audio guide. Chapters 1–6 of data/audio/case-study.script.md teach observability as narrative. This doc is the matching technical reference: every claim points at a real file + line number, every header is named, every failure mode is in the catalog. Read this when you need to change the observability surface or debug why a trace is broken — not when you're listening to the guide for context.

Scope: the observability plane as implemented in the case-study monorepo, end to end: outbound instrumentation in the Next.js product app, header propagation over the wire, the Pyodide-Worker-side tracer, LangSmith ingest, the cost report, and the in-app admin dashboard.

Out of scope: anything taught by the constitution as a generic principle without an implementation here (e.g. RUM-style frontend tracing — not wired). Per-graph evaluator design is in chapters 7–11 of the audio guide.

1. Why observability is its own plane

Traditional HTTP health checks tell you the API returned 200. They say nothing about whether the model answered the user's question correctly. That gap — between a transport-layer success and a semantic-layer failure — is the problem the observability plane exists to close.

The three planes are non-negotiable in the constitution (specs/case-study/mission.md:17-27):

The observability plane has two non-negotiables of its own:

1. One user action = one debuggable thing across the TS ↔ Python hop. A request that enters the Next.js handler and fans out across worker pools shows up as a single tree, not as N orphaned spans.
2. Zero per-request latency overhead in dev / when disabled. All instrumentation is gated by env vars (OTEL_EXPORTER_OTLP_ENDPOINT, LANGSMITH_TRACING); unset → strict no-op, zero network, zero PII egress.

2. The wire protocol — exact headers across the boundary

Every cross-process trace hop carries up to four header families. None of them are invented for this codebase; they're standards LangGraph + OTel + LangSmith already understand.

Outbound (TS → Worker)

Inbound parsing (Worker)

The worker's src/_tracing.py:parse_inbound(headers) reads the same two LangSmith-family headers — langsmith-trace + baggage — to either continue the caller's run tree or self-root if absent. The dotted-order parse is the contract that makes nested LangSmith runs work.

Outbound (Worker → TS, on response)

Inbound parsing (TS)

After the fetch returns, langgraph-client.ts:244-249 reads those three headers and writes them onto the active OTel span:

const runId = res.headers.get("x-langsmith-run-id");
const runUrl = res.headers.get("x-langsmith-run-url");
const peerTrace = res.headers.get("x-trace-id");
if (runId) span.setAttribute("langsmith.run_id", runId);
if (runUrl) span.setAttribute("langsmith.run_url", runUrl);
if (peerTrace) span.setAttribute("peer.trace_id", peerTrace);

The pivot rule: any operator looking at a trace in the company OTel UI can click langsmith.run_url and land directly in LangSmith on the matching Python-side run. There is no manual correlation step.

3. TypeScript side — langgraph-client.ts walkthrough

3.1 Bootstrap (env-gated, zero-overhead off)

OpenTelemetry bootstrap is via @vercel/otel's registerOTel. It is only called when OTEL_EXPORTER_OTLP_ENDPOINT is set. With it unset, local development and unconfigured deploys pay no cost — the SDK is never loaded, no spans are created, no fetches are wrapped.

When enabled, registerOTel wires three things:

1. OTLP/HTTP trace exporter — sends spans to the configured collector.
2. W3C trace-context + baggage propagators — so any outbound fetch automatically gets traceparent/tracestate/baggage injected.
3. Global fetch auto-instrumentation — every fetch() call becomes a child span of whatever span is active at the call site. No per-call-site code changes. This catches Neon (@neondatabase/serverless) database queries, LangGraph /runs/wait calls, DeepSeek LLM calls — all of them.

Sampling is env-driven via OTEL_TRACES_SAMPLER (defaults to always_on while volume is low; flip to parentbased_traceidratio and set OTEL_TRACES_SAMPLER_ARG when scaling).

3.2 The boundary call — runGraphWithMeta

langgraph-client.ts:194-271 is the single seam where every outbound graph call lives. Annotated walkthrough:

return tracer.startActiveSpan(`langgraph.${assistantId}`, async (span) => {
  span.setAttribute("langgraph.assistant_id", assistantId);
  try {
    const route = routeFor(assistantId);                      // ← data plane
    const headers: Record<string, string> = {
      "Content-Type": "application/json",
    };
    if (route.token) headers.Authorization = `Bearer ${route.token}`;
    propagation.inject(context.active(), headers);            // ← W3C
    injectLangSmithHeaders(headers);                          // ← LangSmith
    // … fetch /runs/wait, classify response, parse headers …
  } catch (err) {
    if (err instanceof Error) span.recordException(err);
    span.setStatus({ code: SpanStatusCode.ERROR });
    throw err;
  } finally {
    span.end();
  }
});

Three contracts this enforces:

• Span name = langgraph.<assistantId>. Stable across graph versions so cross-time queries don't fragment. Do NOT include version suffixes.
• langgraph.assistant_id attribute is set on every span. It's the join key for rolling cost / latency up by capability.
• Exceptions are recorded onto the span before re-throwing, so a thrown LangGraphError doesn't break the tree shape.

3.3 LangGraphError — typed failures with run-tree metadata

langgraph-client.ts:137-161:

export type LangGraphErrorKind = "auth" | "timeout" | "backend" | "client" | "unknown";

export class LangGraphError extends Error {
  readonly kind: LangGraphErrorKind;
  readonly status: number;
  readonly assistantId: string;
  readonly bodyText: string;
  // …
}

Status classification (classifyStatus, lines 180-186):

sanitizeBody (lines 163-178) caps response bodies at 500 chars and unwraps FastAPI {"detail": "…"} shapes so the error message is readable. Callers catch LangGraphError to decide retry vs. fallback vs. UI-visible message (grep: apps/agentic-sales/src/app/api/**/*.ts, src/apollo/resolvers/**).

3.4 injectLangSmithHeaders — best-effort distributed trace

langgraph-client.ts:23-33:

function injectLangSmithHeaders(headers: Record<string, string>): void {
  try {
    const rt = getCurrentRunTree(true); // permitAbsentRunTree → undefined, no throw
    if (!rt) return;
    const ls = rt.toHeaders();
    if (ls["langsmith-trace"]) headers["langsmith-trace"] = ls["langsmith-trace"];
    if (ls.baggage) headers.baggage = ls.baggage;
  } catch {
    // no run tree / SDK shape change — worker self-roots
  }
}

Why best-effort: the Next.js app does NOT wrap every request in a traceable()/wrapOpenAI() context — observability is OTel-first. The LangSmith run tree only exists in a request scope when the route handler explicitly opened one (e.g. inside an ainvoke chain). When absent, getCurrentRunTree(true) returns undefined and the worker becomes the root of its own LangSmith run. That's the common case, and it's fine — it just means the LangSmith UI shows the worker's tree as a standalone run not nested under a Next.js parent.

3.5 Async ingestion — startGraphRun

For long-running graphs (recall the audio chapter on "Why synchronous ingestion fails"), there's an async variant: POST /threads then POST /threads/<id>/runs with multitask_strategy: "enqueue". The response returns three handles — app run ID, LangGraph run ID, thread ID — that the caller uses to reconstruct the run tree later. Same OTel + LangSmith header injection applies.

4. Python worker side — _tracing.py walkthrough

4.1 Canonical source

services/_shared/tracing/_tracing.py is the single source of truth. Every worker's src/_tracing.py is a synced copy (setup_pyodide_deps.sh in each worker's scripts/ does the rsync). Do NOT edit the copies — edit the shared file and re-run npm run sync in each worker.

The synced copies live at:

• services/case-study-discovery-worker/src/_tracing.py
• services/case-study-classify-worker/src/_tracing.py
• services/research-worker/src/_tracing.py

(There is also services/case-study-intel-worker/src/_tracing.py, kept as a rollback handle after the 2026-05-28 merge — see specs/case-study/2026-05-28-merge-intel-into-discovery/.)

4.2 Why hand-rolled, not the langsmith SDK

The official langsmith Python SDK depends on orjson, zstandard, xxhash, uuid-utils, websockets — all native/compiled extensions with no Cloudflare Workers Python wheels. So this module rolls a parent/child run tree by hand and POSTs to the LangSmith ingest REST API over httpx.

The implementation is < 700 lines, talks two endpoints (/runs/batch, /feedback), and runs in Pyodide unchanged.

4.3 Gating — strict no-op unless enabled

_tracing.py:90-94:

def _enabled(env) -> bool:
    return (
        (_env(env, "LANGSMITH_TRACING", "") or "").strip().lower() == "true"
        and bool((_env(env, "LANGSMITH_API_KEY", "") or "").strip())
    )

Both env vars must be set. With either unset:

• Zero network egress.
• Zero PII egress (no run inputs/outputs leave the worker).
• trace_run becomes a transparent context manager that yields a stub.

4.4 Auth — workspace tenancy matters

_tracing.py:97-107:

def _post_headers(cfg: dict) -> dict:
    h = {"x-api-key": cfg.get("api_key", ""), "Content-Type": "application/json"}
    tenant = (cfg.get("tenant", "") or "").strip()
    if tenant:
        h["X-Tenant-Id"] = tenant
    return h

Gotcha: org-scoped service keys (the ones with prefix lsv2_sk_) 403 on every ingest without X-Tenant-Id. Personal keys (lsv2_pt_) work without it. The LANGSMITH_WORKSPACE_ID env var (or its alias LANGSMITH_TENANT_ID) is the workspace UUID. In the email worker config this is 6095afb1-3101-4972-af8e-d457099fe9f4 — every worker that traces must have the same value or you get silent 403 storms.

4.5 parse_inbound — continue or self-root

Reads langsmith-trace + baggage from the worker's inbound request headers. Returns either:

{"trace_id": ..., "parent_run_id": ..., "parent_dotted_order": ...,
 "tags": [...], "metadata": {...}}

…or None when no LangSmith headers are present (the worker self-roots). Called once per /runs/wait in entry.py:

inbound = parse_inbound(request.headers)
async with trace_run(env, assistant_id, graph_input, service=SERVICE,
                     inbound=inbound, ctx=self.ctx) as run:
    result = await graph.ainvoke(graph_input)
    run.outputs = result
return _json(..., headers=response_headers(run))

4.6 trace_run — the root context manager

_tracing.py:483-522:

@asynccontextmanager
async def trace_run(env, name, inputs, *, run_type="chain",
                    service="lead-gen", inbound=None,
                    post_pending=False, ctx=None):
    enabled = _enabled(env)
    cfg = _cfg(env, service=service, ctx=ctx)
    root = _Run(...)
    if post_pending:
        await _flush(cfg, post=[root._run_create(pending=True)])
        root.posted = True
    try:
        yield root
    except Exception as e:
        root.error = f"{type(e).__name__}: {e}"
    finally:
        if root.posted:
            await _flush(cfg, post=root._buf or None, patch=[root._run_patch()])
        else:
            root._finalize()
            await _flush(cfg, post=root._buf)

Three behavior modes:

1. Disabled (default in dev). No POSTs. The yielded run is a stub with no-op set_llm_usage, outputs, etc. The graph code never branches on whether tracing is on.
2. Enabled, fast path. Build the run tree in memory, batch all children
   • root into a single POST /runs/batch at exit. One HTTP call per request.
3. Enabled, durable path (post_pending=True). Used for jobs that might be evicted by Worker CPU limit / lose connectivity mid-run. POSTs the root as status=pending immediately, then PATCHes it on close. Even if the worker dies mid-graph the run shows up in LangSmith as a pending run that never closed — enough to debug.

4.7 submit_feedback — production quality signals

_tracing.py exposes a feedback POST against the root run:

await submit_feedback(env, root.id, "gate_passed", score=1.0)
await submit_feedback(env, root.id, "no_ai_markers", score=0.0,
                      comment="markers found in body")

The key naming convention is load-bearing. Production feedback keys must match the offline evaluator keys in apps/agentic-sales/backend/scripts/langsmith_run_experiments.py. Same namespace = the LangSmith pass-rate view aggregates both signals.

5. The three load-bearing span attributes

Chapter 5 of the audio guide ("What goes on a span") names three classes. Every other attribute is decoration that inflates cardinality.

5.1 Request identity

5.2 Call shape (Python worker side, set_llm_usage)

On any run_type="llm" span the worker records:

• model_name (set via set_llm_usage(model=...)) — e.g. deepseek-chat, @cf/baai/bge-m3. Surfaces as extra.metadata.ls_model_name so LangSmith's price table can auto-compute $.
• provider (set via set_llm_usage(provider="cloudflare"|"deepseek")) — extra.metadata.ls_provider.
• Prompt version — recorded by the graph code (not _tracing.py) as outputs.metadata.prompt_version. You cannot drop this field. Without it you have a recorded LLM call you can never reproduce because the template might have been edited since.

5.3 Cost (token counts)

run.set_llm_usage(input_tokens=N, output_tokens=N) emits outputs.usage_metadata in the shape LangSmith's price table consumes. The worker does NOT record dollar amounts — those would rot as pricing changes. LangSmith multiplies tokens × model rate at query time.

5.4 What you must NOT record

• Raw model output on the span. PII leak risk — a generated email might contain a phone number, address, or SSN. Outputs go in run.outputs (which gets ingested by LangSmith with full PII controls), never on searchable span attributes.
• Prompt text as an attribute. Use the prompt version instead.
• Per-prompt-variant attribute keys. Keep the attribute name stable, put the variation into the value. Otherwise cardinality blows up your observability bill.

6. LangSmith configuration — the env-var matrix

Every worker reads the same envelope. Set in wrangler.jsonc vars (for non-secrets) and wrangler secret put (for the API key).

Two distinct trace stores by design:

• LangSmith — owns the run tree of the Python-side graph (one tree per /runs/wait call). Holds inputs, outputs, token counts, feedback. This is the "what did the model decide" store.
• OTel collector (when wired) — owns the cross-process distributed trace. Holds the Next.js HTTP span → fetch span → Python span (continued via traceparent) → any other downstream calls. This is the "what happened across the system" store.

They cross-link via x-langsmith-run-id (set by worker → read by TS span) and traceparent (set by TS → continued by Python OTel runtime, if configured on the worker side).

7. The cost report — langsmith_cost_report.py

apps/agentic-sales/backend/scripts/langsmith_cost_report.py is the batch-mode summary that turns LangSmith runs into dollars + quality percentages.

7.1 Projects it consumes

Lines 26-31:

DEFAULT_PROJECTS = [
    "lead-gen",            # discovery-worker (absorbed intel 2026-05-28) + container
    "lead-gen-gateway",    # ai-gateway choke point
    "research-pipeline",
]

Add a new project here when you spin up a new worker. Otherwise its spend rolls up into nothing.

7.2 Fields read per run

• prompt_tokens, completion_tokens, total_tokens
• total_cost (LangSmith's auto-computed $ from model + tokens)
• error (truthy if the run failed)
• feedback_stats (the production quality signals submitted via submit_feedback)
• extra.metadata.ls_model_name (for per-model breakdown)

7.3 Cost attribution rule

Attributes cost only to run_type == "llm" leaf runs. Parent chain/tool runs are skipped to avoid double-counting (LangSmith's roll-up already does that on the server side, but the report re-implements it for batch CSV output).

7.4 Quality gates parsed

feedback_stats["gate_passed"]    # [n, avg]
feedback_stats["no_ai_markers"]  # [n, avg]

Pass-rate = sum(avg * n) / sum(n) across runs. Surfaced in the admin dashboard as colored badges per project.

8. In-app observability surface — admin/langsmith/page.tsx

Lives at apps/agentic-sales/src/app/admin/langsmith/page.tsx. Renders data from a langsmithAnalytics GraphQL resolver that runs the cost-report logic on demand.

What's there:

• Cost / token rollup per project.
• Per-model breakdown table (model name, run count, tokens, $).
• Top-failures table — each row links out to the LangSmith trace URL via the persisted langsmith_run_id (see §9).
• Gate pass-rate + no-AI-marker pass-rate badges per project.

What's NOT there:

• Embedded trace tree visualization. Trace trees are consumed exclusively in the LangSmith UI. The in-app surface is metrics + deep-links.

Why no embedded trace UI: LangSmith's UI is the production-grade tool for run-tree exploration. Embedding our own would duplicate years of work and immediately diverge. Deep-linking via x-langsmith-run-url is the cheap-and-correct path. (specs/case-study/roadmap.md Phase 8.3 proposes a deep-dive page but not an embedded trace viewer.)

9. Persisting LangSmith run IDs to D1 — closing the feedback loop

The email-compose path is the canonical example. Migration:

-- apps/agentic-sales/migrations/0009_email_langsmith_run_id.sql
ALTER TABLE contact_emails ADD COLUMN langsmith_run_id TEXT;

When an email is composed, the langsmith_run_id from the x-langsmith-run-id response header (set by the worker via response_headers(run)) gets written to D1 alongside the email. When the recipient later replies, the inbound-reply handler:

1. Looks up the original contact_emails row.
2. Reads its langsmith_run_id.
3. Calls submit_feedback(env, run_id, "reply_received", score=1.0).

That's the production feedback loop chapter 12 of the audio guide teaches — the recipient's reply becomes a reward signal on the original LLM call, months after the fact, with no manual joining.

Where else this pattern is used: any graph whose outcome can be measured later (after the synchronous run closes) should persist its langsmith_run_id on the same D1 row that holds the artifact. The pattern is shaped, not codified — add new graphs by mirroring the email migration.

10. Sampling — composing three strategies

Audio chapter 6 ("Sampling without going blind"). The strategies layer:

This repo only owns head sampling explicitly (via OTEL_TRACES_SAMPLER_ARG). Tail sampling and tag-based retention belong in the OTel collector or in LangSmith's project retention settings. Neither is wired in code; both are config decisions made out-of-band.

Per-assistant override: the WORKER_ROUTES table is the natural seam for differential sampling — high-stakes assistants (email_compose, gh_lead_research) stay on always_on; bulk assistants (country_classify_bulk) flip to a low ratio. Today this is enforced manually via env vars per assistant; there is no in-code switch.

11. Failure-mode catalog

Every failure mode mentioned in the audio guide chapters 1-6, mapped to where it actually hits in this codebase.

11.1 Propagation breakages

11.2 Tracer disabled when it shouldn't be

11.3 Cardinality / cost blowup

11.4 Reproducibility breakage

11.5 PII / safety

11.6 Async-ingestion gotchas

12. What's deferred (not wired)

The audio guide chapters 12–15 teach feedback loops, online evaluators, and PSI drift detection. The current implementation status:

• gate_passed + no_ai_markers feedback: LIVE on email_compose finalize node. Pattern is portable to other graphs by mirroring the submit_feedback calls; not yet replicated elsewhere.
• Online evaluators (chapter 14): NOT WIRED. The pattern is to run an evaluator graph in the data plane against recent production runs and submit scores as feedback. The wiring would go in a new worker (services/case-study-evaluator-worker/?) — not yet planned.
• PSI drift detection (chapter 15): NOT WIRED. The audio guide teaches it as a pattern (computeKS, computePSI); no implementation exists in this repo yet. Roadmap-side phase TBD.
• Embedded trace UI in the admin panel: NOT WIRED. See §8 — design decision is to deep-link to LangSmith, not embed.
• canary and ramp automation (chapter 19): NOT WIRED. The WORKER_ROUTES table supports manual per-assistant ramp by swapping *_WORKER_URL env vars; no automatic rollback on metric regression yet.

13. References

Audio guide chapters that teach the observability plane

Code anchors

• apps/agentic-sales/src/lib/langgraph-client.ts — TS-side client + spans + headers
• services/_shared/tracing/_tracing.py — canonical Python tracer
• services/case-study-{discovery,classify,email}-worker/src/_tracing.py — synced copies
• services/case-study-{discovery,classify,email}-worker/src/entry.py — trace_run usage at the worker entry
• services/case-study-{discovery,classify,email}-worker/wrangler.jsonc — env-var configuration
• apps/agentic-sales/backend/scripts/langsmith_cost_report.py — batch cost roll-up
• apps/agentic-sales/src/app/admin/langsmith/page.tsx — in-app dashboard
• apps/agentic-sales/migrations/0009_email_langsmith_run_id.sql — D1 persistence example

Constitution / spec anchors

• apps/ai-engineer-roadmap/specs/case-study/mission.md:17-27 — three-plane definition
• apps/ai-engineer-roadmap/specs/case-study/tech-stack.md — stack, commands, forbidden patterns
• apps/ai-engineer-roadmap/specs/case-study/roadmap.md Phase 8.3 — /case-study/deep-dive page idea (this doc is its companion content)

External authorities

• W3C Trace Context — https://www.w3.org/TR/trace-context/
• LangSmith ingest REST API — see _tracing.py:_flush() for the actual endpoints used (/runs/batch, /feedback).
• OpenTelemetry semantic conventions for span attributes — used as the shape for langgraph.*, langsmith.*, peer.* attributes.
• LangSmith dotted-order parent header format — see RunTree.to_headers() in the LangSmith JS SDK; matches what parse_inbound reads.

Last verified against code: 2026-05-28. If apps/agentic-sales/src/lib/langgraph-client.ts or services/_shared/tracing/_tracing.py has been edited since, line numbers in §3-4 may have drifted — re-grep for the symbol names rather than chasing the line.

Companion to the audio guide. The case-study guide is a 26-chapter, ~130-minute narration whose MP3s live on Cloudflare R2. This doc is the matching technical reference for how those bytes reach the player: the same-origin /api/audio proxy, HTTP Range/206 streaming, the service-worker contract, and the cross-route migration (backlog A41) that brought /langgraph, /langsmith and the per-lesson [slug] pages onto the same path. Read this when you need to change audio delivery or debug a stall — not when you're just listening.

Scope: the audio delivery plane end to end: the URL rewrite in lib/audio.ts, the proxy route at app/api/audio/[...path]/route.ts, the service worker in public/sw.js, and the cross-route parity work. Every claim points at a real file.

Out of scope: TTS generation and R2 upload (the Rust finalize_audio pipeline), and the audio content itself. This doc is about transport, not production.

1. Why audio delivery is its own problem

A 5-minute chapter MP3 is ~5 MB. A web <audio> element does not download it and then play — it streams, seeking to byte offsets on demand as the user scrubs. The mechanism the browser relies on for that is HTTP Range requests: the player sends Range: bytes=1048576- and expects a 206 Partial Content response carrying exactly that slice plus a Content-Range header. Get this wrong and the seek bar either does nothing or stalls the whole tab.

There are two non-negotiables for the delivery plane:

1. Progressive streaming, never full-download-then-play. The user must be able to start a chapter and scrub it without waiting for the whole file.
2. Same-origin. Cross-origin audio drags in CORS, an extra DNS/TLS handshake, and — critically — a service-worker code path that broke streaming (see §5).

Both are satisfied by routing every chapter MP3 through one same-origin proxy endpoint that faithfully relays Range semantics from R2 to the browser.

2. The three pieces

The contract that ties them together is the key path: knowledge/<slug>/NN.mp3. R2 stores objects under this prefix; the rewrite produces /api/audio/knowledge/<slug>/NN.mp3; the route allowlist accepts exactly that shape and nothing else.

3. The URL rewrite (lib/audio.ts)

R2 MP3s are public behind tts.vadim.blog. getAudioMeta does not hand those absolute URLs to the player — it rewrites them through toSameOriginAudioUrl:

export function toSameOriginAudioUrl(url: string | undefined): string | undefined {
  if (!url) return url;
  try {
    const { pathname } = new URL(url);
    if (/^\/knowledge\/[A-Za-z0-9/_-]+\.mp3$/.test(pathname)) {
      return `/api/audio${pathname}`;
    }
  } catch {
    // Relative URL or unparseable — leave it as-is.
  }
  return url;
}

Two properties matter:

• It is conservative. Only paths matching ^/knowledge/…\.mp3$ are rewritten; anything else passes through unchanged. A non-R2 URL, or a path that doesn't match the key shape, is never turned into a proxy request.
• It runs at metadata-build time, not render time. Both the stitched and single-file audio modes are rewritten as getAudioMeta assembles the AudioMeta object (lib/audio.ts), so by the time the player sees a URL it is already same-origin.

Note. The case-study page is a client component, so it cannot import from @/lib/audio (that module pulls in Node fs/path and would break the client bundle). It keeps a local copy of toSameOriginAudioUrl with a comment to keep it in sync. The duplication is deliberate — a runtime import of the server module is the bug it avoids.

4. The proxy route (app/api/audio/[...path]/route.ts)

The route runs as a Vercel Node function (runtime = "nodejs", dynamic = "force-dynamic"). It is a streaming reverse proxy for one narrow class of objects.

4.1 The allowlist (SSRF guard)

const KEY_RE = /^knowledge\/[A-Za-z0-9\/_-]+\.mp3$/;

Every request is checked against KEY_RE and rejected if it contains ... This is what stops the route from becoming an open proxy / SSRF vector — it can only ever fetch our own audio objects, never an arbitrary URL. A non-matching key returns 404 and logs [audio] reject key="…".

4.2 Range forwarding — the heart of seeking

The browser drives seeking with Range/If-Range; R2 answers with 206 and a Content-Range. The route's job is to relay both directions faithfully:

• Request headers forwarded upstream: range, if-range, if-none-match, if-modified-since. The first two drive seeking; the rest let R2 answer 304 on a full-object reload.
• Response headers copied back: content-type, content-length, content-range, accept-ranges, etag, last-modified, expires. The upstream status is preserved — 206 (partial) or 200 (full) passes through verbatim.

The status table the route handles explicitly:

4.3 Header normalization

Two fixups the route always applies, regardless of what R2 returned:

• accept-ranges: bytes is always set, even on a 200, so the player offers a seek bar.
• content-type is normalized: R2 sometimes serves application/octet-stream for objects uploaded without an explicit type, which breaks <audio> in some browsers; the route coerces that (and binary/text/plain) to audio/mpeg.
• vary: Range is set so any cache keys on the Range header — a cached full body is never replayed for a partial request, and vice-versa.

4.4 cache: "no-store" is mandatory

const upstream = await fetch(url, { method, headers, cache: "no-store", … });

Next's fetch cache keys on URL only. Without no-store, two different byte-ranges of the same object would collide on that key and the proxy would serve the wrong bytes for a seek. CF's edge still caches the immutable upstream object independently, so no-store costs nothing in practice — it only disables the per-request fetch dedup that would corrupt Range responses.

4.5 Streaming, retries, timeouts

• No buffering. GET returns new Response(upstream.body, …) — the body is piped straight through, so a multi-MB MP3 is never fully materialized in Worker memory.
• One retry on transient upstream failures (408/429/5xx or a network error), with a 250ms × attempt backoff. The first body is cancel()ed so the connection can be reused.
• 20s per-attempt timeout via AbortController. A timeout surfaces as 504; a connection failure as 502.
• HEAD and OPTIONS are supported: HEAD returns headers only; OPTIONS answers 204 with Allow: GET, HEAD, OPTIONS + Accept-Ranges for the occasional player/extension that probes before streaming.

4.6 Observability

One structured log line per request — key, range, upstream status, length, Content-Range, content-type, elapsed ms — visible via wrangler tail and the CF dashboard. A range request that R2 answered with a full 200 is tagged (range-ignored-by-upstream) so a degraded seek is diagnosable from the logs.

5. Why cross-origin stalled (public/sw.js)

Before the proxy, MP3s were served straight from tts.vadim.blog. The service worker matched cross-origin audio (url.hostname === AUDIO_HOST) and ran a CacheFirst-with-Range-reconstruction strategy: the Cache API cannot satisfy a Range request, so the worker had to fetch and cache the whole MP3, then slice the cached body into a synthetic 206. That eager full-download is exactly what stalled playback — the user pressed play and waited for the entire file before the first byte sounded.

The fix is the same-origin path. Because the proxy lives under /api/, the worker passes it through with a dedicated, deliberately non-buffering branch:

// Same-origin streaming audio (`/api/audio/…mp3`): online playback is a plain
// network passthrough — it does NOT eagerly download the whole file first
// (that was the stall on the cross-origin path).
if (url.pathname.startsWith("/api/audio/") && url.pathname.endsWith(".mp3")) {
  event.respondWith(handleSameOriginAudio(request, url));
  return;
}

Offline caching becomes opt-in: a full copy is only saved when explicitly requested for offline use, and online playback never pre-downloads. The legacy cross-origin AUDIO_HOST branch is retained only for any URL that still escapes the rewrite. Bumping CACHE_VERSION (now v4) invalidates every cache on the next activate, which is how an sw.js change ships to clients that already registered an older worker.

6. The cross-route migration (A41)

Originally only /case-study streamed same-origin; /langgraph, /langsmith and the per-lesson [slug] pages still pointed at tts.vadim.blog and inherited the stall. Because the rewrite lives in getAudioMeta rather than in any one page, routing those routes onto the proxy was a data-plane change, not a per-page edit: once getAudioMeta emits /api/audio URLs, every page that reads its metadata inherits the proxy for free. The proxy allowlist already accepts any knowledge/<slug>/NN.mp3 key, so no route-specific allowlisting was needed — langgraph, langsmith and lesson slugs all fit the one key shape.

Verification (Phase 7.4): on each route, DevTools Network must show audio requests going to /api/audio/knowledge/<slug>/NN.mp3 (not tts.vadim.blog), returning 206 on seek, with scrubbing that doesn't stall.

7. Failure-mode catalog

8. Invariants to preserve when changing this surface

1. The key shape is the contract. knowledge/<slug>/NN.mp3 is shared by the rewrite regex, the route allowlist, and R2's object layout. Change one, change all three.
2. cache: "no-store" on the upstream fetch stays. Removing it silently corrupts Range responses (§4.4).
3. Preserve the upstream status. Never synthesize a 206 from a 200 or vice-versa in the route — relay what R2 sent; the browser handles a 200 answer to a Range request per RFC 7233.
4. No eager full-download in the service worker. Online playback must stay a passthrough; offline caching stays opt-in (§5).
5. Bump CACHE_VERSION on any sw.js change, or clients keep the old worker.

Companion to the fleet. The case-study stack is ~50 LangGraph graphs that discover, classify, research, compose, send, and apply on their own. The single question "may an agent act without a human?" is answered in exactly one place. This doc is the technical reference for that place — read it when you need to change the posture, reason about blast radius, or explain to an auditor why a fully autonomous fleet is defensible.

Scope: the autonomy posture as implemented in apps/agentic-sales/backend/infra/autonomy.py — the one module every gate consults, the functions it exposes, the gate matrix it enforces, the runtime switch that was removed, and how the posture is stamped onto every LangSmith run so full autonomy stays auditable.

Out of scope: the individual graphs' business logic (what a discovery/classify/email agent does once it's cleared to act), and LLM_KILL_SWITCH (a separate lever that halts every LLM path — see §6). The observability plumbing that carries the autonomy tags is its own deep dive (observability.md).

1. Why one module owns the posture

A fleet this size has autonomy decisions scattered across every graph: an outreach queue could check its own env var, a CV-application node could read its own flag, a vertical-activation step could hardcode its own default. That is how posture drifts — one graph ships human-in-the-loop, another ships fully autonomous, and nobody can answer "is the fleet approving its own sends?" without auditing fifty files.

autonomy.py exists to make that question answerable in one read. It is the single chokepoint every interrupt / dry-run / plan-pending gate calls into. The call sites keep their gate seams — nothing is deleted — but the answer they get comes from here. That gives two properties:

1. Consistency across ~50 graphs. Change the posture once and the whole fleet moves together; there is no per-graph flag to forget.
2. A single audit target. Reviewing the fleet's safety posture means reviewing this ~110-line module, not the entire backend.

The history matters for anyone reading the call sites: the fleet was originally draft-first / human-in-the-loop by design. Every campaign touch, outreach queue, vertical activation, and CV application paused for an operator. Autonomy existed only as opt-in flags (auto_approve / auto_confirm / dry_run=False) that defaulted off. Per the owner decision dated 2026-06-30, that inverted: the fleet is now fully autonomous, hardcoded.

2. autonomy_mode() — what "full" means

def autonomy_mode() -> str:
    return "full"

def is_autonomous() -> bool:
    return autonomy_mode() == "full"

autonomy_mode() returns the string "full" unconditionally. is_autonomous() is the boolean the gates branch on.

"Full" means end-to-end, no human approval. An agent that discovers a lead, classifies it, researches it, composes an email, and sends it — or one that finds a role and submits a CV application — does the whole chain without an operator in the loop. There is no draft state waiting for a click, no outreach queue pending review, no plan-pending interrupt that blocks.

Two deliberate design choices live in these two functions:

• They are functions, not constants. Tests monkeypatch autonomy_mode() → "gated" to exercise the human-in-the-loop branches that still live at the call sites. Those branches are dead in production but reachable in tests, which is why the seams were kept rather than ripped out.
• The "full" / "gated" split is the only vocabulary. There is no third mode, no per-capability posture. The fleet is uniform.

3. The gate matrix — what's off, what's on

def gate_enabled(name: str) -> bool:
    if not is_autonomous():
        return True
    return name.strip().lower() == "suppression"

Every fail-closed gate asks gate_enabled("<name>"). Under full autonomy the answer is off for everything except suppression.

The if not is_autonomous() seam is retained on purpose: a test that monkeypatches autonomy_mode() to "gated" gets every gate back (return True for all names) — full HITL. In production is_autonomous() is always true, so the function collapses to the single suppression check.

Suppression is not a gate you can flip

Suppression has its own dedicated function so it can never be switched off by the same mechanism as the others:

def keep_suppression() -> bool:
    return True

Even under full autonomy, unsubscribes and do_not_contact are always honored. Sending to a suppressed or unsubscribed recipient can violate CAN-SPAM, so this is the one safety check that is hardcoded rather than gate-controlled, with no env var to disable it. It is legal-compliance plumbing, not an autonomy dial — which is why it lives outside the gate matrix entirely and is also reachable via gate_enabled("suppression") for call sites that consult the generic seam.

4. The removed runtime switch

Earlier the posture was a runtime decision. A cluster of env vars governed it:

• AGENT_AUTONOMY — the master switch (full / gated).
• AGENT_AUTONOMY_KEEP_SUPPRESSION — kept suppression on independently.
• AGENT_AUTONOMY_GATE_<NAME> — per-gate overrides.
• a per-run autonomy override — supplied on an individual invocation.

All of these were removed in the 2026-06-30 decision. There is no toggle back to human-in-the-loop in production, no per-gate env override, and no per-run override. The posture is not configuration anymore; it is code.

Why hardcode instead of leaving the switch defaulting to full? A live switch is a live risk: a stray env var in one worker's wrangler.jsonc, a copy-paste of an old config, or an operator "just testing gated mode" would silently re-introduce a HITL pause somewhere in the fleet and desync the posture the chokepoint exists to keep uniform. Removing the switch makes the posture a reviewable code change with a git history, not an ambient runtime state you have to go discover across ~50 deploys.

5. Making full autonomy auditable, not reckless

Full autonomy without a paper trail would be reckless. Two things make it defensible.

5.1 Blast-radius containment

The posture governs whether agents act without approval — it does not remove the guardrails that bound what an action can do. Suppression is permanent (§3). The kill switch still halts the model outright (§6). The fleet still routes through per-capability worker pools with their own auth. The autonomy decision is narrow: it removes the approval step, not the containment.

5.2 Every run is stamped

def autonomy_trace_fields(*, decision=None, mode=None) -> dict[str, Any]:
    resolved = (mode or autonomy_mode()).strip().lower()
    meta = {"autonomy_mode": resolved}
    if decision:
        meta["autonomy_decision"] = decision
    return {"tags": [f"autonomy:{resolved}"], "metadata": meta}

Every LangSmith run/span folds in this fragment, producing:

{"tags": ["autonomy:full"],
 "metadata": {"autonomy_mode": "full", "autonomy_decision": "auto_sent"}}

• tags: ["autonomy:full"] lets an operator slice all traces by posture in the LangSmith UI — "show me everything the fleet did fully autonomously" is one filter.
• metadata.autonomy_mode records the posture as it was at run time, so a historical trace is self-describing even after the posture changes.
• metadata.autonomy_decision records what the agent actually did at the gate, drawn from a small fixed enum:

Because every autonomous action carries autonomy:full plus a decision label, "the agent did X on its own" is always attributable to a specific run, with inputs and outputs, after the fact. Auditability is what converts full autonomy from reckless into accountable.

6. What autonomy is not — the kill switch

autonomy.py is independent of LLM_KILL_SWITCH. They answer different questions:

An operator who wants to stop the fleet does not reach for autonomy — that lever is hardcoded. They reach for the kill switch, which is the emergency stop. Autonomy is about approval, the kill switch is about execution. Conflating them is the most common misreading of this module.

7. References

Code anchors

• apps/agentic-sales/backend/infra/autonomy.py — the chokepoint (this doc's subject)
• infra.langsmith_setup.agent_run_span — consumes decision= + autonomy_trace_fields()
• Call sites: every interrupt / dry-run / plan-pending gate that calls is_autonomous() / gate_enabled(...) (grep the backend for those symbols)

Function reference

Constitution / spec anchors

• specs/case-study/mission.md — three-plane definition; autonomy is a control-plane concern
• specs/case-study/deep-dives/observability.md — the trace plumbing that carries autonomy:full

Last verified against code: 2026-06-30 (owner decision that hardcoded full autonomy). If apps/agentic-sales/backend/infra/autonomy.py has been edited since, re-read it — the whole module is the source of truth and it is short enough to read in full rather than trusting quoted excerpts here.

Companion to the LlamaIndex primer. /how-it-works/llamaindex teaches how the RAG engine works — chunks, embeddings, the vector index, grounding. This doc is the matching serving reference: how that engine actually runs in production, on what hardware, under what memory budget, with which retrieval features on and off, and what to do when it breaks. Every claim points at a real file; every number was measured on the live system (2026-07-14, the day this architecture was rebuilt from scratch after the hosting service and the vector collection were both lost).

Status (2026-07-14): the hosted demo is retired; localhost is the only serving profile. The stack uses exactly one model provider — DeepSeek — called directly, and one embedder — FastEmbed, in-process. No Cloudflare AI, no AI Gateway, no Workers AI. That rules the Render free tier out entirely: FastEmbed needs ~616Mi against its 512Mi cap (measured — the box pegs at 512Mi and never opens a port), the only fit was delegating embeddings to Workers AI, and DeepSeek publishes no embeddings endpoint (404) to fill the gap. The Render service is suspended, LLAMAINDEX_SERVICE_URL is unset in prod (so the AI features degrade instantly — 2.2s, not a 30s hang), and the keepwarm cron is gone. Everything below about the hosted profile is kept as the record of what was measured and why — read §4 as "the product vs the demo that could not be".

Scope: the serving plane end to end — the Next.js app on Cloudflare Workers, the Python RAG service on Render's free tier, the Qdrant Cloud vector store, the Cloudflare AI Gateway (LLM and embeddings), the keepwarm cron, the daily backup cron, and the recovery runbook.

Out of scope: how the index is built (the primer and /rag-pipeline own that), retrieval-quality technique design (/llamaindex/patterns), and observability (its own deep dive).

1. Why serving is its own plane

The engine and the serving substrate obey different constraints. The engine wants the best retrieval money can buy: hybrid dense+BM25 fusion, a cross-encoder reranker, a model-driven strategy router — and on localhost, the machine this project is actually built for, it gets all of them. The demo substrate is a free Render instance with a hard 512Mi memory ceiling — and every one of those engine features, left at its default, kills that process:

• In-process FastEmbed (ONNX runtime + bge-small weights + tokenizers) idles the service at 496Mi of 512Mi — measured via Render's memory metrics. Health checks pass; the first real query is an OOM kill.
• The cross-encoder reranker (LLAMAINDEX_CE_RERANK, default ON in rag_service/settings.py) lazy-loads a second ONNX model on the first scored query — a ~1.8GB transient spike, measured locally.
• The hybrid BM25 leg (rag_service/techniques/hybrid.py → indexing.py::store_nodes) hydrates every node in the collection (22,210 as of the rebuild) out of Qdrant into local RAM to build its keyword index.
• The LOOK strategy router (LOOK_ROUTER, default on) can select the hybrid strategy per-query, re-triggering that hydration even with LLAMAINDEX_HYBRID=0.

So production serving is a deliberately narrowed profile of the same codebase: dense-only retrieval over a remote vector store, with query embeddings served over HTTP instead of in-process. Dev keeps the full ladder. That split — not a smaller corpus, not a bigger box — is the architecture.

2. The request path, end to end

browser ── select text / ask
   │
   ▼
Next.js on Vercel
   │  lib/llm-service.ts — the single TS gateway to LLM completions.
   │  Every call carries AbortSignal.timeout(30s): a dead upstream must
   │  fail fast into the feature's degrade path (friendly 503), never
   │  hang the route.
   ▼
FastAPI RAG service — Render free tier, https://llamaindex-rag.onrender.com
   │  rag_service/main.py imports build the process singletons at startup;
   │  /health, /query, /chat, /complete in rag_service/routes.py.
   │
   ├──► Qdrant Cloud (eu-central-1) — collection `ai_engineer_roadmap`,
   │    22,210 nodes (lessons + AST-chunked case-study code), 384-dim
   │    cosine. QDRANT_LOAD_ONLY=1: the service only ever READS this
   │    collection; a controlled build job owns writes (§5).
   │
   └──► Cloudflare AI Gateway (one endpoint, two jobs):
        • synthesis — DeepSeek chat via the OpenAI-compat route; the model
          id MUST be provider-prefixed (`deepseek/deepseek-chat`) or the
          gateway answers 400 "Chat completion bad format"
        • query embeddings — `workers-ai/@cf/baai/bge-small-en-v1.5` via
          the same compat `/embeddings` route (EMBED_REMOTE=1;
          rag_service/factories.py::_GatewayEmbedding). Same credentials,
          zero extra secrets.

Two auxiliary Cloudflare crons complete the plane:

• Keepwarm — workers/edge-tasks/ pings the Render /health every 10 minutes (free instances idle out after ~15). The ping has a bounded 100s timeout (src/entry.py::PING_TIMEOUT_MS): generous enough for Render's ~50s cold wake, bounded because a dead Render service hangs rather than fast-fails (observed: 180s+ with zero bytes), which used to wedge every cron firing.
• Backup — daily at 04:00 UTC the main worker's scheduled handler (workers/app/index.ts) self-calls /api/cron/backup: a D1 .sql dump plus a server-side Qdrant snapshot, both pushed to the db-backups R2 bucket (§6).

3. The memory budget, with receipts

The 2026-07-14 rebuild replayed every failure mode in sequence; the numbers below are from Render's events/metrics APIs and local RSS sampling of the identical profile.

The five load-bearing env vars, each with its reason:

All five are encoded in services/llamaindex/render.yaml so a blueprint recreation reproduces the working service, not the crash-loop.

4. The product (localhost) vs the demo (Render free)

One codebase, two profiles. The flags compose per environment (rag_service/settings.py reads them all):

run_local.sh enforces the left column rather than trusting the ambient env: it exports EMBED_REMOTE=0 (an inherited EMBED_REMOTE=1 would otherwise quietly ship local queries to Cloudflare) and pins the LLM to api.deepseek.com with an unprefixed model id, reading DEEPSEEK_API_KEY from $DEEPSEEK_API_KEY, the monorepo-root .env, or the app's .env.local — whichever has it. LLM_LOCAL=1 goes further and swaps DeepSeek for a local llama.cpp proxy: embeddings, retrieval, and generation then run with zero network egress.

Why the demo can't just do what localhost does: in-process FastEmbed costs ~290Mi of ONNX runtime (baseline 19Mi → 308Mi after model load; threads=1 changes nothing), which puts the whole service at 616Mi measured — 104Mi over the free tier's ceiling, untunable. And DeepSeek publishes no embeddings endpoint (/v1/embeddings → 404), so "embed with DeepSeek" is not an option that exists. The demo either borrows Cloudflare's copy of the same bge-small model or it does not serve uncached queries at all.

The rule that keeps the two profiles compatible: an index is built and queried with vectors from compatible runtimes. FastEmbed and Workers AI serve the same bge-small model, but pooling differences put identical text at cosine ~0.96 across runtimes — measured 22/25 top-5 retrieval overlap on the live corpus, fine for query-time use. Never mix runtimes within one collection build.

5. State: the three caches and the one collection

• Vector collection (ai_engineer_roadmap, Qdrant Cloud) — the ONLY copy of the production retrieval index. Written exclusively by the build job: services/llamaindex/run_local.sh with QDRANT_URL + QDRANT_COLLECTION=ai_engineer_roadmap + CASE_STUDY_DIR exported boots the service locally, which builds and uploads the code-inclusive corpus (~22k nodes), then serves it for smoke-testing.
• Semantic cache (explain_semantic_cache, same cluster) — near-match answer reuse at 0.95 cosine. Because it lives in Qdrant, it survives Render redeploys and even service deletion (it did — it was the surviving artifact that identified the original cluster during recovery).
• Explanation cache (SQLite on the Render disk) — exact-match, ephemeral by design; rebuilt from cold on each deploy.
• Embedding cache (.embed-cache/embeddings.db, app root) — build-time infrastructure shared with the roadmap-kg grounding lanes. With EMBED_REMOTE=1 it plays no role at serve time.

6. Ops runbook

Deploys. Render autoDeploy is OFF. The only sanctioned trigger is the deploy hook (RENDER_DEPLOY_HOOK in the credentials vault) — pushes to main do not deploy the service. The Next app deploys with pnpm run deploy (scripts/deploy.sh: local build → prebuilt tgz to Vercel → verify).

Backups. /api/cron/backup (app/api/cron/backup/route.ts) runs daily at 04:00 UTC from the worker's scheduled handler and needs four worker secrets: CRON_SECRET (Bearer auth the handler sends itself), QDRANT_URL

• QDRANT_API_KEY (snapshot source), and CLOUDFLARE_D1 (D1 dump token). Keys land in R2 db-backups as knowledge/<date>/d1-<stamp>.sql and qdrant/<date>/<collection>-<snapshot>, 30-day retention. A missing secret degrades silently ("skipped" / 500 inside a cron nobody watches) — after any worker recreation, run bash scripts/sync-worker-creds.sh and check npx wrangler secret list. This bit us: the cron had never produced a single backup until 2026-07-14.

Restore / rebuild. Cluster lost or collection wiped:

1. Fastest: restore the newest qdrant/<date>/… snapshot from R2 (Qdrant accepts snapshot uploads on collection create).
2. Otherwise rebuild from source (~30 min): run the build job (§5) against a fresh cluster, then update QDRANT_URL/QDRANT_API_KEY on the Render service and in the worker secrets, and redeploy via the hook.
3. If Render itself is gone: recreate from services/llamaindex/render.yaml (dashboard → New → Blueprint, or POST /v1/services), set the dashboard secrets it declares sync: false, and confirm the env profile of §3.

Health. GET /health reports the wired backends — vectorStore: qdrant:<url>/<collection>, embedBackend: workers-ai-gateway | fastembed, semanticCache — so remote-vs-local embedding and disk-vs-cloud vectors are observable without dashboard access.

7. Failure catalog

8. What we deliberately did NOT do

• No paid tier. Standard (2GB) would restore hybrid+rerank in prod with zero code changes; the free-tier profile was chosen deliberately.
• No qdrant-client bump (1.14.3 vs server 1.18.2 warns but works) — pinned deps change on their own test pass, not during an outage.
• No mixed-runtime collection: the corpus is always built by one embedding runtime end-to-end (§4).