Langfuse

Two runtimes, one trace shape

LLM work happens in two places here — the Next.js/Workers app (TypeScript) and the pipeline lanes (Python). Both trace to Langfuse, and the Python uploader's REST payload is parity-pinned to the TypeScript client's shape: same trace structure from either runtime, so a dashboard doesn't care which side of the stack produced a span.

Offline-first: capture is not shipping

Pipelines don't take a hard dependency on an observability SaaS being up. Every lane writes trace events as JSONL locally, always — even with no Langfuse credentials configured. Tracing can never block or break a run. A separate uploader ships batches to Langfuse's ingestion API later, and it's idempotent: a per-directory ledger tracks what shipped, so re-running the uploader never duplicates events.

That decoupling buys durability (traces survive network failures), replay (reprocess old runs into a fresh project), and honest structured logging — append-only, schema'd, machine-readable — as a side effect.

What a trace contains

Per-call: model, tokens, latency, cost. Per-lane: spans for each stage — retrieve → rerank → synthesize → judge — so a quality regression is attributable to a stage, not to "the pipeline". Judge scores land next to the traces they evaluate, which turns "which prompt version regressed faithfulness?" into a query instead of an investigation — the closing of the loop the Evaluation & Feedback guide builds toward.

The wider net

The case-study system layers LangSmith over OpenTelemetry on a single trace across Next.js, Python, and Rust — with eval failures routed to annotation queues, so failures become labeled data rather than log lines. Cost telemetry feeds per-workflow daily budgets with a global kill switch: observability closing the loop into control, not just visibility.

The invariants

• Two runtimes, one parity-pinned trace shape — a dashboard never cares which side of the stack produced a span.
• Capture is JSONL-locally-always, even with no credentials configured — tracing can never block or break a run.
• The uploader is idempotent by ledger: re-running it duplicates zero events.
• Every lane run is spanned per stage (retrieve → rerank → synthesize → judge), so regressions attribute to a stage.