Evaluation & Feedback — Transcript

Evaluating generation graphs with separate faithfulness and answer relevancy metrics, adaptable to retrieval or free-generation contexts.

def make_generation_metrics(
    judge: Any = None,
    *,
    with_faithfulness: bool = True,
    strict: bool = False,
) -> list[Any]:
    from deepeval.metrics import AnswerRelevancyMetric, FaithfulnessMetric

    j = judge if judge is not None else make_judge()
    faith_threshold = THRESHOLD_STRICT if strict else THRESHOLD_DEFAULT
    metrics: list[Any] = []
    if with_faithfulness:
        metrics.append(
            FaithfulnessMetric(threshold=faith_threshold, model=j, include_reason=True)
        )
    metrics.append(
        AnswerRelevancyMetric(threshold=THRESHOLD_DEFAULT, model=j, include_reason=True)
    )
    return metrics

Imagine you're a quality inspector at a busy kitchen, and the head chef has dozens of different recipes running at once. You can't just taste everything and give one overall "good enough" rating — a soup might be perfectly salty but still have a burned bottom, and a steak could be cooked right but missing the side dish. That's exactly the problem this subsystem solves for language‑model "recipes" (called graphs). It's a per‑graph checklist that catches where a specific mistake happens, so a single high score never hides a broken part.

For a generation graph (like a chatbot that writes an email), the inspector runs two independent checks. First, the FaithfulnessMetric verifies every factual claim in the output is backed by the source documents — no invented facts. Second, the AnswerRelevancyMetric confirms the output actually addresses what the user asked. If you only used one average score, a brilliant but hallucinated claim could still pass. For an extraction graph (which pulls key facts from a document), the inspector is even stricter: a FaithfulnessMetric at THRESHOLD_STRICT locks down that nothing extracted is made up, and a ContextualPrecisionMetric ensures the extracted pieces are the most relevant ones, not just any random correct fact.

The non‑obvious rule is that each graph type has its own required pair of metrics, and if a graph has no LLM calls at all (a pure data‑routing graph), it's exempt — you don't run a taste test on a conveyor belt. Without this subsystem, a single accuracy number would let a smooth‑talking model pass while quietly inventing false claims, and a beginner would feel the result: a cold email that references a product feature that never existed, or an extraction that lists a wrong contact name — trust broken instantly.

The evaluation subsystem operationalized in run_evals.py follows a precisely ordered pipeline. First, it discovers every *_graph.py module in the agentic_sales package. For each graph it runs static analysis—checking against compiled regex patterns that match LLM call sites such as llm.ainvoke or llm.bind—and classifies the graph into one of four buckets: covered, no-llm, delegates, or exempt. Graphs that call an LLM and are not in the explicit DELEGATES or exempt lists must be covered, meaning at least one test file imports the graph module. Next, for every covered graph, the system runs the full DeepEval suite built by make_generation_metrics, which by default includes a FaithfulnessMetric (threshold THRESHOLD_STRICT or THRESHOLD_DEFAULT) and an AnswerRelevancyMetric. If any graph that requires an eval is found in the missing bucket, or if the aggregate DeepEval accuracy falls below the ≥0.80 bar, the gate fails and the script returns exit code 1, printing the specific problem (e.g., “MISSING eval: <graph>” or the activation-pipeline failure messages from _check_activation_pipeline_coverage).

The core invariant this design preserves is the Eval-First coverage guarantee: every language-model graph that produces an output subject to human or automated judgment must have at least one associated evaluation test, and that test must clear the ≥0.80 accuracy threshold. This invariant is enforced mechanically, not through documentation or developer discipline. The _ACTIVATION_PIPELINE_GRAPHS and _UK_UNIVERSITIES_GATED_GRAPHS registries strengthen it by requiring that even graphs that delegate their LLM work (like vertical_activation_graph) are explicitly tracked—removing a graph or letting its eval coverage silently rot causes a loud failure in the gate.

The key trade-off is between automated static classification and a hand-maintained manifest. The obvious rejected alternative is a manually curated list of every graph and its eval status. That approach would be brittle: any addition, renaming, or removal of a graph would require a human to update the list, and a stale manifest would silently mask missing evals. The chosen design instead derives coverage from source-code structure—regex detection of LLM calls provides a baseline, and explicit DELEGATES and exempt dictionaries serve as escape hatches for exceptional cases. The cost avoided is the ongoing maintenance burden and the risk of silent regressions that a static list introduces. The downside is that the regex heuristic may misclassify a graph that builds prompts but never directly calls an LLM method, so the escape hatches are necessary—and each escape must carry a written reason to prevent abuse.

A concrete failure mode: a developer introduces pipeline_orchestrator_graph.py that internally calls llm.invoke to generate a final summary but forgets to write a test_pipeline_orchestrator_graph.py. When run_evals.py executes, it detects the LLM call via the regex pattern re.compile(r"\bllm\.(a?invoke|a?stream|bind\w*)\b"), classifies the graph as needing an eval, and finds no test imports it—so it lands in the missing bucket. The operator sees the following signal printed to stderr: “MISSING eval: pipeline_orchestrator_graph (calls an LLM or is unclassified — add tests/test_pipeline_orchestrator_graph.py or DELEGATES entry)”, followed by a non-zero exit code that breaks the CI pipeline. This forces the developer to either write the required DeepEval test (which must subsequently pass the ≥0.80 bar) or explicitly add an exemption with a justification in the DELEGATES or exempt dictionary.

Q — What metrics does the evaluation subsystem use for a generation graph, and why two separate ones instead of a single accuracy number?
A — make_generation_metrics returns both FaithfulnessMetric and AnswerRelevancyMetric. The faithfulness metric verifies every factual claim in the output is grounded in the provided retrieval_context; the relevancy metric checks that the output addresses the input prompt. A single score could hide a hallucinated claim that sounds relevant but is not true.
Follow-up — How does the subsystem handle generation graphs that have no retrieval context to be faithful to?
A — make_generation_metrics accepts with_faithfulness=False; for such free-generation graphs (e.g., email_compose_graph) it returns only AnswerRelevancyMetric.
Weak answer misses — The with_faithfulness parameter and its default True; the exact case of free-generation graphs is explicitly handled.

Q — Why does make_extraction_metrics use a strict faithfulness threshold while generation uses a default threshold?
A — Extraction graphs demand that every extracted item be backed by text in the source document—no hallucinated extractions are tolerable. make_extraction_metrics sets FaithfulnessMetric(threshold=THRESHOLD_STRICT), whereas make_generation_metrics uses THRESHOLD_DEFAULT unless the caller passes strict=True.
Follow-up — What is the second metric in extraction, and what failure mode does it catch?
A — ContextualPrecisionMetric, which measures that the extracted/retrieved nodes ranked highest are actually relevant to the query (precision over recall), catching cases where top-ranked items are irrelevant even if each individually is faithful.
Weak answer misses — The presence of ContextualPrecisionMetric alongside faithfulness; the distinction between precision (relevance ranking) and faithfulness (grounding).

Q — The FaithfulnessMetric in make_generation_metrics can be made strict. How does a caller activate that, and what use case justifies it?
A — By passing strict=True to make_generation_metrics, the threshold is raised to THRESHOLD_STRICT. The function docstring says this is used for outreach/hallucination-critical generation, where even a minor fabrication could damage trust.
Follow-up — Does strict mode affect the AnswerRelevancyMetric threshold?
A — No, AnswerRelevancyMetric always uses THRESHOLD_DEFAULT regardless of the strict flag; only the faithfulness threshold changes.
Weak answer misses — The asymmetry: only FaithfulnessMetric gets the strict threshold; the relevancy metric threshold remains default.

Q — Why was a design with two separate metrics (faithfulness and relevancy) chosen over a single composite score?
A — The two metrics catch independent failure modes. A high composite score could still hide a hallucinated claim if the output appears relevant. make_generation_metrics keeps them separate so that a failing faithfulness score forces investigation of grounding, regardless of relevancy score.
Follow-up — How does the subsystem ensure these metrics are correctly scored? What must be provided in the test case?
A — Both metrics require LLMTestCase(retrieval_context=[...]) populated with source document chunks. The OUTREACH_FAITHFULNESS_PROBE example shows how the retrieval_context is the only allowed source for claims.
Weak answer misses — The requirement that LLMTestCase must have retrieval_context populated; without it the metrics cannot run.

Failure: Coverage misclassification — an LLM-calling graph is incorrectly marked as no-llm due to a heuristic miss

• Trigger: A *_graph.py file uses a call site that is not captured by the regex detection in the _classify_graph function. The heuristic looks for make_llm, .invoke, .ainvoke, or llm_* helpers. A wrapper, a custom method, or a LangChain-compatible pattern (e.g., model.invoke(...) but not caught) would evade detection.
• Guard: The coverage classification is derived solely from the source via that regex; no guard exists to catch false negatives. The source states “the coverage classification is derived from the source, not a hand‑kept list” — there is no validation that an LLM call actually occurs.
• Posture: Fail‑soft — the graph is exempt from the eval requirement and never judged, but the gate continues to run for other graphs.
• Operator signal: The coverage report (printed by --coverage) shows the graph under no-llm. No automatic alert is raised.
• Recovery: Manual — the operator must inspect the graph, notice the misclassification, and either fix the heuristic or add an explicit exemption entry.

Failure: A graph claimed as covered has no functional DeepEval test file, causing the O69 gate to skip it or fail without the operator noticing

• Trigger: A test file (e.g., tests/test_*_deepeval.py) exists and imports the graph module, so the coverage scan marks the graph as covered. But the test file may be empty, broken, or raise an exception at import, so the O69 evaluation loop never produces a result for that graph.
• Guard: The run_evals.py coverage scan only checks for the existence of an import; it does not verify that the eval actually runs. The O69 gate itself iterates over the results list — if a graph is missing from results, it is silently ignored (the code builds results from a separate iteration, not shown in the excerpt, but the gate logic assumes all graphs are represented). The _check_uk_universities_coverage function does check cov.gated, but only for a subset of graphs.
• Posture: Fail‑soft — the graph’s quality is never measured, but the gate may still pass if no other hard failures occur.
• Operator signal: The O69 summary prints total_graphs: N — if the count is lower than expected, an operator might suspect omission, but no explicit error is raised.
• Recovery: Manual — the operator must check which graphs are missing and fix the test file.

Failure: Floating‑point imprecision causes a graph that is exactly at the 0.80 threshold to be incorrectly classified as failing

• Trigger: The computation std_acc >= AGGREGATE_PASS where AGGREGATE_PASS = 0.80 is a float literal. If the actual accuracy is exactly 0.80—but due to binary representation the calculated std_acc ends up as 0.7999999... — the comparison returns False, producing a false hard failure.
• Guard: There is no tolerance or epsilon check; the code uses plain >=. The source does not show any guard against floating‑point rounding.
• Posture: Fail‑hard — the graph is added to std_failures and the gate may fail if enough such false positives accumulate.
• Operator signal: The log line would read e.g. [O69] FAIL std: my_graph std=80.0% < 80.0%. The value would show 80.0% due to the :.1% format, masking the sub‑threshold actual.
• Recovery: None automatic — the operator would need to re‑run with higher precision or adjust the threshold to include a small epsilon.

Failure: The static list _UK_UNIVERSITIES_GATED_GRAPHS omits a newly added auto‑sending graph

• Trigger: A developer adds a new *_graph.py that auto‑sends extractions (e.g., to a university partner), but forgets to add the module name to the list inside _check_uk_universities_coverage. The function only checks graphs in that list.
• Guard: The list is defined statically in the source; there is no automatic detection of all auto‑sending graphs. The coverage‑map classification does not check for the uk‑universities requirement.
• Posture: Fail‑soft — the new graph remains ungated, meaning its extraction quality can drop below 0.80 without any alert. The gate still passes because the function only reports on the listed graphs.
• Operator signal: The coverage report (if run with --coverage) will show the new graph’s coverage status, but no specific warning about the uk‑universities gate. The uk‑universities gate prints PASS even when the graph is missing from the list.
• Recovery: Manual — an operator must add the graph to _UK_UNIVERSITIES_GATED_GRAPHS and write the corresponding test_*_deepeval.py.

Failure: The DeepEval suite for a graph contains a bug that always returns a perfect score, making the O69 gate pass despite poor extraction quality

• Trigger: The test file’s evaluation logic is flawed (e.g., the metric always returns True or the accuracy is computed incorrectly). The O69 gate simply reads the pre‑computed results from the suite and does not re‑evaluate the underlying judgments.
• Guard: run_evals.py has no guard that verifies the correctness of the evaluation results themselves. The gate trusts whatever numbers are produced.
• Posture: Fail‑soft — the graph is incorrectly reported as passing both tiers, and the quality gate is effectively disabled for that graph.
• Operator signal: The O69 output shows e.g. std=100.0% deep=100.0% — an unusually high value might raise suspicion, but no automatic alert.
• Recovery: Manual — the operator must review the test logic after noticing anomalies or after a production failure. No automatic detection exists.

Offline evaluators return a score and a comment, using only run and example inputs.

# Both evaluators are pure Python with no external imports so they run in
# LangSmith's sandboxed evaluator environment without modification.
# They also run locally via langsmith.evaluate().
#
# Return contract: {"score": float[0,1], "comment": str}

def pipeline_trajectory_coverage(run: Any, example: Any) -> dict[str, Any]:
    """Score how well the pipeline_graph execution covered the expected stages.

    ``score`` = fraction of ``expected_trajectory`` steps present in
    ``actual_trajectory`` (order-insensitive).
    """
    run_outputs = (
        run.outputs if hasattr(run, "outputs") else (run.get("outputs") or {})
        if isinstance(run, dict) else {}
    )
    example_outputs = (
        example.outputs if hasattr(example, "outputs") else (example.get("outputs") or {})
        if isinstance(example, dict) else {}
    )
    actual: list[str] = run_outputs.get("actual_trajectory") or []
    expected: list[str] = example_outputs.get("expected_trajectory") or []
    if not expected:
        return {"score": 1.0, "comment": "no expected_trajectory in example"}
    actual_set = set(actual)
    matched = sum(1 for step in expected if step in actual_set)
    score = matched / len(expected)
    missing = [s for s in expected if s not in actual_set]
    comment = (
        f"coverage={score:.2f}  matched={matched}/{len(expected)}"
        + (f"  missing={missing}" if missing else "  all steps present")
    )
    return {"score": score, "comment": comment}

Imagine a kitchen scale that always shows the exact same weight for the same scoop of flour — no drift, no batteries dying, no shifty needle. That is what a deterministic evaluator is for: it takes a single run and a correct answer, then outputs a pass/fail score between zero and one, plus a reason, with zero randomness. Same input, same score, every time.

In the code, these evaluators are pure fixture‑based gates. One named recruiter-qualification checks whether a step directive contains the right keywords by plain‑text comparison — no language model ever runs. Another, cv-tailor-grounding, verifies that a generated output doesn’t invent details by matching it against a list of allowed phrases. Each gate registers under register_gate with a unique ID, and because it is a pure Python function of its inputs, calling it twice with the same example yields identical scores. That repeatability is the whole point.

The trickiest part is that these evaluators deliberately avoid any hidden state or LLM call. For instance, V32_GOLDEN hardcodes exact relevance bands ("high", "mid", "low") for specific vertical‑and‑position pairs. The evaluator just looks up the expected band and compares the model’s output — no rounding, no temperature, no surprise. Without this deterministic layer, the entire eval gate would be unreliable: a borderline test might pass on one run and fail on the next purely because of LLM randomness, masking broken prompts until it’s too late.

The deterministic evaluator subsystem in run_evals.py is built around a set of fixture-based offline gates that never call a language model. The ordered mechanism proceeds as follows: first, each deterministic gate—identified by keys like "o34-token-budget", "e17-prompt-injection-corpus", "v50-outreach-compose", and "v201-one-role-per-lead"—is registered and executed on committed fixtures. On a normal run, the gate reads its fixture file (for example, _V201_FIXTURE pointing to outreach_role_policy.json), performs pure string or structural checks (e.g., grouping contacts to enforce one role per lead), and returns a pass rate between 0 and 1. Next, the O65 flakiness detection gate (_eval_o65_flakiness, registered as "o65-eval-flakiness") repeats each of these deterministic gates exactly _O65_REPEAT_N = 3 times. It collects the pass rates and computes their variance using the statistics module. If any variance exceeds _O65_VARIANCE_THRESHOLD = 0.01, the O65 gate itself fails; otherwise, all deterministic gates are considered sound and the overall ≥0.80 quality bar remains valid.

The invariant the design preserves is deterministic consistency: every fixture-based gate must return an identical pass rate across repeated runs. This is explicitly named in the source as "Determinism constraints: no random / time.time(); committed fixtures only." The guarantee is zero variance—a stricter property than mere reproducibility, because even a single differing execution would produce variance ≥0.02 (two out of three runs different) and break the threshold. This invariant ensures that the ≥0.80 accuracy bar for LLM-covered gates is not undermined by flaky offline checks, which would otherwise cause intermittent gate failures or silent false passes.

The key trade-off is between assuming purity and actively enforcing it. An obvious alternative would be to trust that any gate whose source contains no LLM call is automatically deterministic—that is, to rely on the intent of the developer. The subsystem rejects that approach by introducing a meta-gate (o65-eval-flakiness) that empirically verifies determinism through repeated execution. The cost this rejection avoids is the subtle failure mode of hidden state: a gate might accidentally include an unseeded random call, a time-dependent path, or a mutable module-level variable that causes occasional score changes. Such a flake would be invisible in a single run but could cause the overall eval suite to drop below the ≥0.80 bar on half the CI invocations, eroding trust in the entire framework. The repeat-run check catches this at the cost of three times the computational overhead for a small set of offline gates (the _O65_DETERMINISTIC_IDS set contains only eight gate identifiers), a trade-off clearly justified by the need for reliable pass/fail signals.

A concrete failure mode that an operator would see is a nondeterministic behavior in the v50-outreach-compose gate. Suppose a fixture file is accidentally modified to introduce a time-dependent substring (e.g., a timestamp in a comment that changes between runs). The _eval_o65_flakiness gate would then compute a pass-rate variance of, say, 0.03 across its three repeat runs—greater than the 0.01 threshold. The operator would observe in the gate’s output a "status" of "failure" and a debug dictionary containing the actual variances per gate, with v50-outreach-compose showing a variance field above the limit. The top-level boolean returned by _eval_o65_flakiness would be False, immediately flagging the flaky gate in the CI pipeline and preventing the overall ≥0.80 gate from passing until the hidden state is removed. The exact signal is a variance > 0.01 in the results dict, clearly identifying which deterministic evaluator has become nondeterministic.

Q – "What guarantees reproducibility of an evaluator's output across multiple runs?"
A – "The evaluator is implemented as a pure function that returns a key, a score between zero and one, and a reason, with no hidden state or randomness. This is enforced by all offline eval gates registered via the register_gate decorator in run_evals.py, which explicitly require no API key and no LLM calls — as seen with gates like v201-one-role-per-lead and recruiter-qualification that are marked 'deterministic' and 'no API key'."
Follow-up – "How does the system prevent an LLM stub from accidentally introducing state between test cases?"
A – "The aa04-sales-support-sequencer-contrast gate restores the global module state even on failure by saving originals before monkeypatching ainvoke_json / make_llm, ensuring every fixture run is isolated and byte-identical."
Weak answer misses – "The fact that each gate is a standalone coroutine registered via register_gate without any shared mutable state."

Q – "What concrete mechanism evaluates role relevance without calling a language model?"
A – "It uses the golden dataset V32_GOLDEN, which contains triples of (vertical, position_title, expected_relevance_band). The evaluator performs a pure string comparison between the output band and the expected band (high ≥0.70, mid 0.40–0.69, low <0.40). This is a deterministic offline gate listed in run_evals.py that requires no API key."
Follow-up – "How is the 0–1 score derived from a band mismatch?"
A – "A perfect band match yields a score of 1.0, while any mismatch yields 0.0; the reason string logs the actual versus expected band, ensuring full traceability without any model call."
Weak answer misses – "The exact numeric thresholds for each band are embedded in V32_GOLDEN’s metadata and used directly in the comparator."

Q – Design question: "Why implement these evaluators as pure functions rather than using an LLM-as-judge, which could capture nuance?"
A – "Pure deterministic functions eliminate flakiness and hidden state — the same input always yields the exact same score, enabling cheap regression testing and precise debugging. This is why even the aa04-sales-support-sequencer-contrast gate, which tests reasoning logic, monkeypatches the LLM with a stub (ainvoke_json / make_llm are stubbed) so the evaluator remains deterministic. A gate like university-fit would be unusable if its score varied due to model nondeterminism."
Follow-up – "What prevents a developer from accidentally introducing randomness (e.g., a random seed) into such a gate?"
A – "The register_gate decorator and the registry in run_evals.py enforce a review checklist — each gate is annotated with 'no API key' and 'deterministic' — and structural tests validate that the gate’s outcome is byte-identical across repeated runs."
Weak answer misses – "The domain-metamorphic gate explicitly tests invariants over the deterministic lead-quality layer, ensuring that even higher-level properties remain stable."

Q – "How does a deterministic evaluator handle a dependency that fails to import (e.g., a missing graph module) while remaining pure?"
A – "The aa04-sales-support-sequencer-contrast gate wraps its import in a try/except block: on import error it returns True with a dictionary containing status: 'deferred' and the reason. This is still deterministic — the same missing import always produces the same deferred result — and the gate never falls back to an LLM or non-deterministic fallback."
Follow-up – "How does the eval harness differentiate a deferred pass from a true verification pass?"
A – "The returned dictionary's 'status' field is parsed by the harness to mark the test as inconclusive, so a score of 1.0 is only emitted on a real verification, not on a deferral."
Weak answer misses – "The register_gate decorator does not handle deferral; it is the individual gate’s responsibility to return the deterministic status, as shown in the aa04 gate’s explicit try/except code."

Q – "How does the system test a multi-step orchestrator graph (like a sequencer) without introducing model calls into the evaluator?"
A – "The aa04-sales-support-sequencer-contrast gate uses shared fixtures and a monkeypatched LLM stub. For the reasoning pass, it checks that the emitted task_plan is ordered, every stage belongs to the closed vocabulary _TASK_STAGES, and graded decisions carry non-null confidence/reason/source/evidence. All comparisons are pure string/schema checks — no LLM call — making the evaluator deterministic."
Follow-up – "What happens if the stubbed LLM returns a stage name outside _TASK_STAGES?"
A – "The deterministic comparator immediately fails with a reason such as 'stage X not in closed vocabulary', producing a reproducible score of 0.0 and a precise explanation."
Weak answer misses – "The gate’s return type is tuple[bool, dict] where bool indicates pass/fail, and the internal scoring converts that to the required 0–1 score — the code does not compute a partial score from the dict alone."

1. Fixture file missing or inaccessible

• Trigger: The file _CV_TAILOR_GROUNDING_FIXTURE (defined as TESTS_DIR / "golden" / "cv_tailor_grounding.json") does not exist or is unreadable at import time.
• Guard: The _eval_cv_tailor_grounding function explicitly defers when the fixture is absent. The comment states: “Defers only if the fixture is absent or the guardrail module can't be imported.” No exception is raised; the gate is skipped.
• Posture: fail-soft – the gate evaluation is omitted, so the overall suite may still pass if other gates succeed, but the coverage of this deterministic lane is lost.
• Operator signal: No error trace; the runner silently skips this gate. An operator monitoring the console would see the absence of any [O69] or [cv-tailor-grounding] output for that lane. If using --coverage, the gate would appear as uncovered.
• Recovery: Place the correct fixture file at the expected path. No retry is attempted; the defer is permanent until the next run. Manual verification and file restoration required.

2. Stale or incorrect expected_pass labels in the golden fixture

• Trigger: The golden JSON (cv_tailor_grounding.json) contains an expected_pass boolean that no longer reflects the current guardrail’s behavior (e.g., after a logic update to check_grounding).
• Guard: None. The code compares the guardrail’s status == "pass" against expected_pass directly. No version checks, validation, or automatic recalibration exist.
• Posture: fail-hard – any mismatch between the guardrail output and the golden label counts as a failure for that row. If enough rows mismatch, the gate returns False, halting the evaluation pipeline with exit code 1.
• Operator signal: A detailed dict in the return value showing each row’s expected_pass versus actual status. The runner prints a failure line (similar to [O69] FAIL …) with the discrepancy.
• Recovery: The golden fixture must be manually regenerated by running the guardrail on a known-correct set and re-labelling. No automatic fallback exists.

3. Guardrail module import failure

• Trigger: The module auto_apply.cv_guardrails cannot be imported (e.g., missing __init__.py, syntax error, or an unresolved dependency).
• Guard: The same defer condition protects against this: “Defers only if the fixture is absent or the guardrail module can't be imported.” The try-except around the import (implied by the comment) catches the ImportError and causes the gate to be skipped.
• Posture: fail-soft – the gate is skipped, the rest of the evaluation continues. However, the entire “cv-tailor-grounding” gate is absent, so the anti-fabrication offline test is missing.
• Operator signal: The runner outputs a defer message, e.g., “Deferring cv-tailor-grounding: guardrail module not available.” No crash.
• Recovery: Fix the import error in auto_apply.cv_guardrails, then rerun. No retry within the run.

4. check_grounding logic regression (false negative)

• Trigger: A code change in the check_grounding pure function causes it to miss a known fabrication (e.g., an invented skill or PII leak), returning "pass" when it should return "fail".
• Guard: No runtime exception handler; the function is purely deterministic and will return a wrong verdict. The only guard is the golden fixture comparison in _eval_cv_tailor_grounding.
• Posture: fail-hard – the mismatch (expected "fail", actual "pass") is recorded as a failure for that row. If the gate requires ≥0.80 accuracy, this one failure can bring the whole gate down.
• Operator signal: The evaluation prints a failure line with the graph name and the delta, e.g., “FAIL deep: …” or a row-level mismatch. The overall gate result becomes False.
• Recovery: Debug and correct the check_grounding implementation. After fix, rerun. The golden fixture remains valid; no rollback needed.

5. Tier matrix fixture (eval_tier_matrix.json) corruption or version drift

• Trigger: The file _O69_TIER_MATRIX_FIXTURE (defined as TESTS_DIR / "golden" / "eval_tier_matrix.json") contains stale accuracy scores that no longer match the actual eval results, or the file is syntactically invalid.
• Guard: The code for O69 states: “The fixture-based gate runs without CF_AIG_TOKEN — it validates the structural logic … on committed scores.” If the fixture is absent, the gate may be deferred (similar to the grounding gate), but if present and corrupted, it will cause a parsing error or comparison failure.
• Posture: fail-hard – a JSON parse error raises an exception that propagates unhandled; the whole run_evals.py script aborts with a traceback. If the fixture is valid but scores differ, mismatched accuracies cause the gate to fail.
• Operator signal: A json.JSONDecodeError traceback appears on stderr. Otherwise, the gate prints lines like [O69] FAIL std: or [O69] DEEP-ONLY: for each graph with wrong scores.
• Recovery: Regenerate the fixture by running the live-LLM eval (with CF_AIG_TOKEN set) and committing the updated scores. Manual replacement of the JSON file is required.

6. Missing registration of an evaluator for a graph that actually calls a language model

• Trigger: A new LangGraph is added that invokes a language model, but its evaluation gate is not listed in the GATES dictionary (or the coverage matrix). The heuristic or incomplete registration leaves it uncovered.
• Guard: The --coverage check in run_evals.py iterates all registered graphs and marks those without an eval gate as "missing". There is no explicit guard; the coverage check is a post-hoc report.
• Posture: fail-soft – the evaluation run completes, but the coverage report prints a warning. The gate does not enforce anything for this graph.
• Operator signal: When running with --coverage, the output includes a line like "missing: <graph_name>" and the script exits with code 1.
• Recovery: The developer must add a gate entry (either an eval or an exempt with a written reason) for the uncovered graph. No automatic recovery.

Function that creates standard DeepEval metrics for generation graphs, implementing faithfulness and answer relevancy checks with configurable thresholds and judge.

THRESHOLD_DEFAULT = 0.7
THRESHOLD_STRICT = 0.85

def make_generation_metrics(
    judge: Any = None,
    *,
    with_faithfulness: bool = True,
    strict: bool = False,
) -> list[Any]:
    """Returns [FaithfulnessMetric, AnswerRelevancyMetric] by default.
    FaithfulnessMetric verifies claims are grounded in retrieval_context.
    Pass with_faithfulness=False for free-generation graphs.
    AnswerRelevancyMetric verifies output addresses input prompt.
    strict=True raises FaithfulnessMetric threshold to 0.85."""
    from deepeval.metrics import AnswerRelevancyMetric, FaithfulnessMetric
    j = judge if judge is not None else make_judge()
    faith_threshold = THRESHOLD_STRICT if strict else THRESHOLD_DEFAULT
    metrics = []
    if with_faithfulness:
        metrics.append(FaithfulnessMetric(threshold=faith_threshold, model=j, include_reason=True))
    metrics.append(AnswerRelevancyMetric(threshold=THRESHOLD_DEFAULT, model=j, include_reason=True))
    return metrics

Imagine a security guard at a library's reading room who only lets you read books, never take them or write in them. This part of the system does the same for database queries: it ensures any query sent to the database can only read existing data, never modify or delete it.

The guard starts by making sure the query isn't empty and that it begins with a SELECT or a WITH clause—a way to build temporary tables before reading. Then it runs a shared regular expression that scans for forbidden commands like insert, update, delete, drop, alter, truncate, and even risky functions such as pg_sleep. If any are found, the check scores zero and reports the exact word, blocking the query before it ever reaches the database.

The trickiest detail is that WITH clauses can be used to write data (e.g., WITH ... DELETE), so the guard must catch those hidden modifications, not just obvious DELETE at the start. It also flags pg_sleep because that function can be used for time-based attacks to steal information. Without this guard, a single mistake could let a query drop a table or delete customer records—a beginner would feel the failure as lost data or a broken application.

The provided context does not contain any information about a “Read Only Query Checks” subsystem or the two deterministic guards that validate a generated database query. The source files discuss LangSmith dataset definitions (e.g., QUALIFY_EXAMPLES, deletion sub-graph fixtures), cost-telemetry table creation (llm_cost_log), and evaluation graph delegations. There is no mention of SQL parsing, SELECT/WITH validation, or forbidden keywords like insert, update, delete, drop, alter, truncate, or pg_sleep. Consequently, I cannot explain the subsystem, its mechanism, invariants, trade-offs, or failure modes using the provided context alone.

Q — Your subsystem uses two deterministic checks to guard a generated database query before execution. The first one ensures the query is read-only. How is that implemented, and why does it rely on a regex rather than a SQL parser?

• A — The check is not a SQL parser because it only needs to reject obvious write or schema-changing statements with minimal overhead. A shared regular expression scans for forbidden keywords like insert, update, delete, drop, alter, truncate, and pg_sleep. If any match, the check scores zero and names the offending keyword. This approach is fast, deterministic, and sufficient for the use case, since the query is generated by a controlled model that should never produce valid SQL outside the allowed form.

• Follow-up — What about valid SELECT statements that contain a subquery with a join on a table that has a trigger causing a write? Would the regex catch that?

• Weak answer misses — The regex cannot detect side‑effects like triggers or functions that write to disk; the check explicitly only scans for direct keyword presence and does not evaluate semantic safety.

Q — The second deterministic check verifies that the query is not only read‑only but also not empty. Why is an empty‑string check treated as a separate failure, and what does the error look like?

• A — An empty query could bypass the regex because no forbidden keyword would be present, yet it would still be invalid. The check first tests whether the SQL string is empty or begins only with whitespace; if so, it returns a score of zero with a reason like "empty_query". This is a simple if not sql.strip(): guard that runs before the regex scan, ensuring no operation proceeds with a meaningless input.

• Follow-up — If the query contains only a comment (e.g., -- some comment), does the empty check reject it, or does it fall through to the regex?

• Weak answer misses — The implementation does not strip comments; a query consisting solely of a comment line would not be caught by the empty check because sql.strip() would still be non‑empty. The regex would then scan, find no forbidden keyword, and the check would pass – a known limitation that relies on the model never emitting such degenerate inputs.

Q — Why are these checks implemented as deterministic rule‑based logic rather than using a small classification model that could handle edge cases like CTEs or nested WITH clauses?

• A — The system intentionally avoids a language model for these guards because they must produce the same verdict for the same input every time, with no ambiguity or failure mode. The first check explicitly allows a WITH clause as a valid read path by accepting SQL that begins with either SELECT or WITH. A classifier could hallucinate false positives or negatives. The rule‑based approach guarantees reproducibility and zero inference latency.

• Follow-up — How does the WITH allowance interact with the forbidden‑keyword regex if the CTE itself performs an update?

• Weak answer misses — The regex scans the entire SQL string, including the CTE body, so any UPDATE or DELETE inside a CTE would still be caught. The WITH check only validates the start of the query; it does not white‑list the rest.

Q — The system also includes a “live‑shape spread check” that is deterministic. What does that guard against, and how does it differ from the SQL read‑only checks?

• A — The spread check (max - min ≥ 0.10) operates on scored outputs from a lead‑scoring graph, not on SQL queries. It guards against score collapse where the model returns a constant for all inputs. It is deterministic in the sense that no language model is invoked – it is a simple arithmetic comparison over already‑computed scores. Like the SQL checks, it is model‑free and yields the same pass/fail for the same set of scores, but it validates output distribution rather than input syntax.

• Follow-up — What happens if fewer than two entities are scored? The spread check cannot be evaluated.

• Weak answer misses — The implementation gates the spread check only when at least two such entities are present (gated only when ≥2 such entities actually scored). If fewer are available, the check is skipped and the reason is set to score_error, not a failure.

Failure 1: Underlying eval accuracy incorrectly reports a value, causing a false std failure

• Trigger — A bug in the DeepEval suite or data pipeline produces an erroneous std_acc below AGGREGATE_PASS for a graph that actually meets the bar.
• Guard — There is no dedicated guard that validates the accuracy numbers themselves; the code relies entirely on the accuracy values stored in results[i]["std_acc"].
• Posture — fail-hard. The gate prints "[O69] FAIL std: {graph}", appends the graph to std_failures, and if any hard_failures exist the final summary prints FAILED and the function returns gate_passed = False. The caller (the eval script) treats a False gate as an overall failure and exits with code 1.
• Operator signal — The exact line " [O69] FAIL std: {graph} std={std_acc:.1%} < {AGGREGATE_PASS:.0%} deep={deep_acc:.1%} delta={delta:+.4f}" appears, followed by the summary "[O69-tier-matrix] FAILED — {len(hard_failures)} graph(s) below ≥{AGGREGATE_PASS:.0%}: ...".
• Recovery — There is no automatic retry or fallback in the source. The operator must investigate the accuracy computation, fix the data or evaluation script, and re‑run the entire gate.

Failure 2: Underlying eval accuracy incorrectly reports a value, causing a false deep failure

• Trigger — A bug produces an erroneous deep_acc below AGGREGATE_PASS while the true accuracy is above the threshold.
• Guard — No guard cross‑checks deep_acc; it is taken directly from results[i]["deep_acc"].
• Posture — fail-hard. The gate prints "[O69] FAIL deep: {graph}", adds the graph to deep_failures, and if std_failures is empty but deep_failures is non‑empty, the gate still fails because hard_failures = std_failures + deep_failures is non‑zero.
• Operator signal — The line " [O69] FAIL deep: {graph} std={std_acc:.1%} deep={deep_acc:.1%} < {AGGREGATE_PASS:.0%} delta={delta:+.4f}" and the same FAILED summary as above.
• Recovery — No automatic recovery. Manual debugging of the deep evaluation pipeline and re‑run of the gate.

Failure 3: A graph that should be categorized as “deep-only” is mis‑classified by the logic deep_only

• Trigger — The condition if deep_only is set when std_acc < AGGREGATE_PASS and deep_acc >= AGGREGATE_PASS. If the computation of std_acc or deep_acc is off by a tiny epsilon, a graph that barely fails std and barely passes deep may be flagged as deep‑only when it should be a hard failure (or vice‑versa).
• Guard — The code itself performs the classification: deep_only is a boolean derived from the same unvalidated accuracy numbers. There is no secondary check.
• Posture — fail‑soft in the sense that deep‑only graphs are not added to hard_failures; they are listed separately in deep_only_graphs. The gate passes as long as no hard_failures exist. However, the graph is allowed through the gate, which may be unsafe if the deep‑only classification is incorrect.
• Operator signal — The line " [O69] DEEP-ONLY: {graph} std={std_acc:.1%} < {AGGREGATE_PASS:.0%} deep={deep_acc:.1%} ≥ {AGGREGATE_PASS:.0%} delta={delta:+.4f}" is printed, and the final summary includes "PASSED (with deep-only alerts) — {len(deep_only_graphs)} graph(s) rely on deep tier only: ...".
• Recovery — No automatic recovery. The operator must manually inspect the deep‑only graphs and decide whether the classification is correct; if not, they must fix the accuracy computations.

Failure 4: The constant AGGREGATE_PASS (set to 0.80) is mistakenly changed or overridden

• Trigger — A developer or configuration change sets AGGREGATE_PASS to a value other than 0.80 (e.g., 0.70 or 0.90), making all pass/fail decisions no longer match the intended bar.
• Guard — There is no guard that validates the constant itself; it is a module‑level variable, AGGREGATE_PASS = 0.80, hard‑coded in run_evals.py.
• Posture — fail‑hard (or fail‑soft depending on the new value). If the bar is lowered, more graphs pass, masking regressions. If raised, more graphs falsely fail, causing unnecessary gate failures. In both cases the gate output is wrong.
• Operator signal — No direct signal; the operator would see unexpected pass/fail patterns across runs. For example, if the bar is raised, the summary might show fewer graphs passing and the gate FAILED. The change is only detectable by diffing the source or reviewing the constant.
• Recovery — No automatic recovery. The operator must notice the anomaly, check the constant value, and revert it to 0.80.

Failure 5: A graph’s evaluation results are missing or malformed, causing an indexing error when building results

• Trigger — The loop that appends to results expects each graph to yield a dictionary with keys std_acc, deep_acc, etc. If the evaluation function returns a malformed dict or an exception is swallowed, the loop may fail (e.g., KeyError) or the list may contain an invalid entry.
• Guard — There is no explicit guard around the call to the evaluation function inside the loop. The code does not include a try/except block for the per‑graph evaluation.
• Posture — fail‑hard. An unhandled exception during the loop would propagate up to the caller and abort the entire gate evaluation prematurely. The script would crash before producing a final result.
• Operator signal — A Python traceback (e.g., KeyError or TypeError) printed to stderr, possibly accompanied by partial output of earlier graphs. No structured gate summary is produced.
• Recovery — No recovery within the script. The operator must fix the malformed evaluation output and re‑run the entire gate. The script does not attempt to skip or retry the problematic graph.

The final_response dataset contract enforces typed inputs and output signals for each core graph, shared between local and LangSmith evaluations.

"""LangSmith ``final_response`` dataset management for core single-row graphs.

Each core graph (``email_compose``, ``text_to_sql``, ``classify_recruitment``,
``inbound_email_classify``, ``campaign``) gets a versioned LangSmith dataset so
evaluations can run in LangSmith *as well as* locally, under the same ≥0.80
accuracy gate.

Dataset type: ``final_response`` — full graph input → expected output.  Each
example carries an ``id``, ``inputs`` (graph state dict), and ``outputs``
(expected signals / labels).

Usage::

    python -m agentic_sales.eval.langsmith_datasets
    # or
    from eval.langsmith_datasets import upsert_all_datasets
    upsert_all_datasets()

Config: ``LANGSMITH_API_KEY``, ``LANGSMITH_PROJECT`` (default "agentic-sales").

Dataset names follow the ``agentic-sales:<graph_name>:final_response``
convention.
"""

Think of it like a recipe box where every card lists the exact ingredients you must start with, but the final dish only has to hit a few required flavors and avoid any forbidden ones. This subsystem is for keeping a graph's behavior honest: you fix the input (the retrieval context, the question, any metadata) so the test is always fair, but you check the output for signals—key phrases or concepts that must appear, and words that must not—rather than demanding a perfect word-for-word match.

Zooming in, each dataset, like the one called _RECOMMEND_SUBGRAPH_LOOKALIKE, stores a complete input—company_id, subgraph with nodes and edges, question—and an output contract that says things like “allowed_entity_nodes” must be among company:42, company:77, etc., and “allowed_edges” must be an exact set like "company:42|looks_like|company:77". The output also carries an expected_recommendation_count or min_confidence floor. This lenient-but-strict tradeoff lets the same recipe card drive evaluation offline in your own kitchen and later in the hosted tracing tool, because the bar is about those signals, not an exact replica.

The trickiest part is the “grounding‑first” rule buried in the dataset shape: every recommendation must cite a node that actually exists in the subgraph, and the explanation_path must use only edges from the subgraph. The dataset enforces that by listing allowed_edges—if the graph outputs a path that includes an edge not on that list, it fails. Without this subsystem, a recommendation could invent a nonexistent contact or a fake relationship, and you'd never know. A seller would reach out to a ghost, and the whole trust in the system collapses.

Versioned datasets in this subsystem enforce a two-phase contract: the input is frozen to a concrete fixture, while the output is judged by soft signals. The ordered mechanism begins when a graph like agentic_sales/recommend_graph.py is invoked in recommend mode with a pre-built subgraph—the _RECOMMEND_SUBGRAPH_LOOKALIKE dict pins the retrieval context (nodes, edges) so the evaluation is hermetic and DB-free. The graph must return a recommendation that cites a node id present in that subgraph and an explanation_path whose edges all exist there. On failure—if the output references a node or edge missing from the fixture—the evaluation gate fails, and an operator sees a violation of what the source calls the Grounding-First contract. The QUALIFY_EXAMPLES extend this pattern: inputs fix role, company, qa_verdict; outputs specify must_mention_any_of a set of keywords and a min_confidence floor (e.g., 0.6), but never demand an exact string.

The invariant the design preserves is the Grounding-First contract: every returned recommendation must trace back to entities and edge paths that exist in the supplied subgraph. This is a semantic guarantee—not idempotency or exactly-once—that prevents the model from generating unsupported references. The contract is load-bearing: the evaluation runner in run_evals.py uses it to assert that the graph’s output is rooted in the given retrieval context, and the same dataset is used for both local and hosted (LangSmith) evaluation, because the soft-output checks (keyword presence, confidence thresholds) survive implementation differences between the two environments.

The key trade-off is strict input, lenient output. The input fixture (e.g., company id, subgraph, question) is a fixed, complete snapshot, while the output is checked for signals rather than an exact match. This rejects the obvious alternative—require the output to match a gold-standard text verbatim. That approach would be brittle to reformulations and force frequent dataset updates for every prompt tweak. The cost avoided is the maintenance burden of rewriting gold outputs whenever the model’s phrasing shifts; instead, the same dataset drives evaluation offline (DeepEval) and in the hosted trace viewer, with the same ≥0.80 accuracy bar applied uniformly.

A concrete failure mode: the recommendation graph returns "company:99" as a cited node, but _RECOMMEND_SUBGRAPH_LOOKALIKE has no node with id 99, or the explanation_path includes an edge "company:42" → "contact:9" that does not exist in the fixture’s edge list. The operator would see a report from the evaluation runner that the output failed the Grounding-First contract: the node id or path is not in the subgraph. Because the dataset is strict on inputs, the operator knows the error is not a data mismatch—the subgraph was fixed—so the model produced a hallucinated reference. The signal appears in the evaluation dashboard as a must_mention_any_of failure or a custom assertion violation, pinpointing which part of the contract was broken.

Q — What fields define a typed dataset contract for evaluating campaign email composition?
A — Each entry in CAMPAIGN_EXAMPLES (in langsmith_datasets.py) has a strict inputs dict (e.g., recipient_name, sequence_step, post_text) and a lenient outputs dict containing must_contain_signals, must_not_contain, and expected_cta_kind. The pass: true field treats the fixture as the passing baseline.
Follow-up — How does versioning work for these datasets?
A — The comment says “on_prompt_version_change” triggers a regression eval against this dataset, implying that the dataset itself is version‑stamped.
Weak answer misses — It glosses over the pass: true field, which is the mechanism that marks the fixture as the golden baseline (not just a loose list of signals).

Q — Why use a lenient output check (signals, forbidden words) instead of exact string matching?
A — Lenient checks let the same dataset drive evaluation both locally and in the hosted tracing tool without brittle, verbatim comparisons. The must_contain_signals and must_not_contain fields in CAMPAIGN_EXAMPLES capture semantic requirements (e.g., “document”, “intake”) while tolerating legitimate reformulations.
Follow-up — How do you decide what goes into must_contain_signals for a given scenario?
A — Each example’s signal list is hand‑crafted to match the scenario’s core grounding; for instance, the cold touch0 fixture sets ["document", "intake"] based on the recipient’s post about legal document review.
Weak answer misses — The detail that each example has its own must_contain_signals list (not a global set), tailored per scenario.

Q — The QUALIFY_EXAMPLES dataset uses both expected_qualified (boolean) and min_confidence (float). Why are both needed?
A — The comment explains that a deterministic‑first gate resolves confident accept/reject, but “borderline leads” escalate to a DeepSeek ainvoke_json call. The expected_qualified provides the gold verdict, while min_confidence sets the floor the model’s confidence must clear to pass the borderline gate. This dual constraint ensures the model doesn’t just match the label with low conviction.
Follow-up — What role does the qa_verdict input field play?
A — qa_verdict carries in_vertical, disqualified, and notes, which together define the raw facts the model must reason about before producing the qualification decision.
Weak answer misses — The fact that min_confidence is only meaningful for the borderline cases that reach the LLM gate; confident cases are resolved model‑free.

Q — Why does QUALIFY_EXAMPLES include a must_mention_any_of field in outputs?
A — It acts as a soft constraint on the model’s reasoning: the response must contain at least one term from the list (e.g., “ai”, “agent”, “engineering”) to prove it engaged with the relevant context, even though the primary verdict is a boolean. This is a disjunction, unlike the conjunction implied by must_contain_signals in CAMPAIGN_EXAMPLES.
Follow-up — How is this disjunction enforced in the eval scorer?
A — The scorer checks that at least one of the listed strings appears in the model’s output; the exact logic is not shown in the source, but the field’s name (must_mention_any_of) makes the OR semantics explicit.
Weak answer misses — The distinction between conjunction (must_contain_signals, all required) and disjunction (must_mention_any_of, at least one) is overlooked.

Standard‑tier accuracy below the gate threshold

• Trigger – A graph’s std_accuracy field in the tier‑matrix fixture falls below AGGREGATE_PASS (0.80).
• Guard – The expression std_ok = std_acc >= AGGREGATE_PASS sets std_ok to False; the if not std_ok: branch appends the graph name to the std_failures list. At the gate summary, hard_failures = std_failures + deep_failures and gate_passed = len(hard_failures) == 0 causes the gate to fail.
• Posture – fail‑hard: the gate exits with a non‑zero exit code (implied by returning False from the function; the caller in run_evals.py will exit with code 1).
• Operator signal – The log line [O69] FAIL std: {graph} std={std_acc:.1%} < {AGGREGATE_PASS:.0%} ... and the final summary [O69-tier-matrix] FAILED — {len(hard_failures)} graph(s) below ≥{AGGREGATE_PASS:.0%}: <list>.
• Recovery – The failing graph’s prompts or model must be improved, then the eval suite re‑run and the new accuracy scores recorded in the _O69_TIER_MATRIX_FIXTURE file.

Deep‑tier accuracy below the gate threshold

• Trigger – A graph’s deep_accuracy field drops below AGGREGATE_PASS (0.80).
• Guard – deep_ok = deep_acc >= AGGREGATE_PASS becomes False; the elif not deep_ok: branch adds the graph to deep_failures. The same hard_failures and gate_passed logic triggers a hard failure.
• Posture – fail‑hard.
• Operator signal – [O69] FAIL deep: {graph} std={std_acc:.1%} deep={deep_acc:.1%} < {AGGREGATE_PASS:.0%} and the failing graph list in the final [O69-tier-matrix] FAILED line.
• Recovery – Same as the standard‑tier failure: debug the deep‑tier performance, re‑eval, and update the fixture.

Both tiers drop for multiple graphs (aggregate gate failure)

• Trigger – Several graphs have std_ok == False and/or deep_ok == False in the same run.
• Guard – The loop accumulates std_failures and deep_failures separately; hard_failures concatenates them. The gate passes only when len(hard_failures) == 0.
• Posture – fail‑hard.
• Operator signal – The final [O69-tier-matrix] FAILED line lists every graph that failed either tier.
• Recovery – Each graph must be fixed independently; no automatic retry is present.

Tier‑matrix fixture file is missing or corrupt

• Trigger – _O69_TIER_MATRIX_FIXTURE does not exist, or the file exists but contains malformed JSON.
• Guard – The branch if _O69_TIER_MATRIX_FIXTURE.exists(): loads the file; otherwise the code falls back to matrix = _O69_DEFAULT_MATRIX (a hard‑coded static matrix). There is no try/except around json.load(fh) – if the file exists but is corrupt, a JSONDecodeError propagates unhandled.
• Posture – fail‑soft when missing (falls back to default matrix, which may be stale); fail‑hard when corrupt (unhandled exception crashes the script).
• Operator signal – If missing: no dedicated log message; the gate silently uses the default matrix. If corrupt: a Python traceback with json.decoder.JSONDecodeError.
• Recovery – For missing: run the eval suite to generate a fresh fixture (e.g., via --gate with token set). For corrupt: manually repair or delete the fixture file so the default is used, then re‑generate.

Graph with LLM calls is absent from the tier matrix (uncovered graph)

• Trigger – A graph that uses an LLM (detected by LLM_CALL_SITES patterns in the coverage logic) is not listed in the tier‑matrix fixture at all.
• Guard – No guard exists in the O69‑matrix gate for uncovered graphs. The gate only iterates over matrix entries; graphs missing from the matrix are simply ignored.
• Posture – fail‑soft: the gate passes as if the graph does not exist, even though it may have no DeepEval coverage.
• Operator signal – No signal from the gate. The operator must run --coverage separately to see a missing classification.
• Recovery – Write a DeepEval test for the graph, register it in the tier matrix, and re‑run the gate to enforce the 0.80 bar.

“Deep‑only” alert raised but not treated as a hard failure

• Trigger – A graph has std_accuracy < AGGREGATE_PASS and deep_accuracy >= AGGREGATE_PASS, causing deep_only = (not std_ok) and deep_ok to be True.
• Guard – The if deep_only: branch adds the graph to deep_only_graphs. At the summary, if gate_passed is True and deep_only_graphs is non‑empty, a “PASSED (with deep‑only alerts)” message is printed instead of a failure.
• Posture – fail‑soft: the gate passes, but the operator is warned via a non‑blocking informational message.
• Operator signal – Per‑graph line [O69] DEEP-ONLY: {graph} std={std_acc:.1%} < {AGGREGATE_PASS:.0%} deep={deep_acc:.1%} ≥ {AGGREGATE_PASS:.0%} delta={delta:+.4f}.
• Recovery – The operator should investigate why the standard tier fails and improve the routing or prompt to make the cheaper tier also pass the bar. No automatic retry.

The make_generation_metrics and make_extraction_metrics functions build reusable evaluation metric lists with a shared judge model.

def make_generation_metrics(judge=None, *, with_faithfulness=True, strict=False):
    from deepeval.metrics import AnswerRelevancyMetric, FaithfulnessMetric
    j = judge if judge is not None else make_judge()
    faith_threshold = THRESHOLD_STRICT if strict else THRESHOLD_DEFAULT
    metrics = []
    if with_faithfulness:
        metrics.append(FaithfulnessMetric(threshold=faith_threshold, model=j, include_reason=True))
    metrics.append(AnswerRelevancyMetric(threshold=THRESHOLD_DEFAULT, model=j, include_reason=True))
    return metrics

def make_extraction_metrics(judge=None):
    from deepeval.metrics import ContextualPrecisionMetric, FaithfulnessMetric
    j = judge if judge is not None else make_judge()
    return [
        FaithfulnessMetric(threshold=THRESHOLD_STRICT, model=j, include_reason=True),
        ContextualPrecisionMetric(threshold=THRESHOLD_DEFAULT, model=j, include_reason=True),
    ]

Imagine a teacher who never memorizes answer keys but instead reads every student essay, checks that each fact in the essay can be traced back to the textbook, and confirms the essay actually answers the question. That is what this subsystem does: it reuses one consistent teacher—a language model—to grade every open-ended answer the system produces, ensuring both truthfulness and relevance.

The teacher—built once by _build_deepseek_judge()—wraps the team’s own language model so that DeepEval’s grading tools can call it. For each test, the system selects the right rubric: make_generation_metrics returns two scoring guides—FaithfulnessMetric checks every claim against the provided source documents, while AnswerRelevancyMetric checks the answer addresses the prompt. The faithfulness score has a default passing bar of 70%, but for critical tasks like outreach or data extraction, the system raises it to 85% via the strict flag. The relevancy score always uses the default bar. All tests together must clear an 80% overall pass rate—the ≥0.80 gate. The same teacher is reused across every graph test, keeping evaluation consistent.

The trickiest rule is the stricter threshold for faithfulness. The source shows faith_threshold = THRESHOLD_STRICT if strict else THRESHOLD_DEFAULT. Why 85% for extraction? Because if a outreach agent hallucinates a single extracted detail—like a company’s industry—the whole pitch derails. The threshold demands that nearly every claim be grounded. Without this subsystem, there would be no way to catch hallucinations; a user would receive confident but fabricated answers, eroding trust. The concrete failure: a seller confidently asserts a prospect uses a specific technology based on nothing, leading to an awkward, trust-breaking call.

The deep evaluation judge is materialised once by make_judge() and then reused across every generation graph through the make_generation_metrics() factory. In the ordered mechanism, classification first happens: each *_graph.py is assigned to a coverage bucket (covered, no-llm, delegates, or exempt) by static source analysis in run_evals.py. Only graphs that contain an LLM call site and are not explicitly delegated or exempt are required to clear the DeepEval gate. For those, make_generation_metrics(judge, with_faithfulness=True, strict=False) produces a FaithfulnessMetric (threshold default 0.7, raised to THRESHOLD_STRICT = 0.85 for critical graphs) and an AnswerRelevancyMetric (always at THRESHOLD_DEFAULT). Evaluation runs on each test input; if any metric falls below its threshold, or if the aggregate pass rate across the whole DeepEval suite drops below 0.80, the gate fails.

The core invariant is the “≥ 0.80 aggregate bar” – the entire DeepEval suite must pass at 80% accuracy or higher. This is enforced by the gate runner in run_evals.py, which explicitly fails when the suite drops below that bound. Additionally, for high‑risk graphs (e.g. university_staff_graph in the _UK_UNIVERSITIES_GATED_GRAPHS registry), the graph must both exist and be DeepEval‑gated; the registry ensures that removing the eval while keeping a structural test triggers a loud failure rather than silent under‑coverage. Faithfulness metrics themselves have a floor of 0.7 (default) or 0.85 (strict), but the overall invariant is the aggregate rate.

The trade‑off is that a single, shared wrapper with fixed thresholds is used instead of per‑graph customisation. The obvious rejected alternative is to hand‑craft thresholds and metrics for every graph – which would be brittle, require constant manual tuning, and reintroduce the “grepping by hand” the script explicitly avoids. By centralising the judge and standardising thresholds, the team gains consistency and self‑maintenance: adding a new graph automatically appears as MISSING until its eval is written, and the 0.80 gate smooths over individual judge variance. The cost avoided is the maintenance burden of a hand‑kept coverage list and the risk of silent quality regression when a graph’s eval is accidentally removed while its structural test still passes.

One concrete failure mode occurs with university_staff_graph. If a developer removes the DeepEval test (test_university_staff_graph_deepeval.py) but keeps the structural unit test (test_university_staff_graph.py), the source‑analysis heuristic would mark the graph as covered (because some test imports it). However, the _UK_UNIVERSITIES_GATED_GRAPHS registry explicitly requires both presence and gating – the gate runner detects that the graph is not DeepEval‑gated and fails loudly. The operator sees a message like Graph 'university_staff_graph' in gated registry but no DeepEval gate found (or similar in the coverage map output). Alternatively, if the aggregate pass rate dips below 0.80, the gate prints a failure line indicating the suite dropped below the bar, with no individual metric breakdown – forcing the operator to inspect the per‑graph DeepEval results.

Q1 (warm-up)

Q — How does the subsystem create a judge to evaluate graph outputs, and which function containers the standard metric list for generation graphs?

A — The helper make_judge() is called inside make_generation_metrics() to instantiate a single judge model, then that judge is passed into both FaithfulnessMetric and AnswerRelevancyMetric. The function returns [FaithfulnessMetric, AnswerRelevancyMetric] by default.

Follow-up — What determines whether a generation graph uses two metrics or just one?
A — The with_faithfulness parameter: make_generation_metrics(…, with_faithfulness=True) adds the faithfulness metric; passing False returns only [AnswerRelevancyMetric] (e.g., for free‑generation graphs like email_compose_graph).

Weak answer misses — A shallow answer would omit that the strict parameter raises the FaithfulnessMetric threshold to THRESHOLD_STRICT (0.85) while AnswerRelevancyMetric always uses THRESHOLD_DEFAULT (0.7).

Q2 (warm-up)

Q — How does the subsystem validate that the judge is capable of detecting hallucinations before the actual graph gate runs?

A — The function run_faithfulness_probe() executes the FaithfulnessMetric on two canned variants: a clean_output (all claims grounded in the probe’s retrieval_context) and a hallucinated_output (claims not in the context). It returns score, success, and reason for each, confirming the judge can separate faithful from hallucinated answers.

Follow-up — What is the difference between a strict and a default probe invocation?
A — With strict=True the FaithfulnessMetric uses THRESHOLD_STRICT (0.85); with strict=False it uses THRESHOLD_DEFAULT (0.7). The source comment says strict mode is “appropriate for outreach” tasks.

Weak answer misses — A weak answer would not mention that the probe relies on the predefined dictionaries OUTREACH_FAITHFULNESS_PROBE and OUTREACH_HALLUCINATION_PROBE (the latter is named but shown as a literal in the context; the actual identifier is OUTREACH_HALLUCINATION_PROBE in the source snippet, though the context shows a hallucinated_output key inside a larger dict; the exact probe name is OUTREACH_HALLUCINATION_PROBE per the source header comment).

Q3 (design question)

Q — Why did the team design the judge as a single shared wrapper reused across every graph test, rather than letting each metric create its own judge instance?

A — All metric‑factory functions (make_generation_metrics, make_extraction_metrics) accept an optional judge parameter; if not provided they call make_judge() once. This pattern ensures consistent model configuration (same model, same prompt template) across all metrics in a suite, preventing evaluation drift. The source shows j = judge if judge is not None else make_judge(), then passes model=j into every metric.

Follow-up — Isn’t that a single point of failure if the judge model goes down? How is the system still evaluated?
A — The evaluation mode uses a _NoopConn database stub and delegates all judgment to the external judge model; there is no fallback, so a model outage would block the gate. The design prioritises consistency and centralised control over resilience.

Weak answer misses — A shallow answer would fail to mention that extraction graphs use a different metric list ([FaithfulnessMetric(strict), ContextualPrecisionMetric]) but still call the same make_judge() to create the judge.

Q4 (hard)

Q — The system enforces an overall pass rate of ≥0.80 for the DeepEval suite. How does run_evals.py determine which graphs must have an eval and which are exempt?

A — run_evals.py classifies each *_graph.py into one of four buckets by scanning the source: covered (any test imports it), no-llm (no make_llm/.invoke call sites), delegates (explicitly listed wrappers of covered graphs), or exempt (escape hatch with written reason). Only covered graphs run through the DeepEval gate; no-llm and delegates are considered judged by delegate and thus exempt from the ≥0.80 bar.

Follow-up — If a graph is misclassified as no-llm because the heuristic missed an LLM call hidden in a helper, how does the system catch that?
A — The classification is derived from the source at runtime: the script greps for call sites such as make_llm and .invoke. If a helper function calls an LLM but the graph module only calls that helper (not the LLM directly), the heuristic could miss it. The exempt escape hatch exists for manual override, but no automated cross‑check is shown.

Weak answer misses — A weak answer would omit that the coverage map is self‑maintaining (“adding a graph automatically shows up as MISSING”), meaning the classification is not a static list but computed from the file system.

Q5 (hard)

Q — How does the subsystem handle the evaluation of extraction graphs differently from generation graphs, and why?

A — The function make_extraction_metrics() returns [FaithfulnessMetric(threshold=THRESHOLD_STRICT), ContextualPrecisionMetric]. The stricter faithfulness threshold (0.85) ensures no hallucinated extractions, and ContextualPrecisionMetric measures whether the highest‑ranked extracted nodes are actually relevant to the query (precision over recall). Both require retrieval_context populated with source document chunks.

Follow-up — Why is ContextualPrecisionMetric used only for extraction, not for generation?
A — The source comment says it measures “that the extracted/retrieved nodes ranked highest are actually relevant to the query”. In generation graphs the output is free‑form text, not ranked node selections, so precision of a node ranking is not applicable.

Weak answer misses — A shallow answer would not reference the exact metric names (ContextualPrecisionMetric) or the fact that make_extraction_metrics forces the strict threshold for faithfulness, unlike generation where it’s optional via the strict parameter.

Judge Model API Timeout or Connection Failure

• Trigger — The underlying language model API (called inside make_judge()) is unreachable, returns a timeout, or exceeds its rate limit during a DeepEval metric computation (e.g., FaithfulnessMetric or ContextualPrecisionMetric using model=j).
• Guard — No exception handler, retry, or fallback is visible in the provided source. The code simply calls the judge and assumes it succeeds; an uncaught requests.HTTPError or asyncio.TimeoutError would propagate up the call stack.
• Posture — fail‑hard – the entire eval run aborts because no error recovery exists.
• Operator signal — An unhandled Python traceback ending with something like requests.exceptions.ConnectionError: [Errno 111] Connection refused is printed to stderr; no structured log entry is written.
• Recovery — Manual intervention: the operator must restart the eval after the API is available again. There is no retry count or backoff in the source.

Malformed or Non‑numeric Judge Score

• Trigger — The judge model returns a string, null, or a value that cannot be coerced to float (e.g., "excellent" instead of 0.95). This would occur inside the matrix-iteration loop of gate() when float(entry.get("std", 0.0) or 0.0) is called.
• Guard — No explicit validation or exception handler for ValueError or TypeError is shown. The conversion is performed raw, and a malformed value would raise an exception.
• Posture — fail‑hard – the gate computation aborts because the float() call fails.
• Operator signal — A traceback such as ValueError: could not convert string to float: 'excellent' printed to stderr. The results list may be incomplete.
• Recovery — Manual: fix the judge’s output format or add a wrapper that coerces scores and catches exceptions (not present in the source).

Threshold Violation Causing Gate Failure

• Trigger — One or more graphs in the tier matrix have std_accuracy or deep_accuracy below AGGREGATE_PASS (0.80). For example, a faithfulness metric using THRESHOLD_STRICT (0.85) may consistently score below 0.80.
• Guard — The condition std_ok = std_acc >= AGGREGATE_PASS and deep_ok = deep_acc >= AGGREGATE_PASS catches this. Graphs that fail either check are appended to std_failures or deep_failures. The variable hard_failures collects both, and gate_passed is set to False when len(hard_failures) > 0.
• Posture — fail‑hard – the gate() function returns False and prints a FAILED summary, aborting the run with exit code 1 (from the script’s main).
• Operator signal — The log line [O69-tier-matrix] FAILED — <N> graph(s) below ≥80%: <graph_names>.
• Recovery — Manual: improve the graph’s accuracy until it clears the bar. There is no automatic retry.

Deep‑Only Flag (Non‑blocking Alert)

• Trigger — A graph has std_accuracy below AGGREGATE_PASS but deep_accuracy at or above AGGREGATE_PASS. This is allowed but flagged as “deep‑only”.
• Guard — The boolean deep_only = (not std_ok) and deep_ok is computed. Graph names are appended to the list deep_only_graphs. The gate still passes as long as no graph fails both tiers (hard_failures is empty).
• Posture — fail‑soft – the gate passes, but two different summary lines are printed: one for deep‑only graphs and one for the overall PASSED (with deep-only alerts).
• Operator signal — The line [O69] DEEP-ONLY: <graph> std=…% < 80% deep=…% ≥ 80% per graph, plus the final summary [O69-tier-matrix] PASSED (with deep-only alerts) — <N> graph(s) rely on deep tier only: <names>.
• Recovery – Automatic: the run continues normally. The operator should investigate the routing mismatch, but no retry or fallback is triggered.

Empty Tier Matrix (No Graphs Registered)

• Trigger — The fixture file _O69_TIER_MATRIX_FIXTURE does not exist (or is empty) and _O69_DEFAULT_MATRIX is also empty. The matrix = [] branch is taken.
• Guard – The explicit check if not matrix: prints [O69-tier-matrix] DEFERRED — empty tier matrix (no graphs registered) and returns True with status deferred.
• Posture — fail‑soft – the gate reports deferred and passes (returns True), so the overall script continues.
• Operator signal – The log line [O69-tier-matrix] DEFERRED — empty tier matrix (no graphs registered).
• Recovery – Automatic: the run proceeds; the operator later populates the matrix. No retry needed.

Judge Initialization Failure (make_judge)

• Trigger – make_judge() (called inside make_extraction_metrics or directly by DeepEval metrics) fails because of missing API key, invalid model name, or authentication error.
• Guard – No try/except is shown around the judge creation. The source does not define any fallback or validation.
• Posture – fail‑hard – an unhandled exception (e.g., deepeval.utils.ApiKeyError) propagates, aborting the eval.
• Operator signal – A Python traceback mentioning make_judge or the underlying SDK error is printed.
• Recovery – Manual: correct the environment variables (e.g., CF_AIG_TOKEN) or model name and re‑run. No automatic retry or fallback exists.

Trajectory coverage evaluator for pipeline_graph.

def pipeline_trajectory_coverage(run: Any, example: Any) -> dict[str, Any]:
    run_outputs = (
        run.outputs if hasattr(run, "outputs")
        else (run.get("outputs") or {}) if isinstance(run, dict)
        else {}
    )
    example_outputs = (
        example.outputs if hasattr(example, "outputs")
        else (example.get("outputs") or {}) if isinstance(example, dict)
        else {}
    )
    actual = run_outputs.get("actual_trajectory") or []
    expected = example_outputs.get("expected_trajectory") or []
    if not expected:
        return {"score": 1.0, "comment": "no expected_trajectory in example"}
    actual_set = set(actual)
    matched = sum(1 for step in expected if step in actual_set)
    score = matched / len(expected)
    missing = [s for s in expected if s not in actual_set]
    comment = (
        f"coverage={score:.2f}  matched={matched}/{len(expected)}"
        + (f"  missing={missing}" if missing else "  all steps present")
    )
    return {"score": score, "comment": comment}

Imagine you have a map of a city with only certain roads and landmarks drawn on it—everything outside that map is off-limits. When you follow a driver’s recorded route, you check that every street they used appears on that map and that they didn’t invent a shortcut through an unmarked field. That is exactly what this trajectory evaluation subsystem does for a graph: it verifies that the actual path a graph runs through its nodes and tools matches a golden, expected route—no made-up stops or fictional shortcuts allowed.

Here’s how it works in practice. Test cases define exactly which “nodes” (like landmarks or companies) and which “edges” (the relationships between them) are allowed—these are listed in fields like allowed_entity_nodes and allowed_edges. When the graph runs, two metrics from the _deepeval_utils module check the route: FaithfulnessMetric (with a strict threshold) ensures that every piece of the output is actually backed by a piece of text in the source documents—like verifying a landmark name appeared on the map before you say you visited it. ContextualPrecisionMetric checks that the highest-ranked nodes are genuinely relevant to the query, not just noise. Both metrics require a retrieval_context filled with the original source chunks, so they stay grounded.

The trickiest rule is the “Grounding-First contract” seen in fixtures like _RECOMMEND_SUBGRAPH_LOOKALIKE: every recommendation must cite a node ID that exists in the subgraph, and every step in its explanation path must use only edges that appear in that same subgraph. If the subgraph is empty (as in fixture-recommend-empty-subgraph-failopen), the system fails open to zero recommendations—no made-up output allowed. Without this subsystem, a graph could confidently recommend a path referencing a nonexistent entity or a relationship that doesn’t actually exist, leading to a rep trying to contact a company that was never in the database—a concrete waste of time and trust.

The trajectory evaluation subsystem operates through a deterministic ordered mechanism. First, when a graph runs in recommend mode, it is invoked with a pre‑built subgraph—for example, _RECOMMEND_SUBGRAPH_LOOKALIKE—which seeds the KG‑retrieval deterministically, keeping the evaluation hermetic and DB‑free. Next, the output must contain a recommendation that cites a node id present in that subgraph, and an explanation_path whose edges all exist in the subgraph. On failure—if the cited node is missing or the path includes an edge not in the subgraph—the evaluation metrics detect the violation. The standard metrics from make_extraction_metrics (FaithfulnessMetric at THRESHOLD_STRICT and ContextualPrecisionMetric) require a retrieval_context populated with the source document chunks; here the subgraph nodes and edges serve as that context, so a mismatch immediately lowers the faithfulness or precision score below the gate threshold.

The invariant this design preserves is the load‑bearing Grounding‑First contract: every returned recommendation must be grounded to a node id present in the subgraph and an explanation_path whose edges all exist in the subgraph. No output may reference entities or relationships outside the provided synthetic graph. This guarantee is enforced by the combination of a strict FaithfulnessMetric (every extracted item must be backed by text in the subgraph) and ContextualPrecisionMetric (the highest‑ranked nodes must be relevant to the query). Together they ensure that each step of the trajectory is verifiably derived from the allowed entity set, mirroring the real‑world requirement that recommendations are built only from known nodes and edges.

The key trade‑off is the rejection of a live KG database in favor of a pre‑built, fictitious subgraph for evaluation. The obvious alternative—running against the production graph database—would introduce non‑determinism, database dependencies, and the risk of flaky tests due to data drift or PII exposure. By using a synthetic fixture like _RECOMMEND_SUBGRAPH_LOOKALIKE, the subsystem avoids the cost of maintaining a test database, eliminates connectivity failures, and makes every test run perfectly repeatable. The price is that the eval only exercises reasoning over a small fixed set of nodes and edges, but the grounding mechanism itself is exhaustively tested in a hermetic environment.

A concrete failure mode occurs when the graph outputs a recommendation citing node id 99 (which is absent from the subgraph). The FaithfulnessMetric at THRESHOLD_STRICT will flag this as unfaithful because the extracted node is not backed by any text in the subgraph document. An operator monitoring the DeepEval suite would see the FaithfulnessMetric score drop below the strict threshold (e.g., 0.8), and the gate script (run_evals.py) would report that the graph’s eval failed. The same violation would appear in the test runner as a failing assertion that the explanation_path cannot be constructed from edges present in _RECOMMEND_SUBGRAPH_LOOKALIKE.

Q – What function assembles the standard evaluation metrics for extraction nodes in a graph trajectory, and what two metrics does it return?
A – The function make_extraction_metrics(judge) in _deepeval_utils.py builds a list containing FaithfulnessMetric with THRESHOLD_STRICT and ContextualPrecisionMetric with THRESHOLD_DEFAULT. Both metrics are instantiated with include_reason=True and use a judge model obtained from make_judge() if none is provided.
Follow-up – Why must every extraction metric’s retrieval_context be populated?
A – Both FaithfulnessMetric and ContextualPrecisionMetric require an LLMTestCase that has a retrieval_context list containing the source document chunks; without it the metrics cannot verify grounding or relevance.
Weak answer misses – The detail that ContextualPrecisionMetric focuses on the ranking of nodes (precision over recall), not just binary relevance.

Q – How does the system distinguish between generation-graph evaluation and extraction-graph evaluation, and what threshold difference marks that distinction?
A – make_generation_metrics returns [FaithfulnessMetric, AnswerRelevancyMetric] (optionally without faithfulness for free-generation graphs), while make_extraction_metrics returns [FaithfulnessMetric(strict), ContextualPrecisionMetric]. The extraction-specific FaithfulnessMetric uses THRESHOLD_STRICT (a higher bar) to enforce every extracted claim must be textually grounded; generation graphs use a default threshold by default.
Follow-up – In the test fixtures for a recommend subgraph, what exact fields define the expected trajectory?
A – The fixtures in langsmith_datasets.py include allowed_entity_nodes (e.g., ["company:42", "company:77"]) and allowed_edges (e.g., "company:42|looks_like|company:77") in the outputs dictionary, which the eval harness compares against the actual path.
Weak answer misses – That make_generation_metrics also accepts a strict flag that raises the faithfulness threshold to THRESHOLD_STRICT for hallucination-critical generation.

Q (design question) – Why does make_extraction_metrics apply THRESHOLD_STRICT to FaithfulnessMetric but THRESHOLD_DEFAULT to ContextualPrecisionMetric—why not the same threshold for both?
A – Extraction nodes must never hallucinate; every extracted item must be 100% grounded in the source document, so faithfulness demands a strict threshold. Contextual precision measures how well the system ranks relevant nodes higher than irrelevant ones—a softer, gradual property where a default threshold is sufficient to catch regressions without over‑penalizing reasonable ranking variations. This design mirrors the principle that fabrication is unacceptable, while ranking precision can tolerate minor noise.
Follow-up – How does the evaluation framework ensure that a borderline extraction (near the threshold boundary) still triggers a reasoned failure?
A – Both metrics are created with include_reason=True, so the judge model returns a textual explanation alongside the binary pass/fail, allowing the test runner to diagnose near‑threshold cases without retuning the threshold itself.
Weak answer misses – The fact that make_judge() is called when no judge is passed, defaulting to a standard evaluation model that could be swapped for domain‑specific judges.

Q – When a graph’s trajectory includes a “golden expected” path (defined by allowed entity nodes and edges), what mechanism actually scores whether the realized path matches that expectation?
A – The evaluation code (excerpts from run_evals.py) implements ordering checks where a “stronger” candidate must score higher than a “weaker” one, and a live‑shape spread check that requires at least a 0.10 difference (max - min) over a subset of entities to guard against a constant‑score regression. These checks operate on the scores produced by the two metrics (faithfulness and contextual precision), which are computed per extraction step in the trajectory.
Follow-up – What happens in the test fixture fixture-recommend-empty-subgraph-failopen when the subgraph has no nodes or edges?
A – The expected output "expected_recommendation_count": 0 indicates a fail‑open path that skips the LLM call entirely, meaning no extraction metrics are even applied because there are zero recommendations to evaluate.
Weak answer misses – That the ordering checks require both the stronger and weaker score to exist; if one is missing the check records "passed": False with reason "score_error".

Q – How does the concept of “strict” vs. “default” threshold manifest in the actual test datasets for a borderline qualification example?
A – In the QUALIFY_EXAMPLES fixture qualify-fit-ai-eng-lead, the expected output includes "min_confidence": 0.6, which is a floor on the model’s confidence score. This is a different notion of “strictness” from the metric thresholds; it gates the overall qualification verdict at the graph level, while the metric thresholds (e.g., THRESHOLD_STRICT for extraction faithfulness) operate per evaluation step. The two layers together enforce that both the extracted facts and the final qualification decision meet separate confidence bars.
Follow-up – What field in that same fixture ensures the LLM’s reasoning touches specific concepts?
A – The "must_mention_any_of": ["ai", "agent", "engineering", "ml"] field is used by the evaluator to check that the model’s output contains at least one of those terms, acting as a soft lexical constraint on the trajectory’s output.
Weak answer misses – That the THRESHOLD_STRICT constant applies to the FaithfulnessMetric’s threshold parameter, not to the graph‑level confidence floor; they are separate mechanisms.

Failure 1: Faithfulness (std) Accuracy Falls Below 0.80

• Trigger — The graph extracts an entity or relationship that is not textually grounded in the source document. This causes std_acc to drop below AGGREGATE_PASS (0.80).
• Guard — The variable std_ok is set to False when std_acc < AGGREGATE_PASS. The graph name is appended to the list std_failures, making it a hard failure.
• Posture — fail-hard — The overall gate (gate_passed) becomes False because hard_failures (which includes std_failures) is non‑empty. The script exits with code 1.
• Operator signal — The console prints [O69] FAIL std: <graph> std=… < 0.80 deep=… delta=… and the final summary lists <graph> in the std_failure_graphs field.
• Recovery — No automated retry or fallback. The operator must inspect the extracted item against the source document, fix the graph (or the golden fixture), and re‑run the eval.

Failure 2: Contextual Precision (deep) Accuracy Falls Below 0.80

• Trigger — The highest‑ranked nodes in the extracted trajectory are not relevant to the query, so deep_acc drops below AGGREGATE_PASS.
• Guard — deep_ok is set to False when deep_acc < AGGREGATE_PASS. The graph is added to deep_failures, a hard‑failure list.
• Posture — fail-hard — The gate fails (gate_passed = False). The evaluation run exits with code 1.
• Operator signal — The line [O69] FAIL deep: <graph> std=… deep=… < 0.80 delta=… appears, and the summary lists <graph> in deep_failure_graphs.
• Recovery — No retry. The operator must examine the ranking order, adjust the retrieval context or the relevance model, and re‑evaluate.

Failure 3: Both Metrics Fail, but Only the std Failure Is Reported as a Hard Failure

• Trigger — The graph’s extracted entities are unfaithful (low std_acc) and also poorly ranked (low deep_acc). Because the code checks std_ok before deep_ok, the branch if not std_ok fires, adding the graph only to std_failures; the deep failure is never added to deep_failures.
• Guard — No guard exists for this scenario; the logic masks the deep failure. The variable deep_only is False because deep_ok is also False, so the “deep‑only” branch is not taken. The result is that hard_failures is non‑empty (due to the std failure), so the gate still fails — but the deep failure is invisible in the summary’s deep_failure_graphs list.
• Posture — fail-hard (the gate fails), but with silent degradation of diagnostic information.
• Operator signal — The console prints only [O69] FAIL std: <graph> …; there is no deep‑failure line. The delta value appears (likely negative), but the operator may not realise a deep miss also occurred.
• Recovery — The operator must compare the deep_acc and delta fields in the JSON output (or inspect the raw results) to discover the deep failure manually. No automated recovery exists.

Failure 4: “Deep‑Only” Scenario — std Fails but deep Passes

• Trigger — Standard accuracy drops below 0.80 while deep accuracy remains ≥0.80. The graph’s faithfulness is weak, but its relevance ranking is acceptable.
• Guard — The variable deep_only is set to True (derived from the graph’s properties). The code explicitly treats this as not a hard failure: std_failures and deep_failures are not appended. The graph is added to deep_only_graphs.
• Posture — fail‑soft — The gate passes overall (gate_passed = True), but a warning is emitted. The final summary includes a note: “PASSED (with deep‑only alerts) — …”.
• Operator signal — The line [O69] DEEP-ONLY: <graph> std=… < 0.80 deep=… ≥ 0.80 delta=… is printed. The summary lists the graph in the deep_only_graphs list.
• Recovery — No automated recovery is performed; the operator is alerted that the graph relies solely on the deep tier and may need a faithfulness improvement.

Failure 5: Retrieval Context Is Empty or Malformed

• Trigger — The retrieval context (source document chunks) supplied to the faithfulness and contextual precision metrics is missing, incomplete, or corrupt. Both std_acc and deep_acc are computed as 0 or very low.
• Guard — The source code contains no explicit guard for missing or invalid context. The metrics will silently produce low accuracy scores, which then trigger either a std or deep failure (or both). The deep_only logic does not apply because both tiers fail. The graph becomes a hard failure.
• Posture — fail‑hard — The gate fails. However, the root cause (context issue) is not distinguished from a genuine model regression.
• Operator signal — The console shows a failure message (e.g. [O69] FAIL std: … or [O69] FAIL deep: …). The deep_acc and std_acc are both near zero, but the logs do not explicitly say “context missing”.
• Recovery — The operator must examine the fixture data pipeline, ensure the golden set includes valid source chunks, and re‑run. No retry is attempted.

Failure 6: Threshold Drift Between run_evals.py and _eval_utils

• Trigger — The constant AGGREGATE_PASS in run_evals.py is set to 0.80, mirroring _eval_utils.AGGREGATE_PASS. If the utilities module is updated to a different threshold (e.g. 0.85) but run_evals.py is not, the gate enforces the old 0.80 bar while the evaluation logic uses the new bar. The result is either false passes or false failures.
• Guard — No guard detects this mismatch. The two constants are defined independently; there is no code to assert they are equal.
• Posture — fail‑soft in the sense that the gate runs, but its decision may be inconsistent with the intended OPTIMIZATION‑STRATEGY bar. The script exits with code 0 or 1 depending on which threshold is stricter.
• Operator signal — No direct signal. The threshold field in the output dict contains the run_evals.py value (0.80), but the operator would need to manually compare it with the utility’s threshold.
• Recovery — The operator must synchronise the two constants manually. No automated recovery exists. A linting rule or runtime assertion would be required to prevent this failure.

Standard generation metrics setup: faithfulness for retrieval graphs, answer relevancy for all, with a stricter threshold option for critical tasks.

def make_generation_metrics(
    judge: Any = None,
    *,
    with_faithfulness: bool = True,
    strict: bool = False,
) -> list[Any]:
    from deepeval.metrics import AnswerRelevancyMetric, FaithfulnessMetric

    j = judge if judge is not None else make_judge()
    faith_threshold = THRESHOLD_STRICT if strict else THRESHOLD_DEFAULT
    metrics: list[Any] = []
    if with_faithfulness:
        metrics.append(
            FaithfulnessMetric(threshold=faith_threshold, model=j, include_reason=True)
        )
    metrics.append(
        AnswerRelevancyMetric(threshold=THRESHOLD_DEFAULT, model=j, include_reason=True)
    )
    return metrics

Imagine a safety inspector walking through a factory with a clipboard, checking every machine that could potentially produce a dangerous product. The coverage gate is exactly that clipboard: a list that shows which machines—here, "graphs" that generate text or extract data—have been tested for quality and which have not. This system exists so that every machine that uses a smart model (the part that can make things up) must have a written test before it is allowed to run in production. The clipboard is not written by hand; it reads the factory floor automatically, flagging any new machine that calls a model as "missing a test" and demanding someone write one.

When the inspector finds a machine that generates text, the standard test checks two things: faithfulness (every claim the machine makes must be traceable back to the original source materials) and answer relevancy (the output must actually respond to the question it was given). These are real checks—functions like FaithfulnessMetric and AnswerRelevancyMetric—and they live in the toolbox file _deepeval_utils.py. For machines that extract facts instead of generating prose, the tests are stricter: strict faithfulness bans any made-up extractions entirely, and contextual precision ensures that only the most relevant chunks are pulled from the source. These come from make_extraction_metrics.

The trickiest part is that the checklist is self-updating. If an engineer adds a new machine that uses a model, it automatically appears as missing on the clipboard—even if the engineer forgot to mention it. This prevents silent gaps. The code also has escape hatches for machines that are just orchestrators or delegates, but every exception must include a written reason. Without this coverage gate, a new graph could slip into production with no quality check. The result would be recommendations that cite fake nodes or emails that answer the wrong question, and the beginner would first notice when customers complain about weird, untrustworthy responses that no one tested.

The coverage gate subsystem in run_evals.py applies a deterministic, source-derived classification to every *_graph.py module, then enforces a quality invariant. The ordered mechanism begins when a developer invokes python scripts/run_evals.py --coverage (or the default mode which runs both coverage and gate). First, the script scans the source tree for all graph modules and, for each, applies a set of heuristics—most critically, the regex re.compile(r"\bllm\.(a?invoke|a?stream|bind\w*)\b") and a companion pattern—to decide whether the graph contains an LLM call site. Each graph is then placed into exactly one of five buckets: covered, no-llm, delegates, exempt, or missing. After the full matrix is built, two explicit registries are checked: _check_uk_universities_coverage walks _UK_UNIVERSITIES_GATED_GRAPHS and fails if any declared graph is absent or not gated (i.e., lacks a DeepEval test); _check_activation_pipeline_coverage verifies every graph in _ACTIVATION_PIPELINE_GRAPHS is non‑missing. If any graph lands in the missing bucket—meaning it calls an LLM but has no eval and no exemption—the gate exits with code 1, printing each offending module and its reason.

The core invariant the design preserves is the “Eval-First” pillar: every prompt–model change must clear a ≥0.80 accuracy bar (AGGREGATE_PASS = 0.80). The coverage gate ensures this by requiring that every graph that emits model output (detected by the LLM call‑site heuristics) has an associated eval—either a structural test that imports it (counts as covered) or a DeepEval‑quality gate (for auto‑sending verticals). The stronger variant for uk‑universities demands the gated flag, meaning the graph must appear in the DeepEval runner suite, not merely be imported by a unit test. This invariant is maintained automatically as new graphs are added, because the classification comes from the source itself, not a hand‑kept roster.

The key trade‑off is self‑maintaining auto‑classification versus a manually curated list. The explicit alternative rejected is a static enum of “graphs‑with‑evals” that a developer would have to update by hand. That approach would silently miss new graphs that call an LLM, letting them slip through without any eval while the coverage report still shows green. By deriving the classification from source‑level heuristics, the gate forces a new graph to show up as missing until someone writes an eval or adds an explicit exemption in DELEGATES or to the exempt bucket. The cost of this choice is that heuristics can miscategorise—e.g., an orchestrator that only delegates to sub‑graphs might be flagged as no‑llm when it actually routes to LLM‑calling children—so the design includes the explicit DELEGATES dictionary (with written reasons like "pipeline_graph" pointing to "orchestrates discover/enrich/contacts/qa/outreach stage graphs; …") and exempt` escape hatches to correct those cases.

A concrete failure mode is adding a new graph candidate_scoring_graph.py that calls llm.invoke(...) and has no test file. When the operator runs python scripts/run_evals.py, the coverage report will place it in the missing bucket. The script then prints:
FAIL [coverage]: 1 graph(s) require eval coverage or exemption: followed by NOT FOUND/ NOT GATED: candidate_scoring_graph (calls an LLM or is unclassified — add tests/test_candidate_scoring_graph.py or DELEGATES entry). The exit code is 1, and the CI pipeline would reject the change. For the uk‑universities vertical, if someone deletes test_university_staff_deepeval.py but keeps the structural test, _check_uk_universities_coverage will report NOT GATED: university_staff_graph (auto‑sending vertical — needs a DeepEval ≥80% quality gate …), failing the build even though the graph is still “covered” in the generic sense. The signal an operator sees is the precise module name, its bucket, and the remediation hint—no guesswork.

Interview Q&A: The Coverage Gate

Pair 1 (Warm-up)

Q
When a new *_graph.py is added, how does the system determine whether it needs an eval?

A
The coverage map in run_evals.py scans every *_graph.py for LLM call sites using regex patterns like r"\bllm\.(a?invoke|a?stream|bind\w*)\b". If no such pattern is found, the graph is classified as no-llm and exempt from the eval bar. Otherwise, it’s checked against two explicit lists: DELEGATES (graphs that delegate judged behaviour to a covered sibling) and an explicit exempt bucket. Anything that doesn’t match those and has zero test imports becomes MISSING.

Follow-up
What happens if a graph does have LLM calls but also appears in DELEGATES?

Follow-up answer
It is classified as delegates — the judged behaviour is attributed to the delegate graph listed in DELEGATES (e.g. classify_recruitment_bulk_graph → classify_recruitment_graph).

Weak answer misses
Omission of the exact regex patterns (.invoke / .ainvoke / bind*) and the no-llm category; a shallow answer would just say “it checks if an LLM is used” without naming the source-scanning mechanism.

Pair 2 (Medium)

Q
For a generation graph that has a retrieval context, what two DeepEval metrics are standard and how are they configured?

A
The function make_generation_metrics returns [FaithfulnessMetric, AnswerRelevancyMetric] by default. FaithfulnessMetric verifies that every factual claim in the output is grounded in the provided retrieval_context; its threshold is THRESHOLD_DEFAULT unless strict=True is passed, which raises it to THRESHOLD_STRICT. AnswerRelevancyMetric checks that the output addresses the input prompt and always uses THRESHOLD_DEFAULT. Both include a reason.

Follow-up
What happens when with_faithfulness=False?

Follow-up answer
Only AnswerRelevancyMetric is returned — used for free-generation graphs like email_compose_graph that have no retrieval document to be faithful to.

Weak answer misses
Leaving out the include_reason=True parameter and the strict flag that changes only the faithfulness threshold; a shallow answer might say “faithfulness and relevancy” but not the default vs strict distinction.

Pair 3 (Hard – design question)

Q
Why does the UK‑universities vertical use an explicit _UK_UNIVERSITIES_GATED_GRAPHS registry instead of relying on the auto‑discovery coverage model?

A
Auto‑discovery marks a graph “covered” the moment any test imports it — a structural unit test like test_university_staff_graph.py would already satisfy the coverage check, silently masking the absence of a real quality gate (≥0.80 DeepEval). The explicit frozenset requires the graph to be present and have an actual DeepEval gate (i.e., be gated). If a future change removes the deepeval test but leaves the structural test, the gate fails loudly instead of quietly dropping the bar.

Follow-up
How does the coverage map treat UK universites differently from other graphs?

Follow-up answer
The registry forces the graph to be in the gated column; without it, the coverage map would show it as covered (from unit tests) and exempt it from the ≥0.80 gate — masking the missing evaluation.

Weak answer misses
Failing to mention that auto‑discovery uses test‑import as the sole criterion, and that structural tests alone would create a false sense of coverage. A shallow answer would say “it’s a safety list” but not explain why import-based coverage is insufficient.

Pair 4 (Hard)

Q
The V89 outreach faithfulness gate asserts claim‑support and hard‑negative absence. How does it operate without an LLM judge?

A
It is fixture‑based — it loads a JSON file (_V89_FIXTURE = "golden/outreach_faithfulness_v89.json") containing six entries, each with claim_evidence_pairs and hard_negatives. For every pair it checks: (1) the evidence_span is a substring of the supplied company_evidence (claim‑support), and (2) each unsupported_claim does not appear in company_evidence (hard‑negative absent). The aggregate pass rate across all entries must be ≥0.80.

Follow-up
What does this gate cover in the coverage matrix?

Follow-up answer
It surfaces the email_outreach_graph as gated by a faithfulness check, so the coverage matrix records the outreach graph as having a real quality bar.

Weak answer misses
Omitting the exact file path (outreach_faithfulness_v89.json) and the two‑part assertion logic (substring for supported claims, absence for hard negatives). A shallow answer might just say “it uses a fixture and checks faithfulness” without the concrete mechanism.

Pair 5 (Hard)

Q
What’s the practical difference between a graph being covered and being gated?

A
covered means at least one test imports the graph module — it could be a structural test, a node‑slice test, or a DeepEval test. gated specifically means a DeepEval suite passes the ≥0.80 aggregate bar on a relevant metric. The UK‑universities registry illustrates this: a graph can be covered by a unit test but still not clear the quality gate. The coverage map shows both columns so a reviewer can see at a glance whether coverage is merely structural or backed by accuracy guarantees.

Follow-up
How does the DELEGATES list blur that distinction?

Follow-up answer
A delegate graph (e.g. email_opportunity_graph) is marked covered because its judged behaviour lives in the delegated graph (email_compose_graph), which must itself be gated. The coverage map doesn’t re‑guard the delegate separately.

Weak answer misses
Failing to mention that covered is just a test‑import check, not a quality check; leaving out the UK‑universities registry as the canonical example of the gap. A shallow answer would conflate “has a test” with “has an eval gate.”

New Graph Missing an Evaluation

• Trigger — A developer adds a new *_graph.py file that contains an LLM call site (e.g., .invoke, make_llm, llm_* helper) but does not create any corresponding test file under tests/.
• Guard — The --coverage mode of run_evals.py classifies the graph as missing using the source-derived coverage logic; the script then exits with code 1.
• Posture — Fail‑hard: the coverage report terminates the run with a non‑zero exit code, blocking any pipeline that relies on a clean gate.
• Operator signal — A line like MISSING: <graph> printed in the coverage matrix, followed by exit(1) if --coverage is used.
• Recovery — The developer must write an evaluation test (e.g., a DeepEval suite) that imports the graph module, then re‑run the coverage check.

DeepEval Accuracy Drops Below the Aggregated Bar

• Trigger — A change to the graph’s prompt, model, or retrieval logic causes the standard or deep accuracy to fall below AGGREGATE_PASS = 0.80.
• Guard — The gate logic in run_evals.py compares each accuracy against AGGREGATE_PASS: the variables std_ok = std_acc >= AGGREGATE_PASS and deep_ok = deep_acc >= AGGREGATE_PASS; failure of either adds the graph to hard_failures.
• Posture — Fail‑hard: the gate sets gate_passed = False and the script exits with code 1, stopping the release.
• Operator signal — Lines like FAIL std: <graph> std=0.75 deep=0.85 or FAIL deep: <graph> are printed.
• Recovery — The changed graph must be reverted or improved until both accuracies meet AGGREGATE_PASS; the gate is re‑run.

Auto‑Sending Graph Not DeepEval‑Gated (UK Universities)

• Trigger — A graph listed in _UK_UNIVERSITIES_GATED_GRAPHS either does not exist in the module registry or exists but lacks a DeepEval test (i.e., cov.gated is False).
• Guard — The function _check_uk_universities_coverage iterates over the static list and returns 1 (fail) if any graph is absent or not gated.
• Posture — Fail‑hard: the function returns 1, which is intended to fail the overall coverage gate.
• Operator signal — A line such as NOT GATED: <graph> (auto‑sending vertical – needs a DeepEval ≥0.80 quality gate…) or NOT FOUND: <graph> printed to stdout.
• Recovery — A test file named tests/test_<graph>_deepeval.py must be created (or the graph added to the registry), then the coverage check is re‑run.

Deep‑Only Graph Silently Passes the Gate

• Trigger — A graph’s standard accuracy is below AGGREGATE_PASS but its deep accuracy is at or above the threshold, and the graph is correctly identified as deep_only.
• Guard — The deep_only flag, set when std_ok is false and deep_ok is true, explicitly excludes the graph from hard_failures (the comment states “deep-only is a flag, not a hard fail”).
• Posture — Fail‑soft: the gate passes (gate_passed = True) and a warning is printed. The run continues, but the standard accuracy deficiency is known.
• Operator signal — A line like DEEP‑ONLY: <graph> std=0.75 < 0.80 deep=0.90 ≥ 0.80 is printed.
• Recovery — The operator is expected to investigate the routing‑accuracy warning; if needed, they can manually revert or tighten the standard tier.

Graph Miscategorized as no-llm Due to Undetected LLM Call

• Trigger — A graph uses an LLM in a pattern not caught by the heuristics (e.g., a custom wrapper that does not contain .invoke, make_llm, or llm_*), so the source‑derived classification falsely labels it no-llm.
• Guard — None. The classification logic is derived entirely from static source analysis with no independent validation; no retry, fallback, or validation exists for this scenario.
• Posture — Fail‑soft: the graph is not reported as missing and the coverage gate passes, but no evaluation tests are enforced. The model output remains unjudged.
• Operator signal — There is no direct signal; the graph will simply appear in the coverage report as no-llm (e.g., not flagged as missing). The mistake becomes visible only through manual audit or downstream failures.
• Recovery — A developer must manually add the graph to the exempt list with a written reason, or update the heuristic detection to cover the new pattern.

The generation metrics factory with default and strict faithfulness thresholds and answer relevancy.

def make_generation_metrics(
    judge: Any = None,
    *,
    with_faithfulness: bool = True,
    strict: bool = False,
) -> list[Any]:
    from deepeval.metrics import AnswerRelevancyMetric, FaithfulnessMetric
    j = judge if judge is not None else make_judge()
    faith_threshold = THRESHOLD_STRICT if strict else THRESHOLD_DEFAULT
    metrics = []
    if with_faithfulness:
        metrics.append(
            FaithfulnessMetric(threshold=faith_threshold, model=j, include_reason=True)
        )
    metrics.append(
        AnswerRelevancyMetric(threshold=THRESHOLD_DEFAULT, model=j, include_reason=True)
    )
    return metrics

Imagine a quality control manager at a car factory who collects daily reports from the engine test, brake test, and paint test stations. This manager doesn’t run new tests—they just look at each station’s numbers. If the brake test says “all good” but actual cars keep having brake failures, the manager hits a big red alarm that stops the line until something is fixed. That manager is the feedback gate: it turns separate feedback reports into a single hard check that continuous integration can act on.

Stepping deeper, the gate runs every feedback dimension by reusing the exact reports from real scripts like feedback_email, feedback_applications, and feedback_replies. It rolls up “over‑confident calibration bands” from email and application feedback—where high reported confidence doesn’t match real positive rates. It also flags reply‑classifier classes that are chronically low‑confidence (from feedback_replies). When any of these contradictions between reported and actual outcomes surface, the gate exits with code 1, not 0. This turns a passive report into an enforceable pass/fail step.

The trickiest part is that the gate adds no new analysis—it literally calls the same functions (confidence_profile) and uses the same thresholds as the individual scripts. This design means the gate can never disagree with them; it only elevates a warning to a stop‑the‑build signal. Without it, a team might ignore early signs of classifier drift—like confidence staying high while real performance sinks—until a major failure reaches production. The concrete failure a beginner would feel: a customer receives a wildly wrong reply because the model’s confidence was never re‑checked, and no automated safety net caught it.

The feedback gate, implemented in feedback_gate.py, operates as an ordered enforcement step that runs after the individual feedback dimensions (email, applications, replies) have produced their reports. First, the collect function is called—it reuses the exact reports and thresholds from feedback_email, feedback_applications, and feedback_replies, aggregating any over‑confident calibration bands or chronically low‑confidence reply‑classifier classes into a list of flags and a separate list of source errors. Next, the gate evaluates these lists: if flags are present, it exits with code 1; if no flags but a source errored (e.g., D1 unreachable), it exits with code 2; only when both lists are empty does it exit with code 0. The --warn‑only flag suppresses the non‑zero exit for flags, turning enforcement into a soft warning.

The design preserves a consistency invariant: the gate can never disagree with the individual feedback scripts, because it does not introduce any new analysis or thresholds—it reuses the exact reports and dimensions those scripts already produce. This guarantees that the gate’s signal is always a roll‑up of the same ground truth that the reports display, eliminating any possible conflict between the monitoring layer and the enforcement layer.

The key trade‑off is turning a passive report into an actionable CI gate rather than a separate monitoring alert. The obvious alternative is a standalone observability system (e.g., Grafana alert) that sends a notification when calibration drifts. That approach is rejected because it would rely on human response and could be silently ignored; the feedback gate instead exits non‑zero inside continuous integration, making a miscalibration block a deployment pipeline. The cost avoided is the drift‑acceptance debt that accumulates when alerts are acknowledged but not acted upon—the gate forces a hard check that preserves the ≥0.80 quality bar that the repo’s other evals enforce.

A concrete failure mode is a source error such as an unreachable D1 database during the collect call. An operator would see exit code 2 (distinguishable from a real miscalibration’s exit code 1) and a logged error string from the errors list. The signal is unambiguous: the gate did not find a calibration problem but could not complete the check, so the infrastructure—not the classifier—needs attention. This separation of concern means the gate reliably distinguishes “model is miscalibrated” from “the check itself failed,” enabling quick triage in the deployment pipeline.

Q — What is the purpose of the feedback gate, and how does it differ from the other feedback scripts?
A — The feedback gate, defined in feedback_gate.py, enforces a hard check: it runs every feedback dimension (email, applications, replies) and exits non‑zero when production contradicts a classifier. Unlike the individual feedback_*.py scripts which only report, the gate gates – turning the analysis into a CI‑actionable signal. It prints a one‑screen digest and returns exit code 1 if any miscalibration is flagged, or exit code 2 if a source errors (e.g., D1 unreachable).
Follow-up — Why does the gate exit with code 2 instead of code 1 for source errors?
Answer — So the caller (cron/CI) can distinguish a real miscalibration from an infrastructure failure; the docstring explicitly states “distinguishable from a real miscalibration.”
Weak answer misses — The gate also supports a --warn-only flag that prevents non‑zero exit, but a shallow answer might omit the exact exit‑code semantics (0 clean, 1 flag, 2 error) documented in the file.

Q — Why does the feedback gate avoid performing its own analysis and instead reuse results from other scripts?
A — The gate adds no new analysis – it reuses the exact reports, dimensions, and thresholds that feedback_email, feedback_applications, and feedback_replies already produce. This is a deliberate design choice stated in the docstring of feedback_gate.py: “it reuses the exact reports, dimensions, and thresholds the individual scripts already produce, so the gate can never disagree with them.” Doing otherwise would risk inconsistency between the reporting and enforcement layers.
Follow-up — How does the gate obtain those reports at runtime?
Answer — It calls the confidence_profile function from feedback_common and imports the modules (import feedback_email as email, etc.) to invoke their top-level analysis.
Weak answer misses — The gate also rolls up two distinct kinds of flags: “over‑confident calibration bands” (from email and applications) and “chronically low‑confidence” reply‑classifier classes (from replies), not just a single check.

Q — Walk through the collect() function: what does it return and how does it differentiate between a clean run and a flagged one?
A — The function feedback_gate.collect(args) returns a tuple (flags, errors). flags is a list of dicts each describing one miscalibration; errors is a list of human‑readable strings for sources that failed (e.g., D1 unreachable). The gate then checks: if flags exist (and not --warn-only) → exit code 1; if a source errored but nothing was flagged → exit code 2; otherwise clean exit 0. Flags take precedence — exit 2 is returned only when errors and not flags. This design ensures the exit code directly reflects the content of collect()’s output, making the gate deterministic and testable.
Follow-up — Does collect() run the feedback dimensions in parallel or sequentially?
Answer — It runs them sequentially inside collect(): email.fetch_rows(...) + email.build_reports(...), then apps.fetch_rows(...) + apps.stamp_fit_verdict(...) + apps.build_reports(...), then replies.fetch_rows(...) fed through feedback_common.confidence_profile — each source wrapped in its own try/except so one failure doesn’t stop the others.
Weak answer misses — A shallow answer might overlook that the gate requires a --min-n threshold (default 10, defined in the gate’s own argparse) to avoid flagging on insufficient data; the docstring lists --min-n N as an argument.

Q — The gate is described as “enforceable” and meant for CI. How does it ensure that its thresholds and logic stay synchronized with the individual feedback scripts?
A — The gate explicitly reuses the same thresholds (e.g., gap‑threshold, low‑threshold defaults) and the same analysis functions from the individual scripts. It does not recompute or hardcode any new constants. The docstring states: “it reuses the exact reports, dimensions, and thresholds the individual scripts already produce, so the gate can never disagree with them.” The synchronization is at the function level, not the constant level: the gate calls the same build_reports / confidence_profile functions, passing in its own argparse defaults (--gap-threshold 0.25, --low-threshold 0.6).
Follow-up — What would happen if a developer added a new feedback dimension without updating the gate?
Answer — The gate would not check that dimension, since collect() only calls the three imported modules; it would silently ignore the new dimension until the gate’s imports are updated.
Weak answer misses — A shallow answer might forget that the gate also has --gap-threshold and --low-threshold overrides for runtime tuning, but the default values are defined locally in feedback_gate.py’s argparse, not inherited from the individual scripts.

1. D1 Database Unreachable (Feedback Source Error)

• Trigger – The database that feeds feedback_email, feedback_applications, or feedback_replies is unreachable (e.g., network outage, credentials expired).
• Guard – The collect function’s exception handler (not shown explicitly in provided source) appends a human-readable error string to the errors list. The gate then checks errors and calls sys.exit(2), as documented: “2 = a source errored (e.g. D1 unreachable)”.
• Posture – fail-hard – The gate exits non‑zero (code 2) and produces no calibration flags; the entire feedback gate aborts.
• Operator signal – The error string printed to stderr/stdout, e.g. “D1 unreachable”.
• Recovery – An operator must restore database connectivity and re‑run the gate. No retry is built in.

2. Over‑confident Calibration Band (Email or Applications)

• Trigger – feedback_email or feedback_applications computes a confidence band (e.g., 0.8–0.9) where mean confidence is high but the realised positive rate falls below the configured gap-threshold (default from confidence_profile).
• Guard – The comparison logic in confidence_profile (from feedback_common) and the per‑script threshold checks produce a flag dict; the flags list returned by collect contains it. The gate then exits with sys.exit(1) because flags are non‑empty.
• Posture – fail-hard – The gate exits code 1, blocking the CI pipeline (unless --warn-only is set).
• Operator signal – Printed flag detail, e.g. “confidence_gap 0.25 for band 0.8–0.9”.
• Recovery – The operator reviews the flagged dimension, adjusts thresholds, verifies data quality, or sets --warn-only to allow the gate to pass but still see the warning.

3. Chronically Low‑Confidence Reply Classifier

• Trigger – feedback_replies detects that a reply‑classifier class has a mean confidence below the low-threshold (e.g., 0.40).
• Guard – The low‑confidence check in feedback_replies (using the same confidence_profile logic) creates a flag; collect returns it in the flags list; the gate exits with sys.exit(1).
• Posture – fail-hard – Exit code 1 unless --warn-only overrides.
• Operator signal – Flag printed, e.g. “reply_class low_confidence: 0.35”.
• Recovery – Investigate classifier training data or model drift. The gate can be re‑run with --warn-only to bypass the block while corrective work is done.

4. Parsing or Data Malformation in a Feedback Script

• Trigger – One of the imported scripts (feedback_email, feedback_applications, feedback_replies) encounters malformed input (e.g., non‑numeric confidence, missing fields). This raises an unhandled exception inside that script’s execution.
• Guard – The collect function’s exception handler catches the exception and appends a descriptive error string to the errors list. The gate then exits with sys.exit(2).
• Posture – fail-hard – Exit code 2; no calibration flags are produced.
• Operator signal – Error output (e.g., “ValueError: invalid literal for float in band_data”), printed in the digest.
• Recovery – An operator must correct the malformed data or fixture and rerun the gate. No automatic retry.

5. Missing Configuration or Import Dependency

• Trigger – The gate or one of its dependencies (e.g., feedback_common, feedback_email) cannot be imported because a required module is missing, or a configuration file (such as a database URL environment variable) is absent.
• Guard – No explicit guard is shown in the provided source; the Python interpreter will raise an ImportError or a runtime exception at startup, causing the script to crash before collect is even called. The crash exits with a non‑zero code (typically 1) but not via the documented exit‑code scheme.
• Posture – fail-hard – Unhandled crash; the gate never executes its logic.
• Operator signal – Python traceback with the missing‑module or missing‑variable message.
• Recovery – The operator must install the missing package or set the required environment variable, then rerun. No retry is built in.

The make_generation_metrics function returns the two standard metrics (FaithfulnessMetric and AnswerRelevancyMetric) with configurable thresholds, reason inclusion, and an optional switch to skip faithfulness for free-generation graphs.

def make_generation_metrics(
    judge: Any = None,
    *,
    with_faithfulness: bool = True,
    strict: bool = False,
) -> list[Any]:
    """Standard DeepEval metrics for generation graphs.
    Returns [FaithfulnessMetric, AnswerRelevancyMetric] by default.
    Pass with_faithfulness=False for free-generation graphs.
    Pass strict=True for outreach/hallucination-critical generation.
    """
    from deepeval.metrics import AnswerRelevancyMetric, FaithfulnessMetric

    j = judge if judge is not None else make_judge()
    faith_threshold = THRESHOLD_STRICT if strict else THRESHOLD_DEFAULT
    metrics: list[Any] = []
    if with_faithfulness:
        metrics.append(
            FaithfulnessMetric(threshold=faith_threshold, model=j, include_reason=True)
        )
    metrics.append(
        AnswerRelevancyMetric(threshold=THRESHOLD_DEFAULT, model=j, include_reason=True)
    )
    return metrics

Picture a taxi company that installs a meter in every car, logging every trip’s fare and fuel usage into a central logbook. This system treats cost as a first‑class metric — every time a graph runs (like a taxi trip), its token usage and cost are automatically recorded into a dedicated database table, so the company knows exactly what each ride costs.

Now step inside the taxi: as soon as a trip starts, the meter fires off a cost log to a permanent table called llm_cost_log, but the meter is “best effort” — if the logbook rips, the taxi keeps driving without crashing. Before the car even leaves, a master kill switch (LLM_KILL_SWITCH) can instantly shut down all engines, stopping every trip before it spends a single token. When the kill switch is active, no cost log is written because no fare was generated. The company also has a daily budget alert: every morning it reads today’s total from the logbook and, if the sum exceeds a set threshold, fires a webhook notification with just the numbers. Each route (graph) gets its own per‑trip budget — for example, the text_to_sql route is capped at 3,000 tokens — so the dispatcher can spot a runaway ride in real time.

The trickiest guard is the kill switch’s short‑circuit: it doesn’t just block the engine — it also skips the cost log entirely, because a kill‑switched trip made no calls, so logging zero would be misleading. Without this subsystem, a single leaking graph could silently burn through the entire token budget, no one would get an alert, and the company would only notice after the bill arrived — leaving the system unable to stop or even measure its own spending.

The subsystem begins with record_graph_cost, the single entry point that orchestrates cost logging after every graph run. Its ordered mechanism is: first, it calls _emit_cost_log to persist a row into the D1 llm_cost_log table — this function internally fires _ensure_cost_log_table to idempotently create the table and indexes, then invokes _d1_sink_cost to insert the record. Only after that best-effort D1 write does record_graph_cost proceed to check_token_budget for per-graph token enforcement (observe-only by default), and finally it handles OpenTelemetry span annotation via annotate_current_span or a standalone cost_span. A kill switch (LLM_KILL_SWITCH) short-circuits before any D1 write: if engaged, _d1_sink_cost is skipped entirely because a kill-switched run made no LLM calls and thus has zero cost to log.

The invariant the design preserves is explicitly named: “Best-effort: a D1 failure NEVER blocks or errors the graph.” Every D1 operation is wrapped in a try/except that swallows exceptions and logs them at DEBUG level, and the global _d1_schema_ensured flag makes _ensure_cost_log_table idempotent — the guarantee is that a transient or permanent D1 outage will never propagate an error to the caller, no matter when it occurs during the write pipeline. The kill switch further ensures that when no LLM calls happened, no spurious zero-cost row is written, avoiding clutter in the cost table.

The key trade-off is availability over consistency: the design deliberately rejects a blocking, synchronous D1 write that would fail the graph if the database were unreachable. The obvious alternative — making the cost log a required, retryable commit that blocks the graph until success — is avoided because it would turn a telemetry feature into a dependency that could take down production runs. The cost of the chosen approach is that cost data can be silently lost during a D1 outage; the risk is accepted in favor of guaranteeing graph execution never stalls. No retry logic, no exactly-once guarantees — the system is fire-and-forget.

A concrete failure mode is a D1 connectivity blip during _d1_sink_cost. The operator will see zero user-facing errors; the only signal is a DEBUG log line like "cost log emit skipped: <exception>" (the exception message from the D1 client). If the outage persists, the llm_cost_log table grows stale or empty, and subsequent daily spend budget alerts (fired by the webhook reading today’s total) may report zero cost, masking real spend. The operator would need to correlate missing cost rows with D1 health dashboards or the log stream to detect the silent drop.

1 (Warm-up)
Q — How does the system ensure that cost logging never blocks or crashes a graph run?
A — The function _emit_cost_log and the D1 sink _d1_sink_cost both wrap their work in broad try/except blocks that catch Exception and fallback to log.debug(...) on failure. This “best-effort, fire‑and‑forget” design means a database hiccup or any logging error is swallowed, never stopping the graph.
Follow-up — What about a failure in the token‑budget check — is it handled the same way?
A — Yes, check_token_budget is also wrapped in a similar guard so a budget‑breach exception never crashes the run.
Weak answer misses — The exact catch‑all pattern (except Exception as exc: log.debug(...)) and the comment “telemetry must never crash a run” are the critical details a shallow answer omits.

2 (Medium)
Q — Why is no cost log written when the kill switch has been activated?
A — The code in _d1_sink_cost explicitly checks the kill‑switch state: “Only write when the kill switch is off (a kill‑switch run made no LLM calls, so cost=0 anyway; skipping avoids a spurious row).” Since no tokens were consumed, writing a zero‑cost row would be misleading, so the sink simply returns early.
Follow-up — Where exactly is that check performed?
A — It is a comment‑guided guard inside the same _d1_sink_cost function, right before the D1 insert logic.
Weak answer misses — A shallow answer might say “because there are no calls” but miss the explicit guard comment and the reasoning that a zero‑cost row is spurious.

3 (Hard – design question)
Q — Why does record_graph_cost use two distinct code paths for HTTP versus CLI/cron runs instead of always writing a standalone span?
A — For HTTP (FastAPI) an enclosing server span already exists, so annotate_current_span stamps cost attributes directly onto that active span, keeping the cost trace alongside the request timeline. For CLI/cron there is no parent span, so a dedicated one‑shot span (cost_span) is opened per graph call to avoid clobbering a shared parent. This dual‑path design is documented in the record_graph_cost docstring and is controlled by the return values "annotated", "span", or "noop".
Follow-up — What happens if OpenTelemetry is completely absent?
A — The method returns "noop" and all attribute‑setting operations are silently dropped.
Weak answer misses — A naive answer may propose a single approach for all runtimes, ignoring the explicit runtime distinction (annotate_current_span vs. cost_span) and the three‑way return convention.

4 (Hard)
Q — How are per‑workflow token budgets defined and enforced at runtime?
A — A static dictionary _TOKEN_BUDGETS maps graph names (e.g. "text_to_sql": 3000, "agentic_search": 15000) to token caps. The function check_token_budget is called inside record_graph_cost with the graph’s totals and defaults to soft_fail=False (observe‑only). When soft_fail=True, a budget breach marks the span ERROR and adds an over‑budget event. The fallback "_default" cap of 25 000 tokens catches any unlisted graph.
Follow-up — How is the graph name mapped to the right budget entry?
A — feature_for_graph applies a prefix‑based lookup (e.g. "score_" → "lead_gen"), but the budget dictionary itself is keyed on the exact graph name, so the mapping is a direct name‑to‑cap dictionary lookup.
Weak answer misses — Purely naming the dictionary misses the soft_fail parameter nuance (observe‑only by default, ERROR only when explicitly set to True) and the fact that the budget lookup is name‑based, not feature‑based.

D1 Sink Write Failure (Best‑Effort Database Log)

• Trigger – The _d1_sink_cost call raises an exception (database connection lost, schema mismatch, constraint violation).
• Guard – No exception handler is visible in the provided source around the _d1_sink_cost invocation. The comment “never raises” suggests internal catching inside _d1_sink_cost, but no identifier for that handler is exposed. The guard is therefore absent from the source shown.
• Posture – Fail‑soft if the function internally silences the error (the cost row is dropped); fail‑hard if it ever propagates, which would contradict the system’s stated best‑effort guarantee. Relying on the source alone, the posture is indeterminate, but the design intent is fail‑soft.
• Operator signal – Silent absence of the expected cost row in the D1 table. No log or alert is generated by the subsystem.
• Recovery – No automatic retry. Manual investigation of the database and re‑running the graph is required to restore cost data.

Structured Cost Log Emission Failure

• Trigger – _emit_cost_log raises an exception (logging backend unavailable, serialization error).
• Guard – except Exception as exc: log.debug("cost log emit skipped: %s", exc) in graph_meta.
• Posture – Fail‑soft: the structured log is skipped; the graph continues unimpeded.
• Operator signal – Debug‑level log line "cost log emit skipped: <exception>" in the application logs.
• Recovery – No retry. The cost information is lost from the structured log but may still flow to the D1 database and telemetry spans.

Token Budget Check Failure (O56)

• Trigger – check_token_budget raises an exception (OTel exporter endpoint unreachable, budget configuration parse error).
• Guard – except Exception as exc: log.debug("token budget check skipped: %s", exc) in graph_meta.
• Posture – Fail‑soft: the over‑budget span event is not emitted; production monitoring loses the alert for this run.
• Operator signal – Debug log line "token budget check skipped: <exception>". No agentic_sales.over_budget span event in the telemetry.
• Recovery – No automatic retry. The budget violation goes unobserved; the same graph run could exceed its budget without any operator notification.

Tier Ratio Emission Failure (HTTP Path)

• Trigger – emit_tier_ratio raises an exception inside the active‑span branch (OTel span context missing, telemetry client error).
• Guard – except Exception as exc: log.debug("emit_tier_ratio (active span) skipped: %s", exc) in the HTTP (annotated) code path.
• Posture – Fail‑soft: the tier ratio metric is not stamped on the current span; the HTTP request trace continues without the ratio.
• Operator signal – Debug log line "emit_tier_ratio (active span) skipped: <exception>". The ratio is absent from the span attributes.
• Recovery – No retry. The tier‑ratio metric is lost for that graph invocation. Subsequent runs will attempt to emit again.

Standalone Cost Span Failure (CLI/Cron Path)

• Trigger – The with cost_span(...) block raises an exception (tracer creation fails, attribute set error) or the span emission itself fails.
• Guard – except Exception as exc: log.debug("record_graph_cost standalone span skipped: %s", exc) followed by return "noop" in graph_meta.
• Posture – Fail‑soft: the standalone span is dropped entirely; the graph run returns "noop" and continues.
• Operator signal – Debug log line "record_graph_cost standalone span skipped: <exception>". No agentic_sales.cost.<graph> span is exported for that run.
• Recovery – No automatic retry. Cost telemetry for CLI/cron invocations is missing; the daily cost report may be incomplete.

D10: recruiter cv_tailor anti-fabrication grounding gate using golden set of clean and fabricated resumes.

_CV_TAILOR_GROUNDING_FIXTURE = TESTS_DIR / "golden" / "cv_tailor_grounding.json"

@register_gate(
    "cv-tailor-grounding",
    "cv_guardrails.check_grounding: clean CV passes / fabricated-stat·PII·lead-leak CV fails ≥0.80 (deterministic)",
)
async def _eval_cv_tailor_grounding() -> tuple[bool, dict[str, Any]]:
    """Run check_grounding() over the labelled golden set and check decisions match.
    A row passes when the guardrail's ``status == "pass"`` matches the golden
    ``expected_pass``. No CF_AIG_TOKEN required — check_grounding is a pure
    deterministic function over the source candidate + anonymised resume (no LLM,
    no network). Defers only if the fixture is absent or module can't be imported.
    """
    # Implementation runs deterministic verdicts against golden expectations
    ...

Thresholds that define warning and critical drift for latency, error rate, score, and trajectory length.

# moves more than the warn fraction (and clears the absolute floor where one
# applies) is a finding; more than the critical fraction escalates severity.
_LATENCY_WARN_FRAC = 0.5      # +50% p95 latency
_LATENCY_CRIT_FRAC = 1.0      # +100% p95 latency
_ERROR_WARN_DELTA = 0.10      # +10 percentage points error rate
_ERROR_CRIT_DELTA = 0.25      # +25 percentage points error rate
_SCORE_WARN_DELTA = 0.10      # -0.10 absolute mean-score / pass-rate drop
_SCORE_CRIT_DELTA = 0.25      # -0.25 absolute drop
_TRAJ_WARN_FRAC = 0.5         # +/-50% mean trajectory length

This function identifies failing runs by checking for errors, gate metrics below 1.0, or judge metrics below 0.7.

def _fail_reasons(run) -> list[str]:
    """Return the failing-signal names for a run, or [] if it looks healthy."""
    reasons: list[str] = []
    if getattr(run, "error", None):
        reasons.append("error")
    stats = getattr(run, "feedback_stats", None) or {}
    for key, st in stats.items():
        if not isinstance(st, dict):
            continue
        avg = st.get("avg")
        if avg is None:
            continue
        if key in _GATE_KEYS and avg < 1.0:
            reasons.append(key)
        elif key not in _GATE_KEYS and avg < CELL_PASS:  # judge-style metric
            reasons.append(key)
    return reasons

Metamorphic invariant checks define deterministic, offline invariants for domain functions without LLM.

"""Metamorphic relations for the deterministic lead-quality domain layer.

The qualifier, entity scorer and dedup clusterer have golden *correctness* gates,
but correctness goldens can't catch ... **metamorphic relations**: it transforms a
seed input in a way that should *not* change the relevant output, runs the function
on both, and checks they agree. Pure + deterministic — `score_entity` is called
with `use_llm=False`, so there is no network/LLM and no API key needed.
"""
from __future__ import annotations

@register_gate(
    "domain-metamorphic",
    "metamorphic invariants of qualify/score_entity/lead_dedup hold 100% (deterministic, offline)",
)
async def _eval_domain_metamorphic() -> tuple[bool, dict[str, Any]]:
    """Run every metamorphic relation; pass iff zero invariant violations.
    Seeds reuse the qualification + entity-score golden fixtures ... but check
    *relations across transformed inputs*, not labels.
    No CF_AIG_TOKEN required.
    """

Three perturbation families for mutation robustness checks.

def get_perturbations(
    text: str, seed: int = FIXED_SEED
) -> list[tuple[str, str]]:
    """Return ``[(name, mutated_text), ...]`` for all perturbation families.

    Families are always returned in the same order:
    ``whitespace``, ``synonym``, ``reorder``.
    """
    return [
        ("whitespace", whitespace_perturbation(text)),
        ("synonym", synonym_perturbation(text, seed=seed)),
        ("reorder", sentence_reorder_perturbation(text, seed=seed)),
    ]

Repeated deterministic gate execution to detect flakiness via variance of pass rates across runs

for gid, fn in det_gates:
    rates: list[float] = []
    for run_idx in range(_O65_REPEAT_N):
        try:
            passed, details = await fn()
            if details.get("status") == "deferred":
                continue
            rate = details.get("pass_rate")
            if rate is None:
                rate = 1.0 if passed else 0.0
            rates.append(float(rate))
        except Exception:
            rates.append(0.0)
    if not rates:
        results.append({"gate": gid, "rates": [], "variance": 0.0, "status": "deferred"})
        continue
    variance = _stats.variance(rates) if len(rates) >= 2 else 0.0
    flaky = variance > _O65_VARIANCE_THRESHOLD
    entry = {"gate": gid, "rates": rates, "variance": round(variance, 6),
             "status": "flaky" if flaky else "stable"}
    results.append(entry)

The span‑coverage audit classifies each graph node as covered, graph‑span, or dark using static source analysis.

def audit() -> list[GraphCoverage]:
    registry = _load_registry()
    results = []
    for assistant_id, module in registry:
        graph_file = BACKEND_DIR / (module.replace(".", "/") + ".py")
        src = graph_file.read_text(encoding="utf-8")
        stripped = _strip_comments_and_strings(src)
        has_graph_span = _graph_has_graph_span(stripped)
        node_map = _extract_node_map(src)
        nodes = []
        for node_name, func_name in node_map.items():
            if _node_has_span(src, func_name):
                bucket = "covered"
            elif has_graph_span:
                bucket = "graph-span"
            else:
                bucket = "dark"
            nodes.append(NodeCoverage(node=node_name, func=func_name, bucket=bucket))
        results.append(GraphCoverage(assistant_id, module, has_graph_span, nodes))
    return results

@dataclass
class GraphCoverage:
    assistant_id: str
    module: str
    has_graph_span: bool
    nodes: list[NodeCoverage] = field(default_factory=list)
    @property
    def covered_count(self) -> int:
        return sum(1 for n in self.nodes if n.bucket == "covered")
    @property
    def graph_span_count(self) -> int:
        return sum(1 for n in self.nodes if n.bucket == "graph-span")
    @property
    def dark_count(self) -> int:
        return sum(1 for n in self.nodes if n.bucket == "dark")

The evaluation matrix is implemented by the GraphCoverage dataclass and print_coverage function, which classify graphs into buckets and track the DeepEval gated flag for the ≥0.80 quality gate.

@dataclass
class GraphCoverage:
    module: str
    has_llm: bool
    tests: list[str] = field(default_factory=list)
    bucket: str = "missing"  # covered | no-llm | delegates | exempt | missing
    note: str = ""
    gated: bool = False

def print_coverage(rows: list[GraphCoverage]) -> int:
    by_bucket = {b: [] for b in ("covered","missing","no-llm","delegates","exempt")}
    for r in rows:
        by_bucket[r.bucket].append(r)
    covered = len(by_bucket["covered"])
    missing = len(by_bucket["missing"])
    needs_eval = covered + missing
    pct = (covered / needs_eval * 100) if needs_eval else 100.0
    print(f"Coverage: {covered}/{needs_eval} = {pct:.0f}%")
    print(f"Gated: {len([r for r in rows if r.bucket=='covered' and r.gated])}")
    return 1 if missing else 0

Standard DeepEval metrics for generation graphs — FaithfulnessMetric and AnswerRelevancyMetric — with optional strict threshold for hallucination-critical outputs.

def make_generation_metrics(
    judge: Any = None,
    *,
    with_faithfulness: bool = True,
    strict: bool = False,
) -> list[Any]:
    from deepeval.metrics import AnswerRelevancyMetric, FaithfulnessMetric

    j = judge if judge is not None else make_judge()
    faith_threshold = THRESHOLD_STRICT if strict else THRESHOLD_DEFAULT
    metrics: list[Any] = []
    if with_faithfulness:
        metrics.append(
            FaithfulnessMetric(threshold=faith_threshold, model=j, include_reason=True)
        )
    metrics.append(
        AnswerRelevancyMetric(threshold=THRESHOLD_DEFAULT, model=j, include_reason=True)
    )
    return metrics

Imagine a factory inspector who checks each product against its blueprint and the customer's order. If a single part is wrong, the inspector flags it but does not stop the entire assembly line—that one item gets reworked or scrapped while the rest keep moving. This is exactly what the team’s evaluation-first philosophy does: before any change is shipped, a set of automated inspectors verify every output. One inspector makes sure every claim in the response actually appears in the source material—like the grounding check that catches a hallucinated “$3M cost saving” that never existed in the LinkedIn post. Another inspector confirms the output actually answers the original input, so a reply about “migrating to Go” doesn’t slip through when the question was about latency reduction. These checks are built into the code as deterministic gates, like the _eval_university_sequence_quality function that scans directives for banned phrases and required keywords without using any language model. The real subtlety is that each inspector fails independently: a single ungrounded claim raises a red flag for that particular piece (the code returns a “deferred” status if the test fixture is missing) but does not block the pipeline. That means if one cold email fails the grounding check, the rest of the outreach continues—GTM isn’t stopped by one mistake. Without this subsystem, a single hallucination would sail through and land in a customer’s inbox, eroding trust with a fabricated statistic, while the whole workflow ground to a halt fixing it.

The system's evaluation-first architecture is anchored by a deterministic, zero-cost guardrail named check_grounding inside the cv_guardrails module. When a CV is submitted for tailoring, the first action is to run this guardrail over the anonymised candidate text against the source candidate profile. It uses a labelled golden set (_CV_TAILOR_GROUNDING_FIXTURE) that includes both faithful anonymisations — which must pass — and canonical fabrications (invented skills, PII leaks, altered tenure, etc.) — which must fail. If the guardrail returns a "pass" verdict, processing proceeds; on failure it raises a red flag and blocks the PDF render and submission for that specific CV, isolating the error to that component. Only after this deterministic gate passes does the system escalate borderline leads — those captured in the QUALIFY_EXAMPLES dataset — to a single DeepSeek ainvoke_json call that judges role-fit, confidence, and vertical alignment. This ordered mechanism ensures the cheapest, most reliable check runs first and stops false positives without consuming LLM budget.

The invariant the design preserves is stated explicitly in the source: ats-only-no-synthetic-decisions. Every fact in the final anonymised resume must be grounded in the source candidate; no new information may be synthesised, even if it appears plausible. This is a write boundary: the guardrail enforces that the tailored output contains exactly the candidate’s own claims, nothing invented by a model. The invariant is checked deterministically — no LLM, no network — so it is both verifiable and repeatable in CI. The gate is the safety floor that prevents fabricated data from ever reaching the ATS.

The key trade-off is between cost and expressiveness. The obvious alternative would be to use an LLM for grounding checks, which could catch subtle semantic mismatches but would introduce latency, token cost, and the risk of the LLM itself hallucinating a false verdict. The design rejects that approach because check_grounding is a pure, deterministic function over the source candidate and the anonymised text — it runs fully offline with zero LLM calls. The cost of this rejection is reduced flexibility: some borderline or paraphrase-based fabrications might slip past a simple deterministic rule. But in exchange, the team avoids the unacceptable cost of an LLM‑based guardrail that could itself generate false negatives (failing a clean CV) or false positives (passing a fabricated one) due to model drift or prompt brittleness. This is why the system labels the guardrail the “safety floor”: it favours a hard, verifiable invariant over a probabilistic one.

A concrete failure mode occurs when the check_grounding function encounters a modified fixture or an import error for cv_guardrails. The evaluation entry point (_eval_cv_tailor_grounding) defers execution if the golden fixture file is absent or the module cannot be imported. An operator would see a deferred status in the evaluation dashboard for the "cv-tailor-grounding" gate — the lane does not report pass/fail, it simply skips. This signal tells the operator that the safety floor is not active for that deployment, which should immediately trigger an investigation into the missing fixture or broken module import, because without it the pipeline loses its guarantee against synthetic decisions.

Q – What evaluation metrics does the team attach to generation graphs by default, and what does each check?

A – The function make_generation_metrics returns a list containing FaithfulnessMetric and AnswerRelevancyMetric by default. The FaithfulnessMetric verifies every factual claim in the output is grounded in the provided retrieval_context, while the AnswerRelevancyMetric checks that the output addresses the input prompt. Both are created with include_reason=True and use a common judge model.

Follow-up – If I need only the answer‑relevancy check—say for a free‑form email that has no retrieval context—how would the code change?

Answer – Set with_faithfulness=False in make_generation_metrics. That skips the FaithfulnessMetric and returns a list containing only the AnswerRelevancyMetric. The default mode includes both; the with_faithfulness flag controls the inclusion.

Weak answer misses – The real function name make_generation_metrics and the existence of the with_faithfulness parameter.

Q – Why did the team choose to combine two separate metrics (faithfulness and relevancy) rather than using a single composite score?

A – The design isolates two orthogonal failure modes: a fact‑grounding error (hallucination) and a relevance error (off‑target answer). The FaithfulnessMetric catches claims not in retrieval_context; the AnswerRelevancyMetric catches responses that ignore the prompt. By keeping them separate, the system can flag which dimension fails without conflating the two. This mirrors the principle that each component fails on its own error without blocking downstream pipeline steps.

Follow-up – Could you just use the faithfullness metric alone for generation graphs that do have context?

Answer – No, because even a perfectly grounded output might not answer the user’s question. The AnswerRelevancyMetric independently ensures the output addresses the input; dropping it would miss relevance regressions. The team’s philosophy is “never ship a change without first proving it passes the bar” – both bars must be met.

Weak answer misses – The explicit function make_generation_metrics returns a list; the two metrics are appended separately, not fused. The phrase “each component fails on its own error” is echoed in the subsystem description but not cited as a function name – the relevant mechanism is the list construction that keeps metrics independent.

Q – How does the evaluation pipeline handle a case where FaithfulnessMetric fails but the system should still continue processing the rest of the workflow?

A – The subsystem treats each metric as a gate that raises a red flag without a global halt. The FaithfulnessMetric returns a reason string when claims are ungrounded, but the pipeline does not stop. This is visible in the _eval_cv_tailor_grounding lane: the deterministic check_grounding guardrail returns a verdict (pass or fail) and the evaluation records whether the verdict matches expected_pass; a failure does not abort the entire run. The design ensures that a failing check logs the error and allows subsequent stages (e.g., PDF generation) to proceed with a warning rather than a crash.

Follow-up – Is there a scenario where the system would block the entire pipeline, not just flag?

Answer – In high‑risk areas like extraction, the team uses a stricter version: strict=True in make_generation_metrics raises the FaithfulnessMetric threshold to THRESHOLD_STRICT. Even then, the metric only raises its own failure – the pipeline continues, but the stricter threshold means more red flags are raised. The only true block is the deterministic guardrail in cv_guardrails.check_grounding, which blocks PDF submission if fabricated facts are detected; but that is a safety floor, not a metric.

Weak answer misses – The exact parameter strict in make_generation_metrics and the existence of a dedicated grounding gate _eval_cv_tailor_grounding that runs purely deterministically without LLM.

Q – Why is the _eval_cv_tailor_grounding evaluation lane described as “PURE, no‑LLM/no‑network”, and what design advantage does that provide?

A – The function runs check_grounding() over a labelled golden set; check_grounding is a pure deterministic function that operates over the source candidate and anonymised resume without any LLM or network call. This guarantees zero cost, zero latency and full reproducibility – it can run offline. The advantage is that the safety floor for PDF submission (blocking fabricated skills, PII leaks, lead leaks) never depends on external model behavior or API availability.

Follow-up – If the guardrail is deterministic, why is it part of an evaluation suite at all? Why not just embed it directly in the production pipeline?

Answer – It already is embedded in production as cv_guardrails.check_grounding. The evaluation lane (_eval_cv_tailor_grounding) is a regression test that proves the guardrail still passes clean CVs and still fails canonical fabrications (invented skill, tenure alteration, etc.). Without the evaluation, a code change could silently break the deterministic logic.

Weak answer misses – The exact fixture file _CV_TAILOR_GROUNDING_FIXTURE = TESTS_DIR / "golden" / "cv_tailor_grounding.json" and the phrase “no CF_AIG_TOKEN required” confirming its offline nature.

Fixture File Missing for the Grounding Gate (D10)

• Trigger — The file at _CV_TAILOR_GROUNDING_FIXTURE does not exist or is unreadable when _eval_cv_tailor_grounding is invoked.
• Guard — The function contains explicit deferral logic: “Defers only if the fixture is absent or the guardrail module can't be imported.” This is a passive guard; no active handler is shown in the snippet.
• Posture — fail-soft: the gate is skipped entirely (deferred), and the evaluation run continues with the remaining gates.
• Operator signal — No error line is emitted for this gate; it is silently absent from the output. The --coverage report (see below) would later flag it as “missing”, but only if that run is performed.
• Recovery — No automatic retry. The operator must manually restore the fixture file or investigate why it is missing. The evaluation run does not halt for this gate.

Graph Accuracy Falls Below the 0.80 Threshold

• Trigger — For a given graph, both std_accuracy and deep_accuracy (from the _O69_TIER_MATRIX_FIXTURE or the static default) are below AGGREGATE_PASS = 0.80.
• Guard — The logic appends the graph name to both std_failures and deep_failures, then collects them into hard_failures. The final check (gate_passed = len(hard_failures) == 0) causes the gate to fail. No retry or fallback is provided.
• Posture — fail-hard: the gate result is False, and the evaluation script exits with code 1 (assuming --gate is used).
• Operator signal — The console prints: [O69-tier-matrix] FAILED — {n} graph(s) below ≥80%: {graph names}.
• Recovery — The failed graph’s accuracy must be improved (e.g., by updating prompts or model calls) and the fixture updated. The evaluation run stops; the change cannot be shipped until this gate passes.

Deep‑Only Alert (Standard Tier Below Bar, Deep Tier Above)

• Trigger — A graph’s std_accuracy < 0.80 while deep_accuracy ≥ 0.80. The graph is not a hard failure; it is added to deep_only_graphs.
• Guard — No guard prevents this; it is an intentional warning. The condition if deep_only triggers a print flagging the graph as “DEEP-ONLY”, but the gate does not fail because deep_only is not a hard failure.
• Posture — fail-soft: the overall gate passes (gate_passed = True), but an alert is raised in the output.
• Operator signal — A line per graph: [O69] DEEP-ONLY: {graph} std={val} < 80% deep={val} ≥ 80% delta={delta:+.4f}. At summary: [O69-tier-matrix] PASSED (with deep-only alerts) — {n} graph(s) rely on deep tier only: ....
• Recovery — The operator is expected to investigate why the cheaper standard tier fails and either improve it or adjust the fixture. No automated rerun or fallback occurs.

Coverage Gap: a Graph That Needs Eval Is Not Registered

• Trigger — A LangGraph graph that emits model output (detected by the LLM call‑site heuristics) is not covered by any registered @register_gate evaluation, nor listed in the exempt section of the coverage matrix.
• Guard — The --coverage subcommand (run by default) compares the set of known graphs against registered gates and the exempt list. It outputs a report and exits with code 1 if any graphs are missing.
• Posture — fail-hard for the coverage check: the script prints the missing graphs and terminates with exit code 1, blocking the full evaluation run.
• Operator signal — Terminal output includes missing: {graph names} and a summary line indicating coverage failed. The --json flag would produce a structured report with a missing field.
• Recovery — The operator must either register a new evaluation gate for the graph (or add it to the exempt list with a written reason) and rerun. No automatic fix exists.

Guardrail Module Import Failure

• Trigger — The cv_guardrails.check_grounding module cannot be imported (e.g., syntax error, missing dependency) when _eval_cv_tailor_grounding attempts to load it.
• Guard — The same deferral logic as the fixture‑missing case: the function defers if the guardrail module can't be imported.
• Posture — fail-soft: the gate is skipped, and the rest of the evaluation proceeds.
• Operator signal — No explicit error line is printed for this gate; the --coverage report would show it as “missing” because the gate never completed.
• Recovery — The operator must fix the import issue manually (e.g., correct the module path or install missing packages) and rerun the evaluation. No automatic retry is attempted.