Evaluation & Feedback — Audio Guide

Evaluating a fleet of language model graphs means looking at separate checks, not just one number. Each stage of the pipeline can fail on its own. A single score would hide those independent failures. For example, a graph that uses retrieved documents needs a faithfulness metric. That metric checks every factual claim against the provided context. Another graph that generates freely only needs an answer relevancy metric. That metric verifies the output addresses the input prompt. If you only used one number, you might miss when the output is relevant but makes up facts. Or when it is faithful but does not answer the question. The evaluation uses two metrics by default. The first is faithfulness. The second is answer relevancy. For critical outreach graphs, a stricter threshold is applied. That is a tighter check. For graphs without retrieval documents, the faithfulness metric is skipped. There is nothing to be faithful to. So the evaluation adapts to the graph's structure. Each stage has its own independent failure point. One metric would hide the specific problem. You need separate checks for separate risks. That is why a single number cannot diagnose a multi stage pipeline.

These evaluators take a run and an example. They return a key, a score from zero to one, and a reason. The score tells you how well the output matches expectations. The reason explains why.

One such evaluator is the faithfulness metric. It checks that every factual claim in the output comes from the provided context. Another is the answer relevancy metric. It checks that the output addresses the input prompt.

Both use a judge model and a fixed threshold. Because the model and threshold never change, the same input always gives the same score. That consistency means the evaluator runs unchanged in continuous integration and in production. You do not need different settings for testing versus live use.

The threshold can be set to strict if you need higher accuracy. But the core logic stays the same. This makes the evaluators reliable and simple to maintain. They produce a clear pass-or-fail signal for each generation.

There are two deterministic checks on a generated text. The first check is called faithfulness. It verifies every factual claim in the output is grounded in the provided retrieval context. That means the model must not invent facts. The second check is answer relevancy. It verifies the output addresses the input prompt. So the response must be on topic. Both checks use a judge model. The judge can be passed in or created automatically. The faithfulness check has a threshold. A strict threshold is available for critical cases. By default, the threshold is a standard value. The answer relevancy check always uses a default threshold. Both include a reason for their verdict. This gives transparency. The checks are designed for generation graphs. If the generation has no retrieval document, the faithfulness check is skipped. Then only answer relevancy runs. That makes sense for free generation tasks. For example, composing an email with no source material. Faithfulness would be meaningless there. So the system adapts. The two checks together ensure quality. They catch hallucinations and off-topic responses. They are standard in the evaluation library. They work with any language model. The judge can be a custom model or the default. This flexibility is useful. The checks are deterministic in the sense that they follow a fixed process. They do not rely on randomness. Each run produces the same result given the same inputs. That is important for evaluation consistency. The combination of both checks gives a solid evaluation. One checks grounding. The other checks relevance. Together they cover the main failure modes.

For each core graph, the evaluation uses a set of test examples. Each example has a list of inputs and a list of outputs. The input section holds specific details like the recipient name, company, and instructions. The output section does not give the exact answer. Instead, it lists signals that must appear and signals that must be avoided. This makes the input side very strict. The output side is much more flexible. The same test cases are used when you run checks on your local machine. They also appear in the LangSmith platform. LangSmith is a hosted tracing tool. The accuracy standard stays the same in both places. For instance, the cold outreach graph has an example with a recipient named Alex Rivera. The output says the email must mention the words rag and eval. It must not say "I hope this finds you well." The metrics check faithfulness and relevancy. The faithfulness metric uses a threshold. That threshold can be strict or default. If you set strict, the tolerance for errors is lower. That is the trade-off. You lose some flexibility but gain confidence in the output. Another graph handles email replies. Its output signals include must validate the objection. The same shared bar applies. That way, every developer knows what good looks like, whether they run tests locally or on the hosted platform.

A function builds evaluation metrics for generation graphs. It uses a judge model to check the output. The same judge is reused for every metric in the test suite. There are two main metrics: faithfulness and answer relevancy. Faithfulness verifies that every factual claim comes from the provided text. Answer relevancy verifies that the output directly addresses the input prompt. You can set a stricter threshold for critical generation cases. For extraction graphs, a third metric appears. Contextual precision measures whether the highest-ranked results are truly relevant to the query. The judge is created once and passed to all metrics. That keeps the evaluation consistent across tests. The default threshold works for most cases. The stricter threshold is for outreach tasks where hallucinations are especially harmful. Both thresholds are built into the function. This shared design reduces duplication. It also ensures every metric uses the same evaluation model. The result is a reliable, repeatable test suite for generation quality.

When you run a graph, you can capture every node and tool that actually fired. Then you compare that ordered list to the expected golden path. This catches regressions that a simple final answer check would miss. Why? Because the final answer might seem fine, but the steps to get there could have broken. For example, let's say the output must contain specific signals. The ordered path check verifies each claim is grounded. It looks at every factual statement. A final relevancy check only asks if the output answers the prompt. That is not enough. If a claim is made up, but the overall answer still addresses the question, you might not catch the error. The path check catches those made up statements. It requires every claim to match the evidence. This is like using a faithfulness metric. It checks each step against the provided context. The final answer check is like answer relevancy. It only checks the surface. So for reliable evaluations, you need both. The ordered sequence gives you a deeper look at the process. That is why an ordered path check catches regressions that a single final check would miss.

Some generation graphs include evaluation metrics. Others do not. The standard setup gives two metrics to graphs that use retrieval. One is a faithfulness metric. It checks every claim against the provided documents. The other is an answer relevancy metric. It verifies the output addresses the prompt. For free generation graphs without retrieval, only answer relevancy applies. A new graph that calls a model starts with no evaluation. It stays missing until someone writes its evaluation. The system uses a threshold to decide if a result passes. For critical tasks like outreach, the threshold becomes stricter. That turns it into a gate. When a graph that has evaluation drops below that bar, the gate fails. This prevents bad results from going through. That is how the coverage works. Graphs with evaluation are covered. Graphs without it are not. The gate protects quality for the ones that are covered.

A faithfulness metric checks each claim in the output against the provided background information. If a claim is not supported, the metric returns a non-zero signal. This signal acts as a gate for production feedback. The metric has a default threshold. For critical tasks, you can set a strict threshold. The strict threshold catches more hallucinations. There is also an answer relevancy metric. It verifies the output addresses the input prompt. Together, these metrics form a feedback loop. The trade-off is between safety and precision. A strict threshold reduces risk but may flag correct claims as wrong. A default threshold balances coverage and errors. The feedback signal is continuous. It helps improve the system over time. The gate rolls up these checks into one actionable result. This result tells you when the output contradicts the provided context. The gate exits non-zero only when there is a problem. That makes the feedback easy to act on. The source material includes concrete examples. For instance, a clean output stays grounded in the given context. A hallucinated output invents facts not in the text. The metric catches that difference. It turns a vague problem into a clear signal. That signal can drive improvements in your pipeline. The process is straightforward. You run the metric after generating each output. If the signal is non-zero, you know something is wrong. You can then review and fix the output. This keeps your system honest.

You have two metrics. The faithfulness metric verifies every claim is grounded in the retrieval context. The answer relevancy metric verifies the output addresses the input prompt. You set a threshold. The default threshold is balanced. The strict threshold demands higher accuracy. You can include a reason from the judge. That reason explains the score. This is how you control the system. You can choose not to use faithfulness. That is for free generation. Then you only have the answer relevancy metric. The judge is a model you provide. You can create a judge using a make judge function. That judge evaluates everything. The threshold for faithfulness can be strict or default. Strict means you require very accurate claims. Default allows more flexibility. The system gives you a clear control. You decide how strict to be. That decision shapes the output. The metric records the reason. That reason helps you understand the result. This is the main control you have. You set it per workflow. There is no global kill switch here. But the threshold acts as a limit. It determines the trade off between accuracy and cost. The strict threshold pushes for perfect answers. The default threshold is more forgiving. That is the key difference. You use this control to balance your needs. The system is straightforward. You have two metrics and a threshold. That is all.

The guardrail tests itself with a golden set of labeled resumes. The set includes one clean, anonymised resume that must pass. It also includes resumes with fabricated details. Those should fail. That negative control proves the guard actually fires. Without it, you might block everything. With a clean example passing, you know the guard works. The test is fully deterministic. It uses no language model and no network request. That means it runs offline, fast, and the same every time. The verdict either blocks or allows the final PDF submission. The golden set covers specific types of fabrication. It checks for invented skills, leaked personal information, altered tenure, and lead company leaks. Each test case has an expected pass or fail. The guardrail matches its verdict to that expectation. This proves resistance to attacker-controlled input. Even though cooperative evaluators never send such input, the test still ensures the guard holds. The guardrail is a pure enforcement point. It acts as a safety floor. Every real submission must carry only facts grounded in the source candidate. The golden set proves the guardrail passes a faithfully anonymised CV and fails the canonical fabrications. That makes the gate reliable and trustworthy.

This monitor compares a graph’s recent field behavior to its own past behavior. It reads live production traces from LangSmith. The monitor uses two moving windows. A short window covers about one day. A longer baseline window spans seven days. Both windows pull root runs from the same project. The monitor folds those runs into per-graph signals. It tracks error rate, latency, verdict pass rate, and trajectory length. Trajectory length is the count of child runs. It computes each signal for both windows separately. Then it compares the recent values against the baseline values. If a metric moves beyond a warning threshold, a finding is raised. Larger moves escalate to a critical severity. The monitor does not call any language model. It only reads and compares arithmetic aggregates.

This approach solves a key problem. Offline golden datasets cannot track field drift. Latency and error rates depend on the environment and change over time by design. A fixed golden answer would become stale. A rolling self-baseline adapts to natural shifts. It lets the monitor detect real anomalies without false alarms from expected changes. The monitor’s baseline is the graph’s own recent history. This mirrors how auto‑curation scans use a moving window. The result is a live check that does not need static golden files.

A system can mine its production runs to find failures. It looks for runs where a gate signal like gate passed or no ai markers is below one point zero. It also checks if a judge metric falls below zero point seven. Those runs are not healthy. Their inputs become new examples in a separate captured dataset. That dataset never mixes with the graded evaluation set until a human reviews the output. The human review happens in an annotation queue. Reviewed examples can then be promoted into the graded dataset. The trade off is clear. If a system only tests its original examples, it never sees real world failures. It cannot improve from the mistakes it makes in production. This closed loop gives it a way to learn from actual errors. It captures the model's actual output, which is often wrong, and turns that into a regression candidate. The system stays fresh because it keeps adding new examples from real traffic. This avoids stale testing. Without this loop, there is no source of new challenge data. The system would keep passing its old tests but fail on new situations. So the loop turns production failures into a path for continuous improvement.

Evaluating a system is tough when you lack expected answers for every input. Instead, you assert invariants — rules that must always hold. The system here uses a deterministic gate for token budgets. Normal inputs with low token counts must pass. Over-budget inputs must be rejected. This relation catches bugs on inputs for which no one wrote a golden answer. Another example is the grounding guardrail. It checks that a curriculum vitae, or C V, contains only facts from the candidate. Fabricated skills or leaked personal data must fail. That invariant protects against invented information. The guardrail verdict acts as a safety floor. It blocks any PDF render and submission when it fails. These deterministic functions run offline without any large language model, or L L M. They use only the source candidate and the anonymised resume. No network is required. This makes them fast and reliable. By asserting such relations, you catch errors in many scenarios. You do not need a labelled expected output for each case. The invariants define correct behavior. The trade-off is you must design the invariants carefully. But once set, they run automatically and guard against many failures.

A robustness evaluation checks if a system keeps its answer when input changes in harmless ways. Three kinds of changes are used. The first is whitespace adjustments. The second is swapping words with synonyms. The third is reordering whole sentences. Each change preserves the original meaning. Synonyms come from a carefully curated list. Every swap keeps the same part of speech and same positive or negative tone. The list avoids any shift in intensity or scope. The synonym map includes greetings like hello and hi. It has affirmatives like yes and sure. Negatives like not a fit. Modifiers like immediately. Verbs like learn and send. Nouns like details and list. Adjectives like interested and great. Polite phrases like please and thanks. The map is case insensitive but preserves original casing. Multi word phrases are matched before single words. Patterns are compiled longest first. The reorder function splits the text into sentences and shuffles them. Whitespace changes do not alter the text content. A fixed seed makes every run identical and repeatable. The three kinds are always applied in the same order. A system that gives a different answer after these cosmetic changes is not actually reliable. Its decision should stay the same. If a small swap of a synonym causes a different result, the system is not trustworthy. The evaluation catches that weakness. The goal is consistent performance. Meaning preserving changes should not confuse the system. A robust system passes all three tests. It returns the same classification regardless of whitespace, synonyms, or reordering.

A good evaluation needs to test itself. The system repeats a gate several times and measures how much its pass rate changes. It compares today's score to a past baseline and to a trend of scores over time. This catches any slow decline. For example, the offline gate for university sequences has a threshold of zero point eight zero. It checks the copy for personalization and banned phrases. If the score starts to drop, the trend will show it. The trend is stored as a JSON line list, each entry with a run ID and a timestamp. Tools like the jq command can pull the time series of covered and missing counts. The O34 token budget gate uses normal fixtures that must pass and over-budget fixtures that must be rejected. Running these gates many times shows if they are stable. When a gate is not stable, its pass rate wobbles. When its score slowly slides down, no one notices until it is too late. A gate that fails silently or inconsistently is worse than no gate at all. It gives a false sense of safety. The deterministic design prevents flakiness, but the trend check remains essential.

This audit measures whether the system is observable, not whether it behaves correctly. Each graph node gets grouped into one of three categories. A covered node contains a manual span call. A graph span node has no per node span but the whole graph sends a cost signal. A dark node has no span at all. The tool then prints a matrix showing these results. It also offers a gate that fails the check if any graph is fully dark. A fully dark graph has zero covered nodes and no graph level span. That means the graph emits no telemetry. Such a gap is a real blind spot. No correctness test can see it. The audit looks at every graph listed in the registry file. It scans the source code for span patterns like start underscore as underscore current underscore span. It also checks for phase, annotate current span, and cost span. The analysis is purely static. It never runs the code. The machine readable report comes out as JSON. The human readable table shows each assistant ID and how many nodes are covered or dark. The goal is to catch any graph that is invisible to monitoring. That is a problem no other test can find.

Every graph runs through the evaluation matrix. The matrix assigns each graph a bucket. Buckets include covered, missing, no large language model, delegates, and exempt. The aggregate pass mark is zero point eight zero. Graphs marked as gated carry the Deep Eval marker. That marker means the graph contributes to the quality gate. If a graph only passes when using that deeper marker, it is flagged. The reason is straightforward. Honest routing depends on proving the standard evaluation is good enough. The matrix ensures no graph can hide behind the more thorough check. It forces each graph to clear the same bar. This builds trust that the normal test path works for every graph. The evaluation runs determined decisions. There are no random numbers or timestamps inside the script. This keeps each run reproducible. The final report writes a line per run. That line holds the run identifier, a timestamp, the aggregate pass mark, and the graph details. Each graph's details include its bucket and a note on why it is exempt or missing. The gate results list each gate's score and status. This setup catches regressions before they reach production. It makes the multi tier routing rely on proven cheap paths. The matrix flags any graph that only passes with the deep evaluation. This ensures the simpler path is always sufficient.

The system follows an evaluate-first approach. No prompt or model change is released until it clears a set of metrics. One metric checks that every claim in the output is backed by the provided context. Another metric verifies the output directly addresses the input prompt. For outreach or hallucination-critical generation, the threshold is stricter. Each test example has signals that must appear. It also has signals that must not appear. If a change introduces a regression in a key segment, the system catches it. Catching that regression matters more than keeping the overall average high. The metrics are applied to every candidate output. This ensures that even a single false claim in a critical email is flagged. The system does not let a weak average mask a serious problem. Every factual check is independent. A failure in one claim means the whole output fails that metric. This philosophy keeps the generation grounded and relevant.