FaithfulnessEvaluator gating

Faithfulness-Drop Gating for Glossary Generation

This worked example shows how to use LlamaIndex's FaithfulnessEvaluator to gate a generated definition — dropping the answer when the retrieved documents do not support it. This ensures only grounded definitions enter the glossary, preventing hallucinated or unsupported claims.

The Problem

When a retriever returns irrelevant nodes, the LLM may produce a plausible-sounding but ungrounded answer. In a glossary, such noise is unacceptable. The repository uses a faithfulness-drop discipline: after generating a definition from retrieved contexts, it evaluates faithfulness. If the definition is not faithful to the contexts, the answer is discarded. Additionally, a transient judge error (e.g., LLM hiccup) must not crash the whole run — it should be treated as a drop-and-continue.

The Worked Solution

Below is the complete implementation. The function generate_with_faithfulness_gate:

1. Retrieve relevant nodes using retriever.retrieve(query).
2. Synthesize an answer using synth.synthesize(query, nodes), producing a raw string.
3. Clean the answer (optional, but shown as in the source).
4. Check for non-answer markers (e.g., retrieval miss) – skip early if detected.
5. Extract contexts from the retrieved nodes: contexts = [n.node.get_content() for n in nodes].
6. Evaluate faithfulness using faithfulness.evaluate(response=definition, contexts=contexts).passing.
7. Handle errors in the evaluation call: wrap in try/except, print a warning, and treat as a drop (return None).
8. Drop if passing is False; otherwise return the cleaned definition.

The function returns None when the answer is dropped, making it easy for the caller to skip that term.

Code Walkthrough

def generate_with_faithfulness_gate(term: str, retriever, synth, faithfulness) -> str | None:
    """Return a faithful definition for `term`, or None if unsupported."""
    # 1. Retrieve nodes
    nodes = retriever.retrieve(f"{term} in LlamaIndex")
    if not nodes:
        print(f"WARNING no nodes for {term!r}")
        return None

    # 2. Synthesize answer
    raw = str(synth.synthesize(f"{term} in LlamaIndex", nodes))
    if not raw:
        return None

    # 3. Clean the answer (simple whitespace flatten; full _clean in source is more thorough)
    definition = " ".join(raw.split()).strip()

    # 4. Check for non-answer patterns (simplified)
    import re
    if re.search(r"(does not|don't|not.*(contain|mention|found|covered))", definition, re.IGNORECASE):
        print(f"WARNING non-answer for {term!r}; skipped")
        return None

    # 5. Extract contexts for faithfulness evaluation
    contexts = [n.node.get_content() for n in nodes]

    # 6. Evaluate faithfulness with error handling
    try:
        passing = faithfulness.evaluate(response=definition, contexts=contexts).passing
    except Exception as exc:
        print(f"WARNING faithfulness eval error for {term!r}: {exc}; skipped")
        return None

    # 7. Drop if not faithful
    if not passing:
        print(f"WARNING unfaithful definition for {term!r}; skipped")
        return None

    # 8. Return the faithful definition
    return definition

Key Points

• The evaluator is instantiated once and reused: FaithfulnessEvaluator(llm=llm).
• The response parameter is the cleaned definition string.
• The contexts parameter is a list of strings (one per retrieved node).
• The return value .passing is a boolean; True means the answer is faithful.
• The try/except ensures a single term's evaluation failure does not abort the entire pipeline.

Now examine the starter code and fill in the missing parts to implement the same logic.