FaithfulnessEvaluator gating

Task: Implement the Faithfulness-Drop Discipline

In the provided LlamaIndex + Qdrant glossary pipeline, a definition is only accepted if it is faithful to the retrieved document contexts. You must write a function that gates a generated answer using a FaithfulnessEvaluator instance.

Specification

Write a function:

def apply_faithfulness_gate(
    definition: str,
    contexts: list[str],
    faithfulness: FaithfulnessEvaluator
) -> tuple[bool, bool]:

• Inputs:
  • definition: the generated answer string (e.g., a glossary definition).
  • contexts: a list of text strings from the retrieved nodes (the source documents).
  • faithfulness: an instance of llama_index.core.evaluation.FaithfulnessEvaluator (already initialised with an LLM).
• Return: a tuple (is_faithful: bool, error_occurred: bool):
  • is_faithful: True if the evaluation passes (the answer is supported by the contexts), False otherwise.
  • error_occurred: True if the evaluation call raised an exception (e.g., transport/parse error), False otherwise.

Behaviour

1. Use faithfulness.evaluate(response=definition, contexts=contexts).passing to check faithfulness.
2. If the call succeeds, return (passing, False).
3. If the call raises any Exception (catch with except Exception as exc:), print the following warning (exactly as shown) using an f-string:
   f"[glossary-li]   WARNING faithfulness eval errored for term: {exc}; skipped"
   

   Then return (False, True) — treat the error as a drop.
4. Do not crash the calling program; the error is handled gracefully.

Important

• Use exactly the real class FaithfulnessEvaluator and method .evaluate(...).passing.
• The function must be self-contained and not rely on any global state.
• This is a blank-slate task; you start with an empty file.