Back to Practice

FaithfulnessEvaluator gating

Completion
evals

Write a function gate_with_faithfulness(definition, nodes, llm) that implements the faithfulness-drop discipline seen in the LlamaIndex/Qdrant glossary pipeline. The function should:

  1. Extract the text content from each node in nodes (using n.node.get_content()).
  2. Use a FaithfulnessEvaluator (from llama_index.core.evaluation) instantiated with the provided llm to check whether the definition is faithful to those contexts.
  3. If the evaluation passes (evaluate(...).passing is True), return the definition unchanged.
  4. If the evaluation fails (not passing) OR the evaluation call raises an exception (e.g., local LLM hiccup), return None to drop the definition and continue — do not crash the run.

The starter_code below has three marked # TODO: blanks at the critical lines. Fill them in to complete the implementation.

Your code
Sources
  • roadmap-kg/kg/glossary_llamaindex.py:215-223
  • roadmap-kg/kg/glossary_llamaindex.py:267-311
  • roadmap-kg/kg/glossary_llamaindex.py:384-420