FaithfulnessEvaluator gating
CompletionWrite a function gate_with_faithfulness(definition, nodes, llm) that implements the faithfulness-drop discipline seen in the LlamaIndex/Qdrant glossary pipeline. The function should:
- Extract the text content from each node in
nodes(usingn.node.get_content()). - Use a
FaithfulnessEvaluator(fromllama_index.core.evaluation) instantiated with the providedllmto check whether thedefinitionis faithful to those contexts. - If the evaluation passes (
evaluate(...).passingis True), return thedefinitionunchanged. - If the evaluation fails (not passing) OR the evaluation call raises an exception (e.g., local LLM hiccup), return
Noneto drop the definition and continue — do not crash the run.
The starter_code below has three marked # TODO: blanks at the critical lines. Fill them in to complete the implementation.
Your code
Sources
- roadmap-kg/kg/glossary_llamaindex.py:215-223
- roadmap-kg/kg/glossary_llamaindex.py:267-311
- roadmap-kg/kg/glossary_llamaindex.py:384-420