Verify code against real source
FullYou are given the following real source code (from the LlamaIndex + Qdrant project). Your task is to implement a function check_code_grounding(code_excerpt: str, source_text: str) -> list[str] that validates whether the identifiers in a code excerpt (the content inside a single fenced code block) are grounded in the real source.
Rules:
- Extract the only fenced code block from
code_excerpt(expect exactly one triple-backtick block; if zero or more than one, return a violation). - Tokenize the extracted code block by finding all tokens matching the pattern
[A-Za-z_][A-Za-z0-9_]{2,}(at least 3 characters). - Remove stop words from that set: Python keywords and generic builtins listed in
_CODE_STOPWORDS(provided below). - Tokenize the
source_textthe same way (using the same pattern) to build the set of real source identifiers. - Compute the ratio of grounded identifiers (those present in both sets) vs total non-stopword identifiers in the excerpt.
- If the ratio is less than 0.667 (two thirds), return a violation string listing the unknown identifiers (up to 6 examples).
- Also return any structural violations (e.g., wrong number of code blocks, no identifiers).
Why whole-identifier membership? A substring check (e.g., "execute" in source_text) would falsely count a hallucinated execute as grounded if the real source contains execute_tool. Whole-token matching prevents that loophole.
You are given the helper constants and functions from the real repo:
_FENCE_RE = re.compile(r"```(?:[a-zA-Z]+)?\s*\n(.*?)```", re.DOTALL)_IDENT_RE = re.compile(r"[A-Za-z_][A-Za-z0-9_]{2,}")_CODE_STOPWORDS = { ... }(same set as in the source)_source_identifiers(source_text) -> set[str](returns the set of whole identifiers from source)_code_grounding_violations(markdown, source_idents) -> list[str](the production function).
Reimplement _code_grounding_violations (or call it) in your check_code_grounding function. Your solution must not rely on any external library besides re.
Finally, in a docstring or comment, explain the failure mode that a substring check would allow (give a concrete example).
Starter code (empty):
- roadmap-kg/kg/ground.py:2259-2287
- roadmap-kg/kg/ground.py:2233-2256
- roadmap-kg/kg/ground.py:2290-2316