Back to Practice

Verify code against real source

Full

You are given the following real source code (from the LlamaIndex + Qdrant project). Your task is to implement a function check_code_grounding(code_excerpt: str, source_text: str) -> list[str] that validates whether the identifiers in a code excerpt (the content inside a single fenced code block) are grounded in the real source.

Rules:

  1. Extract the only fenced code block from code_excerpt (expect exactly one triple-backtick block; if zero or more than one, return a violation).
  2. Tokenize the extracted code block by finding all tokens matching the pattern [A-Za-z_][A-Za-z0-9_]{2,} (at least 3 characters).
  3. Remove stop words from that set: Python keywords and generic builtins listed in _CODE_STOPWORDS (provided below).
  4. Tokenize the source_text the same way (using the same pattern) to build the set of real source identifiers.
  5. Compute the ratio of grounded identifiers (those present in both sets) vs total non-stopword identifiers in the excerpt.
  6. If the ratio is less than 0.667 (two thirds), return a violation string listing the unknown identifiers (up to 6 examples).
  7. Also return any structural violations (e.g., wrong number of code blocks, no identifiers).

Why whole-identifier membership? A substring check (e.g., "execute" in source_text) would falsely count a hallucinated execute as grounded if the real source contains execute_tool. Whole-token matching prevents that loophole.

You are given the helper constants and functions from the real repo:

  • _FENCE_RE = re.compile(r"```(?:[a-zA-Z]+)?\s*\n(.*?)```", re.DOTALL)
  • _IDENT_RE = re.compile(r"[A-Za-z_][A-Za-z0-9_]{2,}")
  • _CODE_STOPWORDS = { ... } (same set as in the source)
  • _source_identifiers(source_text) -> set[str] (returns the set of whole identifiers from source)
  • _code_grounding_violations(markdown, source_idents) -> list[str] (the production function).

Reimplement _code_grounding_violations (or call it) in your check_code_grounding function. Your solution must not rely on any external library besides re.

Finally, in a docstring or comment, explain the failure mode that a substring check would allow (give a concrete example).

Starter code (empty):

Your code
Sources
  • roadmap-kg/kg/ground.py:2259-2287
  • roadmap-kg/kg/ground.py:2233-2256
  • roadmap-kg/kg/ground.py:2290-2316