Verify code against real source

Completion

Exercise: Grounding code excerpts via whole-identifier membership

In the LlamaIndex + Qdrant project, emitted code snippets are verified against the real source code to catch hallucinated APIs. The key insight: substring checks (e.g., "index" in source_text) are too permissive because they match parts of longer real identifiers (index_documents). Instead, the check must use whole-token membership: collect all identifiers from the code block (using regex), then compute what fraction belong to the set of identifiers appearing in the real source.

Your task: implement the function verify_code_excerpt(source_text: str, code_markdown: str) -> list[str] that:

Extracts exactly one fenced code block from code_markdown (if not exactly one, return a violation).
Extracts all identifiers (tokens matching [A-Za-z_][A-Za-z0-9_]{2,}, case-sensitive) from that block, excluding a set of stopwords.
Computes the set of source identifiers: all identifiers appearing in source_text.
Calculates the ratio of grounded identifiers (those found in the source set) to total identifiers.
If ratio < 0.667, adds a violation with details about unknown identifiers.

Also, in a comment at the end of the function, explain why a substring check (ident in source_text) would incorrectly accept identifiers like "execute" when the real source contains "execute_tool".

The starter code below has # TODO: placeholders. Fill them to complete the implementation.

Your code

import re

# Regex to find exactly one fenced code block (language optional)
_FENCE_RE = re.compile(r"```(?:[a-zA-Z]+)?\s*\n(.*?)```", re.DOTALL)
# Regex to find whole identifiers (3+ alphanumeric characters, starting with letter/underscore)
_IDENT_RE = re.compile(r"[A-Za-z_][A-Za-z0-9_]{2,}")
# Stopwords (generic Python keywords / builtins) not counted for grounding
_STOPWORDS = {
    "def", "return", "self", "none", "true", "false", "and", "not", "for",
    "the", "from", "import", "async", "await", "class", "with", "str", "int",
    "list", "dict", "float", "bool", "else", "elif", "try", "except", "raise",
    "python", "result", "state", "get", "data", "text", "type", "any", "len",
    "print", "while", "yield", "lambda", "pass", "continue", "break",
}

def verify_code_excerpt(source_text: str, code_markdown: str) -> list[str]:
    """Return list of violation strings if the code excerpt is insufficiently grounded."""
    violations = []

# TODO 1: Extract fenced code blocks; if not exactly one, add violation and return.
    # Use _FENCE_RE.findall on code_markdown.
    blocks = None  # replace None

# TODO 2: Extract identifiers from the code block (first block, since we already checked exactly one).
    # Use _IDENT_RE.findall on the block string; filter out stopwords (lowercase comparison).
    code_idents = None  # replace None

# TODO 3: Build the set of source identifiers using _IDENT_RE.findall on source_text.
    source_idents = None  # replace None

# If no code identifiers, add violation and return early.
    if not code_idents:
        violations.append("no identifiers in the code block")
        return violations

# Find ungrounded identifiers
    ungrounded = sorted(t for t in code_idents if t not in source_idents)
    grounded = len(code_idents) - len(ungrounded)
    ratio = grounded / len(code_idents)

if ratio < 0.667:
        sample = ", ".join(ungrounded[:6])
        violations.append(
            f"only {grounded}/{len(code_idents)} identifiers ({ratio:.0%}) trace to the "
            f"source — quote real symbols, do not invent APIs (unknown: {sample})"
        )

# TODO 4 (comment): Explain why a substring check would fail.
    # Write your explanation as a comment below (use #).

return violations

Sources

roadmap-kg/kg/ground.py:2259-2287
roadmap-kg/kg/ground.py:2233-2256
roadmap-kg/kg/ground.py:2290-2316