Verify code against real source

Worked Example: Grounding Code Excerpts via Whole-Identifier Membership

In this exercise, you will implement a grounding check that ensures an LLM-emitted code excerpt only uses symbols that actually appear in the real source code. This is how the production system prevents hallucinated API calls like execute when the source only has execute_tool.

The Problem

• A naive substring check (if identifier in source_text) would count execute as grounded because execute_tool contains the substring execute.
• We must match whole tokens only – the identifier must appear as a standalone token in the source.
• We also ignore common Python keywords/generic builtins (the _CODE_STOPWORDS set) because they don't prove grounding.
• A code block is considered faithful if at least 2/3 of its identifiers are found in the source.

Your Task

Write a function verify_code_grounding(markdown: str, source_text: str) -> list[str] that:

1. Builds the set of source identifiers using _source_identifiers(source_text) (which extracts all whole tokens matching [A-Za-z_][A-Za-z0-9_]{2,}).
2. Calls _code_grounding_violations(markdown, source_idents) to check the excerpt.
3. Returns the list of violation messages (empty if the code block passes).

Throughout the solution, you must use the exact same constants and helper functions from the real repository:

import re

_FENCE_RE = re.compile(r"```(?:[a-zA-Z]+)?\s*\n(.*?)```", re.DOTALL)
_IDENT_RE = re.compile(r"[A-Za-z_][A-Za-z0-9_]{2,}")
_CODE_STOPWORDS = {
    "def", "return", "self", "none", "true", "false", "and", "not", "for",
    "the", "from", "import", "async", "await", "class", "with", "str", "int",
    "list", "dict", "float", "bool", "else", "elif", "try", "except", "raise",
    "python", "result", "state", "get", "data", "text", "type", "any", "len",
    "print", "while", "yield", "lambda", "pass", "continue", "break",
}

Step-by-Step Reasoning (the Worked Solution)

1. Extract the code block – Use _FENCE_RE.findall(markdown). Exactly one fence expected; if not, report violation and return.
2. Tokenize the code – Find all identifiers in the code block with _IDENT_RE.findall(fences[0]), then filter out stopwords (case-insensitive).
3. Build source identifier set – Apply _IDENT_RE.findall(source_text) and store as a set (no stopword filter needed; they won't harm but aren't harmful either – the source set includes them, but they are filtered out of the code idents).
4. Identify ungrounded tokens – Any identifier in the code that is not in source_idents is ungrounded.
5. Compute ratio – grounded = total - len(ungrounded); ratio = grounded / total. If ratio < 0.667, produce a violation listing up to 6 unknown symbols.
6. Return violations – Return the list (empty if all good).

Why This Works

By comparing whole tokens (not substrings), a hallucinated documents will not be fooled by index_documents. The 2/3 threshold allows minor generic symbols (e.g., open, write) that might exist in the source but aren't part of the code – but forces the majority to come from the source.

Example Usage

source = '''
def index_documents(docs):
    vector_store.add(docs)
'''


markdown_ok = """```python
vector_store.add(index_documents)
```"""
print(verify_code_grounding(markdown_ok, source))   # []

# Bad code (execute is not in source):
markdown_bad = """```python
result = execute_tool(params)
```"""
print(verify_code_grounding(markdown_bad, source))
# -> ['only 0/2 identifiers (0%) trace to the source — quote real symbols, do not invent APIs (unknown: execute_tool, params)']

Now study the starter_code below, then read the solution_code to see the full implementation.