Chunking with SentenceSplitter

Completion

Task: Chunk Documents with SentenceSplitter

In the LlamaIndex-based retrieval system you're building, raw files and narration are too large to embed as single vectors. Instead, each document is split into smaller, overlapping nodes using a SentenceSplitter. The chunk size and overlap are controlled by environment variables CHUNK_SIZE (default 512) and CHUNK_OVERLAP (default 64) – exactly as the _default_splitter helper does in the real codebase.

Your job: complete the function chunk_documents(docs) so it:

Reads CHUNK_SIZE and CHUNK_OVERLAP from the environment (using the same defaults as the source).
Creates a SentenceSplitter with those values.
Splits every Document in the docs list into overlapping nodes.
Returns the list of all nodes.

The docstring and partial code already import Document and SentenceSplitter. Look for the # TODO: lines – that's where you must write the missing logic.

Hint: The real source (e.g., _default_splitter, _load_nodes, and build_cached_index) all follow this pattern:

python

from llama_index.core.node_parser import SentenceSplitter
splitter = SentenceSplitter(
    chunk_size=int(os.environ.get("CHUNK_SIZE", "512")),
    chunk_overlap=int(os.environ.get("CHUNK_OVERLAP", "64")),
)
nodes = splitter.get_nodes_from_documents(docs)

After completing the code, explain in a comment or docstring why:

A whole file (e.g., 10 000 characters) cannot be directly embedded.
Using a small overlap (e.g., 64 characters) preserves a piece of information that straddles the boundary between two consecutive chunks.

Your code

import os
from llama_index.core import Document
from llama_index.core.node_parser import SentenceSplitter

def chunk_documents(docs: list[Document]) -> list:
    """
    Split every Document into overlapping SentenceSplitter nodes.
    Reads CHUNK_SIZE and CHUNK_OVERLAP from environment (defaults: 512, 64).

Args:
        docs: list of LlamaIndex Document objects.

Returns:
        list of TextNode objects (the split nodes).
    """
    # TODO: Create the SentenceSplitter with env-controlled parameters
    splitter = ???

# TODO: Split all documents into nodes
    nodes = ???

return nodes

if __name__ == "__main__":
    # Example usage (not graded)
    sample_docs = [
        Document(text="First long document. " * 50, metadata={"file": "doc1.txt"}),
        Document(text="Second long document. " * 100, metadata={"file": "doc2.txt"}),
    ]
    result = chunk_documents(sample_docs)
    print(f"Created {len(result)} nodes")
    if result:
        print(f"First node length: {len(result[0].text)}")
        print(f"Overlap example: ...{result[0].text[-20:]} ... {result[1].text[:20]}...")

Sources

roadmap-kg/kg/ground_content.py:218-261
roadmap-kg/kg/memory_common.py:757-804
roadmap-kg/kg/memory_common.py:683-728