Chunking with SentenceSplitter

Worked Example: Chunking Documents with SentenceSplitter

In LlamaIndex, raw documents (e.g., entire files, chapters) are often too large to embed directly. Embedding models have a fixed context window — a single vector cannot capture the meaning of a 10,000‑token file. Instead, we split documents into smaller, overlapping chunks (nodes). Each chunk becomes its own retrieval unit.

Why overlap? When a sentence or fact straddles a chunk boundary, overlapping text ensures both chunks include the relevant context, so the fact is not lost. Overlap preserves boundary‑straddling facts.

The _default_splitter Pattern

The real repository uses a SentenceSplitter controlled by two environment variables:

• CHUNK_SIZE (default 512)
• CHUNK_OVERLAP (default 64)

This exact pattern appears in _default_splitter() and _build_query_engine() in the source. We will implement a function that respects these knobs, splits a list of Document objects, and returns a list of TextNode objects.

Complete Worked Solution

Below is the full implementation. Every step is annotated with the reasoning.

import os
from llama_index.core import Document
from llama_index.core.node_parser import SentenceSplitter
from llama_index.core.schema import TextNode

def chunk_documents(docs: list[Document]) -> list[TextNode]:
    """Split documents into overlapping chunks using SentenceSplitter.

    Reads CHUNK_SIZE and CHUNK_OVERLAP from environment (defaults 512 and 64).
    Each Document is split independently; the resulting nodes are returned in one list.
    """
    # 1. Read configuration from environment (same as _default_splitter)
    chunk_size = int(os.environ.get("CHUNK_SIZE", "512"))
    chunk_overlap = int(os.environ.get("CHUNK_OVERLAP", "64"))

    # 2. Instantiate SentenceSplitter with these values
    splitter = SentenceSplitter(
        chunk_size=chunk_size,
        chunk_overlap=chunk_overlap,
    )

    # 3. Split all documents. get_nodes_from_documents returns a flat list.
    nodes = splitter.get_nodes_from_documents(docs)

    # 4. Return the list of TextNode objects (ready for embedding)
    return nodes

What each step accomplishes:

• Step 1: The environment variables allow operators to tune chunk size without code changes; the defaults match the repository’s production settings.
• Step 2: SentenceSplitter respects sentence boundaries where possible, producing coherent chunks rather than arbitrary cuts.
• Step 3: get_nodes_from_documents processes each Document and returns a list of TextNode objects, each with a portion of the original text and inherited metadata (e.g., file name). The splitter automatically creates overlapping segments — the last tokens of one chunk are repeated at the start of the next, preserving context.

Why not split manually? LlamaIndex’s SentenceSplitter is battle‑tested, handles edge cases (short docs, punctuation), and integrates natively with other LlamaIndex components (e.g., embedding caches). The env‑knob pattern makes it reusable across the whole project.

Now proceed to the starter code and implement the same function.