Chunking with SentenceSplitter
FullTask: Chunking with SentenceSplitter
Write a Python function split_documents_into_chunks(docs: list) that takes a list of Document objects (from llama_index.core) and returns a list of TextNode objects obtained by splitting each document using a SentenceSplitter. The splitter must be configured using the environment variables CHUNK_SIZE (default 512) and CHUNK_OVERLAP (default 64). Use os.environ.get() to read these values.
Inside the function, include a comment (or docstring) that explains:
- Why embedding an entire file as a single vector is suboptimal (the chunk size limit and loss of granularity).
- How overlap helps preserve facts that straddle chunk boundaries.
Your implementation must use the exact same identifiers and imports as shown in the repository source (e.g., SentenceSplitter, get_nodes_from_documents).
Your code
Sources
- roadmap-kg/kg/ground_content.py:218-261
- roadmap-kg/kg/memory_common.py:757-804
- roadmap-kg/kg/memory_common.py:683-728