Code-aware splitting (CodeSplitter)

Completion

Exercise: Code-Aware Splitting for Grounded Retrieval

In the LlamaIndex + Qdrant retrieval pipeline you've been studying, source files are split into nodes for indexing. The repository uses two different splitters depending on whether the file is code (.py, .ts) or prose (everything else).

Your task is to implement the function split_source_file(file_path: Path, text: str) -> list[BaseNode] that mimics the logic in _load_nodes and _code_splitter from the real source above. Specifically:

Determine the programming language using the mapping from file suffix (.py → "python", .ts/.tsx → "typescript"). If the suffix does not match, treat the file as prose.
For code files: build a CodeSplitter with an explicit tree-sitter Parser (obtained from tree_sitter_language_pack.get_language) and chunk settings CODE_CHUNK_LINES=60, CODE_CHUNK_OVERLAP=12. For prose: use a SentenceSplitter with chunk_size=512, chunk_overlap=64.
Wrap the text in a Document with metadata {"file": file_path.name} and split it. If tree-sitter parsing fails, fall back to prose splitting.
Return the list of nodes.

The scaffold below already imports all required libraries and partially fills in the logic. Replace the # TODO: placeholders with the correct code.

Hint: Look at _lang_for, _code_splitter, and the try/except fallback in _load_nodes from the repository source.

Your code

from pathlib import Path
from llama_index.core import Document
from llama_index.core.node_parser import CodeSplitter, SentenceSplitter
from llama_index.core.schema import BaseNode
from tree_sitter import Parser
from tree_sitter_language_pack import get_language
import os

def split_source_file(file_path: Path, text: str) -> list[BaseNode]:
    """Split the given text according to the file type (code or prose).
    Returns a list of LlamaIndex nodes."""
    # TODO 1: Determine the language from the file suffix using a mapping
    #          (like _lang_for in the source). If no mapping, set lang = None.
    lang = None  # Replace this line

doc = Document(text=text, metadata={"file": file_path.name})

if lang is not None:
        # TODO 2: Create a CodeSplitter with an explicit tree-sitter parser
        #         Use chunk_lines=60, chunk_lines_overlap=12.
        #         (Hint: Parser(get_language(lang)))
        try:
            nodes = None  # Replace this line
        except Exception:
            # Fallback to prose splitter on parse failure
            nodes = SentenceSplitter(
                chunk_size=int(os.environ.get("CHUNK_SIZE", "512")),
                chunk_overlap=int(os.environ.get("CHUNK_OVERLAP", "64")),
            ).get_nodes_from_documents([doc])
    else:
        # TODO 3: Use SentenceSplitter with default chunk_size=512, chunk_overlap=64
        nodes = None  # Replace this line

return nodes

Sources

roadmap-kg/kg/rerank.py:151-186
roadmap-kg/kg/glossary_llamaindex.py:215-223
roadmap-kg/kg/ground_content.py:218-261