Code-aware splitting (CodeSplitter)
FullWrite a Python function route_splitter(file_path: str) -> tuple[object, str] that determines whether a given source file should be split using a tree-sitter-backed CodeSplitter (for .py, .ts, .tsx files) or a SentenceSplitter (for other prose‑like files). The function must:
- Use the same logic as the repository’s
_lang_forto map the file extension to a language string ("python"or"typescript") orNonefor prose. - Build the appropriate
CodeSplitterwith an explicitParserfromtree_sitterandget_languagefromtree_sitter_language_pack, using environment variablesCODE_CHUNK_LINES(default 60) andCODE_CHUNK_OVERLAP(default 12). - Build a
SentenceSplitterwithchunk_sizeandchunk_overlapfrom environment variablesCHUNK_SIZE(default 512) andCHUNK_OVERLAP(default 64) when the file is not code. - Return a tuple: (splitter_instance, explanation_string). The explanation must describe why splitting code by sentences (using
SentenceSplitter) breaks identifier grounding, referencing real examples likeroute_for,NativeD1Saver, camelCase, underscores, and the importance of preserving exact tokens for BM25 recall.
You may assume all required imports are available (llama_index.core.node_parser, tree_sitter, tree_sitter_language_pack, os).
Your code
Sources
- roadmap-kg/kg/rerank.py:151-186
- roadmap-kg/kg/glossary_llamaindex.py:215-223
- roadmap-kg/kg/ground_content.py:218-261