Back to Practice

Code-aware splitting (CodeSplitter)

Full

Write a Python function route_splitter(file_path: str) -> tuple[object, str] that determines whether a given source file should be split using a tree-sitter-backed CodeSplitter (for .py, .ts, .tsx files) or a SentenceSplitter (for other prose‑like files). The function must:

  • Use the same logic as the repository’s _lang_for to map the file extension to a language string ("python" or "typescript") or None for prose.
  • Build the appropriate CodeSplitter with an explicit Parser from tree_sitter and get_language from tree_sitter_language_pack, using environment variables CODE_CHUNK_LINES (default 60) and CODE_CHUNK_OVERLAP (default 12).
  • Build a SentenceSplitter with chunk_size and chunk_overlap from environment variables CHUNK_SIZE (default 512) and CHUNK_OVERLAP (default 64) when the file is not code.
  • Return a tuple: (splitter_instance, explanation_string). The explanation must describe why splitting code by sentences (using SentenceSplitter) breaks identifier grounding, referencing real examples like route_for, NativeD1Saver, camelCase, underscores, and the importance of preserving exact tokens for BM25 recall.

You may assume all required imports are available (llama_index.core.node_parser, tree_sitter, tree_sitter_language_pack, os).

Your code
Sources
  • roadmap-kg/kg/rerank.py:151-186
  • roadmap-kg/kg/glossary_llamaindex.py:215-223
  • roadmap-kg/kg/ground_content.py:218-261