rag.splitters.code

Split source code into chunks by structural boundaries with language-specific patterns.

Split source code into chunks by structural boundaries with language-specific patterns.

class CodeChunker(chunk_size=1000, chunk_overlap=200, metadata=None, language='python', split_by='function')[source]

Bases: ChunkerBase

Code-aware chunking that splits by functions/classes.

Parameters:

  • chunk_size (int) – Maximum characters per chunk.

  • chunk_overlap (int) – Overlap between consecutive chunks.

  • metadata (dict[str, Any] | None) – Default metadata for all chunks.

  • language (str) – Programming language for parsing hints.

  • split_by (str) – Strategy: “function”, “class”, or “module”.

LANGUAGE_PATTERNS: dict[str, dict[str, str]] = {'generic': {'class': '^(class|struct|interface)\\s+\\w+', 'comment': '^(#|//|/\\*)', 'function': '^(function|func|def|fn)\\s+\\w+', 'import': '^(import|use|require)'}, 'go': {'class': '^type\\s+\\w+\\s+struct', 'comment': '^(//|/\\*)', 'function': '^func\\s+(\\(\\w+\\s+\\*?\\w+\\)\\s+)?\\w+\\s*\\(', 'import': '^import\\s+'}, 'java': {'class': '^\\s*(public\\s+)?(abstract\\s+)?class\\s+\\w+|interface\\s+\\w+', 'comment': '^(//|/\\*)', 'function': '^\\s*(public|private|protected|static)?\\s*\\w+\\s+\\w+\\s*\\(', 'import': '^import\\s+'}, 'javascript': {'class': '^(export\\s+)?class\\s+\\w+', 'comment': '^(//|/\\*|\\*)', 'function': '^(async\\s+)?function\\s+\\w+|const\\s+\\w+\\s*=\\s*(async\\s+)?\\([^)]*\\)\\s*=>|export\\s+(async\\s+)?function', 'import': '^(import\\s+|export\\s+|require\\s*\\()'}, 'python': {'class': '^class\\s+\\w+[\\(:]', 'comment': '^(#|\'\'\'|\\"\\"\\")', 'decorator': '^@\\w+', 'function': '^(async\\s+)?def\\s+\\w+\\s*\\(', 'import': '^(import\\s+|from\\s+\\S+\\s+import)'}, 'rust': {'class': '^(pub\\s+)?struct\\s+\\w+|impl\\s+\\w+', 'comment': '^(//|/\\*|\\*)', 'function': '^(pub\\s+)?(async\\s+)?fn\\s+\\w+', 'import': '^use\\s+'}, 'typescript': {'class': '^(export\\s+)?(abstract\\s+)?class\\s+\\w+|interface\\s+\\w+|type\\s+\\w+', 'comment': '^(//|/\\*|\\*)', 'function': '^(async\\s+)?(function\\s+\\w+|const\\s+\\w+\\s*=\\s*(async\\s+)?\\([^)]*\\)\\s*(:\\s*\\w+)?\\s*=>|export\\s+(async\\s+)?function)', 'import': '^(import\\s+|export\\s+|require\\s*\\()'}}
chunk(text, metadata=None)[source]

Split code by structural boundaries.

Parameters:

  • text (str) – The code text to split.

  • metadata (dict[str, Any] | None) – Optional metadata to attach to each chunk.

Returns:

List of TextChunk objects.

Return type:

list[TextChunk]

Classes

class CodeChunker(chunk_size=1000, chunk_overlap=200, metadata=None, language='python', split_by='function')[source]

Bases: ChunkerBase

Code-aware chunking that splits by functions/classes.

Parameters:

  • chunk_size (int) – Maximum characters per chunk.

  • chunk_overlap (int) – Overlap between consecutive chunks.

  • metadata (dict[str, Any] | None) – Default metadata for all chunks.

  • language (str) – Programming language for parsing hints.

  • split_by (str) – Strategy: “function”, “class”, or “module”.

LANGUAGE_PATTERNS: dict[str, dict[str, str]] = {'generic': {'class': '^(class|struct|interface)\\s+\\w+', 'comment': '^(#|//|/\\*)', 'function': '^(function|func|def|fn)\\s+\\w+', 'import': '^(import|use|require)'}, 'go': {'class': '^type\\s+\\w+\\s+struct', 'comment': '^(//|/\\*)', 'function': '^func\\s+(\\(\\w+\\s+\\*?\\w+\\)\\s+)?\\w+\\s*\\(', 'import': '^import\\s+'}, 'java': {'class': '^\\s*(public\\s+)?(abstract\\s+)?class\\s+\\w+|interface\\s+\\w+', 'comment': '^(//|/\\*)', 'function': '^\\s*(public|private|protected|static)?\\s*\\w+\\s+\\w+\\s*\\(', 'import': '^import\\s+'}, 'javascript': {'class': '^(export\\s+)?class\\s+\\w+', 'comment': '^(//|/\\*|\\*)', 'function': '^(async\\s+)?function\\s+\\w+|const\\s+\\w+\\s*=\\s*(async\\s+)?\\([^)]*\\)\\s*=>|export\\s+(async\\s+)?function', 'import': '^(import\\s+|export\\s+|require\\s*\\()'}, 'python': {'class': '^class\\s+\\w+[\\(:]', 'comment': '^(#|\'\'\'|\\"\\"\\")', 'decorator': '^@\\w+', 'function': '^(async\\s+)?def\\s+\\w+\\s*\\(', 'import': '^(import\\s+|from\\s+\\S+\\s+import)'}, 'rust': {'class': '^(pub\\s+)?struct\\s+\\w+|impl\\s+\\w+', 'comment': '^(//|/\\*|\\*)', 'function': '^(pub\\s+)?(async\\s+)?fn\\s+\\w+', 'import': '^use\\s+'}, 'typescript': {'class': '^(export\\s+)?(abstract\\s+)?class\\s+\\w+|interface\\s+\\w+|type\\s+\\w+', 'comment': '^(//|/\\*|\\*)', 'function': '^(async\\s+)?(function\\s+\\w+|const\\s+\\w+\\s*=\\s*(async\\s+)?\\([^)]*\\)\\s*(:\\s*\\w+)?\\s*=>|export\\s+(async\\s+)?function)', 'import': '^(import\\s+|export\\s+|require\\s*\\()'}}
chunk(text, metadata=None)[source]

Split code by structural boundaries.

Parameters:

  • text (str) – The code text to split.

  • metadata (dict[str, Any] | None) – Optional metadata to attach to each chunk.

Returns:

List of TextChunk objects.

Return type:

list[TextChunk]

property name: str

Return the chunker strategy name.