rag.splitters

class CharacterChunker(chunk_size=1000, chunk_overlap=200, metadata=None, respect_word_boundaries=True)[source]

Bases: ChunkerBase

Character-based chunking with word-boundary awareness.

Parameters:

  • chunk_size (int) – Maximum characters per chunk.

  • chunk_overlap (int) – Overlap between consecutive chunks.

  • metadata (dict[str, Any] | None) – Default metadata for all chunks.

  • respect_word_boundaries (bool) – Whether to prefer word boundaries for splits.

chunk(text, metadata=None)[source]

Split text into overlapping chunks.

Parameters:

  • text (str) – The text to split.

  • metadata (dict[str, Any] | None) – Optional metadata to attach to each chunk.

Returns:

List of TextChunk objects.

Return type:

list[TextChunk]

class ChunkerBase(chunk_size=1000, chunk_overlap=200, metadata=None)[source]

Bases: ABC

Abstract base class for text chunking strategies.

Parameters:

  • chunk_size (int) – Maximum size of each chunk (interpretation varies by strategy).

  • chunk_overlap (int) – Overlap between consecutive chunks.

  • metadata (dict[str, Any] | None) – Default metadata to attach to all chunks.

abstractmethod chunk(text, metadata=None)[source]

Split text into chunks.

Parameters:

  • text (str) – The text to split.

  • metadata (dict[str, Any] | None) – Optional metadata to attach to each chunk (merged with default).

Returns:

List of TextChunk objects.

Return type:

list[TextChunk]

property name: str

Return the chunker strategy name.

class CodeChunker(chunk_size=1000, chunk_overlap=200, metadata=None, language='python', split_by='function')[source]

Bases: ChunkerBase

Code-aware chunking that splits by functions/classes.

Parameters:

  • chunk_size (int) – Maximum characters per chunk.

  • chunk_overlap (int) – Overlap between consecutive chunks.

  • metadata (dict[str, Any] | None) – Default metadata for all chunks.

  • language (str) – Programming language for parsing hints.

  • split_by (str) – Strategy: “function”, “class”, or “module”.

LANGUAGE_PATTERNS: dict[str, dict[str, str]] = {'generic': {'class': '^(class|struct|interface)\\s+\\w+', 'comment': '^(#|//|/\\*)', 'function': '^(function|func|def|fn)\\s+\\w+', 'import': '^(import|use|require)'}, 'go': {'class': '^type\\s+\\w+\\s+struct', 'comment': '^(//|/\\*)', 'function': '^func\\s+(\\(\\w+\\s+\\*?\\w+\\)\\s+)?\\w+\\s*\\(', 'import': '^import\\s+'}, 'java': {'class': '^\\s*(public\\s+)?(abstract\\s+)?class\\s+\\w+|interface\\s+\\w+', 'comment': '^(//|/\\*)', 'function': '^\\s*(public|private|protected|static)?\\s*\\w+\\s+\\w+\\s*\\(', 'import': '^import\\s+'}, 'javascript': {'class': '^(export\\s+)?class\\s+\\w+', 'comment': '^(//|/\\*|\\*)', 'function': '^(async\\s+)?function\\s+\\w+|const\\s+\\w+\\s*=\\s*(async\\s+)?\\([^)]*\\)\\s*=>|export\\s+(async\\s+)?function', 'import': '^(import\\s+|export\\s+|require\\s*\\()'}, 'python': {'class': '^class\\s+\\w+[\\(:]', 'comment': '^(#|\'\'\'|\\"\\"\\")', 'decorator': '^@\\w+', 'function': '^(async\\s+)?def\\s+\\w+\\s*\\(', 'import': '^(import\\s+|from\\s+\\S+\\s+import)'}, 'rust': {'class': '^(pub\\s+)?struct\\s+\\w+|impl\\s+\\w+', 'comment': '^(//|/\\*|\\*)', 'function': '^(pub\\s+)?(async\\s+)?fn\\s+\\w+', 'import': '^use\\s+'}, 'typescript': {'class': '^(export\\s+)?(abstract\\s+)?class\\s+\\w+|interface\\s+\\w+|type\\s+\\w+', 'comment': '^(//|/\\*|\\*)', 'function': '^(async\\s+)?(function\\s+\\w+|const\\s+\\w+\\s*=\\s*(async\\s+)?\\([^)]*\\)\\s*(:\\s*\\w+)?\\s*=>|export\\s+(async\\s+)?function)', 'import': '^(import\\s+|export\\s+|require\\s*\\()'}}
chunk(text, metadata=None)[source]

Split code by structural boundaries.

Parameters:

  • text (str) – The code text to split.

  • metadata (dict[str, Any] | None) – Optional metadata to attach to each chunk.

Returns:

List of TextChunk objects.

Return type:

list[TextChunk]

class HierarchicalChunker(chunk_size=400, chunk_overlap=100, metadata=None, parent_chunk_size=1500, max_levels=2)[source]

Bases: ChunkerBase

Hierarchical chunking with parent-child relationships.

Parameters:

  • chunk_size (int) – Maximum characters per leaf chunk.

  • chunk_overlap (int) – Overlap between leaf chunks.

  • metadata (dict[str, Any] | None) – Default metadata for all chunks.

  • parent_chunk_size (int) – Size of parent chunks (larger).

  • max_levels (int) – Maximum hierarchy depth.

chunk(text, metadata=None)[source]

Split text into hierarchical chunks with parent-child relationships.

Parameters:

  • text (str) – The text to split.

  • metadata (dict[str, Any] | None) – Optional metadata to attach to each chunk.

Returns:

List of HierarchicalTextChunk objects with parent-child links.

Return type:

list[HierarchicalTextChunk]

get_parent_chunks(chunks)[source]

Filter to return only parent chunks.

Parameters:

chunks (list[HierarchicalTextChunk])

Return type:

list[HierarchicalTextChunk]

get_child_chunks(chunks)[source]

Filter to return only child chunks.

Parameters:

chunks (list[HierarchicalTextChunk])

Return type:

list[HierarchicalTextChunk]

get_chunks_with_parent_context(child_chunks, all_chunks)[source]

Get child chunks with their parent context for retrieval.

Parameters:
Returns:

List of dicts with child content and parent context.

Return type:

list[dict[str, Any]]

class HierarchicalTextChunk(content, chunk_index, start_char, end_char, metadata=None, id='', parent_id=None, child_ids=None, hierarchy_level=0)[source]

Bases: TextChunk

A text chunk that participates in a parent-child hierarchy.

Used by hierarchical chunking strategies that maintain relationships between larger parent chunks and smaller child chunks.

Parameters:
id

Unique identifier for this chunk.

Type:

str

parent_id

Identifier of the parent chunk, or None if this is a root.

Type:

str | None

child_ids

Identifiers of child chunks, or an empty list.

Type:

list[str] | None

hierarchy_level

Depth in the hierarchy (0 = root).

Type:

int

id: str = ''
parent_id: str | None = None
child_ids: list[str] | None = None
hierarchy_level: int = 0
class MarkdownChunker(chunk_size=1000, chunk_overlap=200, metadata=None, split_headers=None, preserve_structure=True, max_chunk_fallback=True)[source]

Bases: ChunkerBase

Markdown-aware chunking that splits by headers.

Parameters:

  • chunk_size (int) – Maximum characters per chunk.

  • chunk_overlap (int) – Overlap between consecutive chunks.

  • metadata (dict[str, Any] | None) – Default metadata for all chunks.

  • split_headers (list[str] | None) – List of header levels to split on (e.g., [“h1”, “h2”]).

  • preserve_structure (bool) – Whether to include header in chunk content.

  • max_chunk_fallback (bool) – Whether to further split large sections.

HEADER_PATTERNS = {'h1': '^# .+', 'h2': '^## .+', 'h3': '^### .+', 'h4': '^#### .+', 'h5': '^##### .+', 'h6': '^###### .+'}
chunk(text, metadata=None)[source]

Split markdown text by headers while respecting size limits.

Parameters:

  • text (str) – The markdown text to split.

  • metadata (dict[str, Any] | None) – Optional metadata to attach to each chunk.

Returns:

List of TextChunk objects.

Return type:

list[TextChunk]

class RecursiveChunker(chunk_size=1000, chunk_overlap=200, metadata=None, separators=None, keep_separator=True)[source]

Bases: ChunkerBase

Recursive chunking that splits hierarchically by separators.

Parameters:

  • chunk_size (int) – Maximum characters per chunk.

  • chunk_overlap (int) – Overlap between consecutive chunks.

  • metadata (dict[str, Any] | None) – Default metadata for all chunks.

  • separators (list[str] | None) – List of separators in order of preference.

  • keep_separator (bool) – Whether to keep the separator with chunks.

DEFAULT_SEPARATORS = ['\n\n\n', '\n\n', '\n', '. ', '! ', '? ', '; ', ', ', ' ', '']
chunk(text, metadata=None)[source]

Split text recursively using hierarchical separators.

Parameters:

  • text (str) – The text to split.

  • metadata (dict[str, Any] | None) – Optional metadata to attach to each chunk.

Returns:

List of TextChunk objects.

Return type:

list[TextChunk]

class TextChunk(content, chunk_index, start_char, end_char, metadata=None)[source]

Bases: object

A single chunk of text produced by a chunker.

Parameters:
content

The text content of the chunk.

Type:

str

chunk_index

Zero-based index of this chunk in the sequence.

Type:

int

start_char

Character offset where this chunk begins in the source.

Type:

int

end_char

Character offset where this chunk ends in the source.

Type:

int

metadata

Optional metadata attached to this chunk.

Type:

dict[str, Any] | None

content: str
chunk_index: int
start_char: int
end_char: int
metadata: dict[str, Any] | None = None
chunk_text(text, strategy='recursive', chunk_size=1000, chunk_overlap=200, metadata=None, **kwargs)[source]

Convenience function to chunk text with a single call.

Parameters:

  • text (str) – Text to chunk.

  • strategy (str) – Chunking strategy name.

  • chunk_size (int) – Maximum chunk size.

  • chunk_overlap (int) – Overlap between chunks.

  • metadata (dict[str, Any] | None) – Metadata to attach to chunks.

  • **kwargs (Any) – Additional strategy-specific parameters.

Returns:

List of TextChunk objects.

Return type:

list[TextChunk]

get_chunker(strategy='recursive', chunk_size=1000, chunk_overlap=200, metadata=None, **kwargs)[source]

Get a chunker instance based on strategy name.

Parameters:

  • strategy (str) – Chunking strategy name (character, recursive, markdown, code, hierarchical).

  • chunk_size (int) – Maximum chunk size.

  • chunk_overlap (int) – Overlap between chunks.

  • metadata (dict[str, Any] | None) – Default metadata for chunks.

  • **kwargs (Any) – Additional strategy-specific parameters.

Returns:

Configured chunker instance.

Raises:

ValueError – If strategy name is not recognized.

Return type:

ChunkerBase

list_chunkers()[source]

List available chunking strategies.

Returns:

List of strategy names.

Return type:

list[str]

Classes

class CharacterChunker(chunk_size=1000, chunk_overlap=200, metadata=None, respect_word_boundaries=True)[source]

Bases: ChunkerBase

Character-based chunking with word-boundary awareness.

Parameters:

  • chunk_size (int) – Maximum characters per chunk.

  • chunk_overlap (int) – Overlap between consecutive chunks.

  • metadata (dict[str, Any] | None) – Default metadata for all chunks.

  • respect_word_boundaries (bool) – Whether to prefer word boundaries for splits.

chunk(text, metadata=None)[source]

Split text into overlapping chunks.

Parameters:

  • text (str) – The text to split.

  • metadata (dict[str, Any] | None) – Optional metadata to attach to each chunk.

Returns:

List of TextChunk objects.

Return type:

list[TextChunk]

property name: str

Return the chunker strategy name.

class ChunkerBase(chunk_size=1000, chunk_overlap=200, metadata=None)[source]

Bases: ABC

Abstract base class for text chunking strategies.

Parameters:

  • chunk_size (int) – Maximum size of each chunk (interpretation varies by strategy).

  • chunk_overlap (int) – Overlap between consecutive chunks.

  • metadata (dict[str, Any] | None) – Default metadata to attach to all chunks.

abstractmethod chunk(text, metadata=None)[source]

Split text into chunks.

Parameters:

  • text (str) – The text to split.

  • metadata (dict[str, Any] | None) – Optional metadata to attach to each chunk (merged with default).

Returns:

List of TextChunk objects.

Return type:

list[TextChunk]

property name: str

Return the chunker strategy name.

class CodeChunker(chunk_size=1000, chunk_overlap=200, metadata=None, language='python', split_by='function')[source]

Bases: ChunkerBase

Code-aware chunking that splits by functions/classes.

Parameters:

  • chunk_size (int) – Maximum characters per chunk.

  • chunk_overlap (int) – Overlap between consecutive chunks.

  • metadata (dict[str, Any] | None) – Default metadata for all chunks.

  • language (str) – Programming language for parsing hints.

  • split_by (str) – Strategy: “function”, “class”, or “module”.

LANGUAGE_PATTERNS: dict[str, dict[str, str]] = {'generic': {'class': '^(class|struct|interface)\\s+\\w+', 'comment': '^(#|//|/\\*)', 'function': '^(function|func|def|fn)\\s+\\w+', 'import': '^(import|use|require)'}, 'go': {'class': '^type\\s+\\w+\\s+struct', 'comment': '^(//|/\\*)', 'function': '^func\\s+(\\(\\w+\\s+\\*?\\w+\\)\\s+)?\\w+\\s*\\(', 'import': '^import\\s+'}, 'java': {'class': '^\\s*(public\\s+)?(abstract\\s+)?class\\s+\\w+|interface\\s+\\w+', 'comment': '^(//|/\\*)', 'function': '^\\s*(public|private|protected|static)?\\s*\\w+\\s+\\w+\\s*\\(', 'import': '^import\\s+'}, 'javascript': {'class': '^(export\\s+)?class\\s+\\w+', 'comment': '^(//|/\\*|\\*)', 'function': '^(async\\s+)?function\\s+\\w+|const\\s+\\w+\\s*=\\s*(async\\s+)?\\([^)]*\\)\\s*=>|export\\s+(async\\s+)?function', 'import': '^(import\\s+|export\\s+|require\\s*\\()'}, 'python': {'class': '^class\\s+\\w+[\\(:]', 'comment': '^(#|\'\'\'|\\"\\"\\")', 'decorator': '^@\\w+', 'function': '^(async\\s+)?def\\s+\\w+\\s*\\(', 'import': '^(import\\s+|from\\s+\\S+\\s+import)'}, 'rust': {'class': '^(pub\\s+)?struct\\s+\\w+|impl\\s+\\w+', 'comment': '^(//|/\\*|\\*)', 'function': '^(pub\\s+)?(async\\s+)?fn\\s+\\w+', 'import': '^use\\s+'}, 'typescript': {'class': '^(export\\s+)?(abstract\\s+)?class\\s+\\w+|interface\\s+\\w+|type\\s+\\w+', 'comment': '^(//|/\\*|\\*)', 'function': '^(async\\s+)?(function\\s+\\w+|const\\s+\\w+\\s*=\\s*(async\\s+)?\\([^)]*\\)\\s*(:\\s*\\w+)?\\s*=>|export\\s+(async\\s+)?function)', 'import': '^(import\\s+|export\\s+|require\\s*\\()'}}
chunk(text, metadata=None)[source]

Split code by structural boundaries.

Parameters:

  • text (str) – The code text to split.

  • metadata (dict[str, Any] | None) – Optional metadata to attach to each chunk.

Returns:

List of TextChunk objects.

Return type:

list[TextChunk]

property name: str

Return the chunker strategy name.

class HierarchicalChunker(chunk_size=400, chunk_overlap=100, metadata=None, parent_chunk_size=1500, max_levels=2)[source]

Bases: ChunkerBase

Hierarchical chunking with parent-child relationships.

Parameters:

  • chunk_size (int) – Maximum characters per leaf chunk.

  • chunk_overlap (int) – Overlap between leaf chunks.

  • metadata (dict[str, Any] | None) – Default metadata for all chunks.

  • parent_chunk_size (int) – Size of parent chunks (larger).

  • max_levels (int) – Maximum hierarchy depth.

chunk(text, metadata=None)[source]

Split text into hierarchical chunks with parent-child relationships.

Parameters:

  • text (str) – The text to split.

  • metadata (dict[str, Any] | None) – Optional metadata to attach to each chunk.

Returns:

List of HierarchicalTextChunk objects with parent-child links.

Return type:

list[HierarchicalTextChunk]

get_parent_chunks(chunks)[source]

Filter to return only parent chunks.

Parameters:

chunks (list[HierarchicalTextChunk])

Return type:

list[HierarchicalTextChunk]

get_child_chunks(chunks)[source]

Filter to return only child chunks.

Parameters:

chunks (list[HierarchicalTextChunk])

Return type:

list[HierarchicalTextChunk]

get_chunks_with_parent_context(child_chunks, all_chunks)[source]

Get child chunks with their parent context for retrieval.

Parameters:
Returns:

List of dicts with child content and parent context.

Return type:

list[dict[str, Any]]

property name: str

Return the chunker strategy name.

class HierarchicalTextChunk(content, chunk_index, start_char, end_char, metadata=None, id='', parent_id=None, child_ids=None, hierarchy_level=0)[source]

Bases: TextChunk

A text chunk that participates in a parent-child hierarchy.

Used by hierarchical chunking strategies that maintain relationships between larger parent chunks and smaller child chunks.

Parameters:
id

Unique identifier for this chunk.

Type:

str

parent_id

Identifier of the parent chunk, or None if this is a root.

Type:

str | None

child_ids

Identifiers of child chunks, or an empty list.

Type:

list[str] | None

hierarchy_level

Depth in the hierarchy (0 = root).

Type:

int

id: str = ''
parent_id: str | None = None
child_ids: list[str] | None = None
hierarchy_level: int = 0
metadata: dict[str, Any] | None = None
content: str
chunk_index: int
start_char: int
end_char: int
class MarkdownChunker(chunk_size=1000, chunk_overlap=200, metadata=None, split_headers=None, preserve_structure=True, max_chunk_fallback=True)[source]

Bases: ChunkerBase

Markdown-aware chunking that splits by headers.

Parameters:

  • chunk_size (int) – Maximum characters per chunk.

  • chunk_overlap (int) – Overlap between consecutive chunks.

  • metadata (dict[str, Any] | None) – Default metadata for all chunks.

  • split_headers (list[str] | None) – List of header levels to split on (e.g., [“h1”, “h2”]).

  • preserve_structure (bool) – Whether to include header in chunk content.

  • max_chunk_fallback (bool) – Whether to further split large sections.

HEADER_PATTERNS = {'h1': '^# .+', 'h2': '^## .+', 'h3': '^### .+', 'h4': '^#### .+', 'h5': '^##### .+', 'h6': '^###### .+'}
chunk(text, metadata=None)[source]

Split markdown text by headers while respecting size limits.

Parameters:

  • text (str) – The markdown text to split.

  • metadata (dict[str, Any] | None) – Optional metadata to attach to each chunk.

Returns:

List of TextChunk objects.

Return type:

list[TextChunk]

property name: str

Return the chunker strategy name.

class RecursiveChunker(chunk_size=1000, chunk_overlap=200, metadata=None, separators=None, keep_separator=True)[source]

Bases: ChunkerBase

Recursive chunking that splits hierarchically by separators.

Parameters:

  • chunk_size (int) – Maximum characters per chunk.

  • chunk_overlap (int) – Overlap between consecutive chunks.

  • metadata (dict[str, Any] | None) – Default metadata for all chunks.

  • separators (list[str] | None) – List of separators in order of preference.

  • keep_separator (bool) – Whether to keep the separator with chunks.

DEFAULT_SEPARATORS = ['\n\n\n', '\n\n', '\n', '. ', '! ', '? ', '; ', ', ', ' ', '']
chunk(text, metadata=None)[source]

Split text recursively using hierarchical separators.

Parameters:

  • text (str) – The text to split.

  • metadata (dict[str, Any] | None) – Optional metadata to attach to each chunk.

Returns:

List of TextChunk objects.

Return type:

list[TextChunk]

property name: str

Return the chunker strategy name.

class TextChunk(content, chunk_index, start_char, end_char, metadata=None)[source]

Bases: object

A single chunk of text produced by a chunker.

Parameters:
content

The text content of the chunk.

Type:

str

chunk_index

Zero-based index of this chunk in the sequence.

Type:

int

start_char

Character offset where this chunk begins in the source.

Type:

int

end_char

Character offset where this chunk ends in the source.

Type:

int

metadata

Optional metadata attached to this chunk.

Type:

dict[str, Any] | None

content: str
chunk_index: int
start_char: int
end_char: int
metadata: dict[str, Any] | None = None

Functions

chunk_text(text, strategy='recursive', chunk_size=1000, chunk_overlap=200, metadata=None, **kwargs)[source]

Convenience function to chunk text with a single call.

Parameters:

  • text (str) – Text to chunk.

  • strategy (str) – Chunking strategy name.

  • chunk_size (int) – Maximum chunk size.

  • chunk_overlap (int) – Overlap between chunks.

  • metadata (dict[str, Any] | None) – Metadata to attach to chunks.

  • **kwargs (Any) – Additional strategy-specific parameters.

Returns:

List of TextChunk objects.

Return type:

list[TextChunk]

get_chunker(strategy='recursive', chunk_size=1000, chunk_overlap=200, metadata=None, **kwargs)[source]

Get a chunker instance based on strategy name.

Parameters:

  • strategy (str) – Chunking strategy name (character, recursive, markdown, code, hierarchical).

  • chunk_size (int) – Maximum chunk size.

  • chunk_overlap (int) – Overlap between chunks.

  • metadata (dict[str, Any] | None) – Default metadata for chunks.

  • **kwargs (Any) – Additional strategy-specific parameters.

Returns:

Configured chunker instance.

Raises:

ValueError – If strategy name is not recognized.

Return type:

ChunkerBase

list_chunkers()[source]

List available chunking strategies.

Returns:

List of strategy names.

Return type:

list[str]