rag.splitters.markdown

Split Markdown documents into chunks by header sections with fallback re-chunking.

class MarkdownChunker(chunk_size=1000, chunk_overlap=200, metadata=None, split_headers=None, preserve_structure=True, max_chunk_fallback=True)[source]

Bases: ChunkerBase

Markdown-aware chunking that splits by headers.

Parameters:

chunk_size (int) – Maximum characters per chunk.
chunk_overlap (int) – Overlap between consecutive chunks.
metadata (dict[str, Any] | None) – Default metadata for all chunks.
split_headers (list[str] | None) – List of header levels to split on (e.g., [“h1”, “h2”]).
preserve_structure (bool) – Whether to include header in chunk content.
max_chunk_fallback (bool) – Whether to further split large sections.

HEADER_PATTERNS = {'h1': '^# .+', 'h2': '^## .+', 'h3': '^### .+', 'h4': '^#### .+', 'h5': '^##### .+', 'h6': '^###### .+'}

chunk(text, metadata=None)[source]

Split markdown text by headers while respecting size limits.

Parameters:

text (str) – The markdown text to split.
metadata (dict[str, Any] | None) – Optional metadata to attach to each chunk.

Returns:

List of TextChunk objects.

Return type:

list[TextChunk]

Classes

class MarkdownChunker(chunk_size=1000, chunk_overlap=200, metadata=None, split_headers=None, preserve_structure=True, max_chunk_fallback=True)[source]

Bases: ChunkerBase

Markdown-aware chunking that splits by headers.

Parameters:

chunk_size (int) – Maximum characters per chunk.
chunk_overlap (int) – Overlap between consecutive chunks.
metadata (dict[str, Any] | None) – Default metadata for all chunks.
split_headers (list[str] | None) – List of header levels to split on (e.g., [“h1”, “h2”]).
preserve_structure (bool) – Whether to include header in chunk content.
max_chunk_fallback (bool) – Whether to further split large sections.

HEADER_PATTERNS = {'h1': '^# .+', 'h2': '^## .+', 'h3': '^### .+', 'h4': '^#### .+', 'h5': '^##### .+', 'h6': '^###### .+'}

chunk(text, metadata=None)[source]

Split markdown text by headers while respecting size limits.

Parameters:

text (str) – The markdown text to split.
metadata (dict[str, Any] | None) – Optional metadata to attach to each chunk.

Returns:

List of TextChunk objects.

Return type:

list[TextChunk]

property name: str: Return the chunker strategy name.