rag.splitters.base

Define the abstract ChunkerBase interface and the TextChunk and HierarchicalTextChunk data classes.

Define the abstract ChunkerBase interface and the TextChunk and HierarchicalTextChunk data classes.

class TextChunk(content, chunk_index, start_char, end_char, metadata=None)[source]

Bases: object

A single chunk of text produced by a chunker.

Parameters:
content

The text content of the chunk.

Type:

str

chunk_index

Zero-based index of this chunk in the sequence.

Type:

int

start_char

Character offset where this chunk begins in the source.

Type:

int

end_char

Character offset where this chunk ends in the source.

Type:

int

metadata

Optional metadata attached to this chunk.

Type:

dict[str, Any] | None

content: str
chunk_index: int
start_char: int
end_char: int
metadata: dict[str, Any] | None = None
class HierarchicalTextChunk(content, chunk_index, start_char, end_char, metadata=None, id='', parent_id=None, child_ids=None, hierarchy_level=0)[source]

Bases: TextChunk

A text chunk that participates in a parent-child hierarchy.

Used by hierarchical chunking strategies that maintain relationships between larger parent chunks and smaller child chunks.

Parameters:
id

Unique identifier for this chunk.

Type:

str

parent_id

Identifier of the parent chunk, or None if this is a root.

Type:

str | None

child_ids

Identifiers of child chunks, or an empty list.

Type:

list[str] | None

hierarchy_level

Depth in the hierarchy (0 = root).

Type:

int

id: str = ''
parent_id: str | None = None
child_ids: list[str] | None = None
hierarchy_level: int = 0
class ChunkerBase(chunk_size=1000, chunk_overlap=200, metadata=None)[source]

Bases: ABC

Abstract base class for text chunking strategies.

Parameters:

  • chunk_size (int) – Maximum size of each chunk (interpretation varies by strategy).

  • chunk_overlap (int) – Overlap between consecutive chunks.

  • metadata (dict[str, Any] | None) – Default metadata to attach to all chunks.

abstractmethod chunk(text, metadata=None)[source]

Split text into chunks.

Parameters:

  • text (str) – The text to split.

  • metadata (dict[str, Any] | None) – Optional metadata to attach to each chunk (merged with default).

Returns:

List of TextChunk objects.

Return type:

list[TextChunk]

property name: str

Return the chunker strategy name.

Classes

class ChunkerBase(chunk_size=1000, chunk_overlap=200, metadata=None)[source]

Bases: ABC

Abstract base class for text chunking strategies.

Parameters:

  • chunk_size (int) – Maximum size of each chunk (interpretation varies by strategy).

  • chunk_overlap (int) – Overlap between consecutive chunks.

  • metadata (dict[str, Any] | None) – Default metadata to attach to all chunks.

abstractmethod chunk(text, metadata=None)[source]

Split text into chunks.

Parameters:

  • text (str) – The text to split.

  • metadata (dict[str, Any] | None) – Optional metadata to attach to each chunk (merged with default).

Returns:

List of TextChunk objects.

Return type:

list[TextChunk]

property name: str

Return the chunker strategy name.

class HierarchicalTextChunk(content, chunk_index, start_char, end_char, metadata=None, id='', parent_id=None, child_ids=None, hierarchy_level=0)[source]

Bases: TextChunk

A text chunk that participates in a parent-child hierarchy.

Used by hierarchical chunking strategies that maintain relationships between larger parent chunks and smaller child chunks.

Parameters:
id

Unique identifier for this chunk.

Type:

str

parent_id

Identifier of the parent chunk, or None if this is a root.

Type:

str | None

child_ids

Identifiers of child chunks, or an empty list.

Type:

list[str] | None

hierarchy_level

Depth in the hierarchy (0 = root).

Type:

int

id: str = ''
parent_id: str | None = None
child_ids: list[str] | None = None
hierarchy_level: int = 0
metadata: dict[str, Any] | None = None
content: str
chunk_index: int
start_char: int
end_char: int
class TextChunk(content, chunk_index, start_char, end_char, metadata=None)[source]

Bases: object

A single chunk of text produced by a chunker.

Parameters:
content

The text content of the chunk.

Type:

str

chunk_index

Zero-based index of this chunk in the sequence.

Type:

int

start_char

Character offset where this chunk begins in the source.

Type:

int

end_char

Character offset where this chunk ends in the source.

Type:

int

metadata

Optional metadata attached to this chunk.

Type:

dict[str, Any] | None

content: str
chunk_index: int
start_char: int
end_char: int
metadata: dict[str, Any] | None = None