rag.indexing.contextual

Contextual embeddings for improved chunk retrieval.

Contextual embeddings for improved chunk retrieval.

class ContextualEmbeddings(context_template=None, max_context_length=200)

Bases: object

Generate embeddings with document context prepended.

Prepends document context (title, summary, or preceding content) to each chunk before embedding, improving semantic understanding.

Parameters:

• context_prefix – Template for context prefix.

• max_context_length (int) – Maximum characters for context prefix.

• context_template (str | None)

DEFAULT_CONTEXT_TEMPLATE = 'Document: {title}\n\nSection: {section}\n\n{chunk}'

prepare_chunk_for_embedding(chunk_content, document_title=None, section_header=None, document_summary=None, preceding_context=None)

Prepare a chunk with context for embedding.

Parameters:

• chunk_content (str) – The chunk text content.

• document_title (str | None) – Document title or name.

• section_header (str | None) – Section header if available.

• document_summary (str | None) – Brief document summary.

• preceding_context (str | None) – Text immediately preceding this chunk.

Returns:

Context-enhanced text for embedding.

Return type:

str

prepare_chunks_batch(chunks, document_title=None, document_summary=None)

Prepare multiple chunks with context for embedding.

Parameters:

• chunks (list[dict[str, Any]]) – List of chunk dictionaries with ‘content’ and optional metadata.

• document_title (str | None) – Document title for all chunks.

• document_summary (str | None) – Document summary (currently unused but available).

Returns:

List of context-enhanced texts for embedding.

Return type:

list[str]

class LateChunkingEmbeddings(embedding_model=None)

Bases: object

Late chunking strategy for token-level embeddings.

Instead of embedding chunks separately, embeds the full document and extracts chunk-level representations from the token embeddings.

This is a placeholder for ColBERT-style late interaction embeddings. Full implementation would require a model that outputs token-level embeddings and late interaction scoring.

Parameters:

embedding_model (Any | None) – The underlying embedding model.

embed_document_with_tokens(document_id, content, chunk_boundaries)

Embed document and extract chunk representations.

Note: This is a simplified implementation. A full ColBERT-style implementation would use a model that outputs token embeddings.

Parameters:

• document_id (str) – Document identifier.

• content (str) – Full document content.

• chunk_boundaries (list[tuple[int, int]]) – List of (start, end) tuples for chunks.

Returns:

List of chunk embeddings.

Return type:

list[list[float]]

Classes

class ContextualEmbeddings(context_template=None, max_context_length=200)

Bases: object

Generate embeddings with document context prepended.

Prepends document context (title, summary, or preceding content) to each chunk before embedding, improving semantic understanding.

Parameters:

• context_prefix – Template for context prefix.

• max_context_length (int) – Maximum characters for context prefix.

• context_template (str | None)

DEFAULT_CONTEXT_TEMPLATE = 'Document: {title}\n\nSection: {section}\n\n{chunk}'

prepare_chunk_for_embedding(chunk_content, document_title=None, section_header=None, document_summary=None, preceding_context=None)

Prepare a chunk with context for embedding.

Parameters:

• chunk_content (str) – The chunk text content.

• document_title (str | None) – Document title or name.

• section_header (str | None) – Section header if available.

• document_summary (str | None) – Brief document summary.

• preceding_context (str | None) – Text immediately preceding this chunk.

Returns:

Context-enhanced text for embedding.

Return type:

str

prepare_chunks_batch(chunks, document_title=None, document_summary=None)

Prepare multiple chunks with context for embedding.

Parameters:

• chunks (list[dict[str, Any]]) – List of chunk dictionaries with ‘content’ and optional metadata.

• document_title (str | None) – Document title for all chunks.

• document_summary (str | None) – Document summary (currently unused but available).

Returns:

List of context-enhanced texts for embedding.

Return type:

list[str]

class LateChunkingEmbeddings(embedding_model=None)

Bases: object

Late chunking strategy for token-level embeddings.

Instead of embedding chunks separately, embeds the full document and extracts chunk-level representations from the token embeddings.

This is a placeholder for ColBERT-style late interaction embeddings. Full implementation would require a model that outputs token-level embeddings and late interaction scoring.

Parameters:

embedding_model (Any | None) – The underlying embedding model.

embed_document_with_tokens(document_id, content, chunk_boundaries)

Embed document and extract chunk representations.

Note: This is a simplified implementation. A full ColBERT-style implementation would use a model that outputs token embeddings.

Parameters:

• document_id (str) – Document identifier.

• content (str) – Full document content.

• chunk_boundaries (list[tuple[int, int]]) – List of (start, end) tuples for chunks.

Returns:

List of chunk embeddings.

Return type:

list[list[float]]