rag.indexing.bm25

BM25 sparse index for hybrid search.

class BM25Index(k1=1.5, b=0.75, epsilon=0.25)[source]

Bases: object

BM25 sparse index for keyword-based retrieval.

Implements the Okapi BM25 algorithm for text relevance scoring. Used alongside vector search for hybrid retrieval.

Parameters:

k1 (float) – BM25 k1 parameter (term frequency saturation).
b (float) – BM25 b parameter (document length normalization).
epsilon (float) – Small value to avoid division by zero.

tokenize(text)[source]

Tokenize text into terms.

Parameters:: text (str) – Text to tokenize.
Returns:: List of lowercase tokens.
Return type:: list[str]

add_document(doc_id, content, metadata=None)[source]

Add a document to the index.

Parameters:

doc_id (str) – Unique document identifier.
content (str) – Document text content.
metadata (dict[str, Any] | None) – Optional metadata dictionary.

Return type:

None

add_documents(documents, id_key='id', content_key='content')[source]

Add multiple documents to the index.

Parameters:

documents (list[dict[str, Any]]) – List of document dictionaries.
id_key (str) – Key for document ID in each dict.
content_key (str) – Key for document content in each dict.

Returns:

Number of documents added.

Return type:

int

delete_document(doc_id)[source]

Delete a document from the index.

Parameters:: doc_id (str) – Document ID to delete.
Returns:: True if document was deleted, False if not found.
Return type:: bool

search(query, n_results=5)[source]

Search for documents using BM25 scoring.

Parameters:

query (str) – Search query.
n_results (int) – Maximum number of results.

Returns:

List of result dicts with doc_id, score, content, metadata.

Return type:

list[dict[str, Any]]

clear()[source]

Clear all documents from the index.

Return type:: None

delete_by_metadata(key, value)[source]

Delete all documents matching a metadata key-value pair.

Parameters:

key (str) – Metadata key to match.
value (Any) – Metadata value to match.

Returns:

Number of documents deleted.

Return type:

int

count()[source]

Get the number of documents in the index.

Return type:: int

get_stats()[source]

Get index statistics.

Return type:: dict[str, Any]

Classes

class BM25Index(k1=1.5, b=0.75, epsilon=0.25)[source]

Bases: object

BM25 sparse index for keyword-based retrieval.

Implements the Okapi BM25 algorithm for text relevance scoring. Used alongside vector search for hybrid retrieval.

Parameters:

k1 (float) – BM25 k1 parameter (term frequency saturation).
b (float) – BM25 b parameter (document length normalization).
epsilon (float) – Small value to avoid division by zero.

tokenize(text)[source]

Tokenize text into terms.

Parameters:: text (str) – Text to tokenize.
Returns:: List of lowercase tokens.
Return type:: list[str]

add_document(doc_id, content, metadata=None)[source]

Add a document to the index.

Parameters:

doc_id (str) – Unique document identifier.
content (str) – Document text content.
metadata (dict[str, Any] | None) – Optional metadata dictionary.

Return type:

None

add_documents(documents, id_key='id', content_key='content')[source]

Add multiple documents to the index.

Parameters:

documents (list[dict[str, Any]]) – List of document dictionaries.
id_key (str) – Key for document ID in each dict.
content_key (str) – Key for document content in each dict.

Returns:

Number of documents added.

Return type:

int

delete_document(doc_id)[source]

Delete a document from the index.

Parameters:: doc_id (str) – Document ID to delete.
Returns:: True if document was deleted, False if not found.
Return type:: bool

search(query, n_results=5)[source]

Search for documents using BM25 scoring.

Parameters:

query (str) – Search query.
n_results (int) – Maximum number of results.

Returns:

List of result dicts with doc_id, score, content, metadata.

Return type:

list[dict[str, Any]]

clear()[source]

Clear all documents from the index.

Return type:: None

delete_by_metadata(key, value)[source]

Delete all documents matching a metadata key-value pair.

Parameters:

key (str) – Metadata key to match.
value (Any) – Metadata value to match.

Returns:

Number of documents deleted.

Return type:

int

count()[source]

Get the number of documents in the index.

Return type:: int

get_stats()[source]

Get index statistics.

Return type:: dict[str, Any]