Memory Vector Recall

FFAI’s memory feature embeds completed Q+A turns and retrieves them by semantic similarity. It is opt-in and disabled by default, matching the rag.enabled convention.

This guide covers configuration, the embedding backend resolution ladder, persistence semantics, the Memory class API, and troubleshooting. For a hands-on walk-through, see Memory Vector Recall: Remember What Was Discussed.

How it works

When memory.enabled is true:

1. Each call to generate_response() with status="success" submits an embed task to a single-worker background thread.

2. The thread concatenates the prompt and response into a Q+A pair, embeds it, and appends it to an in-memory TurnVectorStore.

3. search() embeds the query and returns ranked TurnHit objects.

The embed runs fire-and-forget — it does not block the main generate_response() call. Failures are logged at WARNING and dropped; they never propagate to the caller.

Configuration

All settings live under the memory: key in config/main.yaml:

memory:
  enabled: false                    # opt-in; matches rag.enabled convention
  embedding_model: null             # null -> resolution ladder
  store_backend: memory             # reserved; Tier 1 uses in-memory store
  store_config: {}
  persist_dir: "./ffai_data/memory"
  collection_name: "ffai_turns"
  persist: false                    # ephemeral by default

Each field also has a constructor override on FFAI:

Constructor overrides

Keyword	Effect	Default
memory_enabled	Override config.memory.enabled. True enables; False disables regardless of config.	None (use config)
memory_embeddings	Override config.memory.embedding_model. None triggers the resolution ladder.	None (use config)
memory_persist	Override config.memory.persist.	None (use config)
memory	Supply a pre-built Memory instance. Bypasses the resolution ladder entirely. Useful for tests and for injecting a Memory with a custom embedding backend.	None

Priority order: constructor kwargs > environment variables > YAML.

Environment variables

Pydantic settings reads nested fields via __ delimiter. The env var for memory.enabled is:

MEMORY__ENABLED=true

There is no FFAI_ prefix (the Config class does not set env_prefix).

Embedding backend resolution

When memory.enabled is true and embedding_model is null, FFAI resolves a backend at construction time:

1. If embedding_model is set, try to construct Embeddings with it. ImportError → log a warning and disable memory.

2. Otherwise, try local/all-MiniLM-L6-v2. Works if fastembed or sentence-transformers is importable.

3. Otherwise, if MISTRAL_API_KEY is set, use mistral/mistral-embed.

4. Otherwise, if OPENAI_API_KEY is set, use openai/text-embedding-3-small.

5. Otherwise: log a warning and disable memory (ffai.history.memory becomes None, search returns []).

Local embeddings

Install one of:

pip install "ffai[memory]"    # fastembed only (~80 MB, no torch)
pip install "ffai[rag]"       # fastembed + chromadb (heavier)
pip install sentence-transformers   # alternative local backend

Local models are identified by a local/ prefix:

from ffai.core.embeddings import Embeddings

embeddings = Embeddings(model="local/all-MiniLM-L6-v2")

The first call downloads the model; subsequent calls use the cache.

API embeddings

Any LiteLLM-supported embedding provider works. Configure via embedding_model:

memory:
  enabled: true
  embedding_model: "mistral/mistral-embed"

Common options:

Persistence

When persist: true:

• Startup: FFAI loads <persist_dir>/<collection_name>.parquet if it exists. The loaded TurnVectorStore is wired into the active Memory instance.

• Per embed: after each successful index_turn_text, the store is rewritten to the same path.

The Parquet schema:

Writes are overwrite-on-persist, not append. Each write rewrites the full file — acceptable for Tier 1 scale (<100K turns). For larger workloads, future versions will support Chroma or Qdrant backends via store_backend (currently reserved).

Warning

Writes are not atomic against process death mid-write. If the process crashes during a Parquet write, the file may be corrupted on restart. Acceptable for short-running workflows; long-running services should snapshot-then-rename or implement periodic backups.

Schema versioning

The _schema_version column lets future releases migrate the format. Loaders reject unknown versions with a clear error rather than silently corrupting data:

from ffai.core.memory import load_store

store = load_store("turns.parquet")
# ValueError: Unsupported memory persistence schema version(s): [2].
# This build supports version 1 only.

TurnHit reference

class TurnHit(score, turn, turn_index, text, metadata)

Bases: object

A single ranked result returned by TurnVectorStore.search.

Parameters:

• score (float)

• turn (dict[str, Any])

• turn_index (int)

• text (str)

• metadata (dict[str, Any])

score

Cosine similarity in [-1.0, 1.0] between the query embedding and the stored turn’s embedding. Sorted descending by search().

Type:

float

turn

The raw turn dict stored at index time. Mirrors the PermanentHistory turn shape: {"role": str, "content": [{"type": "text", "text": str}], "timestamp": float, "metadata": dict}.

Type:

dict[str, Any]

turn_index

Position of the entry in TurnVectorStore at the time of the search. Stable within a single search() call for a given store; may shift after clear().

Type:

int

text

Pre-extracted plain text (typically the Q+A pair) that was embedded. Equal to the string passed to add(text=...).

Type:

str

metadata

Caller-provided metadata attached at index time. Always present (possibly empty dict). Tier 2 will populate user_id / session_id / agent_id here.

Type:

dict[str, Any]

score: float

turn: dict[str, Any]

turn_index: int

text: str

metadata: dict[str, Any]

Fields:

score

Cosine similarity in [-1.0, 1.0]. Sorted descending by search().

turn

The raw turn dict stored at index time. Mirrors the PermanentHistory turn shape: {"role": str, "content": [{"type": "text", "text": str}], "timestamp": float, "metadata": dict}.

turn_index

Position of the entry in the store at search time. Stable within a single search call; may shift after clear().

text

The plain text that was embedded. Equal to the Q+A pair (f"{prompt}\\n{response}").

metadata

Caller-provided metadata. Always present (possibly empty dict). Currently carries prompt_name; Tier 2 will add user_id, session_id, agent_id.

Memory class API

The Memory facade is exposed via ffai.history.memory (or None when disabled). You can also use it directly without an FFAI wrapper:

from ffai.core.embeddings import Embeddings
from ffai.core.memory import Memory

embeddings = Embeddings(model="local/all-MiniLM-L6-v2")
memory = Memory(embeddings)

memory.index_turn_text(
    text="user question\nassistant answer",
    turn={"role": "assistant", "content": [...], "timestamp": ...},
    metadata={"prompt_name": "demo"},
)

hits = memory.search("related question", top_k=3, threshold=0.3)

class Memory(embeddings, store=None)

Bases: object

Semantic recall over completed conversation turns.

Wraps an EmbeddingBackend and a TurnVectorStore. Provides two indexing entry points:

• index_turn() / aindex_turn() — extract text from a structured turn dict via turn["content"][0]["text"].

• index_turn_text() / aindex_turn_text() — embed an arbitrary caller-supplied string while still storing the structured turn dict alongside. Used by HistoryRecorder (L4) to embed the Q+A pair (f"{prompt}\n{response}") rather than just the response.

Parameters:

• embeddings (EmbeddingBackend) – Embedding backend (LiteLLM API model or local local/... model). Any object implementing EmbeddingBackend is accepted.

• store (TurnVectorStore | None) – Optional TurnVectorStore. Defaults to a fresh instance. Public and settable so callers can swap in a store loaded from Parquet (L3).

index_turn(turn, metadata=None)

Embed turn["content"][0]["text"] and store the turn.

Parameters:

• turn (dict[str, Any]) – Turn dict mirroring PermanentHistory shape. Must contain content[0]["text"].

• metadata (dict[str, Any] | None) – Optional caller metadata. Defaults to empty dict.

Returns:

The integer index of the stored entry.

Return type:

int

index_turn_text(text, turn, metadata=None)

Embed an arbitrary text and store the structured turn.

Use this when the embedded text differs from turn["content"][0]["text"] — e.g., when embedding the Q+A pair (f"{prompt}\n{response}") while still storing the response as the canonical turn content.

Parameters:

• text (str) – The plain text to embed.

• turn (dict[str, Any]) – The structured turn dict to store alongside.

• metadata (dict[str, Any] | None) – Optional caller metadata.

Returns:

The integer index of the stored entry.

Return type:

int

async aindex_turn(turn, metadata=None)

Async variant of index_turn().

Parameters:

• turn (dict[str, Any])

• metadata (dict[str, Any] | None)

Return type:

int

async aindex_turn_text(text, turn, metadata=None)

Async variant of index_turn_text().

Parameters:

• text (str)

• turn (dict[str, Any])

• metadata (dict[str, Any] | None)

Return type:

int

search(query, top_k=5, threshold=None)

Embed query and return ranked hits from the store.

Parameters:

• query (str) – Plain-text query.

• top_k (int) – Maximum hits to return.

• threshold (float | None) – Optional minimum cosine similarity.

Returns:

List of TurnHit sorted by score descending.

Return type:

list

async asearch(query, top_k=5, threshold=None)

Async variant of search().

Parameters:

• query (str)

• top_k (int)

• threshold (float | None)

Return type:

list

reindex(new_embeddings)

Re-embed all stored texts with a new embedding model.

Reads all entries via TurnVectorStore.iter_entries(), clears the store, and re-adds each entry with the new embedding. The turn dicts and metadata are preserved verbatim. After reindexing, self._embeddings is updated so subsequent search() calls embed queries with the new model.

Parameters:

new_embeddings (EmbeddingBackend) – The new embedding backend to use.

Return type:

None

count()

Number of turns currently indexed.

Return type:

int

clear()

Remove all indexed turns.

Return type:

None

class TurnVectorStore

Bases: object

Append-only in-memory store of embedded turns with cosine search.

Entries are stored in four parallel lists keyed by an integer index. add() returns the index of the appended entry; that index is preserved on the corresponding TurnHit returned by search().

Search is O(N) over all stored entries. Adequate for Tier 1 scale; a vector-index backend (Chroma/Qdrant) is a future concern.

add(text, embedding, turn, metadata=None)

Append an entry and return its integer index.

Parameters:

• text (str) – Plain text that was embedded.

• embedding (list[float]) – Float vector returned by the embedding backend.

• turn (dict[str, Any]) – Raw turn dict (mirrors PermanentHistory turn shape).

• metadata (dict[str, Any] | None) – Caller metadata. Defaults to an empty dict.

Returns:

The integer index of the newly appended entry.

Return type:

int

search(query_embedding, top_k=5, threshold=None)

Return up to top_k turns ranked by cosine similarity.

Parameters:

• query_embedding (list[float]) – Embedded query vector.

• top_k (int) – Maximum hits to return.

• threshold (float | None) – Optional minimum cosine similarity. Hits below this score are excluded. None means no floor.

Returns:

Hits sorted by score descending. Empty list if the store is empty or all hits fall below threshold.

Return type:

list[TurnHit]

iter_entries()

Yield all entries as Entry tuples in insertion order.

Used by Memory.reindex() and persist_store() to read the store’s full contents without exposing the underlying lists.

Return type:

Iterator[Entry]

count()

Number of entries currently in the store.

Return type:

int

clear()

Remove all entries. Subsequent count() returns 0.

Return type:

None

Persistence API

persist_store(store, path)

Write all entries in store to a Parquet file at path.

Overwrites any existing file. Empty stores produce a valid zero-row Parquet file.

Parameters:

• store (TurnVectorStore) – The TurnVectorStore to serialize.

• path (str) – Destination file path. Parent directories are created.

Return type:

None

load_store(path)

Load a Parquet file written by persist_store() into a fresh store.

Parameters:

path (str) – Source file path.

Returns:

A new TurnVectorStore whose entries and ordering match what was persisted.

Raises:

• FileNotFoundError – If path does not exist.

• ValueError – If the file’s _schema_version is not 1.

Return type:

TurnVectorStore

Lifecycle and resource cleanup

The background embed thread pool is owned by HistoryRecorder. It is created when a Memory is wired in and shut down by close():

ffai = FFAI(client, memory_enabled=True)
# ... use it ...
ffai.close()   # shuts down the embed thread pool

close() is idempotent. Calling it at process teardown avoids leaking daemon threads, which matters in long-running services.

If you forget to call close(), the daemon thread will not block interpreter shutdown (it is a daemon thread by default).

Troubleshooting

Memory is None even though I set memory_enabled=True

The resolution ladder fell through to step 5 (no backend available). Check:

1. Is fastembed or sentence-transformers importable? (pip install "ffai[memory]")

2. Is MISTRAL_API_KEY or OPENAI_API_KEY set?

3. Is config.memory.embedding_model pointing to a valid model?

A warning is logged at construction time when this happens:

Memory enabled but no embedding backend available.
Install ffai[memory] (or ffai[rag]) for local embeddings, or set
memory.embedding_model / MISTRAL_API_KEY / OPENAI_API_KEY.

Embeddings are slow

API embeddings add one network call per record(). Local embeddings (local/all-MiniLM-L6-v2) avoid the network but use CPU. The embed runs on a fire-and-forget background thread, so it does not block the main call — but it can fall behind if record() is called faster than embeds can complete.

Symptoms: ffai.history.memory.count() lags behind the number of generate_response() calls.

Search returns nothing relevant

• Lower threshold (default is no floor)

• Increase top_k

• Verify the embedding model matches the one used at index time. If you change embedding_model, existing vectors become invalid; call reindex() with the new backend.

Parquet file is corrupted after a crash

Overwrite-on-persist is not atomic. If your service crashes mid-write, recover from a backup or rebuild the store by re-running the original workflow. Future versions may add atomic writes (write-temp-then-rename).

See also

• Memory Vector Recall: Remember What Was Discussed — hands-on tutorial

• History & Query API — the four other history stores (raw, clean, ordered, permanent)

• Configuration — full configuration reference