Memory Vector Recall: Remember What Was Discussed

In this tutorial you will add semantic recall to an FFAI-powered assistant. By the end, your assistant will retrieve past Q+A turns by meaning, not by exact keyword match — so a later question like “the database issue from earlier” finds the turn about a Postgres migration even though neither “database” nor “issue” appears in that turn.

You’ll learn:

• How to enable memory in FFAI

• How generate_response() eagerly embeds each Q+A pair

• How ffai.history.search() retrieves semantically related turns

• How to filter by similarity threshold

• How to persist memory across process restarts

Prerequisites

• Python >= 3.10

• pip install "ffai[memory]" (pulls fastembed for local embeddings)

• A Mistral API key (or any LiteLLM-supported provider) set as MISTRAL_API_KEY

Note

The companion notebook examples/memory_basics/memory_basics.ipynb runs the same examples with a local embedding backend and no API key. Use it if you want to try the code without setting up a provider.

Step 1: Enable memory

Memory is opt-in. Pass memory_enabled=True to FFAI:

import os

from ffai.Clients import FFLiteLLMClient
from ffai import FFAI

client = FFLiteLLMClient(
    model_string="mistral/mistral-small-latest",
    api_key=os.environ["MISTRAL_API_KEY"],
)

ffai = FFAI(client, memory_enabled=True)

At construction time FFAI resolves an embedding backend via this ladder:

1. config.memory.embedding_model (if set in config/main.yaml)

2. local/all-MiniLM-L6-v2 if fastembed or sentence-transformers is importable

3. mistral/mistral-embed if MISTRAL_API_KEY is set

4. openai/text-embedding-3-small if OPENAI_API_KEY is set

5. Otherwise: memory is disabled with a warning

You can also enable memory via configuration instead of the constructor keyword — see Memory Vector Recall.

Step 2: Generate turns — embedding happens automatically

Every successful generate_response() call embeds the Q+A pair (f"{prompt}\\n{response}") on a fire-and-forget background thread. The call returns immediately; embedding does not block the response.

prompts = [
    ("Let's debug the Postgres migration", "postgres_debug"),
    ("Now update the user table schema", "schema_update"),
    ("What is Python's GIL?", "gil_concept"),
    ("How do I deploy to AWS?", "aws_deploy"),
    ("Write a haiku about autumn", "haiku"),
]

for prompt, prompt_name in prompts:
    result = ffai.workflow.generate_response(prompt=prompt, prompt_name=prompt_name)
    print(f"{prompt_name}: {str(result.response)[:60]}...")

Output (response text is illustrative):

postgres_debug: The migration issue is the missing index on users.email...
schema_update: ALTER TABLE users ADD COLUMN verified_at TIMESTAMP...
gil_concept: The Global Interpreter Lock prevents multiple native...
aws_deploy: Use the AWS CDK to define infrastructure as TypeScript...
haiku: Red leaves whisper down / Branches bare against the sky...

The prompt_name you pass becomes metadata["prompt_name"] on the indexed turn, which is how you’ll trace hits back to specific prompts.

Step 3: Semantic search — find related turns by meaning

Call ffai.history.search(query, top_k) to retrieve turns ranked by cosine similarity to the query:

hits = ffai.history.search("the database issue from earlier", top_k=3)

for i, hit in enumerate(hits, 1):
    print(f"{i}. [{hit.score:.3f}] {hit.metadata['prompt_name']}")
    print(f"   {hit.text[:80]}...")

Output (scores are deterministic for a given embedding model):

1. [0.407] schema_update
   Now update the user table schema
   ALTER TABLE users ADD COLUMN verified_at TIMESTAMP DEFAULT NO...
2. [0.291] postgres_debug
   Let's debug the Postgres migration
   The migration issue is the missing index on users.email...
3. [0.056] haiku
   Write a haiku about autumn
   Red leaves whisper down / Branches bare against the sky...

Notice the query said “database issue” — words that appear in none of the indexed turns. Linear substring search would return nothing. Vector recall catches the schema-update and Postgres turns because they are semantically related to databases.

Step 4: Inspect TurnHit fields

Each hit is a frozen dataclass with five fields:

hit = ffai.history.search("Python threading", top_k=1)[0]

print(f"score:       {hit.score:.3f}  (cosine similarity, [-1.0, 1.0])")
print(f"turn_index:  {hit.turn_index}  (position in the store)")
print(f"text:        {hit.text[:60]}...")
print(f"metadata:    {hit.metadata}")
print(f"turn.role:   {hit.turn['role']}")

Output:

score:       0.566  (cosine similarity, [-1.0, 1.0])
turn_index:  2  (position in the store)
text:        What is Python's GIL?...
metadata:    {'prompt_name': 'gil_concept'}
turn.role:   assistant

metadata is where Tier 2 (auto context injection) will later carry user_id / session_id for cross-user scoping. For now it carries whatever you passed via prompt_name.

Step 5: Filter by similarity threshold

Pass threshold=N to drop hits below a cosine similarity floor. Useful when you’d rather get [] than a noisy long tail:

query = "ship code to production"

for threshold in [None, 0.3, 0.5]:
    hits = ffai.history.search(query, top_k=5, threshold=threshold)
    label = "no floor" if threshold is None else f">= {threshold}"
    print(f"Threshold {label:>10}: {len(hits)} hits")

Output:

Threshold   no floor: 5 hits
Threshold     >= 0.3: 1 hits
Threshold     >= 0.5: 0 hits

A typical starting threshold is 0.3 — high enough to drop unrelated turns, low enough to catch synonym-heavy matches.

Step 6: Persist memory across restarts

By default, memory is ephemeral — the store dies with the process. Set memory_persist=True to load and write a Parquet file:

ffai = FFAI(
    client,
    memory_enabled=True,
    memory_persist=True,
)

At startup, FFAI loads <persist_dir>/<collection_name>.parquet if it exists. After each successful embed, it writes the file. On restart, prior turns are searchable immediately — no re-embedding required.

The defaults (configurable in config/main.yaml):

memory:
  persist_dir: "./ffai_data/memory"
  collection_name: "ffai_turns"

Step 7: Disable search returns empty

When memory is disabled, ffai.history.search() returns [] — it never raises. This means downstream code can call search() unconditionally:

ffai_off = FFAI(client)  # memory_enabled defaults to False

print(ffai_off.history.memory)          # None
print(ffai_off.history.search("x"))     # []

Complete listing

import os

from ffai.Clients import FFLiteLLMClient
from ffai import FFAI

client = FFLiteLLMClient(
    model_string="mistral/mistral-small-latest",
    api_key=os.environ["MISTRAL_API_KEY"],
)

ffai = FFAI(client, memory_enabled=True)

for prompt, name in [
    ("Let's debug the Postgres migration", "postgres_debug"),
    ("Now update the user table schema", "schema_update"),
    ("What is Python's GIL?", "gil_concept"),
]:
    ffai.workflow.generate_response(prompt=prompt, prompt_name=name)

hits = ffai.history.search("the database issue", top_k=2, threshold=0.3)
for hit in hits:
    print(f"[{hit.score:.3f}] {hit.metadata['prompt_name']}")

Next steps

• Memory Vector Recall — configuration reference, embedding backend resolution, persistence semantics, troubleshooting

• Runnable notebooks in examples/:

  • memory_basics/memory_basics.ipynb — Memory class directly with local fastembed (no API key)

  • memory_persistence/memory_persistence.ipynb — Parquet round-trip

  • memory_ffai_integration/memory_ffai_integration.ipynb — full FFAI wiring with a mock client (no API key)

• History & Query API — the four other history stores that complement memory vector recall