Clients

class AsyncFFLiteLLMClient(model_string, config=None, *, api_key=None, api_base=None, api_version=None, system_instructions=None, temperature=None, max_tokens=None, fallbacks=None, retry_config=None, **kwargs)[source]

Bases: BaseLiteLLMClient, AsyncFFAIClientBase

Async LiteLLM-backed AI client implementing AsyncFFAIClientBase.

Key features: - Internal conversation history management - Clone pattern for parallel execution - Model string routing (e.g., “azure/mistral-small-2503”) - Retry and fallback support

Parameters:

  • model_string (str) – LiteLLM model identifier.

  • config (dict[str, Any] | None) – Optional configuration dictionary.

  • api_key (str | None) – API key (overrides env var).

  • api_base (str | None) – API base URL (overrides env var).

  • system_instructions (str) – System prompt.

  • temperature (float) – Sampling temperature (0-2).

  • max_tokens (int) – Maximum tokens to generate.

  • fallbacks (list[str] | None) – List of fallback model strings.

  • retry_config (dict[str, Any] | None) – Retry configuration.

  • api_version (str | None)

  • kwargs (Any)

async generate_response(prompt, model=None, system_instructions=None, temperature=None, max_tokens=None, **kwargs)[source]

Generate a response from the model asynchronously.

Falls back to configured fallback models when the primary call fails.

Parameters:

  • prompt (str) – User’s input text.

  • model (str | None) – Model identifier override (preserves provider prefix if set).

  • system_instructions (str | None) – System prompt override.

  • temperature (float | None) – Sampling temperature override.

  • max_tokens (int | None) – Maximum tokens to generate override.

  • **kwargs (Any) – Additional parameters forwarded to litellm.acompletion().

Returns:

The model’s response text.

Raises:
Return type:

str

async clone()[source]

Create a deep copy of this client with reset usage and empty history.

Returns:

A new AsyncFFLiteLLMClient with identical configuration.

Return type:

AsyncFFLiteLLMClient

class FFLiteLLMClient(model_string, config=None, *, api_key=None, api_base=None, api_version=None, system_instructions=None, temperature=None, max_tokens=None, fallbacks=None, retry_config=None, **kwargs)[source]

Bases: BaseLiteLLMClient, FFAIClientBase

LiteLLM-backed AI client implementing FFAIClientBase.

This client wraps LiteLLM’s completion() function while maintaining the FFAIClientBase contract for compatibility with FFAI wrapper.

Key features: - Internal conversation history management - Clone pattern for parallel execution - Model string routing (e.g., “azure/mistral-small-2503”) - Retry and fallback support

Parameters:

  • model_string (str) – LiteLLM model identifier (e.g., “openai/gpt-4”, “azure/my-deployment”)

  • config (dict[str, Any] | None) – Optional configuration dictionary

  • api_key (str | None) – API key (overrides env var)

  • api_base (str | None) – API base URL (overrides env var)

  • system_instructions (str) – System prompt

  • temperature (float) – Sampling temperature (0-2)

  • max_tokens (int) – Maximum tokens to generate

  • fallbacks (list[str] | None) – List of fallback model strings

  • retry_config (dict[str, Any] | None) – Retry configuration

  • api_version (str | None)

  • kwargs (Any)

Example

>>> client = FFLiteLLMClient(model_string="azure/mistral-small-2503")
>>> response = client.generate_response("Hello!")
>>>
>>> # With fallbacks
>>> client = FFLiteLLMClient(
...     model_string="anthropic/claude-3-opus",
...     fallbacks=["openai/gpt-4", "azure/gpt-4"]
... )
generate_response(prompt, model=None, system_instructions=None, temperature=None, max_tokens=None, **kwargs)[source]

Generate a response from the AI model with retry and fallback logic.

Retries are handled by retry_utils.get_configured_retry_decorator on the inner _call_primary method. If the primary model (and all its retries) fail, fallback models are tried once each.

Parameters:

  • prompt (str) – The user prompt

  • model (str | None) – Override model (appends to provider prefix)

  • system_instructions (str | None) – Override system instructions

  • temperature (float | None) – Override temperature

  • max_tokens (int | None) – Override max tokens

  • **kwargs (Any) – Additional LiteLLM parameters

Returns:

The generated response text

Raises:
Return type:

str

clone()[source]

Create a fresh clone of this client with empty history.

Used for thread-safe parallel execution where each thread needs an isolated client instance with the same configuration.

Returns:

New FFLiteLLMClient with same config, empty history.

Return type:

FFLiteLLMClient

class FFMistralSmall(config=None, **kwargs)[source]

Bases: FFAIClientBase

Mistral Small client with chat completions, function calling, and tool use.

Wraps the mistralai SDK to provide a consistent FFAIClientBase interface with built-in retry, conversation tracking, and tool-call handling for the Mistral Small family of models.

Parameters:

  • config (dict | None) – Optional configuration dictionary.

  • **kwargs – Keyword overrides for model, temperature, max_tokens, etc.

get_conversation_history()[source]

Get the conversation history.

Return type:

list[dict[str, Any]]

set_conversation_history(history)[source]

Set the conversation history.

Parameters:

history (list[dict[str, Any]])

Return type:

None

clone()[source]

Create a fresh clone of this client with empty history.

Return type:

FFMistralSmall

generate_response(prompt, model=None, system_instructions=None, temperature=None, max_tokens=None, max_completion_tokens=None, top_p=None, presence_penalty=None, frequency_penalty=None, stop=None, response_format=None, tools=None, tool_choice=None, safe_mode=None, **kwargs)[source]

Generate a response from the model with robust parameter handling.

Parameters:

  • prompt (str) – User’s input text

  • model (str | None) – Model to use (overrides default)

  • system_instructions (str | None) – System instructions (overrides default)

  • temperature (float | None) – Controls randomness (0-1)

  • max_tokens (int | None) – Maximum tokens to generate

  • max_completion_tokens (int | None) – Alternative name for max_tokens (for compatibility)

  • top_p (float | None) – Nucleus sampling parameter

  • presence_penalty (float | None) – Penalizes repeated tokens

  • frequency_penalty (float | None) – Penalizes frequent tokens

  • stop (list[str] | None) – List of strings that stop generation when encountered

  • response_format (str | dict | type | None) – Format of the response (“text” or “json”)

  • tools (list[dict] | None) – List of tools to make available to the model

  • tool_choice (str | None) – Tool choice strategy (“auto”, “none”, or specific tool)

  • safe_mode (bool | None) – Whether to enable Mistral’s safe mode

  • **kwargs – Any additional parameters that may be passed but aren’t used

Returns:

The model’s response as a string

Return type:

str

add_tool_result(tool_call_id, content)[source]

Add a tool result to the conversation history.

Parameters:

  • tool_call_id (str) – The ID of the tool call this is responding to

  • content (Any) – The content/result from the tool

Return type:

None

clear_conversation()[source]

Clear the conversation history.

test_connection()[source]

Test the connection to the Mistral API endpoint.

Returns:

True if connection is successful, False otherwise

Return type:

bool

Classes

class AsyncFFLiteLLMClient(model_string, config=None, *, api_key=None, api_base=None, api_version=None, system_instructions=None, temperature=None, max_tokens=None, fallbacks=None, retry_config=None, **kwargs)[source]

Bases: BaseLiteLLMClient, AsyncFFAIClientBase

Async LiteLLM-backed AI client implementing AsyncFFAIClientBase.

Key features: - Internal conversation history management - Clone pattern for parallel execution - Model string routing (e.g., “azure/mistral-small-2503”) - Retry and fallback support

Parameters:

  • model_string (str) – LiteLLM model identifier.

  • config (dict[str, Any] | None) – Optional configuration dictionary.

  • api_key (str | None) – API key (overrides env var).

  • api_base (str | None) – API base URL (overrides env var).

  • system_instructions (str) – System prompt.

  • temperature (float) – Sampling temperature (0-2).

  • max_tokens (int) – Maximum tokens to generate.

  • fallbacks (list[str] | None) – List of fallback model strings.

  • retry_config (dict[str, Any] | None) – Retry configuration.

  • api_version (str | None)

  • kwargs (Any)

async generate_response(prompt, model=None, system_instructions=None, temperature=None, max_tokens=None, **kwargs)[source]

Generate a response from the model asynchronously.

Falls back to configured fallback models when the primary call fails.

Parameters:

  • prompt (str) – User’s input text.

  • model (str | None) – Model identifier override (preserves provider prefix if set).

  • system_instructions (str | None) – System prompt override.

  • temperature (float | None) – Sampling temperature override.

  • max_tokens (int | None) – Maximum tokens to generate override.

  • **kwargs (Any) – Additional parameters forwarded to litellm.acompletion().

Returns:

The model’s response text.

Raises:
Return type:

str

async clone()[source]

Create a deep copy of this client with reset usage and empty history.

Returns:

A new AsyncFFLiteLLMClient with identical configuration.

Return type:

AsyncFFLiteLLMClient

add_tool_result(tool_call_id, content)

Append a tool result message to the conversation history.

Parameters:

  • tool_call_id (str) – Provider-specific ID of the tool call being answered.

  • content (str) – The tool’s return value as a string.

Return type:

None

clear_conversation()

Remove all messages from the conversation history.

Return type:

None

configure_retry(retry_config=None)

Configure retry behavior for this client.

Parameters:

retry_config (dict[str, Any] | None) – Optional retry configuration. If None, uses global config.

Return type:

None

get_conversation_history()

Return a shallow copy of the conversation history.

Returns:

List of message dictionaries.

Return type:

list[dict[str, Any]]

static get_default_retry_config()

Get default retry configuration from global config.

Returns:

Dictionary with retry configuration parameters.

Return type:

dict[str, Any]

property last_cost_usd: float

Estimated cost in USD from the most recent generate_response() call.

property last_duration_ms: float | None

Wall-clock duration in ms of the most recent generate_response() call.

property last_usage: TokenUsage | None

Token usage from the most recent generate_response() call.

retry_config: dict[str, Any] | None = None
set_conversation_history(history)

Replace the conversation history with a new list of messages.

Parameters:

history (list[dict[str, Any]]) – List of message dictionaries to set.

Return type:

None

model: str
system_instructions: str
conversation_history: list[dict[str, Any]]
api_key: str | None
api_base: str | None
api_version: str | None
temperature: float
max_tokens: int
class FFLiteLLMClient(model_string, config=None, *, api_key=None, api_base=None, api_version=None, system_instructions=None, temperature=None, max_tokens=None, fallbacks=None, retry_config=None, **kwargs)[source]

Bases: BaseLiteLLMClient, FFAIClientBase

LiteLLM-backed AI client implementing FFAIClientBase.

This client wraps LiteLLM’s completion() function while maintaining the FFAIClientBase contract for compatibility with FFAI wrapper.

Key features: - Internal conversation history management - Clone pattern for parallel execution - Model string routing (e.g., “azure/mistral-small-2503”) - Retry and fallback support

Parameters:

  • model_string (str) – LiteLLM model identifier (e.g., “openai/gpt-4”, “azure/my-deployment”)

  • config (dict[str, Any] | None) – Optional configuration dictionary

  • api_key (str | None) – API key (overrides env var)

  • api_base (str | None) – API base URL (overrides env var)

  • system_instructions (str) – System prompt

  • temperature (float) – Sampling temperature (0-2)

  • max_tokens (int) – Maximum tokens to generate

  • fallbacks (list[str] | None) – List of fallback model strings

  • retry_config (dict[str, Any] | None) – Retry configuration

  • api_version (str | None)

  • kwargs (Any)

Example

>>> client = FFLiteLLMClient(model_string="azure/mistral-small-2503")
>>> response = client.generate_response("Hello!")
>>>
>>> # With fallbacks
>>> client = FFLiteLLMClient(
...     model_string="anthropic/claude-3-opus",
...     fallbacks=["openai/gpt-4", "azure/gpt-4"]
... )
generate_response(prompt, model=None, system_instructions=None, temperature=None, max_tokens=None, **kwargs)[source]

Generate a response from the AI model with retry and fallback logic.

Retries are handled by retry_utils.get_configured_retry_decorator on the inner _call_primary method. If the primary model (and all its retries) fail, fallback models are tried once each.

Parameters:

  • prompt (str) – The user prompt

  • model (str | None) – Override model (appends to provider prefix)

  • system_instructions (str | None) – Override system instructions

  • temperature (float | None) – Override temperature

  • max_tokens (int | None) – Override max tokens

  • **kwargs (Any) – Additional LiteLLM parameters

Returns:

The generated response text

Raises:
Return type:

str

clone()[source]

Create a fresh clone of this client with empty history.

Used for thread-safe parallel execution where each thread needs an isolated client instance with the same configuration.

Returns:

New FFLiteLLMClient with same config, empty history.

Return type:

FFLiteLLMClient

add_tool_result(tool_call_id, content)

Append a tool result message to the conversation history.

Parameters:

  • tool_call_id (str) – Provider-specific ID of the tool call being answered.

  • content (str) – The tool’s return value as a string.

Return type:

None

clear_conversation()

Remove all messages from the conversation history.

Return type:

None

configure_retry(retry_config=None)

Configure retry behavior for this client.

Parameters:

retry_config (dict[str, Any] | None) – Optional retry configuration. If None, uses global config.

Return type:

None

get_conversation_history()

Return a shallow copy of the conversation history.

Returns:

List of message dictionaries.

Return type:

list[dict[str, Any]]

static get_default_retry_config()

Get default retry configuration from global config.

Returns:

Dictionary with retry configuration parameters.

Return type:

dict[str, Any]

property last_cost_usd: float

Estimated cost in USD from the most recent generate_response() call.

property last_duration_ms: float | None

Wall-clock duration in ms of the most recent generate_response() call.

property last_usage: TokenUsage | None

Token usage from the most recent generate_response() call.

retry_config: dict[str, Any] | None = None
set_conversation_history(history)

Replace the conversation history with a new list of messages.

Parameters:

history (list[dict[str, Any]]) – List of message dictionaries to set.

Return type:

None

model: str
system_instructions: str
conversation_history: list[dict[str, Any]]
api_key: str | None
api_base: str | None
api_version: str | None
temperature: float
max_tokens: int
class FFMistralSmall(config=None, **kwargs)[source]

Bases: FFAIClientBase

Mistral Small client with chat completions, function calling, and tool use.

Wraps the mistralai SDK to provide a consistent FFAIClientBase interface with built-in retry, conversation tracking, and tool-call handling for the Mistral Small family of models.

Parameters:

  • config (dict | None) – Optional configuration dictionary.

  • **kwargs – Keyword overrides for model, temperature, max_tokens, etc.

model: str
system_instructions: str
get_conversation_history()[source]

Get the conversation history.

Return type:

list[dict[str, Any]]

set_conversation_history(history)[source]

Set the conversation history.

Parameters:

history (list[dict[str, Any]])

Return type:

None

clone()[source]

Create a fresh clone of this client with empty history.

Return type:

FFMistralSmall

configure_retry(retry_config=None)

Configure retry behavior for this client.

Parameters:

retry_config (dict[str, Any] | None) – Optional retry configuration. If None, uses global config.

Return type:

None

static get_default_retry_config()

Get default retry configuration from global config.

Returns:

Dictionary with retry configuration parameters.

Return type:

dict[str, Any]

property last_cost_usd: float

Estimated cost in USD from the most recent generate_response() call.

property last_duration_ms: float | None

Wall-clock duration in ms of the most recent generate_response() call.

property last_usage: TokenUsage | None

Token usage from the most recent generate_response() call.

retry_config: dict[str, Any] | None = None
generate_response(prompt, model=None, system_instructions=None, temperature=None, max_tokens=None, max_completion_tokens=None, top_p=None, presence_penalty=None, frequency_penalty=None, stop=None, response_format=None, tools=None, tool_choice=None, safe_mode=None, **kwargs)[source]

Generate a response from the model with robust parameter handling.

Parameters:

  • prompt (str) – User’s input text

  • model (str | None) – Model to use (overrides default)

  • system_instructions (str | None) – System instructions (overrides default)

  • temperature (float | None) – Controls randomness (0-1)

  • max_tokens (int | None) – Maximum tokens to generate

  • max_completion_tokens (int | None) – Alternative name for max_tokens (for compatibility)

  • top_p (float | None) – Nucleus sampling parameter

  • presence_penalty (float | None) – Penalizes repeated tokens

  • frequency_penalty (float | None) – Penalizes frequent tokens

  • stop (list[str] | None) – List of strings that stop generation when encountered

  • response_format (str | dict | type | None) – Format of the response (“text” or “json”)

  • tools (list[dict] | None) – List of tools to make available to the model

  • tool_choice (str | None) – Tool choice strategy (“auto”, “none”, or specific tool)

  • safe_mode (bool | None) – Whether to enable Mistral’s safe mode

  • **kwargs – Any additional parameters that may be passed but aren’t used

Returns:

The model’s response as a string

Return type:

str

add_tool_result(tool_call_id, content)[source]

Add a tool result to the conversation history.

Parameters:

  • tool_call_id (str) – The ID of the tool call this is responding to

  • content (Any) – The content/result from the tool

Return type:

None

clear_conversation()[source]

Clear the conversation history.

test_connection()[source]

Test the connection to the Mistral API endpoint.

Returns:

True if connection is successful, False otherwise

Return type:

bool