Clients.AsyncFFLiteLLMClient

Async LiteLLM-backed AI client implementing AsyncFFAIClientBase contract.

Mirrors FFLiteLLMClient but uses litellm.acompletion() for async I/O. Shares all non-I/O logic via BaseLiteLLMClient.

Async LiteLLM-backed AI client implementing AsyncFFAIClientBase contract.

Mirrors FFLiteLLMClient but uses litellm.acompletion() for async I/O. Shares all non-I/O logic via BaseLiteLLMClient.

class AsyncFFLiteLLMClient(model_string, config=None, *, api_key=None, api_base=None, api_version=None, system_instructions=None, temperature=None, max_tokens=None, fallbacks=None, retry_config=None, **kwargs)[source]

Bases: BaseLiteLLMClient, AsyncFFAIClientBase

Async LiteLLM-backed AI client implementing AsyncFFAIClientBase.

Key features: - Internal conversation history management - Clone pattern for parallel execution - Model string routing (e.g., “azure/mistral-small-2503”) - Retry and fallback support

Parameters:

  • model_string (str) – LiteLLM model identifier.

  • config (dict[str, Any] | None) – Optional configuration dictionary.

  • api_key (str | None) – API key (overrides env var).

  • api_base (str | None) – API base URL (overrides env var).

  • system_instructions (str) – System prompt.

  • temperature (float) – Sampling temperature (0-2).

  • max_tokens (int) – Maximum tokens to generate.

  • fallbacks (list[str] | None) – List of fallback model strings.

  • retry_config (dict[str, Any] | None) – Retry configuration.

  • api_version (str | None)

  • kwargs (Any)

async generate_response(prompt, model=None, system_instructions=None, temperature=None, max_tokens=None, **kwargs)[source]

Generate a response from the model asynchronously.

Falls back to configured fallback models when the primary call fails.

Parameters:

  • prompt (str) – User’s input text.

  • model (str | None) – Model identifier override (preserves provider prefix if set).

  • system_instructions (str | None) – System prompt override.

  • temperature (float | None) – Sampling temperature override.

  • max_tokens (int | None) – Maximum tokens to generate override.

  • **kwargs (Any) – Additional parameters forwarded to litellm.acompletion().

Returns:

The model’s response text.

Raises:
Return type:

str

async clone()[source]

Create a deep copy of this client with reset usage and empty history.

Returns:

A new AsyncFFLiteLLMClient with identical configuration.

Return type:

AsyncFFLiteLLMClient

Classes

class AsyncFFLiteLLMClient(model_string, config=None, *, api_key=None, api_base=None, api_version=None, system_instructions=None, temperature=None, max_tokens=None, fallbacks=None, retry_config=None, **kwargs)[source]

Bases: BaseLiteLLMClient, AsyncFFAIClientBase

Async LiteLLM-backed AI client implementing AsyncFFAIClientBase.

Key features: - Internal conversation history management - Clone pattern for parallel execution - Model string routing (e.g., “azure/mistral-small-2503”) - Retry and fallback support

Parameters:

  • model_string (str) – LiteLLM model identifier.

  • config (dict[str, Any] | None) – Optional configuration dictionary.

  • api_key (str | None) – API key (overrides env var).

  • api_base (str | None) – API base URL (overrides env var).

  • system_instructions (str) – System prompt.

  • temperature (float) – Sampling temperature (0-2).

  • max_tokens (int) – Maximum tokens to generate.

  • fallbacks (list[str] | None) – List of fallback model strings.

  • retry_config (dict[str, Any] | None) – Retry configuration.

  • api_version (str | None)

  • kwargs (Any)

async generate_response(prompt, model=None, system_instructions=None, temperature=None, max_tokens=None, **kwargs)[source]

Generate a response from the model asynchronously.

Falls back to configured fallback models when the primary call fails.

Parameters:

  • prompt (str) – User’s input text.

  • model (str | None) – Model identifier override (preserves provider prefix if set).

  • system_instructions (str | None) – System prompt override.

  • temperature (float | None) – Sampling temperature override.

  • max_tokens (int | None) – Maximum tokens to generate override.

  • **kwargs (Any) – Additional parameters forwarded to litellm.acompletion().

Returns:

The model’s response text.

Raises:
Return type:

str

async clone()[source]

Create a deep copy of this client with reset usage and empty history.

Returns:

A new AsyncFFLiteLLMClient with identical configuration.

Return type:

AsyncFFLiteLLMClient

add_tool_result(tool_call_id, content)

Append a tool result message to the conversation history.

Parameters:

  • tool_call_id (str) – Provider-specific ID of the tool call being answered.

  • content (str) – The tool’s return value as a string.

Return type:

None

clear_conversation()

Remove all messages from the conversation history.

Return type:

None

configure_retry(retry_config=None)

Configure retry behavior for this client.

Parameters:

retry_config (dict[str, Any] | None) – Optional retry configuration. If None, uses global config.

Return type:

None

get_conversation_history()

Return a shallow copy of the conversation history.

Returns:

List of message dictionaries.

Return type:

list[dict[str, Any]]

static get_default_retry_config()

Get default retry configuration from global config.

Returns:

Dictionary with retry configuration parameters.

Return type:

dict[str, Any]

property last_cost_usd: float

Estimated cost in USD from the most recent generate_response() call.

property last_duration_ms: float | None

Wall-clock duration in ms of the most recent generate_response() call.

property last_usage: TokenUsage | None

Token usage from the most recent generate_response() call.

retry_config: dict[str, Any] | None = None
set_conversation_history(history)

Replace the conversation history with a new list of messages.

Parameters:

history (list[dict[str, Any]]) – List of message dictionaries to set.

Return type:

None

model: str
system_instructions: str
conversation_history: list[dict[str, Any]]
api_key: str | None
api_base: str | None
api_version: str | None
temperature: float
max_tokens: int