agent.response_validator

LLM-as-judge response validation with automatic retry.

Validates LLM responses by asking a second LLM call to judge PASS/FAIL. On failure, the original prompt is augmented with the rejection reason and re-executed up to max_retries times.

LLM-as-judge response validation with automatic retry.

Validates LLM responses by asking a second LLM call to judge PASS/FAIL. On failure, the original prompt is augmented with the rejection reason and re-executed up to max_retries times.

class ValidationResult(passed, attempts=0, critique=None)

Bases: object

Result of a response validation attempt.

Parameters:

• passed (bool | None)

• attempts (int)

• critique (str | None)

passed

True if PASS, False if FAIL, None if validation itself failed.

Type:

bool | None

attempts

Number of validation attempts made.

Type:

int

critique

Failure reason from the validator, or None if passed.

Type:

str | None

passed: bool | None

attempts: int = 0

critique: str | None = None

class ResponseValidator(client, model=None, temperature=0.1)

Bases: object

Validates LLM responses using a second LLM as judge.

Usage:

validator = ResponseValidator(client) result = validator.validate(

response=”The answer is 42”, criteria=”Must provide a numerical answer”,

) if result.passed:

print(“Response is valid”)

Parameters:

• client (FFAIClientBase) – FFAIClientBase used for validation LLM calls.

• model (str | None) – Model to use for validation (defaults to client model).

• temperature (float) – Temperature for validation calls (low for consistency).

validate(response, criteria, max_retries=2, re_execute_fn=None)

Validate a response against criteria.

Parameters:

• response (str) – The response text to validate.

• criteria (str) – Validation criteria description.

• max_retries (int) – Maximum re-execution attempts on failure.

• re_execute_fn (Any | None) – Optional callable that accepts an augmented prompt and returns a new response string. If provided, failed validations trigger re-execution with the rejection reason.

Returns:

ValidationResult with pass/fail status and critique.

Return type:

ValidationResult

Classes

class ResponseValidator(client, model=None, temperature=0.1)

Bases: object

Validates LLM responses using a second LLM as judge.

Usage:

validator = ResponseValidator(client) result = validator.validate(

response=”The answer is 42”, criteria=”Must provide a numerical answer”,

) if result.passed:

print(“Response is valid”)

Parameters:

• client (FFAIClientBase) – FFAIClientBase used for validation LLM calls.

• model (str | None) – Model to use for validation (defaults to client model).

• temperature (float) – Temperature for validation calls (low for consistency).

validate(response, criteria, max_retries=2, re_execute_fn=None)

Validate a response against criteria.

Parameters:

• response (str) – The response text to validate.

• criteria (str) – Validation criteria description.

• max_retries (int) – Maximum re-execution attempts on failure.

• re_execute_fn (Any | None) – Optional callable that accepts an augmented prompt and returns a new response string. If provided, failed validations trigger re-execution with the rejection reason.

Returns:

ValidationResult with pass/fail status and critique.

Return type:

ValidationResult

class ValidationResult(passed, attempts=0, critique=None)

Bases: object

Result of a response validation attempt.

Parameters:

• passed (bool | None)

• attempts (int)

• critique (str | None)

passed

True if PASS, False if FAIL, None if validation itself failed.

Type:

bool | None

attempts

Number of validation attempts made.

Type:

int

critique

Failure reason from the validator, or None if passed.

Type:

str | None

passed: bool | None

attempts: int = 0

critique: str | None = None