promptfoo--promptfoo/docs/agents/python.md

Python Guidelines

For Python providers, prompts, assertions, and scripts.

Requirements

• Python 3.8 or later
• Follow Google Python Style Guide
• Use type hints for readability and error catching

Linting & Formatting

Use ruff for both:

ruff check --fix           # Lint with auto-fix
ruff check --select I --fix # Sort imports
ruff format                 # Format code

Testing

Use the built-in unittest module for new Python tests.

Best Practices

• Keep dependencies minimal - avoid unnecessary external packages
• When adding examples, update relevant requirements.txt files
• Follow promptfoo API patterns for custom providers/prompts/assertions
• Write unit tests for new Python functions

Provider Pattern

def call_api(prompt: str, options: dict, context: dict) -> dict:
    """
    Args:
        prompt: The prompt string
        options: Provider configuration
        context: Variables from the test case

    Returns:
        dict with 'output' key (and optionally 'error')
    """
    # Implementation
    return {"output": response_text}

Assertion Pattern

def get_assert(output: str, context: dict) -> dict:
    """
    Args:
        output: The provider's output
        context: Test context including vars

    Returns:
        dict with 'pass' (bool) and optionally 'reason'
    """
    return {"pass": True, "reason": "Validation passed"}