chore: import upstream snapshot with attribution

examples/integration-strands-agents/README.md

@@ -0,0 +1,108 @@
+# integration-strands-agents (Strands Agents SDK example)
+
+This example demonstrates how to evaluate [Strands Agents SDK](https://github.com/strands-agents/sdk-python) with [promptfoo](https://promptfoo.dev).
+
+[Strands Agents](https://strandsagents.com/) is an open-source AI agent framework developed by [AWS](https://github.com/strands-agents) that provides a model-driven approach to building AI agents.
+
+You can run this example with:
+
+```bash
+npx promptfoo@latest init --example integration-strands-agents
+cd integration-strands-agents
+```
+
+## Overview
+
+This example showcases:
+
+- Creating a [Strands agent](https://strandsagents.com/latest/user-guide/concepts/agents/) with custom tools
+- Using the [`@tool` decorator](https://strandsagents.com/latest/user-guide/concepts/tools/python-tools/) to define agent capabilities
+- Evaluating agent responses with various [promptfoo assertions](https://promptfoo.dev/docs/configuration/expected-outputs/)
+- Testing tool usage with mock weather and temperature conversion tools
+
+## Prerequisites
+
+- Python 3.9+
+- [OpenAI API key](https://platform.openai.com/api-keys) (default) or other supported provider
+
+## Setup
+
+### 1. Install Python dependencies
+
+```bash
+pip install -r requirements.txt
+```
+
+This installs:
+
+- [`strands-agents[openai]`](https://pypi.org/project/strands-agents/) - The Strands Agents SDK with OpenAI support
+- [`pydantic`](https://docs.pydantic.dev/) - Data validation library required by Strands
+
+### 2. Set environment variables
+
+```bash
+export OPENAI_API_KEY=your-api-key-here
+```
+
+### Alternative: use Anthropic or Bedrock
+
+[Strands supports multiple model providers](https://strandsagents.com/latest/user-guide/concepts/model-providers/). To use [Anthropic](https://www.anthropic.com/):
+
+```bash
+pip install 'strands-agents[anthropic]'
+export ANTHROPIC_API_KEY=your-key
+```
+
+Then modify `agent.py` to use [`AnthropicModel`](https://strandsagents.com/latest/user-guide/concepts/model-providers/anthropic/) instead of [`OpenAIModel`](https://strandsagents.com/latest/user-guide/concepts/model-providers/openai/).
+
+To use [Amazon Bedrock](https://strandsagents.com/latest/user-guide/concepts/model-providers/amazon-bedrock/):
+
+```bash
+pip install 'strands-agents[bedrock]'
+```
+
+## Running the example
+
+```bash
+# Run evaluation
+npx promptfoo eval
+
+# View results in the web UI
+npx promptfoo view
+```
+
+## How it works
+
+### Agent structure
+
+The agent is defined in `agent.py` using the [Strands Agent class](https://strandsagents.com/latest/user-guide/concepts/agents/) with two tools:
+
+- `get_weather`: Returns mock weather data for cities (New York, London, Tokyo, Paris, Seattle, San Francisco)
+- `convert_temperature`: Converts temperatures between Fahrenheit and Celsius
+
+Tools are defined using the [`@tool` decorator](https://strandsagents.com/latest/user-guide/concepts/tools/python-tools/) which automatically exposes them to the LLM based on their docstrings.
+
+### Provider integration
+
+`agent_provider.py` exposes a `call_api` function that [promptfoo's Python provider](https://promptfoo.dev/docs/providers/python/) calls to interact with the Strands agent.
+
+### Test cases and assertion types
+
+The [promptfoo config](https://promptfoo.dev/docs/configuration/guide/) includes 5 test cases that demonstrate different [assertion types](https://promptfoo.dev/docs/configuration/expected-outputs/):
+
+| Test                                | Description                | Assertion types used                    |
+| ----------------------------------- | -------------------------- | --------------------------------------- |
+| Weather query for New York          | Basic tool usage           | `contains-any`, `llm-rubric`, `latency` |
+| Weather query for London            | Verify temperature format  | `contains-any`, `javascript`, `latency` |
+| Weather query for Tokyo             | Case-insensitive matching  | `icontains`, `javascript`, `latency`    |
+| Weather with temperature conversion | Multi-tool chaining        | `llm-rubric`, `javascript`, `latency`   |
+| Weather for unknown city            | Graceful fallback handling | `icontains`, `not-contains`, `latency`  |
+
+#### Assertion types explained
+
+- **[`latency`](https://promptfoo.dev/docs/configuration/expected-outputs/#latency)** - Ensures responses complete within 30 seconds (applied to all tests via `defaultTest`)
+- **[`contains-any`](https://promptfoo.dev/docs/configuration/expected-outputs/#contains)** - Verifies the agent returns expected city names and weather data from the mock tool
+- **[`icontains`](https://promptfoo.dev/docs/configuration/expected-outputs/#contains)** - Case-insensitive matching to verify city names appear regardless of formatting
+- **[`not-contains`](https://promptfoo.dev/docs/configuration/expected-outputs/#not-contains)** - Ensures the agent handles unknown cities gracefully without error messages
+- **[`javascript`](https://promptfoo.dev/docs/configuration/expected-outputs/#javascript)** - Validates temperature format (°F/°C symbols) and response length requirements
+- **[`llm-rubric`](https://promptfoo.dev/docs/configuration/expected-outputs/model-graded/)** - Semantically evaluates whether the agent correctly chains weather lookup with temperature conversion

examples/integration-strands-agents/agent.py

@@ -0,0 +1,122 @@
+"""Strands agent implementation with weather tools.
+
+This module demonstrates how to build a Strands agent with custom tools
+that can be evaluated by promptfoo. It showcases:
+
+- The @tool decorator for defining agent capabilities
+  https://strandsagents.com/latest/user-guide/concepts/tools/python-tools/
+- Multiple tools working together (weather lookup + temperature conversion)
+- Integration with OpenAI models via Strands SDK
+  https://strandsagents.com/latest/user-guide/concepts/model-providers/openai/
+
+Strands Agents SDK is an open-source AI agent framework by AWS:
+- Documentation: https://strandsagents.com/
+- GitHub: https://github.com/strands-agents/sdk-python
+- PyPI: https://pypi.org/project/strands-agents/
+"""
+
+from strands import Agent, tool
+from strands.models.openai import OpenAIModel
+
+# Define tools using the @tool decorator. The docstring becomes the tool's
+# description that the LLM uses to decide when to call it.
+
+
+@tool
+def get_weather(city: str) -> str:
+    """Get current weather for a city.
+
+    Args:
+        city: The name of the city to get weather for
+    """
+    weather_data = {
+        "new york": "72°F, Sunny",
+        "london": "58°F, Cloudy",
+        "tokyo": "68°F, Clear",
+        "paris": "64°F, Partly Cloudy",
+        "seattle": "55°F, Rainy",
+        "san francisco": "62°F, Foggy",
+    }
+    city_lower = city.lower()
+    if city_lower in weather_data:
+        return f"Weather in {city}: {weather_data[city_lower]}"
+    return f"Weather in {city}: 70°F, Clear (default)"
+
+
+@tool
+def convert_temperature(value: float, from_unit: str) -> str:
+    """Convert temperature between Fahrenheit and Celsius.
+
+    Args:
+        value: The temperature value to convert
+        from_unit: The unit to convert from ('F' for Fahrenheit, 'C' for Celsius)
+    """
+    from_unit = from_unit.upper()
+    if from_unit == "F":
+        celsius = (value - 32) * 5 / 9
+        return f"{value}°F = {celsius:.1f}°C"
+    elif from_unit == "C":
+        fahrenheit = (value * 9 / 5) + 32
+        return f"{value}°C = {fahrenheit:.1f}°F"
+    return f"Unknown unit '{from_unit}'. Use 'F' for Fahrenheit or 'C' for Celsius."
+
+
+def create_agent(model_id: str = "gpt-4o-mini") -> Agent:
+    """Create a Strands agent with weather tools.
+
+    Args:
+        model_id: The OpenAI model ID to use
+
+    Returns:
+        A configured Strands Agent instance
+    """
+    model = OpenAIModel(model_id=model_id, params={"temperature": 0.7})
+    return Agent(
+        model=model,
+        tools=[get_weather, convert_temperature],
+        system_prompt=(
+            "You are a helpful weather assistant. "
+            "Use the weather tool to get weather for cities and the temperature "
+            "conversion tool to convert between Fahrenheit and Celsius."
+        ),
+    )
+
+
+def run_agent(prompt: str, model_id: str = "gpt-4o-mini") -> str:
+    """Run the agent with a prompt and return the response.
+
+    Args:
+        prompt: The user's input message
+        model_id: The OpenAI model ID to use
+
+    Returns:
+        The agent's response as a string
+    """
+    import io
+    import sys
+
+    agent = create_agent(model_id)
+
+    # Suppress stdout during agent execution. The Strands SDK prints tool call
+    # information to stdout (e.g., "Tool #1: get_weather"), which interferes with
+    # promptfoo's Python worker protocol that uses stdout for control messages.
+    # Without this suppression, the worker times out waiting for protocol signals.
+    old_stdout = sys.stdout
+    sys.stdout = io.StringIO()
+    try:
+        result = agent(prompt)
+    finally:
+        sys.stdout = old_stdout
+
+    return str(result)
+
+
+if __name__ == "__main__":
+    import os
+
+    if os.getenv("OPENAI_API_KEY"):
+        print("Testing Strands agent...")
+        response = run_agent("What's the weather in New York?")
+        print(f"Response: {response}")
+    else:
+        print("Set OPENAI_API_KEY to test.")

examples/integration-strands-agents/agent_provider.py

@@ -0,0 +1,66 @@
+"""Promptfoo provider for Strands Agents SDK.
+
+This module exposes a call_api function that promptfoo uses to interact
+with the Strands agent for evaluation.
+
+The provider follows promptfoo's Python provider interface:
+https://promptfoo.dev/docs/providers/python/
+
+Usage in promptfooconfig.yaml:
+    providers:
+      - id: 'file://agent_provider.py:call_api'
+        config:
+          model_id: 'gpt-4o-mini'
+
+For more information:
+- Strands Agents SDK: https://github.com/strands-agents/sdk-python
+- Strands Documentation: https://strandsagents.com/
+- promptfoo Python Providers: https://promptfoo.dev/docs/providers/python/
+"""
+
+from typing import Any, Dict
+
+from agent import run_agent
+
+
+def call_api(
+    prompt: str, options: Dict[str, Any], context: Dict[str, Any]
+) -> Dict[str, Any]:
+    """Main entry point for promptfoo to call the Strands agent.
+
+    This function is called by promptfoo for each test case. It receives the
+    rendered prompt and returns the agent's response.
+
+    Args:
+        prompt: The user's input message (rendered from the prompt template)
+        options: Configuration options from the provider config in YAML
+        context: Context information from promptfoo (vars, test metadata, etc.)
+
+    Returns:
+        Dictionary with 'output' key containing the agent's response.
+        On error, includes both 'error' and 'output' keys.
+    """
+    try:
+        # Extract model configuration from provider options
+        config = options.get("config", {})
+        model_id = config.get("model_id", "gpt-4o-mini")
+
+        # Run the Strands agent and get the response
+        result = run_agent(prompt, model_id)
+
+        # Return in promptfoo's expected format
+        return {"output": result}
+    except Exception as e:
+        # Return error in a format promptfoo can display
+        return {"error": str(e), "output": f"Error: {str(e)}"}
+
+
+if __name__ == "__main__":
+    import os
+
+    print("Testing Strands agent provider...")
+    if os.getenv("OPENAI_API_KEY"):
+        result = call_api("What's the weather in New York?", {}, {})
+        print(f"Result: {result}")
+    else:
+        print("Set OPENAI_API_KEY to test.")

examples/integration-strands-agents/promptfooconfig.yaml

@@ -0,0 +1,104 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+# Strands Agents SDK Example - Weather Assistant Evaluation
+#
+# This configuration evaluates a Strands agent that provides weather information
+# and temperature conversions. It demonstrates:
+# - Using a Python provider (file://) to integrate custom agents
+#   https://promptfoo.dev/docs/providers/python/
+# - Testing tool usage with contains-any assertions
+#   https://promptfoo.dev/docs/configuration/expected-outputs/#contains
+# - Using llm-rubric for semantic evaluation of responses
+#   https://promptfoo.dev/docs/configuration/expected-outputs/model-graded/
+# - Performance assertions (latency, cost)
+#   https://promptfoo.dev/docs/configuration/expected-outputs/#latency
+#
+# Strands Agents SDK by AWS: https://github.com/strands-agents/sdk-python
+# Strands Documentation: https://strandsagents.com/
+
+description: Strands Agents SDK evaluation
+
+# The prompt template - {{query}} is replaced with test case variables
+prompts:
+  - '{{query}}'
+
+# Provider configuration pointing to our Python agent
+# The file:// prefix tells promptfoo to load a Python module
+# Format: file://<script>:<function>
+providers:
+  - id: 'file://agent_provider.py:call_api'
+    label: 'Strands Weather Agent'
+    config:
+      # Model ID passed to the Strands agent (can be changed to gpt-4o, etc.)
+      model_id: 'gpt-4o-mini'
+
+# Default assertions applied to all test cases
+# https://promptfoo.dev/docs/configuration/guide/#default-test-cases
+defaultTest:
+  assert:
+    # Performance: Response should complete within 30 seconds
+    - type: latency
+      threshold: 30000
+
+# Test cases to evaluate the agent's responses
+tests:
+  # Test 1: Basic weather query - verifies the get_weather tool works
+  - description: 'Weather query for New York'
+    vars:
+      query: "What's the weather like in New York?"
+    assert:
+      # Check that response contains expected weather data
+      - type: contains-any
+        value: ['New York', 'Sunny', '72']
+      # Use LLM to verify response quality
+      - type: llm-rubric
+        value: 'Response should include weather information for New York City'
+
+  # Test 2: Another city to verify tool handles different inputs
+  - description: 'Weather query for London'
+    vars:
+      query: 'Tell me the current weather in London'
+    assert:
+      - type: contains-any
+        value: ['London', 'Cloudy', '58']
+      # Verify response mentions temperature (custom JavaScript assertion)
+      # https://promptfoo.dev/docs/configuration/expected-outputs/#javascript
+      - type: javascript
+        value: 'output.includes("°F") || output.includes("degrees")'
+
+  # Test 3: Test with icontains (case-insensitive)
+  - description: 'Weather query for Tokyo'
+    vars:
+      query: "What's the weather in Tokyo?"
+    assert:
+      # Case-insensitive check
+      - type: icontains
+        value: 'tokyo'
+      - type: icontains
+        value: 'clear'
+      # Verify response is not too short (at least 20 chars)
+      - type: javascript
+        value: 'output.length >= 20'
+
+  # Test 4: Multi-tool usage - tests both get_weather AND convert_temperature
+  # This verifies the agent can chain multiple tools together
+  - description: 'Weather with temperature conversion'
+    vars:
+      query: "What's the weather in New York? Also tell me the temperature in Celsius."
+    assert:
+      - type: llm-rubric
+        value: 'Response should include New York weather (72°F) AND convert it to Celsius (approximately 22°C)'
+      # Verify both temperature units appear in response
+      - type: javascript
+        value: '(output.includes("°F") || output.includes("Fahrenheit")) && (output.includes("°C") || output.includes("Celsius"))'
+
+  # Test 5: Unknown city - verify graceful handling
+  - description: 'Weather for unknown city'
+    vars:
+      query: "What's the weather in Atlantis?"
+    assert:
+      # Should still return a response (default weather)
+      - type: icontains
+        value: 'atlantis'
+      # Should not error out
+      - type: not-contains
+        value: 'Error'

examples/integration-strands-agents/requirements.txt

@@ -0,0 +1,14 @@
+# Strands Agents SDK Example - Python Dependencies
+#
+# Strands Agents SDK by AWS:
+# - GitHub: https://github.com/strands-agents/sdk-python
+# - Documentation: https://strandsagents.com/
+# - PyPI: https://pypi.org/project/strands-agents/
+
+# Strands Agents SDK with OpenAI model support
+# https://strandsagents.com/latest/user-guide/concepts/model-providers/openai/
+strands-agents[openai]>=1.0.0,<2.0.0
+
+# Pydantic for data validation (required by Strands)
+# https://docs.pydantic.dev/
+pydantic>=2.0.0