chore: import upstream snapshot with attribution

examples/integration-browser/headless/README.md

@@ -0,0 +1,182 @@
+# integration-browser/headless (Headless Browser Automation)
+
+A browser automation example demonstrating how to test web applications using Playwright.
+
+You can run this example with:
+
+```bash
+npx promptfoo@latest init --example integration-browser/headless
+cd integration-browser/headless
+```
+
+## Overview
+
+This example demonstrates how to:
+
+- Test a local Gradio application using browser automation
+- Handle dynamic JavaScript-rendered content
+- Extract data from web interfaces
+- Work with complex UI interactions (forms, tabs, buttons)
+
+## Prerequisites
+
+Ensure you have Python 3 and Node.js installed on your system.
+
+1. **Install Node.js dependencies**:
+
+```bash
+npm install playwright @playwright/browser-chromium playwright-extra puppeteer-extra-plugin-stealth
+```
+
+2. **Install Python dependencies** (for the demo application):
+
+```bash
+pip install -r requirements.txt
+```
+
+That's it! No additional setup scripts or configuration needed.
+
+## Running the Example
+
+1. **Start the Gradio demo application**:
+
+```bash
+python gradio_demo.py
+```
+
+This starts a local server at http://localhost:7860
+
+2. **Run the browser automation tests**:
+
+```bash
+npx promptfoo@latest eval -c promptfooconfig.yaml
+```
+
+3. **View the results**:
+
+```bash
+npx promptfoo@latest view
+```
+
+## Test Results
+
+### Chatbot Example
+
+The main configuration (`promptfooconfig.yaml`) tests a chatbot interface with a 100% pass rate:
+
+```text
+┌─────────────────────────────────────────────────┬─────────────────────────────────────────────────┐
+│ topic                                           │ [browser-provider] Tell me about {{topic}}      │
+├─────────────────────────────────────────────────┼─────────────────────────────────────────────────┤
+│ testing browser automation                      │ [PASS] Test successful! The browser automation  │
+│                                                 │ is working correctly.                           │
+├─────────────────────────────────────────────────┼─────────────────────────────────────────────────┤
+│ how the system works                            │ [PASS] I received your message: 'Tell me about  │
+│                                                 │ how the system works'. This is a simple demo    │
+│                                                 │ response!                                       │
+├─────────────────────────────────────────────────┼─────────────────────────────────────────────────┤
+│ a simple greeting                               │ [PASS] I received your message: 'Tell me about  │
+│                                                 │ a simple greeting'. This is a simple demo       │
+│                                                 │ response!                                       │
+└─────────────────────────────────────────────────┴─────────────────────────────────────────────────┘
+```
+
+### Calculator Example
+
+The `calculator-example.yaml` demonstrates form interactions with a 100% pass rate:
+
+```text
+┌───────────────────┬───────────────────┬───────────────────┬───────────────────┬───────────────────┐
+│ num1              │ num2              │ operation         │ operationSelector │ [browser-provider]│
+├───────────────────┼───────────────────┼───────────────────┼───────────────────┼───────────────────┤
+│ 10                │ 5                 │ Add               │ #operation        │ [PASS] Calculator │
+│                   │                   │                   │ label:nth-child(1)│ interaction       │
+│                   │                   │                   │                   │ successful        │
+├───────────────────┼───────────────────┼───────────────────┼───────────────────┼───────────────────┤
+│ 20                │ 4                 │ Multiply          │ #operation        │ [PASS] Calculator │
+│                   │                   │                   │ label:nth-child(3)│ interaction       │
+│                   │                   │                   │                   │ successful        │
+└───────────────────┴───────────────────┴───────────────────┴───────────────────┴───────────────────┘
+```
+
+This example demonstrates:
+
+- Navigate between tabs in a web application
+- Fill multiple input fields
+- Select radio button options
+- Click buttons and wait for results
+- Extract and verify content from the page
+
+## Configuration Details
+
+The example configurations demonstrate key concepts:
+
+- **Appropriate delays**: 2-3 seconds between actions for reliability
+- **Local testing**: Tests run against localhost:7860
+- **Error handling**: Uses `transformResponse` for data extraction
+- **Clear assertions**: Validates expected outputs
+
+## Selectors Used
+
+The Gradio application provides consistent selectors:
+
+- `textarea[data-testid="textbox"]` - Message input field
+- `button#submit-button` - Submit button
+- `div[data-testid="bot"]:last-of-type .prose` - Latest bot response
+- `button[value="calculator"]` - Calculator tab button
+- `input[type="radio"]` - Operation selection
+
+## Adapting This Example
+
+### Testing Your Own Application
+
+1. Update the `url` in the navigation step
+2. Modify selectors to match your UI elements
+3. Adjust wait times based on your application's response time
+4. Add appropriate assertions for your use case
+
+### Handling Dynamic Content
+
+For single-page applications or AJAX content:
+
+```yaml
+- action: waitForNewChildren
+  args:
+    parentSelector: '#results-container'
+    timeout: 10000
+```
+
+### Complex Interactions
+
+Chain multiple actions for sophisticated workflows:
+
+```yaml
+steps:
+  - action: navigate
+    args:
+      url: 'http://localhost:3000'
+  - action: click
+    args:
+      selector: '#menu-button'
+  - action: wait
+    args:
+      ms: 1000
+  - action: click
+    args:
+      selector: '#dropdown-option-2'
+```
+
+## Debugging Tips
+
+| Issue                   | Solution                                        |
+| ----------------------- | ----------------------------------------------- |
+| Elements not found      | Use browser DevTools to verify selectors        |
+| Timing issues           | Increase wait times or use `waitForNewChildren` |
+| Want to see the browser | Set `headless: false` in the configuration      |
+| Need detailed logs      | Run with `npx promptfoo@latest eval --verbose`  |
+
+## Additional Resources
+
+- [Browser Provider Documentation](/docs/providers/browser)
+- [Playwright Selectors Guide](https://playwright.dev/docs/selectors)
+- [Gradio Documentation](https://www.gradio.app/docs)

examples/integration-browser/headless/calculator-example.yaml

@@ -0,0 +1,85 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: 'Calculator demo for browser automation'
+
+prompts:
+  - 'Calculate {{num1}} {{operation}} {{num2}}'
+
+providers:
+  - id: browser
+    config:
+      headless: false # See the browser in action
+      steps:
+        # Navigate to the demo
+        - action: navigate
+          args:
+            url: 'http://localhost:7860'
+
+        # Wait for initial page load
+        - action: wait
+          args:
+            ms: 3000
+
+        # Navigate to calculator tab
+        - action: click
+          args:
+            selector: 'button[role="tab"]:nth-child(2)'
+
+        # Wait for tab to load
+        - action: wait
+          args:
+            ms: 2000
+
+        # Enter numbers and perform calculation
+        - action: type
+          args:
+            selector: '#num1 input'
+            text: '{{num1}}'
+
+        - action: type
+          args:
+            selector: '#num2 input'
+            text: '{{num2}}'
+
+        # Select operation
+        - action: click
+          args:
+            selector: '{{operationSelector}}'
+
+        # Perform calculation
+        - action: click
+          args:
+            selector: '#calculate'
+
+        # Wait for calculation
+        - action: wait
+          args:
+            ms: 2000
+
+        # Extract entire result area for debugging
+        - action: extract
+          args:
+            selector: '#result'
+          name: resultArea
+
+      # For demo purposes, we'll just check that we got something
+      transformResponse: 'extracted.resultArea && extracted.resultArea.includes("Result") ? "Calculator interaction successful" : "No result found"'
+
+# Simple demo tests
+tests:
+  - vars:
+      num1: '10'
+      num2: '5'
+      operation: Add
+      operationSelector: '#operation label:nth-child(1)'
+    assert:
+      - type: contains
+        value: 'successful'
+
+  - vars:
+      num1: '20'
+      num2: '4'
+      operation: Multiply
+      operationSelector: '#operation label:nth-child(3)'
+    assert:
+      - type: contains
+        value: 'successful'

examples/integration-browser/headless/gradio_demo.py

@@ -0,0 +1,141 @@
+import time
+
+import gradio as gr
+
+
+# Simple chatbot function
+def chatbot_response(message, history):
+    """Return deterministic keyword replies for browser automation tests."""
+    time.sleep(1)  # Simulate processing time
+
+    responses = {
+        "hello": "Hello! How can I help you today?",
+        "how are you": "I'm doing great, thank you for asking! How are you?",
+        "test": "Test successful! The browser automation is working correctly.",
+        "demo": "This is a demo Gradio application for testing browser automation.",
+        "help": "I can respond to simple greetings and questions. Try saying 'hello' or asking 'how are you'!",
+    }
+
+    # Simple keyword matching
+    message_lower = message.lower()
+    for keyword, response in responses.items():
+        if keyword in message_lower:
+            return response
+
+    # Default response
+    return f"I received your message: '{message}'. This is a simple demo response!"
+
+
+# Create the Gradio interface
+def create_demo():
+    """Create the chatbot demo used by browser automation examples."""
+    with gr.Blocks(title="Browser Testing Demo") as demo:
+        gr.Markdown(
+            """
+            # 🤖 Browser Automation Test Demo
+            
+            This is a simple Gradio application designed for testing browser automation with promptfoo.
+            
+            **Features:**
+            - Simple chat interface
+            - Predictable responses for testing
+            - Clear element identifiers for automation
+            """
+        )
+
+        chatbot = gr.Chatbot(
+            label="Chat History", elem_id="chat-history", type="messages"
+        )
+
+        msg = gr.Textbox(
+            label="Your Message",
+            placeholder="Type a message and press Enter...",
+            elem_id="user-input",
+        )
+
+        with gr.Row():
+            submit = gr.Button("Submit", elem_id="submit-button")
+            clear = gr.Button("Clear", elem_id="clear-button")
+
+        # Example messages for testing
+        gr.Examples(
+            examples=[
+                "Hello",
+                "How are you?",
+                "This is a test",
+                "Show me a demo",
+                "Help",
+            ],
+            inputs=msg,
+            label="Example Messages",
+        )
+
+        # Handle message submission
+        def respond(message, chat_history):
+            """Append the user message and deterministic bot response."""
+            bot_message = chatbot_response(message, chat_history)
+            # Use the new messages format with role and content
+            chat_history.append({"role": "user", "content": message})
+            chat_history.append({"role": "assistant", "content": bot_message})
+            return "", chat_history
+
+        # Wire up the interface
+        msg.submit(respond, [msg, chatbot], [msg, chatbot])
+        submit.click(respond, [msg, chatbot], [msg, chatbot])
+        clear.click(lambda: None, None, chatbot, queue=False)
+
+    return demo
+
+
+# Simple calculator for testing form inputs
+def create_calculator_demo():
+    """Create the calculator demo used for form input testing."""
+    with gr.Blocks(title="Calculator Demo") as calc_demo:
+        gr.Markdown("## Simple Calculator for Testing")
+
+        with gr.Row():
+            num1 = gr.Number(label="First Number", elem_id="num1")
+            num2 = gr.Number(label="Second Number", elem_id="num2")
+
+        operation = gr.Radio(
+            ["Add", "Subtract", "Multiply", "Divide"],
+            label="Operation",
+            elem_id="operation",
+        )
+
+        calculate_btn = gr.Button("Calculate", elem_id="calculate")
+        result = gr.Textbox(label="Result", elem_id="result")
+
+        def calculate(n1, n2, op):
+            """Calculate the selected arithmetic operation."""
+            if op == "Add":
+                return str(n1 + n2)
+            elif op == "Subtract":
+                return str(n1 - n2)
+            elif op == "Multiply":
+                return str(n1 * n2)
+            elif op == "Divide":
+                return str(n1 / n2) if n2 != 0 else "Error: Division by zero"
+            return "Error: Unknown operation"
+
+        calculate_btn.click(calculate, [num1, num2, operation], result)
+
+    return calc_demo
+
+
+if __name__ == "__main__":
+    # Create a tabbed interface with multiple demos
+    demo1 = create_demo()
+    demo2 = create_calculator_demo()
+
+    tabbed_demo = gr.TabbedInterface(
+        [demo1, demo2],
+        ["Chatbot Demo", "Calculator Demo"],
+        title="Browser Automation Test Suite",
+    )
+
+    print("Starting Gradio demo server...")
+    print("Access the demo at http://localhost:7860")
+    print("Use Ctrl+C to stop the server")
+
+    tabbed_demo.launch(server_name="0.0.0.0", server_port=7860, share=False)

examples/integration-browser/headless/promptfooconfig.yaml

@@ -0,0 +1,68 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: 'Testing browser automation with local Gradio app'
+
+prompts:
+  - 'Tell me about {{topic}}'
+
+providers:
+  - id: browser
+    config:
+      headless: false # set to false to see the browser in action
+      steps:
+        # Always start with a respectful delay
+        - action: wait
+          args:
+            ms: 2000
+
+        # Navigate to local Gradio app
+        - action: navigate
+          args:
+            url: 'http://localhost:7860'
+
+        # Wait for the page to load
+        - action: wait
+          args:
+            ms: 2000
+
+        # Type the message in the Gradio textbox
+        - action: type
+          args:
+            selector: 'textarea[data-testid="textbox"]'
+            text: '{{prompt}}'
+
+        # Submit the message
+        - action: click
+          args:
+            selector: 'button#submit-button'
+
+        # Wait for the response with reasonable timeout
+        - action: wait
+          args:
+            ms: 3000
+
+        # Extract the bot's response from the chatbot - get the last message
+        - action: extract
+          args:
+            selector: 'div[data-testid="bot"]:last-of-type .prose'
+          name: response
+
+      transformResponse: 'extracted.response'
+
+tests:
+  - vars:
+      topic: testing browser automation
+    assert:
+      - type: contains
+        value: 'Test successful'
+
+  - vars:
+      topic: how the system works
+    assert:
+      - type: contains
+        value: 'simple demo response'
+
+  - vars:
+      topic: a simple greeting
+    assert:
+      - type: javascript
+        value: output.length > 0

examples/integration-browser/headless/requirements.txt

@@ -0,0 +1,2 @@
+gradio>=5.49.1
+playwright>=1.55.0