chore: import upstream snapshot with attribution

examples/amazon-bedrock/agents/README.md

@@ -0,0 +1,215 @@
+# amazon-bedrock/agents (AWS Bedrock Agents Example)
+
+This example demonstrates how to use AWS Bedrock Agents with promptfoo to test and evaluate deployed AI agents, including both single-agent and multi-agent scenarios.
+
+You can run this example with:
+
+```bash
+npx promptfoo@latest init --example amazon-bedrock/agents
+cd amazon-bedrock/agents
+```
+
+## Prerequisites
+
+1. An AWS account with Bedrock Agents access
+2. One or more deployed Bedrock agents (get agent IDs from the AWS Console)
+3. AWS credentials configured (via environment variables, AWS CLI, or IAM role)
+4. Install the required AWS SDK:
+
+   ```bash
+   npm install @aws-sdk/client-bedrock-agent-runtime
+   ```
+
+## Setup
+
+1. **Get your Agent ID(s)**:
+   - Go to the AWS Bedrock Console
+   - Navigate to Agents
+   - Copy your agent ID(s) (format: `ABCDEFGHIJ`)
+
+2. **Configure AWS Credentials** (choose one method):
+
+   Via environment variables:
+
+   ```bash
+   export AWS_ACCESS_KEY_ID=your_access_key
+   export AWS_SECRET_ACCESS_KEY=your_secret_key
+   export AWS_REGION=us-east-1
+   ```
+
+   Via AWS CLI profile:
+
+   ```bash
+   aws configure --profile my-bedrock-profile
+   ```
+
+   Via IAM role (if running on EC2/Lambda)
+
+3. **Choose your configuration**:
+   - `promptfooconfig.yaml`: Basic single-agent configuration
+   - `promptfooconfig.multi-agent.yaml`: Advanced multi-agent system configuration
+
+## Running the Examples
+
+### Single Agent Example
+
+```bash
+# Run basic agent evaluation
+npx promptfoo eval -c promptfooconfig.yaml
+
+# View results in the web UI
+npx promptfoo view
+```
+
+### Multi-Agent Example
+
+```bash
+# Run multi-agent system evaluation
+npx promptfoo eval -c promptfooconfig.multi-agent.yaml
+
+# View results in the web UI
+npx promptfoo view
+```
+
+## Configuration Options
+
+### Basic Usage
+
+```yaml
+providers:
+  - bedrock-agent:YOUR_AGENT_ID
+```
+
+### Advanced Single Agent Configuration
+
+```yaml
+providers:
+  - id: bedrock-agent:my-agent
+    config:
+      agentId: YOUR_AGENT_ID
+      agentAliasId: PROD_ALIAS # Optional: specific version/alias
+      region: us-east-1 # AWS region
+      sessionId: session-123 # Maintain conversation state
+      enableTrace: true # Get detailed execution traces
+      memoryId: SHORT_TERM_MEMORY # or LONG_TERM_MEMORY
+```
+
+### Multi-Agent System Configuration
+
+```yaml
+providers:
+  # Technical Support Agent
+  - id: tech-agent
+    provider: bedrock-agent:TECH_AGENT_ID
+    config:
+      agentId: TECH_AGENT_ID
+      agentAliasId: TECH_ALIAS_ID
+      region: us-east-1
+      enableTrace: true
+      memoryId: LONG_TERM_MEMORY
+
+  # Billing Agent
+  - id: billing-agent
+    provider: bedrock-agent:BILLING_AGENT_ID
+    config:
+      agentId: BILLING_AGENT_ID
+      agentAliasId: BILLING_ALIAS_ID
+      region: us-east-1
+      enableTrace: true
+```
+
+## Features
+
+### Session Management
+
+The provider supports maintaining conversation state across multiple interactions:
+
+```yaml
+config:
+  sessionId: my-session-123 # Use the same session ID for related queries
+```
+
+### Memory Integration
+
+Enable agent memory for context-aware responses:
+
+```yaml
+config:
+  memoryId: LONG_TERM_MEMORY # or SHORT_TERM_MEMORY
+```
+
+### Trace Information
+
+Get detailed execution traces including tool calls and reasoning:
+
+```yaml
+config:
+  enableTrace: true # Response will include trace metadata
+```
+
+## Testing Scenarios
+
+### Single Agent Tests
+
+The basic config includes tests for:
+
+- Basic agent responses
+- Tool/function calling (e.g., calculator)
+- Memory retention
+- Multi-turn conversations
+
+### Multi-Agent System Tests
+
+The multi-agent config includes tests for:
+
+- Specialized agent capabilities (technical, billing, product)
+- Cross-functional issue handling
+- Agent collaboration and coordination
+- Escalation management
+- Performance and latency validation
+
+## Multi-Agent System Architecture
+
+The multi-agent example demonstrates a customer support system with specialized agents:
+
+```text
+Customer Query
+     ↓
+[Supervisor Agent] ← Monitors & Routes
+     ↓
+┌─────────────────┬─────────────────┬──────────────────┐
+│  Tech Agent     │  Billing Agent  │  Product Agent   │
+│  (Technical)    │  (Payments)     │  (Recommendations)│
+└─────────────────┴─────────────────┴──────────────────┘
+```
+
+## Troubleshooting
+
+1. **Authentication Error**: Ensure AWS credentials are properly configured
+2. **Agent Not Found**: Verify the agent ID and region
+3. **Permissions Error**: Check IAM permissions for `bedrock:InvokeAgent`
+4. **Timeout**: Large agent responses may take time; adjust timeout if needed
+5. **Multi-Agent Issues**: Ensure all agent IDs and aliases are correct in the config
+
+## IAM Permissions
+
+Your AWS credentials need the following permissions:
+
+```json
+{
+  "Version": "2012-10-17",
+  "Statement": [
+    {
+      "Effect": "Allow",
+      "Action": ["bedrock:InvokeAgent"],
+      "Resource": "arn:aws:bedrock:*:*:agent/*"
+    }
+  ]
+}
+```
+
+## Learn More
+
+- [AWS Bedrock Agents Documentation](https://docs.aws.amazon.com/bedrock/latest/userguide/agents.html)
+- [Promptfoo Documentation](https://promptfoo.dev/docs/providers/bedrock/)
+- [Bedrock Agents Samples](https://github.com/awslabs/amazon-bedrock-agents-samples)

examples/amazon-bedrock/agents/promptfooconfig.multi-agent.yaml

@@ -0,0 +1,239 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+# Multi-Agent System Example with AWS Bedrock Agents
+# This example demonstrates testing multiple specialized agents working together
+
+description: 'Multi-Agent Customer Support System Evaluation'
+
+# Define multiple specialized agents
+providers:
+  # Technical Support Agent - handles technical queries
+  - id: tech-agent
+    provider: bedrock-agent:TECH_AGENT_ID
+    config:
+      agentId: TECH_AGENT_ID
+      agentAliasId: TECH_ALIAS_ID
+      region: us-east-1
+      enableTrace: true
+      memoryId: LONG_TERM_MEMORY # Remember customer issues
+
+  # Billing Agent - handles payment and subscription queries
+  - id: billing-agent
+    provider: bedrock-agent:BILLING_AGENT_ID
+    config:
+      agentId: BILLING_AGENT_ID
+      agentAliasId: BILLING_ALIAS_ID
+      region: us-east-1
+      enableTrace: true
+      memoryId: LONG_TERM_MEMORY
+
+  # Product Specialist Agent - handles product recommendations
+  - id: product-agent
+    provider: bedrock-agent:PRODUCT_AGENT_ID
+    config:
+      agentId: PRODUCT_AGENT_ID
+      agentAliasId: PRODUCT_ALIAS_ID
+      region: us-east-1
+      enableTrace: true
+      memoryId: SHORT_TERM_MEMORY
+
+  # Supervisor Agent - routes queries and escalates issues
+  - id: supervisor-agent
+    provider: bedrock-agent:SUPERVISOR_AGENT_ID
+    config:
+      agentId: SUPERVISOR_AGENT_ID
+      agentAliasId: SUPERVISOR_ALIAS_ID
+      region: us-east-1
+      enableTrace: true
+      sessionId: supervisor-session-001 # Shared session for coordination
+
+# Test scenarios for multi-agent collaboration
+prompts:
+  - id: technical-issue
+    content: |
+      My application keeps crashing when I try to upload files larger than 10MB.
+      Error code: UPLOAD_SIZE_EXCEEDED. How can I fix this?
+
+  - id: billing-question
+    content: |
+      I was charged twice for my subscription last month.
+      Can you help me get a refund for the duplicate charge?
+
+  - id: product-recommendation
+    content: |
+      I need a solution for real-time data processing that can handle
+      100,000 events per second. What product do you recommend?
+
+  - id: complex-issue
+    content: |
+      I upgraded my plan but I'm still seeing the old limits.
+      Also, the API is returning 403 errors even though my payment went through.
+
+  - id: escalation-needed
+    content: |
+      This is the third time I'm contacting support about the same issue.
+      I need to speak with a manager about getting a full refund and account closure.
+
+tests:
+  # Test Technical Agent's capability
+  - description: 'Technical agent handles error troubleshooting'
+    vars:
+      query: '{{prompts.technical-issue}}'
+    providers:
+      - tech-agent
+    assert:
+      - type: javascript
+        value: |
+          const v = output.toLowerCase();
+          ['file size', 'upload limit', 'configuration'].some(s => v.includes(s))
+      - type: javascript
+        value: |
+          // Check if technical solution is provided
+          output.toLowerCase().includes('limit') ||
+          output.toLowerCase().includes('configuration') ||
+          output.toLowerCase().includes('setting')
+
+  # Test Billing Agent's capability
+  - description: 'Billing agent handles refund requests'
+    vars:
+      query: '{{prompts.billing-question}}'
+    providers:
+      - billing-agent
+    assert:
+      - type: javascript
+        value: |
+          const v = output.toLowerCase();
+          ['refund', 'duplicate', 'charge'].some(s => v.includes(s))
+      - type: javascript
+        value: |
+          // Check for customer service tone
+          output.toLowerCase().includes('sorry') ||
+          output.toLowerCase().includes('apologize') ||
+          output.toLowerCase().includes('help')
+
+  # Test Product Agent's recommendations
+  - description: 'Product agent provides relevant recommendations'
+    vars:
+      query: '{{prompts.product-recommendation}}'
+    providers:
+      - product-agent
+    assert:
+      - type: javascript
+        value: |
+          const v = output.toLowerCase();
+          ['stream', 'kafka', 'kinesis', 'event', 'real-time'].some(s => v.includes(s))
+      - type: javascript
+        value: |
+          // Check if throughput requirements are acknowledged
+          output.includes('100,000') ||
+          output.toLowerCase().includes('high throughput') ||
+          output.toLowerCase().includes('scale')
+
+  # Test Multi-Agent Coordination (Complex Issue)
+  - description: 'Multiple agents handle complex cross-functional issue'
+    vars:
+      query: '{{prompts.complex-issue}}'
+    providers:
+      - tech-agent
+      - billing-agent
+    assert:
+      - type: javascript
+        value: |
+          // Tech agent should address API errors
+          const techResponse = outputs['tech-agent'];
+          const billingResponse = outputs['billing-agent'];
+
+          const techAddressed = techResponse && (
+            techResponse.toLowerCase().includes('403') ||
+            techResponse.toLowerCase().includes('permission') ||
+            techResponse.toLowerCase().includes('authorization')
+          );
+
+          const billingAddressed = billingResponse && (
+            billingResponse.toLowerCase().includes('payment') ||
+            billingResponse.toLowerCase().includes('plan') ||
+            billingResponse.toLowerCase().includes('upgrade')
+          );
+
+          return techAddressed || billingAddressed;
+
+  # Test Supervisor Escalation
+  - description: 'Supervisor agent handles escalations appropriately'
+    vars:
+      query: '{{prompts.escalation-needed}}'
+    providers:
+      - supervisor-agent
+    assert:
+      - type: javascript
+        value: |
+          const v = output.toLowerCase();
+          ['escalate', 'manager', 'supervisor', 'priority'].some(s => v.includes(s))
+      - type: javascript
+        value: |
+          // Check for appropriate escalation handling
+          output.toLowerCase().includes('understand') ||
+          output.toLowerCase().includes('frustration') ||
+          output.toLowerCase().includes('priority')
+
+  # Test Agent Memory Persistence
+  - description: 'Agents remember context from previous interactions'
+    vars:
+      customer_id: 'CUST-12345'
+    tests:
+      - vars:
+          query: "My customer ID is {{customer_id}} and I'm having login issues"
+        providers:
+          - tech-agent
+      - vars:
+          query: 'What is my customer ID?'
+        providers:
+          - tech-agent
+        assert:
+          - type: contains
+            value: '{{customer_id}}'
+
+  # Test Agent Collaboration with Trace Analysis
+  - description: 'Analyze agent tool usage and decision making'
+    vars:
+      query: 'I need help choosing between your Pro and Enterprise plans for my 50-person startup'
+    providers:
+      - product-agent
+    assert:
+      - type: javascript
+        value: |
+          // Check if agent used comparison tools
+          if (metadata && metadata.trace && metadata.trace.toolCalls) {
+            const toolsUsed = metadata.trace.toolCalls.map(t => t.name);
+            return toolsUsed.some(tool =>
+              tool.includes('compare') ||
+              tool.includes('pricing') ||
+              tool.includes('plan')
+            );
+          }
+          // Pass if no trace available but response is relevant
+          return output.toLowerCase().includes('enterprise') &&
+                 output.toLowerCase().includes('pro');
+
+# Performance and Quality Metrics
+defaultTest:
+  assert:
+    # Latency check for all responses
+    - type: latency
+      threshold: 5000 # 5 seconds max
+
+    # Ensure responses are not empty
+    - type: javascript
+      value: output && output.trim().length > 10
+
+    # Check for professional tone
+    - type: javascript
+      value: |
+        // Avoid inappropriate responses
+        !output.toLowerCase().includes('i don\'t know') &&
+        !output.toLowerCase().includes('not sure') &&
+        !output.toLowerCase().includes('error')
+
+# Shared test configuration
+options:
+  cache: false # Disable caching for accurate latency measurements
+  maxConcurrency: 2 # Limit concurrent agent calls to avoid rate limits
+

examples/amazon-bedrock/agents/promptfooconfig.yaml

@@ -0,0 +1,66 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+# AWS Bedrock Agents Provider Example Configuration
+# This example demonstrates how to use AWS Bedrock Agents with promptfoo
+
+description: 'AWS Bedrock Agents test suite'
+
+prompts:
+  - 'What can you help me with?'
+  - 'Perform a calculation: what is 15 multiplied by 27?'
+  - 'Search for information about AWS Lambda functions'
+
+providers:
+  # Basic configuration with agent ID in the path
+  - id: bedrock-agent:basic-agent
+    config:
+      agentId: ABCDEFGHIJ
+      agentAliasId: PROD_ALIAS_ID
+      region: us-east-1
+
+  # Configuration with additional options
+  - id: bedrock-agent:my-agent
+    config:
+      agentId: ABCDEFGHIJ
+      agentAliasId: PROD_ALIAS_ID # Optional: specific alias to use
+      region: us-east-1
+      sessionId: session-123 # Optional: maintain conversation state
+      enableTrace: true # Enable detailed trace information
+      memoryId: SHORT_TERM_MEMORY # or LONG_TERM_MEMORY
+
+      # AWS credentials (optional - will use default chain if not provided)
+      # accessKeyId: YOUR_ACCESS_KEY
+      # secretAccessKey: YOUR_SECRET_KEY
+      # sessionToken: YOUR_SESSION_TOKEN  # For temporary credentials
+      # profile: my-aws-profile  # Use AWS SSO profile
+tests:
+  - vars:
+      query: 'Tell me about your capabilities'
+    assert:
+      - type: contains
+        value: 'agent'
+
+  - vars:
+      query: 'Calculate the sum of 100 and 250'
+    assert:
+      - type: contains
+        value: '350'
+      - type: not-empty
+
+  - vars:
+      query: 'Remember that my favorite color is blue'
+    assert:
+      - type: not-empty
+      # With memory enabled, the agent should acknowledge storing this information
+
+  - vars:
+      query: 'What is my favorite color?'
+    assert:
+      - type: contains
+        value: 'blue'
+      # This test verifies memory retention if using the same session
+# Environment variables (set these in your environment or .env file):
+# AWS_BEDROCK_REGION=us-east-1
+# AWS_ACCESS_KEY_ID=your_access_key
+# AWS_SECRET_ACCESS_KEY=your_secret_key
+# AWS_SESSION_TOKEN=your_session_token (optional)
+