chore: import upstream snapshot with attribution

examples/amazon-bedrock/README.md

@@ -0,0 +1,9 @@
+# amazon-bedrock (Amazon Bedrock)
+
+Examples for using promptfoo with [Amazon Bedrock](https://aws.amazon.com/bedrock/).
+
+## Examples
+
+- [models](./models/) - Model evaluations: Claude, Llama, Mistral, Nova, DeepSeek, Qwen, Grok, and more
+- [agents](./agents/) - Bedrock Agents with tool use and knowledge bases
+- [video](./video/) - Video generation with Amazon Nova Reel

examples/amazon-bedrock/agents/README.md

@@ -0,0 +1,215 @@
+# amazon-bedrock/agents (AWS Bedrock Agents Example)
+
+This example demonstrates how to use AWS Bedrock Agents with promptfoo to test and evaluate deployed AI agents, including both single-agent and multi-agent scenarios.
+
+You can run this example with:
+
+```bash
+npx promptfoo@latest init --example amazon-bedrock/agents
+cd amazon-bedrock/agents
+```
+
+## Prerequisites
+
+1. An AWS account with Bedrock Agents access
+2. One or more deployed Bedrock agents (get agent IDs from the AWS Console)
+3. AWS credentials configured (via environment variables, AWS CLI, or IAM role)
+4. Install the required AWS SDK:
+
+   ```bash
+   npm install @aws-sdk/client-bedrock-agent-runtime
+   ```
+
+## Setup
+
+1. **Get your Agent ID(s)**:
+   - Go to the AWS Bedrock Console
+   - Navigate to Agents
+   - Copy your agent ID(s) (format: `ABCDEFGHIJ`)
+
+2. **Configure AWS Credentials** (choose one method):
+
+   Via environment variables:
+
+   ```bash
+   export AWS_ACCESS_KEY_ID=your_access_key
+   export AWS_SECRET_ACCESS_KEY=your_secret_key
+   export AWS_REGION=us-east-1
+   ```
+
+   Via AWS CLI profile:
+
+   ```bash
+   aws configure --profile my-bedrock-profile
+   ```
+
+   Via IAM role (if running on EC2/Lambda)
+
+3. **Choose your configuration**:
+   - `promptfooconfig.yaml`: Basic single-agent configuration
+   - `promptfooconfig.multi-agent.yaml`: Advanced multi-agent system configuration
+
+## Running the Examples
+
+### Single Agent Example
+
+```bash
+# Run basic agent evaluation
+npx promptfoo eval -c promptfooconfig.yaml
+
+# View results in the web UI
+npx promptfoo view
+```
+
+### Multi-Agent Example
+
+```bash
+# Run multi-agent system evaluation
+npx promptfoo eval -c promptfooconfig.multi-agent.yaml
+
+# View results in the web UI
+npx promptfoo view
+```
+
+## Configuration Options
+
+### Basic Usage
+
+```yaml
+providers:
+  - bedrock-agent:YOUR_AGENT_ID
+```
+
+### Advanced Single Agent Configuration
+
+```yaml
+providers:
+  - id: bedrock-agent:my-agent
+    config:
+      agentId: YOUR_AGENT_ID
+      agentAliasId: PROD_ALIAS # Optional: specific version/alias
+      region: us-east-1 # AWS region
+      sessionId: session-123 # Maintain conversation state
+      enableTrace: true # Get detailed execution traces
+      memoryId: SHORT_TERM_MEMORY # or LONG_TERM_MEMORY
+```
+
+### Multi-Agent System Configuration
+
+```yaml
+providers:
+  # Technical Support Agent
+  - id: tech-agent
+    provider: bedrock-agent:TECH_AGENT_ID
+    config:
+      agentId: TECH_AGENT_ID
+      agentAliasId: TECH_ALIAS_ID
+      region: us-east-1
+      enableTrace: true
+      memoryId: LONG_TERM_MEMORY
+
+  # Billing Agent
+  - id: billing-agent
+    provider: bedrock-agent:BILLING_AGENT_ID
+    config:
+      agentId: BILLING_AGENT_ID
+      agentAliasId: BILLING_ALIAS_ID
+      region: us-east-1
+      enableTrace: true
+```
+
+## Features
+
+### Session Management
+
+The provider supports maintaining conversation state across multiple interactions:
+
+```yaml
+config:
+  sessionId: my-session-123 # Use the same session ID for related queries
+```
+
+### Memory Integration
+
+Enable agent memory for context-aware responses:
+
+```yaml
+config:
+  memoryId: LONG_TERM_MEMORY # or SHORT_TERM_MEMORY
+```
+
+### Trace Information
+
+Get detailed execution traces including tool calls and reasoning:
+
+```yaml
+config:
+  enableTrace: true # Response will include trace metadata
+```
+
+## Testing Scenarios
+
+### Single Agent Tests
+
+The basic config includes tests for:
+
+- Basic agent responses
+- Tool/function calling (e.g., calculator)
+- Memory retention
+- Multi-turn conversations
+
+### Multi-Agent System Tests
+
+The multi-agent config includes tests for:
+
+- Specialized agent capabilities (technical, billing, product)
+- Cross-functional issue handling
+- Agent collaboration and coordination
+- Escalation management
+- Performance and latency validation
+
+## Multi-Agent System Architecture
+
+The multi-agent example demonstrates a customer support system with specialized agents:
+
+```text
+Customer Query
+     ↓
+[Supervisor Agent] ← Monitors & Routes
+     ↓
+┌─────────────────┬─────────────────┬──────────────────┐
+│  Tech Agent     │  Billing Agent  │  Product Agent   │
+│  (Technical)    │  (Payments)     │  (Recommendations)│
+└─────────────────┴─────────────────┴──────────────────┘
+```
+
+## Troubleshooting
+
+1. **Authentication Error**: Ensure AWS credentials are properly configured
+2. **Agent Not Found**: Verify the agent ID and region
+3. **Permissions Error**: Check IAM permissions for `bedrock:InvokeAgent`
+4. **Timeout**: Large agent responses may take time; adjust timeout if needed
+5. **Multi-Agent Issues**: Ensure all agent IDs and aliases are correct in the config
+
+## IAM Permissions
+
+Your AWS credentials need the following permissions:
+
+```json
+{
+  "Version": "2012-10-17",
+  "Statement": [
+    {
+      "Effect": "Allow",
+      "Action": ["bedrock:InvokeAgent"],
+      "Resource": "arn:aws:bedrock:*:*:agent/*"
+    }
+  ]
+}
+```
+
+## Learn More
+
+- [AWS Bedrock Agents Documentation](https://docs.aws.amazon.com/bedrock/latest/userguide/agents.html)
+- [Promptfoo Documentation](https://promptfoo.dev/docs/providers/bedrock/)
+- [Bedrock Agents Samples](https://github.com/awslabs/amazon-bedrock-agents-samples)

examples/amazon-bedrock/agents/promptfooconfig.multi-agent.yaml

@@ -0,0 +1,239 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+# Multi-Agent System Example with AWS Bedrock Agents
+# This example demonstrates testing multiple specialized agents working together
+
+description: 'Multi-Agent Customer Support System Evaluation'
+
+# Define multiple specialized agents
+providers:
+  # Technical Support Agent - handles technical queries
+  - id: tech-agent
+    provider: bedrock-agent:TECH_AGENT_ID
+    config:
+      agentId: TECH_AGENT_ID
+      agentAliasId: TECH_ALIAS_ID
+      region: us-east-1
+      enableTrace: true
+      memoryId: LONG_TERM_MEMORY # Remember customer issues
+
+  # Billing Agent - handles payment and subscription queries
+  - id: billing-agent
+    provider: bedrock-agent:BILLING_AGENT_ID
+    config:
+      agentId: BILLING_AGENT_ID
+      agentAliasId: BILLING_ALIAS_ID
+      region: us-east-1
+      enableTrace: true
+      memoryId: LONG_TERM_MEMORY
+
+  # Product Specialist Agent - handles product recommendations
+  - id: product-agent
+    provider: bedrock-agent:PRODUCT_AGENT_ID
+    config:
+      agentId: PRODUCT_AGENT_ID
+      agentAliasId: PRODUCT_ALIAS_ID
+      region: us-east-1
+      enableTrace: true
+      memoryId: SHORT_TERM_MEMORY
+
+  # Supervisor Agent - routes queries and escalates issues
+  - id: supervisor-agent
+    provider: bedrock-agent:SUPERVISOR_AGENT_ID
+    config:
+      agentId: SUPERVISOR_AGENT_ID
+      agentAliasId: SUPERVISOR_ALIAS_ID
+      region: us-east-1
+      enableTrace: true
+      sessionId: supervisor-session-001 # Shared session for coordination
+
+# Test scenarios for multi-agent collaboration
+prompts:
+  - id: technical-issue
+    content: |
+      My application keeps crashing when I try to upload files larger than 10MB.
+      Error code: UPLOAD_SIZE_EXCEEDED. How can I fix this?
+
+  - id: billing-question
+    content: |
+      I was charged twice for my subscription last month.
+      Can you help me get a refund for the duplicate charge?
+
+  - id: product-recommendation
+    content: |
+      I need a solution for real-time data processing that can handle
+      100,000 events per second. What product do you recommend?
+
+  - id: complex-issue
+    content: |
+      I upgraded my plan but I'm still seeing the old limits.
+      Also, the API is returning 403 errors even though my payment went through.
+
+  - id: escalation-needed
+    content: |
+      This is the third time I'm contacting support about the same issue.
+      I need to speak with a manager about getting a full refund and account closure.
+
+tests:
+  # Test Technical Agent's capability
+  - description: 'Technical agent handles error troubleshooting'
+    vars:
+      query: '{{prompts.technical-issue}}'
+    providers:
+      - tech-agent
+    assert:
+      - type: javascript
+        value: |
+          const v = output.toLowerCase();
+          ['file size', 'upload limit', 'configuration'].some(s => v.includes(s))
+      - type: javascript
+        value: |
+          // Check if technical solution is provided
+          output.toLowerCase().includes('limit') ||
+          output.toLowerCase().includes('configuration') ||
+          output.toLowerCase().includes('setting')
+
+  # Test Billing Agent's capability
+  - description: 'Billing agent handles refund requests'
+    vars:
+      query: '{{prompts.billing-question}}'
+    providers:
+      - billing-agent
+    assert:
+      - type: javascript
+        value: |
+          const v = output.toLowerCase();
+          ['refund', 'duplicate', 'charge'].some(s => v.includes(s))
+      - type: javascript
+        value: |
+          // Check for customer service tone
+          output.toLowerCase().includes('sorry') ||
+          output.toLowerCase().includes('apologize') ||
+          output.toLowerCase().includes('help')
+
+  # Test Product Agent's recommendations
+  - description: 'Product agent provides relevant recommendations'
+    vars:
+      query: '{{prompts.product-recommendation}}'
+    providers:
+      - product-agent
+    assert:
+      - type: javascript
+        value: |
+          const v = output.toLowerCase();
+          ['stream', 'kafka', 'kinesis', 'event', 'real-time'].some(s => v.includes(s))
+      - type: javascript
+        value: |
+          // Check if throughput requirements are acknowledged
+          output.includes('100,000') ||
+          output.toLowerCase().includes('high throughput') ||
+          output.toLowerCase().includes('scale')
+
+  # Test Multi-Agent Coordination (Complex Issue)
+  - description: 'Multiple agents handle complex cross-functional issue'
+    vars:
+      query: '{{prompts.complex-issue}}'
+    providers:
+      - tech-agent
+      - billing-agent
+    assert:
+      - type: javascript
+        value: |
+          // Tech agent should address API errors
+          const techResponse = outputs['tech-agent'];
+          const billingResponse = outputs['billing-agent'];
+
+          const techAddressed = techResponse && (
+            techResponse.toLowerCase().includes('403') ||
+            techResponse.toLowerCase().includes('permission') ||
+            techResponse.toLowerCase().includes('authorization')
+          );
+
+          const billingAddressed = billingResponse && (
+            billingResponse.toLowerCase().includes('payment') ||
+            billingResponse.toLowerCase().includes('plan') ||
+            billingResponse.toLowerCase().includes('upgrade')
+          );
+
+          return techAddressed || billingAddressed;
+
+  # Test Supervisor Escalation
+  - description: 'Supervisor agent handles escalations appropriately'
+    vars:
+      query: '{{prompts.escalation-needed}}'
+    providers:
+      - supervisor-agent
+    assert:
+      - type: javascript
+        value: |
+          const v = output.toLowerCase();
+          ['escalate', 'manager', 'supervisor', 'priority'].some(s => v.includes(s))
+      - type: javascript
+        value: |
+          // Check for appropriate escalation handling
+          output.toLowerCase().includes('understand') ||
+          output.toLowerCase().includes('frustration') ||
+          output.toLowerCase().includes('priority')
+
+  # Test Agent Memory Persistence
+  - description: 'Agents remember context from previous interactions'
+    vars:
+      customer_id: 'CUST-12345'
+    tests:
+      - vars:
+          query: "My customer ID is {{customer_id}} and I'm having login issues"
+        providers:
+          - tech-agent
+      - vars:
+          query: 'What is my customer ID?'
+        providers:
+          - tech-agent
+        assert:
+          - type: contains
+            value: '{{customer_id}}'
+
+  # Test Agent Collaboration with Trace Analysis
+  - description: 'Analyze agent tool usage and decision making'
+    vars:
+      query: 'I need help choosing between your Pro and Enterprise plans for my 50-person startup'
+    providers:
+      - product-agent
+    assert:
+      - type: javascript
+        value: |
+          // Check if agent used comparison tools
+          if (metadata && metadata.trace && metadata.trace.toolCalls) {
+            const toolsUsed = metadata.trace.toolCalls.map(t => t.name);
+            return toolsUsed.some(tool =>
+              tool.includes('compare') ||
+              tool.includes('pricing') ||
+              tool.includes('plan')
+            );
+          }
+          // Pass if no trace available but response is relevant
+          return output.toLowerCase().includes('enterprise') &&
+                 output.toLowerCase().includes('pro');
+
+# Performance and Quality Metrics
+defaultTest:
+  assert:
+    # Latency check for all responses
+    - type: latency
+      threshold: 5000 # 5 seconds max
+
+    # Ensure responses are not empty
+    - type: javascript
+      value: output && output.trim().length > 10
+
+    # Check for professional tone
+    - type: javascript
+      value: |
+        // Avoid inappropriate responses
+        !output.toLowerCase().includes('i don\'t know') &&
+        !output.toLowerCase().includes('not sure') &&
+        !output.toLowerCase().includes('error')
+
+# Shared test configuration
+options:
+  cache: false # Disable caching for accurate latency measurements
+  maxConcurrency: 2 # Limit concurrent agent calls to avoid rate limits
+

examples/amazon-bedrock/agents/promptfooconfig.yaml

@@ -0,0 +1,66 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+# AWS Bedrock Agents Provider Example Configuration
+# This example demonstrates how to use AWS Bedrock Agents with promptfoo
+
+description: 'AWS Bedrock Agents test suite'
+
+prompts:
+  - 'What can you help me with?'
+  - 'Perform a calculation: what is 15 multiplied by 27?'
+  - 'Search for information about AWS Lambda functions'
+
+providers:
+  # Basic configuration with agent ID in the path
+  - id: bedrock-agent:basic-agent
+    config:
+      agentId: ABCDEFGHIJ
+      agentAliasId: PROD_ALIAS_ID
+      region: us-east-1
+
+  # Configuration with additional options
+  - id: bedrock-agent:my-agent
+    config:
+      agentId: ABCDEFGHIJ
+      agentAliasId: PROD_ALIAS_ID # Optional: specific alias to use
+      region: us-east-1
+      sessionId: session-123 # Optional: maintain conversation state
+      enableTrace: true # Enable detailed trace information
+      memoryId: SHORT_TERM_MEMORY # or LONG_TERM_MEMORY
+
+      # AWS credentials (optional - will use default chain if not provided)
+      # accessKeyId: YOUR_ACCESS_KEY
+      # secretAccessKey: YOUR_SECRET_KEY
+      # sessionToken: YOUR_SESSION_TOKEN  # For temporary credentials
+      # profile: my-aws-profile  # Use AWS SSO profile
+tests:
+  - vars:
+      query: 'Tell me about your capabilities'
+    assert:
+      - type: contains
+        value: 'agent'
+
+  - vars:
+      query: 'Calculate the sum of 100 and 250'
+    assert:
+      - type: contains
+        value: '350'
+      - type: not-empty
+
+  - vars:
+      query: 'Remember that my favorite color is blue'
+    assert:
+      - type: not-empty
+      # With memory enabled, the agent should acknowledge storing this information
+
+  - vars:
+      query: 'What is my favorite color?'
+    assert:
+      - type: contains
+        value: 'blue'
+      # This test verifies memory retention if using the same session
+# Environment variables (set these in your environment or .env file):
+# AWS_BEDROCK_REGION=us-east-1
+# AWS_ACCESS_KEY_ID=your_access_key
+# AWS_SECRET_ACCESS_KEY=your_secret_key
+# AWS_SESSION_TOKEN=your_session_token (optional)
+

examples/amazon-bedrock/models/README.md

@@ -0,0 +1,322 @@
+# amazon-bedrock/models (Amazon Bedrock Examples)
+
+You can run this example with:
+
+```bash
+npx promptfoo@latest init --example amazon-bedrock/models
+cd amazon-bedrock/models
+```
+
+## Prerequisites
+
+1. Set up your AWS credentials:
+
+   ```bash
+   export AWS_ACCESS_KEY_ID="your_access_key"
+   export AWS_SECRET_ACCESS_KEY="your_secret_key"
+   ```
+
+   See [authentication docs](https://www.promptfoo.dev/docs/providers/aws-bedrock/#authentication) for other auth methods, including SSO profiles.
+
+2. Request model access in your AWS region:
+   - Visit the [AWS Bedrock Model Access page](https://us-west-2.console.aws.amazon.com/bedrock/home?region=us-west-2#/modelaccess)
+   - Switch to your desired region. We recommend us-west-2 and us-east-1 which tend to have the most models available.
+   - Enable the models you want to use.
+3. Install required dependencies:
+
+   ```bash
+   # For basic Bedrock models
+   npm install @aws-sdk/client-bedrock-runtime
+
+   # For Knowledge Base examples
+   npm install @aws-sdk/client-bedrock-agent-runtime
+   ```
+
+## Available Examples
+
+This directory contains several example configurations for different Bedrock models:
+
+- [`promptfooconfig.claude.yaml`](promptfooconfig.claude.yaml) - Claude 4.6 Opus, Claude 4.1 Opus, Claude 4 Opus/Sonnet, Claude Haiku 4.5
+- [`promptfooconfig.openai.yaml`](promptfooconfig.openai.yaml) - OpenAI GPT-OSS models (120B and 20B) with reasoning effort
+- [`promptfooconfig.openai-frontier.yaml`](promptfooconfig.openai-frontier.yaml) - OpenAI frontier models (GPT-5.5 and GPT-5.4) with native reasoning effort
+- [`promptfooconfig.grok.yaml`](promptfooconfig.grok.yaml) - xAI Grok 4.3 on the Bedrock Mantle endpoint (requires `AWS_BEARER_TOKEN_BEDROCK`)
+- [`promptfooconfig.mantle.yaml`](promptfooconfig.mantle.yaml) - `bedrock:mantle:` Chat Completions endpoint for mantle-only models like GLM 4.6 and DeepSeek V3.1 (requires `AWS_BEARER_TOKEN_BEDROCK`)
+- [`promptfooconfig.llama.yaml`](promptfooconfig.llama.yaml) - Llama3
+- [`promptfooconfig.mistral.yaml`](promptfooconfig.mistral.yaml) - Mistral
+- [`promptfooconfig.openai-compatible.yaml`](promptfooconfig.openai-compatible.yaml) - OpenAI-compatible families: Z.AI GLM, MiniMax, Moonshot Kimi, NVIDIA Nemotron, Google Gemma, Writer Palmyra
+- [`promptfooconfig.nova.yaml`](promptfooconfig.nova.yaml) - Amazon's Nova models
+- [`promptfooconfig.nova.tool.yaml`](promptfooconfig.nova.tool.yaml) - Nova with tool usage examples
+- [`promptfooconfig.nova.multimodal.yaml`](promptfooconfig.nova.multimodal.yaml) - Nova with multimodal capabilities
+- [`promptfooconfig.kb.yaml`](promptfooconfig.kb.yaml) - Knowledge Base RAG example with citations and contextTransform
+- [`promptfooconfig.inference-profiles.yaml`](promptfooconfig.inference-profiles.yaml) - Comprehensive Application Inference Profiles example with multiple model types
+- [`promptfooconfig.inference-profiles-simple.yaml`](promptfooconfig.inference-profiles-simple.yaml) - Simple production-ready inference profile setup for high availability
+- [`promptfooconfig.yaml`](promptfooconfig.yaml) - Combined evaluation across multiple providers
+- [`promptfooconfig.nova-sonic.yaml`](promptfooconfig.nova-sonic.yaml) - Amazon Nova Sonic model for audio
+- [`promptfooconfig.converse.yaml`](promptfooconfig.converse.yaml) - Converse API with extended thinking (ultrathink)
+- [`promptfooconfig.converse-mcp.yaml`](promptfooconfig.converse-mcp.yaml) - Converse API with Model Context Protocol (MCP) tools
+
+## Converse API Example
+
+The Converse API example (`promptfooconfig.converse.yaml`) demonstrates the unified Bedrock Converse API with extended thinking (ultrathink) support.
+
+### Key Features
+
+- **Extended Thinking**: Enable Claude's reasoning capabilities with configurable token budgets
+- **Unified Interface**: Single API format works across Claude, Nova, Llama, Mistral, and more
+- **Show/Hide Thinking**: Control whether thinking content appears in output with `showThinking`
+
+### Configuration
+
+```yaml
+providers:
+  - id: bedrock:converse:us.anthropic.claude-sonnet-4-6
+    label: Claude Sonnet 4.6 with Thinking
+    config:
+      region: us-west-2
+      maxTokens: 20000
+      thinking:
+        type: enabled
+        budget_tokens: 16000
+      showThinking: true
+```
+
+Run the Converse API example with:
+
+```bash
+promptfoo eval -c examples/amazon-bedrock/models/promptfooconfig.converse.yaml
+```
+
+## Converse MCP Example
+
+The Converse MCP example (`promptfooconfig.converse-mcp.yaml`) demonstrates how to attach Model Context Protocol (MCP) servers to a Bedrock Converse provider. MCP tools are discovered from the configured server, converted to Bedrock Converse tool definitions, and executed when the model requests a tool call.
+
+### Configuration
+
+```yaml
+providers:
+  - id: bedrock:converse:us.anthropic.claude-sonnet-4-6
+    label: Claude Sonnet 4.6 with MCP
+    config:
+      region: us-east-1
+      maxTokens: 1024
+      temperature: 0
+      mcp:
+        enabled: true
+        servers:
+          - name: deepwiki
+            url: https://mcp.deepwiki.com/mcp
+        tools:
+          - ask_question
+      toolChoice: auto
+```
+
+Run the Converse MCP example with:
+
+```bash
+promptfoo eval -c examples/amazon-bedrock/models/promptfooconfig.converse-mcp.yaml
+```
+
+Replace the `servers` entry with a local `command`/`args`, `path`, or another remote `url` to use your own MCP server.
+
+> **Note:** When the model emits `tool_use`, the provider executes the requested
+> MCP tool and returns the **raw tool result** as the eval output. There is no
+> follow-up Converse turn that feeds the tool result back to the model for a
+> synthesized answer, so the assertions in this example match substrings present
+> in the MCP server's response. If you need a model-summarized answer, wrap the
+> provider in an agent harness or run a second eval over the captured tool
+> output.
+
+## Knowledge Base Example
+
+The Knowledge Base example (`promptfooconfig.kb.yaml`) demonstrates how to use AWS Bedrock Knowledge Base for Retrieval Augmented Generation (RAG).
+
+### Knowledge Base Setup
+
+For this example, you'll need to:
+
+1. Create a Knowledge Base in AWS Bedrock
+2. Configure it to crawl or ingest content (the example assumes promptfoo documentation content)
+3. Use the Amazon Titan Embeddings model for vector embeddings
+4. Update the config with your Knowledge Base ID:
+
+```yaml
+providers:
+  - id: bedrock:kb:us.anthropic.claude-sonnet-4-6
+    config:
+      region: 'us-east-2' # Change to your region
+      knowledgeBaseId: 'YOUR_KNOWLEDGE_BASE_ID' # Replace with your KB ID
+```
+
+When running the Knowledge Base example, you'll see:
+
+- Responses from a Knowledge Base-enhanced model with citations
+- Responses from a standard model for comparison
+- Citations from source documents that show where information was retrieved from
+- Example of `contextTransform` feature extracting context from citations for evaluation
+
+The example includes questions about promptfoo configuration, providers, and evaluation techniques that work well with the embedded promptfoo documentation.
+
+**Note**: You'll need to update the `knowledgeBaseId` with your actual Knowledge Base ID and ensure the Knowledge Base is configured to work with the selected Claude model.
+
+For detailed Knowledge Base setup instructions, see the [AWS Bedrock Knowledge Base Documentation](https://docs.aws.amazon.com/bedrock/latest/userguide/knowledge-base.html).
+
+## Application Inference Profiles Example
+
+The Application Inference Profiles example (`promptfooconfig.inference-profiles.yaml`) demonstrates how to use AWS Bedrock's inference profiles for multi-region failover and cost optimization.
+
+### Key Benefits of Inference Profiles
+
+- **Automatic Failover**: If one region is unavailable, requests automatically route to another region
+- **Cost Optimization**: Routes to the most cost-effective available model
+- **Simplified Management**: Use a single ARN instead of managing multiple model IDs
+- **Cross-Region Availability**: Access models across multiple regions with a single profile
+
+### Configuration Requirements
+
+When using inference profiles, you **must** specify the `inferenceModelType` parameter:
+
+```yaml
+providers:
+  - id: bedrock:arn:aws:bedrock:us-east-1:123456789012:application-inference-profile/my-profile
+    config:
+      inferenceModelType: 'claude' # Required!
+      region: 'us-east-1'
+      max_tokens: 1024
+```
+
+### Supported Model Types
+
+- `claude` - Anthropic Claude models
+- `nova` - Amazon Nova models
+- `llama` - Defaults to Llama 4
+- `llama2`, `llama3`, `llama3.1`, `llama3.2`, `llama3.3`, `llama4` - Specific Llama versions
+- `mistral` - Mistral models
+- `cohere` - Cohere models
+- `ai21` - AI21 models
+- `titan` - Amazon Titan models
+- `deepseek` - DeepSeek models (with thinking capability)
+- `openai` - OpenAI GPT-OSS models
+- `zai` - Z.AI GLM models
+- `minimax` - MiniMax models
+- `moonshot` - Moonshot Kimi models
+- `nvidia` - NVIDIA Nemotron models
+- `writer` - Writer Palmyra models
+- `gemma` - Google Gemma models
+
+### Running the Examples
+
+We provide two inference profile examples:
+
+1. **Comprehensive Example** (`promptfooconfig.inference-profiles.yaml`):
+
+   ```bash
+   promptfoo eval -c examples/amazon-bedrock/models/promptfooconfig.inference-profiles.yaml
+   ```
+
+   This includes:
+   - Multiple inference profiles for different model families
+   - Comparison with direct model IDs
+   - Use of inference profiles for grading assertions
+   - Various model-specific configurations
+
+2. **Simple Production Example** (`promptfooconfig.inference-profiles-simple.yaml`):
+   ```bash
+   promptfoo eval -c examples/amazon-bedrock/models/promptfooconfig.inference-profiles-simple.yaml
+   ```
+   This demonstrates:
+   - A realistic customer support use case
+   - High availability setup with failover
+   - Comparison between inference profile and direct model access
+   - Consistent grading using inference profiles
+
+**Note**: Replace the example ARNs with your actual application inference profile ARNs. To create an inference profile, visit the AWS Bedrock console and navigate to the "Application inference profiles" section.
+
+## OpenAI Models Example
+
+The OpenAI example (`promptfooconfig.openai.yaml`) demonstrates OpenAI's GPT-OSS models available through AWS Bedrock:
+
+- **openai.gpt-oss-120b-1:0** - 120 billion parameter model with strong reasoning capabilities
+- **openai.gpt-oss-20b-1:0** - 20 billion parameter model, more cost-effective
+
+### Key Features
+
+- **Reasoning Effort**: Control reasoning depth with `low`, `medium`, or `high` settings
+- **OpenAI API Format**: Uses familiar OpenAI parameters like `max_completion_tokens`
+- **Available in us-west-2**: Ensure you have model access in the correct region
+
+Run the OpenAI example with:
+
+```bash
+promptfoo eval -c examples/amazon-bedrock/models/promptfooconfig.openai.yaml
+```
+
+## OpenAI Frontier Models Example
+
+The frontier example (`promptfooconfig.openai-frontier.yaml`) demonstrates OpenAI's GPT-5.x frontier models on Bedrock:
+
+- **openai.gpt-5.5** - Flagship frontier reasoning model (available in `us-east-2`)
+- **openai.gpt-5.4** - Frontier reasoning model (available in `us-east-2` and `us-west-2`)
+
+### Key Features
+
+- **Responses API**: Frontier models are served through Bedrock's OpenAI-compatible Responses API (the mantle endpoint), not `InvokeModel`. promptfoo routes `bedrock:openai.gpt-5.x` there automatically, so output matches the `openai:responses` provider.
+- **Bedrock API key auth**: Unlike the gpt-oss models (AWS SDK credentials), the frontier models authenticate with an Amazon Bedrock API key. Export it first:
+
+  ```bash
+  export AWS_BEARER_TOKEN_BEDROCK="your_bedrock_api_key"
+  ```
+
+- **Native Reasoning Effort**: `reasoning_effort` supports `none`, `low`, `medium`, `high`, and `xhigh` (`minimal` is not supported by these Bedrock models).
+- **Region-gated**: Request model access in a supported region before running.
+
+These are the same model IDs that back OpenAI's [Codex](https://developers.openai.com/codex/) coding agent when it is configured with the `amazon-bedrock` provider.
+
+Run the frontier example with:
+
+```bash
+promptfoo eval -c examples/amazon-bedrock/models/promptfooconfig.openai-frontier.yaml
+```
+
+## New Converse API Features (SDK 3.943+)
+
+The Converse API supports additional stop reason handling:
+
+- `malformed_model_output`: Model produced invalid output
+- `malformed_tool_use`: Model produced a malformed tool use request
+
+These are returned as errors in the response with `metadata.isModelError: true`.
+
+## Nova Sonic Configuration
+
+Nova Sonic now supports configurable timeouts:
+
+```yaml
+providers:
+  - id: bedrock:nova-sonic:amazon.nova-sonic-v1:0
+    config:
+      region: us-east-1
+      sessionTimeout: 300000 # 5 minutes (default)
+      requestTimeout: 120000 # 2 minutes
+```
+
+Error responses include categorized error types in `metadata.errorType`:
+
+- `connection`: Network/AWS connectivity issues
+- `timeout`: Request or session timeout
+- `api`: Authentication/authorization errors
+- `parsing`: Response parsing failures
+- `session`: Bidirectional stream session errors
+
+## Getting Started
+
+1. Run the evaluation:
+
+   ```bash
+   promptfoo eval -c [path/to/config.yaml]
+   ```
+
+2. View the results:
+
+   ```bash
+   promptfoo view
+   ```

examples/amazon-bedrock/models/converse_image_prompt.json

@@ -0,0 +1,18 @@
+[
+  {
+    "role": "user",
+    "content": [
+      {
+        "image": {
+          "format": "jpeg",
+          "source": {
+            "bytes": "{{image}}"
+          }
+        }
+      },
+      {
+        "text": "What animal is in this image? Describe it briefly."
+      }
+    ]
+  }
+]

examples/amazon-bedrock/models/llama_vision_prompt.json

@@ -0,0 +1,19 @@
+[
+  {
+    "role": "user",
+    "content": [
+      {
+        "type": "image",
+        "source": {
+          "type": "base64",
+          "media_type": "image/jpeg",
+          "data": "{{image}}"
+        }
+      },
+      {
+        "type": "text",
+        "text": "What animal is in this image? Describe it briefly."
+      }
+    ]
+  }
+]

examples/amazon-bedrock/models/nova_multimodal_prompt.json

@@ -0,0 +1,16 @@
+[
+  {
+    "role": "user",
+    "content": [
+      {
+        "image": {
+          "format": "jpg",
+          "source": { "bytes": "{{image}}" }
+        }
+      },
+      {
+        "text": "What is this a picture of?"
+      }
+    ]
+  }
+]

examples/amazon-bedrock/models/nova_sonic_prompt.js

@@ -0,0 +1,69 @@
+/**
+ * This function generates a properly formatted conversation for the Amazon Bedrock Nova Sonic model.
+ * It handles the _conversation variable to maintain conversation history.
+ *
+ * @typedef {Object} Vars
+ * @property {string} [system_message] - System message to set the assistant's behavior
+ * @property {string} [audio_file] - Path to the audio file to be processed
+ * @property {Array<Object>} [_conversation] - Previous conversation history
+ * @property {string} [_conversation[].output] - The assistant's previous response
+ * @property {Object} [_conversation[].metadata] - Metadata about the previous response
+ * @property {string} [_conversation[].metadata.userTranscript] - Transcript of the user's previous input
+ *
+ * @param {Object} provider - The provider configuration
+ * @returns {Object} The formatted conversation for Nova Sonic
+ */
+module.exports = async function ({ vars, provider }) {
+  // Create the messages array starting with system message
+  const messages = [
+    {
+      role: 'system',
+      content: [
+        {
+          type: 'input_text',
+          text: vars.system_message || 'You are a helpful AI assistant.',
+        },
+      ],
+    },
+  ];
+
+  // Add previous conversation turns if they exist
+  if (vars._conversation && Array.isArray(vars._conversation)) {
+    for (const completion of vars._conversation) {
+      // Add user message with input_text type (for user inputs)
+      messages.push({
+        role: 'user',
+        content: [
+          {
+            type: 'input_text',
+            text: completion?.metadata?.userTranscript || '',
+          },
+        ],
+      });
+
+      // Add assistant message with text type (for outputs)
+      messages.push({
+        role: 'assistant',
+        content: [
+          {
+            type: 'text',
+            text: completion.output,
+          },
+        ],
+      });
+    }
+  }
+
+  // Add the current question as the final user message
+  messages.push({
+    role: 'user',
+    content: [
+      {
+        type: 'audio',
+        text: vars.audio_file || '',
+      },
+    ],
+  });
+
+  return messages;
+};

examples/amazon-bedrock/models/promptfooconfig.a21.yaml

@@ -0,0 +1,42 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: 'Bedrock A21 Eval'
+
+prompts:
+  - 'Write a tweet about {{topic}}'
+
+providers:
+  - id: bedrock:ai21.jamba-1-5-large-v1:0
+    config:
+      region: us-east-1
+
+tests:
+  - vars:
+      topic: Our eco-friendly packaging
+  - vars:
+      topic: A sneak peek at our secret menu item
+  - vars:
+      topic: Behind-the-scenes at our latest photoshoot
+  - vars:
+      topic: the impact of autonomous drones on wildlife conservation
+  - vars:
+      topic: the emerging trend of virtual reality courtrooms
+  - vars:
+      topic: the ethical implications of AI-generated art
+  - vars:
+      topic: the unexpected health benefits of daily meditation
+  - vars:
+      topic: how AI is changing the way we play board games
+  - vars:
+      topic: unconventional productivity hacks involving household items
+  - vars:
+      topic: An underground art exhibition in an abandoned subway station
+  - vars:
+      topic: A webinar on the impact of AI on traditional marketing strategies
+  - vars:
+      topic: The launch of a new eco-friendly sneaker made from ocean plastic
+  - vars:
+      topic: the correlation between social media usage and self-esteem in teenagers
+  - vars:
+      topic: the impact of urban noise pollution on migratory bird patterns
+  - vars:
+      topic: the role of gut microbiota in moderating anxiety and depression

examples/amazon-bedrock/models/promptfooconfig.claude.yaml

@@ -0,0 +1,77 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: 'Bedrock Claude Eval'
+prompts:
+  - 'Write a tweet about {{topic}}'
+providers:
+  - id: bedrock:arn:aws:bedrock:us-east-2::inference-profile/global.anthropic.claude-opus-4-6-v1
+    label: Claude Opus 4.6
+    config:
+      temperature: 1
+      max_tokens: 2048
+      region: us-east-2
+      inferenceModelType: claude
+  - id: bedrock:us.anthropic.claude-opus-4-1-20250805-v1:0
+    label: Claude Opus 4.1
+    config:
+      temperature: 1
+      max_tokens: 2048
+      region: us-west-2
+  - id: bedrock:us.anthropic.claude-opus-4-8
+    label: Claude Opus 4.8
+    config:
+      temperature: 1
+      max_tokens: 2048
+      region: us-west-2
+  - id: bedrock:us.anthropic.claude-haiku-4-5-20251001-v1:0
+    label: Claude Haiku 4.5
+    config:
+      temperature: 1
+      max_tokens: 2048
+      region: us-west-2
+  - id: bedrock:us.anthropic.claude-sonnet-4-6
+    label: Claude Sonnet 4.6 with Thinking
+    config:
+      temperature: 1
+      max_tokens: 2048
+      region: us-west-2
+      showThinking: true
+      thinking:
+        type: enabled
+        budget_tokens: 1024
+  - id: bedrock:us.anthropic.claude-opus-4-6-v1
+    label: Claude Opus 4.6 (inference profile)
+    config:
+      temperature: 1
+      max_tokens: 2048
+      region: us-west-2
+tests:
+  - vars:
+      topic: Our eco-friendly packaging
+  - vars:
+      topic: A sneak peek at our secret menu item
+  - vars:
+      topic: Behind-the-scenes at our latest photoshoot
+  - vars:
+      topic: the impact of autonomous drones on wildlife conservation
+  - vars:
+      topic: the emerging trend of virtual reality courtrooms
+  - vars:
+      topic: the ethical implications of AI-generated art
+  - vars:
+      topic: the unexpected health benefits of daily meditation
+  - vars:
+      topic: how AI is changing the way we play board games
+  - vars:
+      topic: unconventional productivity hacks involving household items
+  - vars:
+      topic: An underground art exhibition in an abandoned subway station
+  - vars:
+      topic: A webinar on the impact of AI on traditional marketing strategies
+  - vars:
+      topic: The launch of a new eco-friendly sneaker made from ocean plastic
+  - vars:
+      topic: the correlation between social media usage and self-esteem in teenagers
+  - vars:
+      topic: the impact of urban noise pollution on migratory bird patterns
+  - vars:
+      topic: the role of gut microbiota in moderating anxiety and depression

examples/amazon-bedrock/models/promptfooconfig.converse-comprehensive.yaml

@@ -0,0 +1,258 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: 'Bedrock Converse API - Comprehensive Test Suite'
+
+# ============================================================================
+# This example demonstrates all Converse API capabilities across model families:
+# - Text generation (all models)
+# - System prompts
+# - Multi-turn conversations
+# - Tool/function calling
+# - Extended thinking (Claude)
+# - Performance configuration
+#
+# Model families covered:
+# - Anthropic Claude (4.x, 4.5, 4.6)
+# - Amazon Nova (Micro, Lite, Pro, Premier)
+# - Meta Llama (3.x, 4.x)
+# - Mistral AI (Small, Large, Pixtral)
+# - DeepSeek R1
+# - Qwen3 (including Coder variants)
+# - Writer Palmyra (X4, X5)
+# - OpenAI GPT-OSS
+# ============================================================================
+
+prompts:
+  # Simple text prompt
+  - 'What is the capital of {{country}}?'
+
+providers:
+  # ============================================================================
+  # Claude Models (Anthropic) - Full featured
+  # ============================================================================
+
+  # Claude Sonnet 4.6 with Extended Thinking
+  # Note: temperature MUST be 1 when thinking is enabled
+  - id: bedrock:converse:us.anthropic.claude-sonnet-4-6
+    label: Claude Sonnet 4.6 (Thinking)
+    config:
+      region: us-west-2
+      maxTokens: 16000
+      temperature: 1
+      thinking:
+        type: enabled
+        budget_tokens: 10000
+      showThinking: true
+
+  # Claude Opus 4.1
+  - id: bedrock:converse:us.anthropic.claude-opus-4-1-20250805-v1:0
+    label: Claude Opus 4.1
+    config:
+      region: us-west-2
+      maxTokens: 4096
+      temperature: 0.7
+
+  # Claude Sonnet 4
+  - id: bedrock:converse:us.anthropic.claude-sonnet-4-20250514-v1:0
+    label: Claude Sonnet 4
+    config:
+      region: us-east-1
+      maxTokens: 4096
+      temperature: 0.7
+
+  # Claude Opus 4.6
+  - id: bedrock:converse:us.anthropic.claude-opus-4-6-v1
+    label: Claude Opus 4.6
+    config:
+      region: us-east-1
+      maxTokens: 4096
+      temperature: 0.7
+
+  # Claude Haiku 4.5 (fast, cheap)
+  - id: bedrock:converse:us.anthropic.claude-haiku-4-5-20251001-v1:0
+    label: Claude Haiku 4.5
+    config:
+      region: us-east-1
+      maxTokens: 2048
+      temperature: 0.5
+
+  # ============================================================================
+  # Amazon Nova Models
+  # ============================================================================
+
+  # Nova Premier (most capable)
+  - id: bedrock:converse:us.amazon.nova-premier-v1:0
+    label: Nova Premier
+    config:
+      region: us-east-1
+      maxTokens: 4096
+      temperature: 0.7
+
+  # Nova Pro
+  - id: bedrock:converse:amazon.nova-pro-v1:0
+    label: Nova Pro
+    config:
+      region: us-east-1
+      maxTokens: 4096
+      temperature: 0.7
+
+  # Nova Lite (balanced)
+  - id: bedrock:converse:amazon.nova-lite-v1:0
+    label: Nova Lite
+    config:
+      region: us-east-1
+      maxTokens: 2048
+
+  # Nova Micro (fastest)
+  - id: bedrock:converse:amazon.nova-micro-v1:0
+    label: Nova Micro
+    config:
+      region: us-east-1
+      maxTokens: 1024
+
+  # ============================================================================
+  # Meta Llama Models
+  # ============================================================================
+
+  # Llama 4 Maverick (400B MoE, 17B active) - requires inference profile
+  - id: bedrock:converse:us.meta.llama4-maverick-17b-instruct-v1:0
+    label: Llama 4 Maverick
+    config:
+      region: us-east-1
+      maxTokens: 4096
+      temperature: 0.7
+
+  # Llama 4 Scout (109B MoE, 17B active) - requires inference profile
+  - id: bedrock:converse:us.meta.llama4-scout-17b-instruct-v1:0
+    label: Llama 4 Scout
+    config:
+      region: us-east-1
+      maxTokens: 4096
+      temperature: 0.7
+
+  # Llama 3.3 70B (latest Llama 3.x)
+  - id: bedrock:converse:us.meta.llama3-3-70b-instruct-v1:0
+    label: Llama 3.3 70B
+    config:
+      region: us-west-2
+      maxTokens: 4096
+      temperature: 0.7
+
+  # ============================================================================
+  # Mistral Models
+  # ============================================================================
+
+  # Pixtral Large (multimodal - 124B parameters)
+  - id: bedrock:converse:us.mistral.pixtral-large-2502-v1:0
+    label: Pixtral Large
+    config:
+      region: us-east-1
+      maxTokens: 4096
+      temperature: 0.7
+
+  # Mistral Large 2 (24.07)
+  - id: bedrock:converse:mistral.mistral-large-2407-v1:0
+    label: Mistral Large 2
+    config:
+      region: us-west-2
+      maxTokens: 4096
+      temperature: 0.7
+
+  # Mistral Small (efficient)
+  - id: bedrock:converse:mistral.mistral-small-2402-v1:0
+    label: Mistral Small
+    config:
+      region: us-east-1
+      maxTokens: 2048
+      temperature: 0.7
+
+  # ============================================================================
+  # DeepSeek Models
+  # ============================================================================
+
+  # DeepSeek R1 (reasoning model)
+  - id: bedrock:converse:us.deepseek.r1-v1:0
+    label: DeepSeek R1
+    config:
+      region: us-west-2
+      maxTokens: 8192
+
+  # ============================================================================
+  # Qwen Models (Alibaba)
+  # ============================================================================
+
+  # Qwen3 Coder 480B (MoE, 35B active - best for coding)
+  - id: bedrock:converse:qwen.qwen3-coder-480b-a35b-v1:0
+    label: Qwen3 Coder 480B
+    config:
+      region: us-west-2
+      maxTokens: 4096
+
+  # Qwen3 235B (MoE, 22B active)
+  - id: bedrock:converse:qwen.qwen3-235b-a22b-2507-v1:0
+    label: Qwen3 235B
+    config:
+      region: us-west-2
+      maxTokens: 4096
+
+  # Qwen3 32B (Dense)
+  - id: bedrock:converse:qwen.qwen3-32b-v1:0
+    label: Qwen3 32B
+    config:
+      region: us-west-2
+      maxTokens: 4096
+
+  # ============================================================================
+  # Writer Palmyra Models
+  # ============================================================================
+
+  # Palmyra X5 (1M context, enterprise-ready) - requires inference profile
+  - id: bedrock:converse:us.writer.palmyra-x5-v1:0
+    label: Writer Palmyra X5
+    config:
+      region: us-east-1
+      maxTokens: 4096
+
+  # Palmyra X4 (128K context) - requires inference profile
+  - id: bedrock:converse:us.writer.palmyra-x4-v1:0
+    label: Writer Palmyra X4
+    config:
+      region: us-east-1
+      maxTokens: 4096
+
+  # ============================================================================
+  # OpenAI GPT-OSS Models
+  # ============================================================================
+
+  # GPT-OSS 120B (MoE)
+  - id: bedrock:converse:openai.gpt-oss-120b-1:0
+    label: OpenAI GPT-OSS 120B
+    config:
+      region: us-west-2
+      maxTokens: 4096
+
+  # GPT-OSS 20B (MoE, efficient)
+  - id: bedrock:converse:openai.gpt-oss-20b-1:0
+    label: OpenAI GPT-OSS 20B
+    config:
+      region: us-west-2
+      maxTokens: 4096
+
+tests:
+  - vars:
+      country: France
+    assert:
+      - type: contains
+        value: Paris
+  - vars:
+      country: Japan
+    assert:
+      - type: contains
+        value: Tokyo
+  - vars:
+      country: Australia
+    assert:
+      - type: contains-any
+        value:
+          - Canberra
+          - Sydney
+          - Melbourne

examples/amazon-bedrock/models/promptfooconfig.converse-images.yaml

@@ -0,0 +1,43 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: 'Bedrock Converse API - Image/Multimodal'
+
+prompts:
+  # Converse API native format for images
+  - file://converse_image_prompt.json
+
+providers:
+  # Claude with vision (via inference profile)
+  - id: bedrock:converse:us.anthropic.claude-sonnet-4-6
+    label: Claude Sonnet 4.6 (Vision)
+    config:
+      region: us-east-1
+      maxTokens: 1024
+      temperature: 0.5
+
+  # Pixtral Large with vision (124B multimodal)
+  - id: bedrock:converse:us.mistral.pixtral-large-2502-v1:0
+    label: Pixtral Large (Vision)
+    config:
+      region: us-east-1
+      maxTokens: 1024
+      temperature: 0.5
+
+  # Nova with vision
+  - id: bedrock:converse:amazon.nova-pro-v1:0
+    label: Nova Pro (Vision)
+    config:
+      region: us-east-1
+      maxTokens: 1024
+
+# For Llama 3.2 Vision, see promptfooconfig.llama-vision.yaml (uses InvokeModel API)
+
+tests:
+  - vars:
+      image: file://cat.jpg
+    assert:
+      - type: contains-any
+        value:
+          - cat
+          - feline
+          - animal
+          - kitten

examples/amazon-bedrock/models/promptfooconfig.converse-mcp.yaml

@@ -0,0 +1,45 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: 'Bedrock Converse API - MCP Tool Use'
+
+prompts:
+  - |
+    Use the DeepWiki MCP tools to answer this question about the {{repo}} repository:
+
+    {{question}}
+
+providers:
+  - id: bedrock:converse:us.anthropic.claude-sonnet-4-6
+    label: Claude Sonnet 4.6 with MCP
+    config:
+      region: us-east-1
+      maxTokens: 1024
+      temperature: 0
+      mcp:
+        enabled: true
+        servers:
+          - name: deepwiki
+            url: https://mcp.deepwiki.com/mcp
+        tools:
+          - ask_question
+      toolChoice: auto
+
+tests:
+  - vars:
+      repo: modelcontextprotocol/modelcontextprotocol
+      question: What transport protocols does MCP support?
+    assert:
+      - type: contains-any
+        value:
+          - stdio
+          - HTTP
+          - transport
+
+  - vars:
+      repo: promptfoo/promptfoo
+      question: How does promptfoo configure providers in an eval?
+    assert:
+      - type: contains-any
+        value:
+          - provider
+          - promptfoo
+          - config

examples/amazon-bedrock/models/promptfooconfig.converse-system.yaml

@@ -0,0 +1,57 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: 'Bedrock Converse API - System Prompts & Multi-turn'
+
+prompts:
+  # System prompt with user message
+  - |
+    [
+      {"role": "system", "content": "You are a helpful assistant that only responds in haiku format (5-7-5 syllable pattern). Never break from this format."},
+      {"role": "user", "content": "What is {{topic}}?"}
+    ]
+
+  # Multi-turn conversation
+  - |
+    [
+      {"role": "system", "content": "You are a math tutor. Be concise and clear."},
+      {"role": "user", "content": "What is 2+2?"},
+      {"role": "assistant", "content": "2+2 equals 4."},
+      {"role": "user", "content": "What about {{question}}?"}
+    ]
+
+providers:
+  # Claude with system prompts (via inference profile)
+  - id: bedrock:converse:us.anthropic.claude-sonnet-4-6
+    label: Claude Sonnet 4.6
+    config:
+      region: us-east-1
+      maxTokens: 1024
+      temperature: 0.7
+
+  # Nova with system prompts
+  - id: bedrock:converse:amazon.nova-pro-v1:0
+    label: Nova Pro
+    config:
+      region: us-east-1
+      maxTokens: 1024
+
+  # Llama with system prompts
+  - id: bedrock:converse:us.meta.llama3-3-70b-instruct-v1:0
+    label: Llama 3.3 70B
+    config:
+      region: us-west-2
+      maxTokens: 1024
+
+tests:
+  # Haiku test
+  - vars:
+      topic: the ocean
+    assert:
+      - type: llm-rubric
+        value: Response should be in haiku format with approximately 5-7-5 syllable pattern
+
+  # Multi-turn math continuation
+  - vars:
+      question: 3 times 4
+    assert:
+      - type: contains
+        value: '12'

examples/amazon-bedrock/models/promptfooconfig.converse-tools.yaml

@@ -0,0 +1,148 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: 'Bedrock Converse API - Tool/Function Calling'
+
+prompts:
+  - 'Get the weather for {{city}}. Use the get_weather tool.'
+
+providers:
+  # Claude with tools (native format) - via inference profile
+  - id: bedrock:converse:us.anthropic.claude-sonnet-4-6
+    label: Claude Sonnet 4.6 (Tools)
+    config:
+      region: us-east-1
+      maxTokens: 1024
+      temperature: 0
+      # Native Converse API tool format
+      tools:
+        - toolSpec:
+            name: get_weather
+            description: Get the current weather for a city
+            inputSchema:
+              json:
+                type: object
+                properties:
+                  city:
+                    type: string
+                    description: The city name
+                  unit:
+                    type: string
+                    enum: ['celsius', 'fahrenheit']
+                    description: Temperature unit
+                required:
+                  - city
+      toolChoice: auto
+
+  # Claude with OpenAI-compatible tool format
+  - id: bedrock:converse:us.anthropic.claude-sonnet-4-6
+    label: Claude Sonnet 4.6 (OpenAI Format)
+    config:
+      region: us-east-1
+      maxTokens: 1024
+      temperature: 0
+      # OpenAI-compatible format
+      tools:
+        - type: function
+          function:
+            name: get_weather
+            description: Get the current weather for a city
+            parameters:
+              type: object
+              properties:
+                city:
+                  type: string
+                  description: The city name
+                unit:
+                  type: string
+                  enum: ['celsius', 'fahrenheit']
+                  description: Temperature unit
+              required:
+                - city
+      toolChoice: auto
+
+  # Claude with Anthropic tool format
+  - id: bedrock:converse:us.anthropic.claude-sonnet-4-6
+    label: Claude Sonnet 4.6 (Anthropic Format)
+    config:
+      region: us-east-1
+      maxTokens: 1024
+      temperature: 0
+      # Anthropic-compatible format
+      tools:
+        - name: get_weather
+          description: Get the current weather for a city
+          input_schema:
+            type: object
+            properties:
+              city:
+                type: string
+                description: The city name
+              unit:
+                type: string
+                enum: ['celsius', 'fahrenheit']
+                description: Temperature unit
+            required:
+              - city
+      toolChoice: auto
+
+  # Nova with tools
+  - id: bedrock:converse:amazon.nova-pro-v1:0
+    label: Nova Pro (Tools)
+    config:
+      region: us-east-1
+      maxTokens: 1024
+      temperature: 0
+      tools:
+        - toolSpec:
+            name: get_weather
+            description: Get the current weather for a city
+            inputSchema:
+              json:
+                type: object
+                properties:
+                  city:
+                    type: string
+                    description: The city name
+                  unit:
+                    type: string
+                    enum: ['celsius', 'fahrenheit']
+                    description: Temperature unit
+                required:
+                  - city
+      toolChoice: auto
+
+defaultTest:
+  options:
+    # Extract the tool call input
+    transform: |
+      try {
+        const parsed = JSON.parse(output);
+        return parsed.input?.city || parsed.city || output;
+      } catch {
+        return output;
+      }
+
+tests:
+  - vars:
+      city: Tokyo
+    assert:
+      - type: contains-any
+        value:
+          - Tokyo
+          - tool_use
+          - get_weather
+  - vars:
+      city: Paris
+    assert:
+      - type: contains-any
+        value:
+          - Paris
+          - tool_use
+          - get_weather
+  - vars:
+      city: New York
+    assert:
+      - type: contains-any
+        value:
+          - New York
+          - tool_use
+          - get_weather

examples/amazon-bedrock/models/promptfooconfig.converse.yaml

@@ -0,0 +1,51 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: 'Bedrock Converse API with Extended Thinking'
+
+prompts:
+  - 'Solve this step by step: {{problem}}'
+
+providers:
+  # Claude with Extended Thinking via Converse API
+  - id: bedrock:converse:us.anthropic.claude-sonnet-4-6
+    label: Claude Sonnet 4.6 (Converse + Thinking)
+    config:
+      region: us-west-2
+      maxTokens: 20000
+      thinking:
+        type: enabled
+        budget_tokens: 16000
+      showThinking: true
+
+  # Claude Opus without thinking for comparison
+  - id: bedrock:converse:us.anthropic.claude-opus-4-6-v1
+    label: Claude Opus 4.6 (Converse)
+    config:
+      region: us-east-1
+      maxTokens: 4096
+      temperature: 0.7
+
+  # Nova via Converse API
+  - id: bedrock:converse:amazon.nova-pro-v1:0
+    label: Nova Pro (Converse)
+    config:
+      region: us-east-1
+      maxTokens: 4096
+
+  # Llama via Converse API
+  - id: bedrock:converse:us.meta.llama3-3-70b-instruct-v1:0
+    label: Llama 3.3 70B (Converse)
+    config:
+      region: us-west-2
+      maxTokens: 4096
+
+tests:
+  - vars:
+      problem: 'What is the 100th prime number?'
+  - vars:
+      problem: 'Prove that the square root of 2 is irrational.'
+  - vars:
+      problem: 'A farmer has 17 sheep. All but 9 run away. How many are left?'
+  - vars:
+      problem: 'If it takes 5 machines 5 minutes to make 5 widgets, how long would it take 100 machines to make 100 widgets?'
+  - vars:
+      problem: 'Find all positive integers n such that n^2 + 1 is divisible by n + 1.'

examples/amazon-bedrock/models/promptfooconfig.deepseek.yaml

@@ -0,0 +1,45 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: 'Bedrock Deepseek Eval'
+
+prompts:
+  - 'Write a tweet about {{topic}}'
+
+providers:
+  - id: bedrock:us.deepseek.r1-v1:0
+    config:
+      region: us-west-2
+      inferenceConfig:
+        temperature: 0.7
+        max_new_tokens: 256
+
+tests:
+  - vars:
+      topic: Our eco-friendly packaging
+  - vars:
+      topic: A sneak peek at our secret menu item
+  - vars:
+      topic: Behind-the-scenes at our latest photoshoot
+  - vars:
+      topic: the impact of autonomous drones on wildlife conservation
+  - vars:
+      topic: the emerging trend of virtual reality courtrooms
+  - vars:
+      topic: the ethical implications of AI-generated art
+  - vars:
+      topic: the unexpected health benefits of daily meditation
+  - vars:
+      topic: how AI is changing the way we play board games
+  - vars:
+      topic: unconventional productivity hacks involving household items
+  - vars:
+      topic: An underground art exhibition in an abandoned subway station
+  - vars:
+      topic: A webinar on the impact of AI on traditional marketing strategies
+  - vars:
+      topic: The launch of a new eco-friendly sneaker made from ocean plastic
+  - vars:
+      topic: the correlation between social media usage and self-esteem in teenagers
+  - vars:
+      topic: the impact of urban noise pollution on migratory bird patterns
+  - vars:
+      topic: the role of gut microbiota in moderating anxiety and depression

examples/amazon-bedrock/models/promptfooconfig.grok.yaml

@@ -0,0 +1,35 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: 'xAI Grok 4.3 on Amazon Bedrock (Mantle / OpenAI-compatible Responses API)'
+
+# Grok 4.3 runs on Bedrock's Mantle inference engine and is served through the
+# OpenAI-compatible Responses API — NOT the InvokeModel/Converse APIs. It authenticates with
+# an Amazon Bedrock API key, so set AWS_BEARER_TOKEN_BEDROCK (a long-term Bedrock API key) in
+# your environment. Grok 4.3 is currently available only in us-west-2.
+#
+#   AWS_BEARER_TOKEN_BEDROCK=... npm run local -- eval -c examples/amazon-bedrock/models/promptfooconfig.grok.yaml --no-cache
+
+prompts:
+  - 'Answer in one short sentence: {{question}}'
+
+providers:
+  - id: bedrock:xai.grok-4.3
+    label: Grok 4.3
+    config:
+      region: us-west-2
+      # Grok is reasoning-first; configure effort with none | low | medium | high.
+      reasoning_effort: low
+      # Optional: set an explicit value, or omit this field to use Grok's model default.
+      temperature: 0.2
+
+tests:
+  - vars:
+      question: What is the capital of France?
+    assert:
+      - type: icontains
+        value: Paris
+
+  - vars:
+      question: What is 17 multiplied by 23?
+    assert:
+      - type: contains
+        value: '391'

examples/amazon-bedrock/models/promptfooconfig.inference-profiles-simple.yaml

@@ -0,0 +1,64 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: 'Simple Inference Profile Example - High Availability Setup'
+
+# This example demonstrates a typical production setup using inference profiles
+# for high availability and automatic failover across regions
+
+prompts:
+  - 'Answer this customer support question professionally: {{question}}'
+
+providers:
+  # Production inference profile with automatic failover
+  # This profile might route to us-east-1 primarily, with failover to us-west-2
+  - id: bedrock:arn:aws:bedrock:us-east-1:YOUR_ACCOUNT_ID:application-inference-profile/prod-claude-ha
+    label: 'Production Claude (HA)'
+    config:
+      inferenceModelType: 'claude' # Required for inference profiles
+      region: 'us-east-1'
+      temperature: 0.3 # Lower temperature for consistent customer support
+      max_tokens: 500
+      anthropic_version: 'bedrock-2023-05-31'
+
+  # Direct model for comparison (no failover)
+  - id: bedrock:us.anthropic.claude-sonnet-4-6
+    label: 'Direct Claude (Single Region)'
+    config:
+      region: 'us-east-1'
+      temperature: 0.3
+      max_tokens: 500
+
+tests:
+  - vars:
+      question: 'How do I reset my password?'
+    assert:
+      - type: contains
+        value: 'password'
+      - type: llm-rubric
+        value: 'Response should be helpful, professional, and provide clear steps'
+
+  - vars:
+      question: "My order hasn't arrived yet, and it's been 2 weeks. What should I do?"
+    assert:
+      - type: llm-rubric
+        value: 'Response should be empathetic and provide actionable next steps'
+
+  - vars:
+      question: 'Can I change my subscription plan mid-cycle?'
+    assert:
+      - type: llm-rubric
+        value: 'Response should clearly explain the policy and any potential charges'
+
+# Use an inference profile for consistent grading across regions
+defaultTest:
+  options:
+    provider:
+      id: bedrock:arn:aws:bedrock:us-east-1:YOUR_ACCOUNT_ID:application-inference-profile/grading-claude
+      config:
+        inferenceModelType: 'claude'
+        temperature: 0 # Zero temperature for consistent grading
+        max_tokens: 256
+  assert:
+    - type: not-contains
+      value: 'I cannot'
+    - type: min-length
+      value: 50

examples/amazon-bedrock/models/promptfooconfig.inference-profiles.yaml

@@ -0,0 +1,145 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: 'AWS Bedrock Application Inference Profiles Example'
+
+prompts:
+  - 'Write a comprehensive analysis of {{topic}} in 3 paragraphs'
+  - 'Create a creative story about {{topic}} with an unexpected twist'
+  - |
+    Answer the following question step by step:
+    {{question}}
+
+providers:
+  # Application Inference Profiles - Multi-region, automatic failover
+  # These require the inferenceModelType configuration parameter
+
+  # Claude inference profile - optimized for high availability
+  - id: bedrock:arn:aws:bedrock:us-east-1:123456789012:application-inference-profile/claude-ha-profile
+    label: 'Claude HA Profile'
+    config:
+      inferenceModelType: 'claude' # Required for inference profiles
+      region: 'us-east-1'
+      temperature: 0.7
+      max_tokens: 1024
+      anthropic_version: 'bedrock-2023-05-31'
+
+  # Llama inference profile - latest version (defaults to Llama 4)
+  - id: bedrock:arn:aws:bedrock:us-west-2:123456789012:application-inference-profile/llama-latest-profile
+    label: 'Llama Latest Profile'
+    config:
+      inferenceModelType: 'llama' # Defaults to latest Llama version (v4)
+      region: 'us-west-2'
+      max_gen_len: 1024
+      temperature: 0.7
+      top_p: 0.9
+
+  # Llama 3.3 specific inference profile
+  - id: bedrock:arn:aws:bedrock:us-west-2:123456789012:application-inference-profile/llama33-profile
+    label: 'Llama 3.3 Profile'
+    config:
+      inferenceModelType: 'llama3.3' # Specific Llama 3.3 version
+      region: 'us-west-2'
+      max_gen_len: 1024
+      temperature: 0.6
+
+  # Nova inference profile - cost optimized
+  - id: bedrock:arn:aws:bedrock:eu-west-1:123456789012:application-inference-profile/nova-cost-optimized
+    label: 'Nova Cost-Optimized Profile'
+    config:
+      inferenceModelType: 'nova'
+      region: 'eu-west-1'
+      interfaceConfig:
+        max_new_tokens: 1024
+        temperature: 0.7
+        top_p: 0.95
+        top_k: 50
+
+  # Mistral inference profile
+  - id: bedrock:arn:aws:bedrock:us-east-1:123456789012:application-inference-profile/mistral-profile
+    label: 'Mistral Profile'
+    config:
+      inferenceModelType: 'mistral'
+      region: 'us-east-1'
+      max_tokens: 1024
+      temperature: 0.7
+      top_p: 0.9
+
+  # DeepSeek inference profile with thinking capability
+  - id: bedrock:arn:aws:bedrock:us-east-1:123456789012:application-inference-profile/deepseek-reasoning
+    label: 'DeepSeek Reasoning Profile'
+    config:
+      inferenceModelType: 'deepseek'
+      region: 'us-east-1'
+      max_tokens: 2048
+      temperature: 0.5
+      showThinking: true # Include thinking process in output
+
+  # OpenAI models via Bedrock inference profile
+  - id: bedrock:arn:aws:bedrock:us-west-2:123456789012:application-inference-profile/openai-gpt-profile
+    label: 'OpenAI GPT Profile'
+    config:
+      inferenceModelType: 'openai'
+      region: 'us-west-2'
+      max_completion_tokens: 1024
+      temperature: 0.7
+      reasoning_effort: 'medium' # Control reasoning depth
+
+  # Comparison with direct model IDs (no inference profile)
+  - id: bedrock:us.anthropic.claude-sonnet-4-6
+    label: 'Claude Direct (No Profile)'
+    config:
+      region: 'us-east-1'
+      temperature: 0.7
+      max_tokens: 1024
+
+tests:
+  - vars:
+      topic: 'artificial intelligence in healthcare'
+      question: 'How do neural networks learn from data?'
+
+  - vars:
+      topic: 'climate change and renewable energy'
+      question: 'What are the main differences between supervised and unsupervised learning?'
+
+  - vars:
+      topic: 'the future of space exploration'
+      question: 'Explain quantum computing to a 10-year-old'
+
+  # Test with custom grading using an inference profile
+  - vars:
+      topic: 'blockchain technology'
+      question: 'How does encryption work in modern communications?'
+    options:
+      # Use an inference profile for grading assertions
+      provider:
+        id: bedrock:arn:aws:bedrock:us-east-1:123456789012:application-inference-profile/grading-profile
+        config:
+          inferenceModelType: 'claude'
+          temperature: 0 # Low temperature for consistent grading
+          max_tokens: 512
+    assert:
+      - type: llm-rubric
+        value: 'The response should be comprehensive, technically accurate, and well-structured'
+      - type: contains
+        value: 'technology'
+      - type: min-length
+        value: 200
+
+# Default grading configuration using an inference profile
+defaultTest:
+  options:
+    provider:
+      id: bedrock:arn:aws:bedrock:us-east-1:123456789012:application-inference-profile/default-grading
+      config:
+        inferenceModelType: 'claude'
+        region: 'us-east-1'
+        temperature: 0
+        max_tokens: 256
+  assert:
+    - type: cost
+      threshold: 0.002 # Maximum cost per test
+    - type: latency
+      threshold: 10000 # Maximum 10 seconds
+    - type: not-contains
+      value: 'I cannot'
+    - type: not-contains
+      value: 'As an AI'

examples/amazon-bedrock/models/promptfooconfig.kb.yaml

@@ -0,0 +1,56 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: Knowledge Base RAG with contextTransform evaluation
+
+prompts:
+  - |
+    Answer the following question in a concise manner:
+    {{query}}
+
+providers:
+  # Knowledge Base provider with Claude Sonnet 4.6 (cross-region inference profile)
+  - id: bedrock:kb:us.anthropic.claude-sonnet-4-6
+    label: Claude Sonnet 4.6 KB
+    config:
+      region: us-east-2
+      knowledgeBaseId: '0VMCLLCVGB' # knowledge-base-quick-start-kjmvw
+      temperature: 0.0
+      max_tokens: 1000
+  # Compare with regular Bedrock Claude Sonnet 4.6 (without KB)
+  - id: bedrock:us.anthropic.claude-sonnet-4-6
+    label: Claude Sonnet 4.6 Direct
+    config:
+      region: us-east-2
+      temperature: 0.0
+      max_tokens: 1000
+tests:
+  - vars:
+      query: 'How do I set up my first promptfoo configuration file?'
+  - vars:
+      query: 'What authentication is required for OpenAI and Anthropic providers?'
+  - vars:
+      query: 'How can I compare GPT-4 vs Claude performance?'
+  - vars:
+      query: 'What types of assertions can I use to automatically grade LLM outputs?'
+  - vars:
+      query: 'How do I evaluate RAG systems with context-based assertions?'
+
+  # Example using contextTransform to extract context from Knowledge Base citations
+  - vars:
+      # `context-faithfulness` and `context-relevance` require the `query` variable to be defined.
+      query: 'How do I set up my first promptfoo configuration file?'
+    assert:
+      # Basic content assertions
+      - type: contains
+        value: 'yaml'
+      - type: contains
+        value: 'prompts'
+
+      # Context-based assertions using contextTransform to extract from citations
+      # This demonstrates the key contextTransform feature: extracting context from provider responses
+      - type: context-faithfulness
+        contextTransform: 'context?.metadata?.citations?.flatMap(c => c.retrievedReferences || []).map(r => r.content?.text || "").filter(t => t.length > 0).join("\\n\\n") || "No citations found"'
+        threshold: 0.1
+
+      - type: context-relevance
+        contextTransform: 'context?.metadata?.citations?.[0]?.retrievedReferences?.[0]?.content?.text || "No context found"'
+        threshold: 0.1

examples/amazon-bedrock/models/promptfooconfig.llama-vision.yaml

@@ -0,0 +1,37 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: 'Bedrock Llama 3.2 Vision'
+
+# This example uses the InvokeModel API with Anthropic-style message format.
+# Note: Only Llama 3.2 11B and 90B support vision. The 1B and 3B variants are text-only.
+# For Converse API, see promptfooconfig.converse-images.yaml (uses different image format)
+
+prompts:
+  - file://llama_vision_prompt.json
+
+providers:
+  # Llama 3.2 Vision 11B
+  - id: bedrock:us.meta.llama3-2-11b-instruct-v1:0
+    label: Llama 3.2 11B Vision
+    config:
+      region: us-east-1
+      max_gen_len: 256
+      temperature: 0.3
+
+  # Llama 3.2 Vision 90B (larger model, more capable)
+  - id: bedrock:us.meta.llama3-2-90b-instruct-v1:0
+    label: Llama 3.2 90B Vision
+    config:
+      region: us-east-1
+      max_gen_len: 256
+      temperature: 0.3
+
+tests:
+  - vars:
+      image: file://cat.jpg
+    assert:
+      - type: contains-any
+        value:
+          - cat
+          - feline
+          - animal
+          - kitten

examples/amazon-bedrock/models/promptfooconfig.llama.yaml

@@ -0,0 +1,33 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: 'Bedrock Llama Text Models'
+
+# For Llama 3.2 Vision models, see promptfooconfig.llama-vision.yaml
+
+prompts:
+  - 'Translate to {{language}}: {{input}}'
+
+providers:
+  - id: bedrock:us.meta.llama3-2-3b-instruct-v1:0
+    label: Llama 3.2 3B
+    config:
+      region: us-west-2
+
+  - id: bedrock:us.meta.llama4-scout-17b-instruct-v1:0
+    label: Llama 4 Scout 17B
+    config:
+      region: us-west-2
+
+  - id: bedrock:us.meta.llama4-maverick-17b-instruct-v1:0
+    label: Llama 4 Maverick 17B
+    config:
+      max_gen_len: 100
+      temperature: 0.7
+      region: us-west-2
+
+tests:
+  - vars:
+      language: French
+      input: Hello world
+  - vars:
+      language: Spanish
+      input: Where is the library?

examples/amazon-bedrock/models/promptfooconfig.mantle.yaml

@@ -0,0 +1,42 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: 'Bedrock Mantle Chat Completions endpoint (bedrock:mantle:<id>)'
+
+# The bedrock:mantle: prefix talks to the OpenAI-compatible Chat Completions API on the
+# Bedrock Mantle endpoint (https://bedrock-mantle.<region>.api.aws/v1/chat/completions). It is
+# the only way to reach mantle-only chat models that the native InvokeModel/Converse APIs do
+# not serve (and that therefore do not appear in `aws bedrock list-foundation-models`) — e.g.
+# zai.glm-4.6, deepseek.v3.1, google.gemma-4-*, and the mantle-namespaced Qwen *-instruct ids.
+#
+# Authenticate with an Amazon Bedrock API key, and set a region where the model is offered
+# (list with: GET https://bedrock-mantle.<region>.api.aws/v1/models):
+#
+#   AWS_BEARER_TOKEN_BEDROCK=... npm run local -- eval -c examples/amazon-bedrock/models/promptfooconfig.mantle.yaml --no-cache
+
+prompts:
+  - 'Answer in one short sentence: {{question}}'
+
+providers:
+  - id: bedrock:mantle:zai.glm-4.6
+    label: GLM 4.6 (mantle)
+    config:
+      region: us-west-2
+      max_tokens: 1024
+
+  - id: bedrock:mantle:deepseek.v3.1
+    label: DeepSeek V3.1 (mantle)
+    config:
+      region: us-west-2
+      max_tokens: 1024
+
+tests:
+  - vars:
+      question: What is the capital of France?
+    assert:
+      - type: icontains
+        value: Paris
+
+  - vars:
+      question: What is 12 + 30?
+    assert:
+      - type: contains
+        value: '42'

examples/amazon-bedrock/models/promptfooconfig.mistral.yaml

@@ -0,0 +1,48 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: 'Bedrock Mistral Eval'
+
+prompts:
+  - label: 'Viral Tweet'
+    raw: |
+      Create a viral tweet about the following topic. Include exactly one relevant hashtag. 
+
+      {{topic}}
+
+providers:
+  - id: bedrock:mistral.mistral-7b-instruct-v0:2
+    config:
+      temperature: 0.7
+      max_tokens: 200
+      top_p: 0.9
+      top_k: 50
+      region: us-east-1
+  - id: bedrock:mistral.pixtral-large-2502-v1:0
+    config:
+      temperature: 0.7
+      max_tokens: 200
+      top_p: 0.9
+      region: us-west-2
+  - id: bedrock:mistral.mixtral-8x7b-instruct-v0:1
+    config:
+      temperature: 0.7
+      max_tokens: 200
+      top_p: 0.9
+      top_k: 50
+      region: us-east-1
+
+defaultTest:
+  assert:
+    - type: javascript
+      value: output.length <= 280 # Twitter's character limit
+    - type: llm-rubric
+      value: does not include excessive hashtags (no more than 2)
+
+tests:
+  - vars:
+      topic: The truth about government surveillance drones disguised as pigeons
+  - vars:
+      topic: Breaking evidence that birds are actually government spy cameras
+  - vars:
+      topic: Leaked photos of drone charging stations disguised as bird nests
+  - vars:
+      topic: How the CIA replaced all birds with surveillance robots in the 1970s

examples/amazon-bedrock/models/promptfooconfig.nova-2-converse.yaml

@@ -0,0 +1,50 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: 'Amazon Nova 2 Lite with Converse API'
+
+prompts:
+  - 'Solve this step by step: {{problem}}'
+
+providers:
+  # Nova 2 Lite with reasoning enabled (high effort) via Converse API
+  # Note: Must use cross-region model ID (us.) for on-demand access
+  # Temperature must not be set when reasoning is enabled
+  - id: bedrock:converse:us.amazon.nova-2-lite-v1:0
+    label: Nova 2 Lite Converse (High Reasoning)
+    config:
+      region: 'us-east-1'
+      maxTokens: 4096
+      reasoningConfig:
+        type: enabled
+        maxReasoningEffort: high
+
+  # Nova 2 Lite with reasoning enabled (medium effort) via Converse API
+  - id: bedrock:converse:us.amazon.nova-2-lite-v1:0
+    label: Nova 2 Lite Converse (Medium Reasoning)
+    config:
+      region: 'us-east-1'
+      maxTokens: 4096
+      reasoningConfig:
+        type: enabled
+        maxReasoningEffort: medium
+
+  # Nova 2 Lite with reasoning disabled (fast mode) via Converse API
+  - id: bedrock:converse:us.amazon.nova-2-lite-v1:0
+    label: Nova 2 Lite Converse (Fast)
+    config:
+      region: 'us-east-1'
+      maxTokens: 2048
+      temperature: 0.7
+      reasoningConfig:
+        type: disabled
+
+tests:
+  - vars:
+      problem: 'A train leaves Station A at 9:00 AM traveling at 60 mph. Another train leaves Station B, 300 miles away, at 10:00 AM traveling toward Station A at 90 mph. At what time will the trains meet?'
+  - vars:
+      problem: 'If a bat and ball cost $1.10 together, and the bat costs $1 more than the ball, how much does the ball cost?'
+  - vars:
+      problem: 'In a room of 23 people, what is the probability that at least two people share a birthday?'
+  - vars:
+      problem: 'A farmer has 17 sheep. All but 9 die. How many sheep does the farmer have left?'
+  - vars:
+      problem: 'If it takes 5 machines 5 minutes to make 5 widgets, how long does it take 100 machines to make 100 widgets?'

examples/amazon-bedrock/models/promptfooconfig.nova-2-reasoning.yaml

@@ -0,0 +1,53 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: 'Amazon Nova 2 Lite with Extended Thinking'
+
+prompts:
+  - 'Solve this step by step: {{problem}}'
+
+providers:
+  # Nova 2 Lite with reasoning enabled (high effort)
+  # Note: Must use cross-region model ID (us.) for on-demand access
+  # Temperature must not be set when reasoning is enabled
+  - id: bedrock:us.amazon.nova-2-lite-v1:0
+    label: Nova 2 Lite (High Reasoning)
+    config:
+      region: 'us-east-1'
+      interfaceConfig:
+        max_new_tokens: 4096
+      reasoningConfig:
+        type: enabled
+        maxReasoningEffort: high
+
+  # Nova 2 Lite with reasoning enabled (medium effort)
+  - id: bedrock:us.amazon.nova-2-lite-v1:0
+    label: Nova 2 Lite (Medium Reasoning)
+    config:
+      region: 'us-east-1'
+      interfaceConfig:
+        max_new_tokens: 4096
+      reasoningConfig:
+        type: enabled
+        maxReasoningEffort: medium
+
+  # Nova 2 Lite with reasoning disabled (fast mode)
+  - id: bedrock:us.amazon.nova-2-lite-v1:0
+    label: Nova 2 Lite (Fast)
+    config:
+      region: 'us-east-1'
+      interfaceConfig:
+        max_new_tokens: 2048
+        temperature: 0.7
+      reasoningConfig:
+        type: disabled
+
+tests:
+  - vars:
+      problem: 'A train leaves Station A at 9:00 AM traveling at 60 mph. Another train leaves Station B, 300 miles away, at 10:00 AM traveling toward Station A at 90 mph. At what time will the trains meet?'
+  - vars:
+      problem: 'If a bat and ball cost $1.10 together, and the bat costs $1 more than the ball, how much does the ball cost?'
+  - vars:
+      problem: 'In a room of 23 people, what is the probability that at least two people share a birthday?'
+  - vars:
+      problem: 'A farmer has 17 sheep. All but 9 die. How many sheep does the farmer have left?'
+  - vars:
+      problem: 'If it takes 5 machines 5 minutes to make 5 widgets, how long does it take 100 machines to make 100 widgets?'

examples/amazon-bedrock/models/promptfooconfig.nova-sonic.yaml

@@ -0,0 +1,60 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: Amazon Bedrock Nova Sonic with a conversation
+
+providers:
+  - id: bedrock:amazon.nova-sonic-v1:0
+    config:
+      inferenceConfiguration:
+        maxTokens: 4096
+        temperature: 0.7
+        topP: 0.95
+      textOutputConfiguration:
+        mediaType: text/plain
+      audioInputConfiguration:
+        mediaType: audio/lpcm
+        sampleRateHertz: 16000
+        sampleSizeBits: 16
+        channelCount: 1
+        encoding: base64
+        audioType: SPEECH
+      audioOutputConfiguration:
+        mediaType: audio/lpcm
+        sampleRateHertz: 24000
+        sampleSizeBits: 16
+        channelCount: 1
+        voiceId: matthew
+        encoding: base64
+        audioType: SPEECH
+      region: us-east-1
+
+prompts:
+  - file://nova_sonic_prompt.js
+
+defaultTest:
+  vars:
+    system_message: You are a helpful assistant. Answer the question based on the audio input.
+  metadata:
+    type: text
+    conversationId: date_thread
+
+# The conversationId is a special metadata field that promptfoo uses to maintain conversation history between tests.
+# When multiple tests share the same conversationId (like "date_thread" here), each test's response is added to the
+# conversation history, allowing the model to reference previous exchanges. This enables testing of multi-turn
+# conversations where context from earlier interactions affects later responses.
+tests:
+  - vars:
+      audio_file: file://assets/hello.wav
+    metadata:
+      type: audio
+      conversationId: date_thread
+    assert:
+      - type: llm-rubric
+        value: the date and time is today
+  - vars:
+      audio_file: file://assets/weather.wav
+    metadata:
+      type: audio
+      conversationId: date_thread
+    assert:
+      - type: llm-rubric
+        value: contains a weather report

examples/amazon-bedrock/models/promptfooconfig.nova.multimodal.yaml

@@ -0,0 +1,17 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: 'Bedrock Nova Eval with Images'
+
+prompts:
+  - file://nova_multimodal_prompt.json
+
+providers:
+  - id: bedrock:amazon.nova-pro-v1:0
+    config:
+      region: 'us-east-1'
+      inferenceConfig:
+        temperature: 0.7
+        max_new_tokens: 256
+
+tests:
+  - vars:
+      image: file://cat.jpg

examples/amazon-bedrock/models/promptfooconfig.nova.tool.yaml

@@ -0,0 +1,110 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: 'Bedrock Nova Tool Use Eval'
+
+prompts:
+  - 'What is the color of {{topic}}? ONLY answer using the "color_json" tool.'
+
+providers:
+  - id: bedrock:amazon.nova-micro-v1:0
+    config:
+      region: 'us-east-1'
+      inferenceConfig:
+        temperature: 0.7
+        max_new_tokens: 256
+      toolConfig:
+        toolChoice:
+          auto: {}
+        tools:
+          - toolSpec:
+              name: color_json
+              description: 'Outputs the color of a thing'
+              inputSchema:
+                json:
+                  type: object
+                  properties:
+                    name:
+                      type: string
+                      description: 'The name of the thing'
+                    color:
+                      type: string
+                      description: 'The color of the thing'
+                  required:
+                    - name
+                    - color
+  - id: bedrock:amazon.nova-lite-v1:0
+    config:
+      region: 'us-east-1'
+      inferenceConfig:
+        temperature: 0.7
+        max_new_tokens: 256
+      toolConfig:
+        toolChoice:
+          auto: {}
+        tools:
+          - toolSpec:
+              name: color_json
+              description: 'Outputs the color of a thing'
+              inputSchema:
+                json:
+                  type: object
+                  properties:
+                    name:
+                      type: string
+                      description: 'The name of the thing'
+                    color:
+                      type: string
+                      description: 'The color of the thing'
+                  required:
+                    - name
+                    - color
+  - id: bedrock:amazon.nova-pro-v1:0
+    config:
+      region: 'us-east-1'
+      inferenceConfig:
+        temperature: 0.7
+        max_new_tokens: 256
+      toolConfig:
+        toolChoice:
+          auto: {}
+        tools:
+          - toolSpec:
+              name: color_json
+              description: 'Outputs the color of a thing'
+              inputSchema:
+                json:
+                  type: object
+                  properties:
+                    name:
+                      type: string
+                      description: 'The name of the thing'
+                    color:
+                      type: string
+                      description: 'The color of the thing'
+                  required:
+                    - name
+                    - color
+
+defaultTest:
+  options:
+    transform: 'JSON.parse(output).input.color'
+tests:
+  - vars:
+      topic: sky
+    assert:
+      - type: equals
+        value: blue
+  - vars:
+      topic: ocean
+    assert:
+      - type: equals
+        value: blue
+  - vars:
+      topic: banana
+    assert:
+      - type: equals
+        value: yellow
+  - vars:
+      topic: grass
+    assert:
+      - type: equals
+        value: green

examples/amazon-bedrock/models/promptfooconfig.nova.yaml

@@ -0,0 +1,51 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: 'Bedrock Nova Models Comparison'
+
+prompts:
+  - 'Write a tweet about {{topic}}'
+
+providers:
+  - id: bedrock:amazon.nova-lite-v1:0
+    config:
+      region: 'us-east-1'
+      inferenceConfig:
+        temperature: 0.7
+        max_new_tokens: 256
+  - id: bedrock:us.amazon.nova-premier-v1:0
+    config:
+      region: 'us-west-2'
+      inferenceConfig:
+        temperature: 0.7
+        max_new_tokens: 256
+
+tests:
+  - vars:
+      topic: Our eco-friendly packaging
+  - vars:
+      topic: A sneak peek at our secret menu item
+  - vars:
+      topic: Behind-the-scenes at our latest photoshoot
+  - vars:
+      topic: the impact of autonomous drones on wildlife conservation
+  - vars:
+      topic: the emerging trend of virtual reality courtrooms
+  - vars:
+      topic: the ethical implications of AI-generated art
+  - vars:
+      topic: the unexpected health benefits of daily meditation
+  - vars:
+      topic: how AI is changing the way we play board games
+  - vars:
+      topic: unconventional productivity hacks involving household items
+  - vars:
+      topic: An underground art exhibition in an abandoned subway station
+  - vars:
+      topic: A webinar on the impact of AI on traditional marketing strategies
+  - vars:
+      topic: The launch of a new eco-friendly sneaker made from ocean plastic
+  - vars:
+      topic: the correlation between social media usage and self-esteem in teenagers
+  - vars:
+      topic: the impact of urban noise pollution on migratory bird patterns
+  - vars:
+      topic: the role of gut microbiota in moderating anxiety and depression

examples/amazon-bedrock/models/promptfooconfig.openai-compatible.yaml

@@ -0,0 +1,74 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: 'Bedrock OpenAI-compatible model families (GLM, MiniMax, Kimi, Nemotron, Gemma, Palmyra)'
+
+# These newer Bedrock families all speak the OpenAI Chat Completions schema over InvokeModel,
+# so they share the same provider config. Confirm each model is enabled in your region with
+# `aws bedrock list-foundation-models --region us-east-1`.
+
+prompts:
+  - 'Answer in one short sentence: {{question}}'
+
+providers:
+  # Z.AI GLM
+  - id: bedrock:zai.glm-5
+    label: GLM 5
+    config:
+      region: us-east-1
+      max_tokens: 1024
+
+  # MiniMax (reasoning model — give it a larger token budget so the answer is not truncated)
+  - id: bedrock:minimax.minimax-m2
+    label: MiniMax M2
+    config:
+      region: us-east-1
+      max_tokens: 2048
+      showThinking: false # strip the <reasoning> block, keep only the final answer
+
+  # Moonshot Kimi
+  - id: bedrock:moonshotai.kimi-k2.5
+    label: Kimi K2.5
+    config:
+      region: us-east-1
+      max_tokens: 1024
+
+  # NVIDIA Nemotron — reasons in-line without tags, so it needs a larger token budget.
+  # To get a direct answer, prepend a `/no_think` system message (NVIDIA's reasoning toggle);
+  # `showThinking` has no tagged block to strip here.
+  - id: bedrock:nvidia.nemotron-nano-9b-v2
+    label: Nemotron Nano 9B
+    config:
+      region: us-east-1
+      max_tokens: 2048
+
+  # Google Gemma 3
+  - id: bedrock:google.gemma-3-12b-it
+    label: Gemma 3 12B
+    config:
+      region: us-east-1
+      max_tokens: 1024
+
+  # Writer Palmyra — on-demand throughput is served through the `us.` inference profile
+  - id: bedrock:us.writer.palmyra-x5-v1:0
+    label: Palmyra X5
+    config:
+      region: us-east-1
+      max_tokens: 1024
+
+tests:
+  - vars:
+      question: What is the capital of France?
+    assert:
+      - type: icontains
+        value: Paris
+
+  - vars:
+      question: What is 2 + 2?
+    assert:
+      - type: contains
+        value: '4'
+
+  - vars:
+      question: Name the largest planet in our solar system.
+    assert:
+      - type: icontains
+        value: Jupiter

examples/amazon-bedrock/models/promptfooconfig.openai-frontier.yaml

@@ -0,0 +1,53 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: 'OpenAI frontier models (GPT-5.5 / GPT-5.4) via AWS Bedrock'
+
+# Frontier models are served through Bedrock's OpenAI-compatible Responses API and
+# authenticate with an Amazon Bedrock API key. Export it before running (promptfoo reads it
+# automatically — no apiKey field needed):
+#   export AWS_BEARER_TOKEN_BEDROCK="..."
+
+prompts:
+  - |
+    Answer the following question concisely and accurately.
+
+    Question: {{question}}
+
+providers:
+  # GPT-5.5 is available in us-east-2 (Ohio). promptfoo routes bedrock:openai.gpt-5.x to
+  # the OpenAI-compatible Responses API on the regional Bedrock mantle endpoint.
+  - id: bedrock:openai.gpt-5.5
+    label: GPT-5.5 (high reasoning)
+    config:
+      region: us-east-2
+      reasoning_effort: high
+      max_output_tokens: 2048
+  # GPT-5.4 is available in us-east-2 (Ohio) and us-west-2 (Oregon).
+  - id: bedrock:openai.gpt-5.4
+    label: GPT-5.4 (low reasoning)
+    config:
+      region: us-east-2
+      # Valid: none | low | medium | high | xhigh (minimal is NOT supported)
+      reasoning_effort: low
+      max_output_tokens: 1024
+
+tests:
+  - vars:
+      question: What is the capital of France? Reply with only the city name.
+    assert:
+      - type: icontains
+        value: Paris
+  - vars:
+      question: 'If a train travels 120 miles in 2 hours, what is its average speed in mph? Reply with just the number and unit.'
+    assert:
+      - type: contains
+        value: '60'
+      - type: icontains-any
+        value: ['mph', 'miles per hour']
+  - vars:
+      question: 'Explain the concept of machine learning in two sentences.'
+    assert:
+      # Deterministic assertions keep this example runnable with only a Bedrock API key.
+      # (A model-graded assertion like `llm-rubric` would require a separate grader provider
+      # with its own credentials — see https://promptfoo.dev/docs/configuration/expected-outputs/model-graded/)
+      - type: icontains-any
+        value: ['data', 'model', 'train', 'learn', 'pattern']

examples/amazon-bedrock/models/promptfooconfig.openai.yaml

@@ -0,0 +1,52 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: 'OpenAI GPT-OSS Models via AWS Bedrock'
+
+prompts:
+  - |
+    You are a helpful AI assistant. Answer the following question concisely and accurately.
+
+    Question: {{question}}
+
+providers:
+  - id: bedrock:openai.gpt-oss-120b-1:0
+    label: OpenAI GPT-OSS 120B
+    config:
+      region: us-west-2
+      max_completion_tokens: 1024
+      temperature: 0.7
+      reasoning_effort: medium
+  - id: bedrock:openai.gpt-oss-20b-1:0
+    label: OpenAI GPT-OSS 20B
+    config:
+      region: us-west-2
+      max_completion_tokens: 1024
+      temperature: 0.7
+      reasoning_effort: low
+  - id: bedrock:openai.gpt-oss-120b-1:0
+    label: OpenAI GPT-OSS 120B High Reasoning
+    config:
+      region: us-west-2
+      max_completion_tokens: 2048
+      temperature: 0.3
+      reasoning_effort: high
+
+tests:
+  - vars:
+      question: What is the capital of France?
+    assert:
+      - type: contains
+        value: Paris
+  - vars:
+      question: Explain the concept of machine learning in simple terms.
+    assert:
+      - type: contains-all
+        value: ['learn', 'data', 'algorithm']
+  - vars:
+      question: 'Solve this step by step: If a train travels 120 miles in 2 hours, what is its average speed?'
+    assert:
+      - type: contains
+        value: '60'
+      - type: contains-any
+        value: ['mph', 'miles per hour']
+      - type: llm-rubric
+        value: 'Response should show step-by-step calculation and arrive at 60 mph'

examples/amazon-bedrock/models/promptfooconfig.qwen.yaml

@@ -0,0 +1,122 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: 'Bedrock Qwen Models Evaluation'
+
+prompts:
+  - file://prompts.txt
+
+providers:
+  # Qwen3 Coder 480B - Large MoE model optimized for coding and agentic tasks
+  - id: bedrock:qwen.qwen3-coder-480b-a35b-v1:0
+    label: Qwen3 Coder 480B
+    config:
+      region: us-west-2
+      max_tokens: 2048
+      temperature: 0.7
+      top_p: 0.9
+      showThinking: true # Set to false to hide thinking content
+
+  # Claude for comparison
+  - id: bedrock:us.anthropic.claude-sonnet-4-6
+    label: Claude Sonnet 4.6
+    config:
+      region: us-west-2
+      max_tokens: 2048
+      temperature: 0.7
+
+  # Qwen3 Coder 30B - Smaller MoE model for efficient coding tasks
+  - id: bedrock:qwen.qwen3-coder-30b-a3b-v1:0
+    label: Qwen3 Coder 30B
+    config:
+      region: us-west-2
+      max_tokens: 1024
+      temperature: 0.5
+      top_p: 0.9
+
+  # Qwen3 235B - General purpose MoE model for reasoning and coding
+  - id: bedrock:qwen.qwen3-235b-a22b-2507-v1:0
+    label: Qwen3 235B
+    config:
+      region: us-west-2
+      max_tokens: 1024
+      temperature: 0.7
+      top_p: 0.9
+
+  # Qwen3 32B Dense - Dense model for consistent performance
+  - id: bedrock:qwen.qwen3-32b-v1:0
+    label: Qwen3 32B Dense
+    config:
+      region: us-east-1
+      max_tokens: 1024
+      temperature: 0.7
+      top_p: 0.9
+
+  # Example with tool calling configuration
+  - id: bedrock:qwen.qwen3-coder-480b-a35b-v1:0
+    label: Qwen3 Coder 480B with Tools
+    config:
+      region: us-west-2
+      max_tokens: 2048
+      temperature: 0.3
+      tools:
+        - type: function
+          function:
+            name: calculate
+            description: Perform arithmetic calculations
+            parameters:
+              type: object
+              properties:
+                expression:
+                  type: string
+                  description: The mathematical expression to evaluate
+              required: ['expression']
+      tool_choice: auto
+
+tests:
+  - vars:
+      question: Write a Python function to calculate the factorial of a number
+    assert:
+      - type: contains
+        value: 'def'
+      - type: contains
+        value: 'factorial'
+
+  - vars:
+      question: Explain the concept of recursion with a simple example
+    assert:
+      - type: contains
+        value: 'recursion'
+      - type: llm-rubric
+        value: Response clearly explains recursion with a concrete example
+
+  - vars:
+      question: What are the benefits of using mixture-of-experts models?
+    assert:
+      - type: contains
+        value: 'efficiency'
+
+  - vars:
+      question: Debug this code and explain the issue - "for i in range(10) print(i)"
+    assert:
+      - type: contains
+        value: 'syntax'
+      - type: contains
+        value: ':'
+
+  - vars:
+      question: Create a REST API endpoint using FastAPI for user authentication
+    assert:
+      - type: contains
+        value: 'FastAPI'
+      - type: contains
+        value: 'authentication'
+
+  # Test tool calling specifically with the tools provider
+  - vars:
+      question: Calculate the result of 15 * 8 + 42
+    assert:
+      - type: contains
+        value: 'Called function calculate'
+      - type: contains
+        value: '15 * 8 + 42'
+    options:
+      provider: bedrock:qwen.qwen3-coder-480b-a35b-v1:0:Qwen3 Coder 480B with Tools

examples/amazon-bedrock/models/promptfooconfig.yaml

@@ -0,0 +1,128 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: 'Bedrock Eval'
+
+prompts:
+  - file://prompts.txt
+
+providers:
+  - id: bedrock:us.anthropic.claude-sonnet-4-6
+    label: Claude Sonnet 4.6
+    config:
+      region: us-west-2
+      max_tokens: 2048
+      temperature: 0
+  - id: bedrock:us.meta.llama4-scout-17b-instruct-v1:0
+    label: Llama 4 Scout 17B
+  - id: bedrock:amazon.nova-pro-v1:0
+    label: Nova Pro
+  - id: bedrock:mistral.pixtral-large-2502-v1:0
+    label: Pixtral Large 25.02
+    config:
+      region: us-west-2
+      max_tokens: 1024
+      temperature: 0
+
+defaultTest:
+  options:
+    provider:
+      # Override the embeddings provider for all assertion types that require it (such as similarity)
+      embedding:
+        id: bedrock:embeddings:amazon.titan-embed-text-v2:0
+        config:
+          region: us-east-1
+
+tests:
+  - vars:
+      question: What's the weather in New York?
+    assert:
+      # This uses the embeddings provider set above
+      - type: similar
+        value: sunny
+  - vars:
+      question: Who won the latest football match between the Giants and 49ers?
+  - vars:
+      question: "Which magazine was started first Arthur's Magazine or First for Women?"
+  - vars:
+      question: 'The Oberoi family is part of a hotel company that has a head office in what city?'
+  - vars:
+      question: 'Musician and satirist Allie Goertz wrote a song about the "The Simpsons" character Milhouse, who Matt Groening named after who?'
+  - vars:
+      question: "What nationality was James Henry Miller's wife?"
+  - vars:
+      question: 'Cadmium Chloride is slightly soluble in this chemical, it is also called what?'
+  - vars:
+      question: 'Which tennis player won more Grand Slam titles, Henri Leconte or Jonathan Stark?'
+  - vars:
+      question: "Which genus of moth in the world's seventh-largest country contains only one species?"
+  - vars:
+      question: 'Who was once considered the best kick boxer in the world, however he has been involved in a number of controversies relating to his "unsportsmanlike conducts" in the sport and crimes of violence outside of the ring.'
+  - vars:
+      question: 'The Dutch-Belgian television series that "House of Anubis" was based on first aired in what year?'
+  - vars:
+      question: 'What is the length of the track where the 2013 Liqui Moly Bathurst 12 Hour was staged?'
+  - vars:
+      question: 'Fast Cars, Danger, Fire and Knives includes guest appearances from which hip hop record executive?'
+  - vars:
+      question: 'Gunmen from Laredo starred which narrator of "Frontier"?'
+  - vars:
+      question: 'Where did the form of music played by Die Rhöner Säuwäntzt originate?'
+  - vars:
+      question: 'In which American football game was Malcolm Smith named Most Valuable player?'
+  - vars:
+      question: 'What U.S Highway gives access to Zilpo Road, and is also known as Midland Trail?'
+  - vars:
+      question: 'The 1988 American comedy film, The Great Outdoors, starred a four-time Academy Award nominee, who received a star on the Hollywood Walk of Fame in what year?'
+  - vars:
+      question: 'What are the names of the current members of  American heavy metal band who wrote the music for  Hurt Locker The Musical?'
+  - vars:
+      question: 'Human Error" is the season finale of the third season of a tv show that aired on what network?'
+  - vars:
+      question: 'Dua Lipa, an English singer, songwriter and model, the album spawned the number-one single "New Rules" is a song by English singer Dua Lipa from her eponymous debut studio album, released in what year?'
+  - vars:
+      question: 'American politician Joe Heck ran unsuccessfully against Democrat Catherine Cortez Masto, a woman who previously served as the 32nd Attorney General of where?'
+  - vars:
+      question: 'Which state does the drug stores, of which the CEO is Warren Bryant, are located?'
+  - vars:
+      question: 'Which  American politician did Donahue replaced '
+  - vars:
+      question: 'Which band was founded first, Hole, the rock band that Courtney Love was a frontwoman of, or The Wolfhounds?'
+  - vars:
+      question: 'How old is the female main protagonist of Catching Fire?'
+  - vars:
+      question: 'Chang Ucchin was born in korea during a time that ended with the conclusion of what?'
+  - vars:
+      question: 'Who is the director of the 2003 film which has scenes in it filmed at the Quality Cafe in Los Angeles?'
+  - vars:
+      question: "Which actress played the part of fictitious character Kimberly Ann Hart, in the franchise built around a live action superhero television series taking much of its footage from the Japanese tokusatsu 'Super Sentai'?"
+  - vars:
+      question: 'Who was born first, Pablo Trapero or Aleksander Ford?'
+  - vars:
+      question: "Are Jane and First for Women both women's magazines?"
+  - vars:
+      question: 'What profession does Nicholas Ray and Elia Kazan have in common?'
+  - vars:
+      question: 'Where is the company that purchased Aixam based in?'
+  - vars:
+      question: 'Which documentary is about Finnish rock groups, Adam Clayton Powell or The Saimaa Gesture?'
+  - vars:
+      question: 'Who was inducted into the Rock and Roll Hall of Fame, David Lee Roth or Cia Berg?'
+  - vars:
+      question: "Zimbabwe's Guwe Secondary School has a sister school in what New York county?"
+  - vars:
+      question: 'Who is the director of the 2003 film which has scenes in it filmed at the Quality Cafe in Los Angeles?'
+  - vars:
+      question: "Which actress played the part of fictitious character Kimberly Ann Hart, in the franchise built around a live action superhero television series taking much of its footage from the Japanese tokusatsu 'Super Sentai'?"
+  - vars:
+      question: 'Who was born first, Pablo Trapero or Aleksander Ford?'
+  - vars:
+      question: "Are Jane and First for Women both women's magazines?"
+  - vars:
+      question: 'What profession does Nicholas Ray and Elia Kazan have in common?'
+  - vars:
+      question: 'Where is the company that purchased Aixam based in?'
+  - vars:
+      question: 'Which documentary is about Finnish rock groups, Adam Clayton Powell or The Saimaa Gesture?'
+  - vars:
+      question: 'Who was inducted into the Rock and Roll Hall of Fame, David LeRoth or Cia Berg?'
+  - vars:
+      question: "Zimbabwe's Guwe Secondary School has a sister school in what New York county?"

examples/amazon-bedrock/models/prompts.txt

@@ -0,0 +1,9 @@
+You are a helpful assistant. Reply with a concise answer to this inquiry: "{{question}}"
+---
+You are a helpful assistant. Reply with a concise answer to this inquiry: "{{question}}"
+
+- Think carefully & step-by-step.
+- Only use information available on Wikipedia.
+- You must answer the question directly, without speculation.
+- You cannot access realtime information. Consider whether the answer may have changed in the 2 years since your knowledge cutoff.
+- If you are not confident in your answer, begin your response with "Unsure".

examples/amazon-bedrock/video/README.md

@@ -0,0 +1,135 @@
+# amazon-bedrock/video (AWS Bedrock Video Generation)
+
+You can run this example with:
+
+```bash
+npx promptfoo@latest init --example amazon-bedrock/video
+cd amazon-bedrock/video
+```
+
+Video generation examples using AWS Bedrock's async invoke API.
+
+## Available Models
+
+| Model            | Config                           | Region         | Duration  |
+| ---------------- | -------------------------------- | -------------- | --------- |
+| Amazon Nova Reel | `promptfooconfig.nova-reel.yaml` | us-east-1      | 6s - 2min |
+| Luma Ray 2       | `promptfooconfig.luma-ray.yaml`  | us-west-2 only | 5s or 9s  |
+
+## Prerequisites
+
+Video generation requires additional AWS setup beyond standard Bedrock access.
+
+### 1. Enable Model Access
+
+Visit the [AWS Bedrock Model Access page](https://console.aws.amazon.com/bedrock/home#/modelaccess) and enable:
+
+- **Nova Reel**: `amazon.nova-reel-v1:1` (us-east-1)
+- **Luma Ray 2**: `luma.ray-v2:0` (us-west-2)
+
+### 2. Create S3 Bucket
+
+Video outputs are written to S3. Create a bucket in the same region as your model:
+
+```bash
+# For Nova Reel (us-east-1)
+aws s3 mb s3://your-bucket-nova-reel --region us-east-1
+
+# For Luma Ray 2 (us-west-2)
+aws s3 mb s3://your-bucket-luma-ray --region us-west-2
+```
+
+### 3. Configure IAM Permissions
+
+```json
+{
+  "Version": "2012-10-17",
+  "Statement": [
+    {
+      "Effect": "Allow",
+      "Action": ["bedrock:InvokeModel", "bedrock:StartAsyncInvoke", "bedrock:GetAsyncInvoke"],
+      "Resource": [
+        "arn:aws:bedrock:*:*:model/amazon.nova-reel-v1:1",
+        "arn:aws:bedrock:*:*:model/luma.ray-v2:0"
+      ]
+    },
+    {
+      "Effect": "Allow",
+      "Action": ["s3:PutObject", "s3:GetObject"],
+      "Resource": "arn:aws:s3:::your-bucket/*"
+    }
+  ]
+}
+```
+
+### 4. Install Dependencies
+
+```bash
+npm install @aws-sdk/client-bedrock-runtime @aws-sdk/client-s3
+```
+
+## Usage
+
+Update `s3OutputUri` in the config file, then run:
+
+```bash
+# Nova Reel
+npx promptfoo@latest eval -c promptfooconfig.nova-reel.yaml
+
+# Luma Ray 2
+npx promptfoo@latest eval -c promptfooconfig.luma-ray.yaml
+```
+
+## Model Comparison
+
+### Amazon Nova Reel
+
+- **Best for**: Longer videos, multi-shot narratives
+- **Resolution**: 1280x720 @ 24 FPS
+- **Duration**: 6 seconds (single shot) or 12-120 seconds (multi-shot)
+- **Features**: TEXT_VIDEO, MULTI_SHOT_AUTOMATED, MULTI_SHOT_MANUAL modes
+- **Typical generation time**: ~90 seconds for 6s video
+
+### Luma Ray 2
+
+- **Best for**: High-quality short clips, image-to-video
+- **Resolution**: 540p or 720p
+- **Aspect ratios**: 1:1, 16:9, 9:16, 4:3, 3:4, 21:9, 9:21
+- **Duration**: 5 or 9 seconds
+- **Features**: Start/end frame keyframes, loop mode
+- **Typical generation time**: ~2-3 minutes
+
+## Configuration Options
+
+### Nova Reel
+
+```yaml
+config:
+  region: us-east-1
+  s3OutputUri: s3://your-bucket/outputs/
+  durationSeconds: 6 # 6 for single, 12-120 for multi-shot
+  taskType: TEXT_VIDEO # or MULTI_SHOT_AUTOMATED, MULTI_SHOT_MANUAL
+  seed: 12345 # Optional, for reproducibility
+  image: file://./start-frame.jpg # Optional, for image-to-video
+```
+
+### Luma Ray 2
+
+```yaml
+config:
+  region: us-west-2
+  s3OutputUri: s3://your-bucket/outputs/
+  duration: '5s' # or '9s'
+  resolution: '720p' # or '540p'
+  aspectRatio: '16:9'
+  loop: false
+  startImage: file://./start.jpg # Optional
+  endImage: file://./end.jpg # Optional
+```
+
+## Resources
+
+- [AWS Bedrock Video Generation](https://docs.aws.amazon.com/bedrock/latest/userguide/video-generation.html)
+- [Nova Reel Documentation](https://docs.aws.amazon.com/nova/latest/userguide/video-generation.html)
+- [Luma Ray Documentation](https://docs.aws.amazon.com/bedrock/latest/userguide/model-parameters-luma.html)
+- [promptfoo AWS Bedrock Provider](https://promptfoo.dev/docs/providers/aws-bedrock)

examples/amazon-bedrock/video/promptfooconfig.luma-ray.yaml

@@ -0,0 +1,36 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: 'Luma Ray 2 Video Generation on AWS Bedrock'
+
+# Luma Ray 2 produces high-quality videos from text prompts.
+# Videos are generated asynchronously and stored in S3.
+#
+# Prerequisites:
+#   1. Enable Luma Ray 2 model access in AWS Bedrock console
+#   2. Create an S3 bucket for video output
+#   3. Configure IAM permissions for bedrock:StartAsyncInvoke,
+#      bedrock:GetAsyncInvoke, s3:PutObject, s3:GetObject
+#
+# Update s3OutputUri below to point to your S3 bucket.
+
+providers:
+  - id: bedrock:video:luma.ray-v2:0
+    config:
+      region: us-west-2 # Luma Ray 2 is currently only available in us-west-2
+      s3OutputUri: s3://your-bucket-name/luma-ray-outputs/
+      duration: '5s'
+      resolution: '720p'
+      aspectRatio: '16:9'
+
+prompts:
+  # Promptfoo red panda mascot themed prompts
+  - 'A cute red panda wearing glasses typing on a glowing laptop in a cozy tech office, soft purple and teal lighting, modern tech aesthetic, the red panda looks focused and intelligent'
+  - 'An adorable red panda in a lab coat examining holographic data visualizations, futuristic tech environment with orange and purple accent lights, cinematic lighting'
+  - 'A friendly red panda mascot waving at the camera in front of a wall of code and security shields, modern office setting, warm lighting, professional but approachable style'
+
+tests:
+  - vars: {}
+    assert:
+      - type: contains
+        value: 'Video:' # Output format is [Video: prompt](url)
+      - type: not-contains
+        value: 'error' # No errors in output

examples/amazon-bedrock/video/promptfooconfig.nova-reel.yaml

@@ -0,0 +1,26 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: Amazon Nova Reel Video Generation
+
+providers:
+  - id: bedrock:video:amazon.nova-reel-v1:1
+    config:
+      region: us-east-1
+      # Required: S3 bucket for video output
+      # Update this to your own S3 bucket
+      s3OutputUri: s3://your-bucket/nova-reel-outputs
+      durationSeconds: 6
+      # Optional: Set seed for reproducible results
+      # seed: 12345
+
+prompts:
+  # Promptfoo red panda mascot themed prompts
+  - 'A cute red panda sitting at a desk reviewing documents, wearing small round glasses, soft warm lighting, camera slowly zooms in on the focused expression'
+  - 'An adorable red panda mascot walking through a futuristic data center with glowing servers, purple and teal ambient lighting, camera follows alongside'
+
+tests:
+  - vars: {}
+    assert:
+      - type: contains
+        value: 'Video:' # Output format is [Video: prompt](url)
+      - type: not-contains
+        value: 'error' # No errors in output