chore: import upstream snapshot with attribution

examples/AGENTS.md

@@ -0,0 +1,88 @@
+# Examples
+
+Self-contained example configurations for `npx promptfoo@latest init --example <name>`.
+
+## CRITICAL: Test with Local Build
+
+```bash
+# Correct - tests your changes
+npm run local -- eval -c examples/my-example/promptfooconfig.yaml
+
+# Wrong - tests published version
+npx promptfoo@latest eval
+```
+
+## Example Structure
+
+Each example needs:
+
+1. Directory with clear name
+2. `README.md` starting with `# folder-name (Human Readable Name)`
+3. `promptfooconfig.yaml` with schema reference
+4. Instructions using `npx promptfoo@latest init --example <name>`
+
+## Configuration Format
+
+```yaml
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: Short description (3-10 words)
+prompts:
+  - ...
+providers:
+  - ...
+tests:
+  - ...
+```
+
+**Field order:** description, env, prompts, providers, defaultTest, scenarios, tests
+
+## Environment Variables in Configs
+
+Use Nunjucks template syntax to reference environment variables:
+
+```yaml
+# ✅ CORRECT - Nunjucks template syntax
+accountId: '{{env.CLOUDFLARE_ACCOUNT_ID}}'
+apiKey: '{{env.OPENAI_API_KEY}}'
+
+# ❌ WRONG - Shell syntax doesn't work in YAML configs
+accountId: ${CLOUDFLARE_ACCOUNT_ID}
+apiKey: $OPENAI_API_KEY
+```
+
+Note: Quotes around `'{{env.VAR}}'` are required in YAML to prevent parsing issues.
+
+## Model Selection
+
+Use current model identifiers (see `site/docs/providers/` for full list):
+
+- OpenAI: `openai:chat:gpt-5.6`, `openai:responses:gpt-5.6-sol`, `openai:responses:gpt-5.6-terra`, `openai:responses:gpt-5.6-luna`, `openai:chat:gpt-5.4-mini`
+- Anthropic: `anthropic:messages:claude-sonnet-4-6`, `anthropic:messages:claude-haiku-4-5-20251001`
+- Google: `google:gemini-3.1-pro-preview`, `google:gemini-2.5-flash`
+
+## Guidelines
+
+- Keep examples simple while demonstrating the concept
+- Use `file://` prefix for external files
+- Make test cases engaging, not boring
+- Document required environment variables in README
+- Test examples before submitting
+- **If adding example for a new provider**, verify docs exist at `site/docs/providers/<provider>.md`
+
+## Chat Format Prompts
+
+Some providers require specific message structures (e.g., exactly one system + one user message). Use a JSON file instead of inline YAML:
+
+```json
+// prompts/chat.json
+[
+  { "role": "system", "content": "You are a helpful assistant." },
+  { "role": "user", "content": "Help with: {{topic}}" }
+]
+```
+
+```yaml
+# promptfooconfig.yaml
+prompts:
+  - file://prompts/chat.json
+```

examples/CLAUDE.md

@@ -0,0 +1 @@
+@AGENTS.md

examples/amazon-bedrock/README.md

@@ -0,0 +1,9 @@
+# amazon-bedrock (Amazon Bedrock)
+
+Examples for using promptfoo with [Amazon Bedrock](https://aws.amazon.com/bedrock/).
+
+## Examples
+
+- [models](./models/) - Model evaluations: Claude, Llama, Mistral, Nova, DeepSeek, Qwen, Grok, and more
+- [agents](./agents/) - Bedrock Agents with tool use and knowledge bases
+- [video](./video/) - Video generation with Amazon Nova Reel

examples/amazon-bedrock/agents/README.md

@@ -0,0 +1,215 @@
+# amazon-bedrock/agents (AWS Bedrock Agents Example)
+
+This example demonstrates how to use AWS Bedrock Agents with promptfoo to test and evaluate deployed AI agents, including both single-agent and multi-agent scenarios.
+
+You can run this example with:
+
+```bash
+npx promptfoo@latest init --example amazon-bedrock/agents
+cd amazon-bedrock/agents
+```
+
+## Prerequisites
+
+1. An AWS account with Bedrock Agents access
+2. One or more deployed Bedrock agents (get agent IDs from the AWS Console)
+3. AWS credentials configured (via environment variables, AWS CLI, or IAM role)
+4. Install the required AWS SDK:
+
+   ```bash
+   npm install @aws-sdk/client-bedrock-agent-runtime
+   ```
+
+## Setup
+
+1. **Get your Agent ID(s)**:
+   - Go to the AWS Bedrock Console
+   - Navigate to Agents
+   - Copy your agent ID(s) (format: `ABCDEFGHIJ`)
+
+2. **Configure AWS Credentials** (choose one method):
+
+   Via environment variables:
+
+   ```bash
+   export AWS_ACCESS_KEY_ID=your_access_key
+   export AWS_SECRET_ACCESS_KEY=your_secret_key
+   export AWS_REGION=us-east-1
+   ```
+
+   Via AWS CLI profile:
+
+   ```bash
+   aws configure --profile my-bedrock-profile
+   ```
+
+   Via IAM role (if running on EC2/Lambda)
+
+3. **Choose your configuration**:
+   - `promptfooconfig.yaml`: Basic single-agent configuration
+   - `promptfooconfig.multi-agent.yaml`: Advanced multi-agent system configuration
+
+## Running the Examples
+
+### Single Agent Example
+
+```bash
+# Run basic agent evaluation
+npx promptfoo eval -c promptfooconfig.yaml
+
+# View results in the web UI
+npx promptfoo view
+```
+
+### Multi-Agent Example
+
+```bash
+# Run multi-agent system evaluation
+npx promptfoo eval -c promptfooconfig.multi-agent.yaml
+
+# View results in the web UI
+npx promptfoo view
+```
+
+## Configuration Options
+
+### Basic Usage
+
+```yaml
+providers:
+  - bedrock-agent:YOUR_AGENT_ID
+```
+
+### Advanced Single Agent Configuration
+
+```yaml
+providers:
+  - id: bedrock-agent:my-agent
+    config:
+      agentId: YOUR_AGENT_ID
+      agentAliasId: PROD_ALIAS # Optional: specific version/alias
+      region: us-east-1 # AWS region
+      sessionId: session-123 # Maintain conversation state
+      enableTrace: true # Get detailed execution traces
+      memoryId: SHORT_TERM_MEMORY # or LONG_TERM_MEMORY
+```
+
+### Multi-Agent System Configuration
+
+```yaml
+providers:
+  # Technical Support Agent
+  - id: tech-agent
+    provider: bedrock-agent:TECH_AGENT_ID
+    config:
+      agentId: TECH_AGENT_ID
+      agentAliasId: TECH_ALIAS_ID
+      region: us-east-1
+      enableTrace: true
+      memoryId: LONG_TERM_MEMORY
+
+  # Billing Agent
+  - id: billing-agent
+    provider: bedrock-agent:BILLING_AGENT_ID
+    config:
+      agentId: BILLING_AGENT_ID
+      agentAliasId: BILLING_ALIAS_ID
+      region: us-east-1
+      enableTrace: true
+```
+
+## Features
+
+### Session Management
+
+The provider supports maintaining conversation state across multiple interactions:
+
+```yaml
+config:
+  sessionId: my-session-123 # Use the same session ID for related queries
+```
+
+### Memory Integration
+
+Enable agent memory for context-aware responses:
+
+```yaml
+config:
+  memoryId: LONG_TERM_MEMORY # or SHORT_TERM_MEMORY
+```
+
+### Trace Information
+
+Get detailed execution traces including tool calls and reasoning:
+
+```yaml
+config:
+  enableTrace: true # Response will include trace metadata
+```
+
+## Testing Scenarios
+
+### Single Agent Tests
+
+The basic config includes tests for:
+
+- Basic agent responses
+- Tool/function calling (e.g., calculator)
+- Memory retention
+- Multi-turn conversations
+
+### Multi-Agent System Tests
+
+The multi-agent config includes tests for:
+
+- Specialized agent capabilities (technical, billing, product)
+- Cross-functional issue handling
+- Agent collaboration and coordination
+- Escalation management
+- Performance and latency validation
+
+## Multi-Agent System Architecture
+
+The multi-agent example demonstrates a customer support system with specialized agents:
+
+```text
+Customer Query
+     ↓
+[Supervisor Agent] ← Monitors & Routes
+     ↓
+┌─────────────────┬─────────────────┬──────────────────┐
+│  Tech Agent     │  Billing Agent  │  Product Agent   │
+│  (Technical)    │  (Payments)     │  (Recommendations)│
+└─────────────────┴─────────────────┴──────────────────┘
+```
+
+## Troubleshooting
+
+1. **Authentication Error**: Ensure AWS credentials are properly configured
+2. **Agent Not Found**: Verify the agent ID and region
+3. **Permissions Error**: Check IAM permissions for `bedrock:InvokeAgent`
+4. **Timeout**: Large agent responses may take time; adjust timeout if needed
+5. **Multi-Agent Issues**: Ensure all agent IDs and aliases are correct in the config
+
+## IAM Permissions
+
+Your AWS credentials need the following permissions:
+
+```json
+{
+  "Version": "2012-10-17",
+  "Statement": [
+    {
+      "Effect": "Allow",
+      "Action": ["bedrock:InvokeAgent"],
+      "Resource": "arn:aws:bedrock:*:*:agent/*"
+    }
+  ]
+}
+```
+
+## Learn More
+
+- [AWS Bedrock Agents Documentation](https://docs.aws.amazon.com/bedrock/latest/userguide/agents.html)
+- [Promptfoo Documentation](https://promptfoo.dev/docs/providers/bedrock/)
+- [Bedrock Agents Samples](https://github.com/awslabs/amazon-bedrock-agents-samples)

examples/amazon-bedrock/agents/promptfooconfig.multi-agent.yaml

@@ -0,0 +1,239 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+# Multi-Agent System Example with AWS Bedrock Agents
+# This example demonstrates testing multiple specialized agents working together
+
+description: 'Multi-Agent Customer Support System Evaluation'
+
+# Define multiple specialized agents
+providers:
+  # Technical Support Agent - handles technical queries
+  - id: tech-agent
+    provider: bedrock-agent:TECH_AGENT_ID
+    config:
+      agentId: TECH_AGENT_ID
+      agentAliasId: TECH_ALIAS_ID
+      region: us-east-1
+      enableTrace: true
+      memoryId: LONG_TERM_MEMORY # Remember customer issues
+
+  # Billing Agent - handles payment and subscription queries
+  - id: billing-agent
+    provider: bedrock-agent:BILLING_AGENT_ID
+    config:
+      agentId: BILLING_AGENT_ID
+      agentAliasId: BILLING_ALIAS_ID
+      region: us-east-1
+      enableTrace: true
+      memoryId: LONG_TERM_MEMORY
+
+  # Product Specialist Agent - handles product recommendations
+  - id: product-agent
+    provider: bedrock-agent:PRODUCT_AGENT_ID
+    config:
+      agentId: PRODUCT_AGENT_ID
+      agentAliasId: PRODUCT_ALIAS_ID
+      region: us-east-1
+      enableTrace: true
+      memoryId: SHORT_TERM_MEMORY
+
+  # Supervisor Agent - routes queries and escalates issues
+  - id: supervisor-agent
+    provider: bedrock-agent:SUPERVISOR_AGENT_ID
+    config:
+      agentId: SUPERVISOR_AGENT_ID
+      agentAliasId: SUPERVISOR_ALIAS_ID
+      region: us-east-1
+      enableTrace: true
+      sessionId: supervisor-session-001 # Shared session for coordination
+
+# Test scenarios for multi-agent collaboration
+prompts:
+  - id: technical-issue
+    content: |
+      My application keeps crashing when I try to upload files larger than 10MB.
+      Error code: UPLOAD_SIZE_EXCEEDED. How can I fix this?
+
+  - id: billing-question
+    content: |
+      I was charged twice for my subscription last month.
+      Can you help me get a refund for the duplicate charge?
+
+  - id: product-recommendation
+    content: |
+      I need a solution for real-time data processing that can handle
+      100,000 events per second. What product do you recommend?
+
+  - id: complex-issue
+    content: |
+      I upgraded my plan but I'm still seeing the old limits.
+      Also, the API is returning 403 errors even though my payment went through.
+
+  - id: escalation-needed
+    content: |
+      This is the third time I'm contacting support about the same issue.
+      I need to speak with a manager about getting a full refund and account closure.
+
+tests:
+  # Test Technical Agent's capability
+  - description: 'Technical agent handles error troubleshooting'
+    vars:
+      query: '{{prompts.technical-issue}}'
+    providers:
+      - tech-agent
+    assert:
+      - type: javascript
+        value: |
+          const v = output.toLowerCase();
+          ['file size', 'upload limit', 'configuration'].some(s => v.includes(s))
+      - type: javascript
+        value: |
+          // Check if technical solution is provided
+          output.toLowerCase().includes('limit') ||
+          output.toLowerCase().includes('configuration') ||
+          output.toLowerCase().includes('setting')
+
+  # Test Billing Agent's capability
+  - description: 'Billing agent handles refund requests'
+    vars:
+      query: '{{prompts.billing-question}}'
+    providers:
+      - billing-agent
+    assert:
+      - type: javascript
+        value: |
+          const v = output.toLowerCase();
+          ['refund', 'duplicate', 'charge'].some(s => v.includes(s))
+      - type: javascript
+        value: |
+          // Check for customer service tone
+          output.toLowerCase().includes('sorry') ||
+          output.toLowerCase().includes('apologize') ||
+          output.toLowerCase().includes('help')
+
+  # Test Product Agent's recommendations
+  - description: 'Product agent provides relevant recommendations'
+    vars:
+      query: '{{prompts.product-recommendation}}'
+    providers:
+      - product-agent
+    assert:
+      - type: javascript
+        value: |
+          const v = output.toLowerCase();
+          ['stream', 'kafka', 'kinesis', 'event', 'real-time'].some(s => v.includes(s))
+      - type: javascript
+        value: |
+          // Check if throughput requirements are acknowledged
+          output.includes('100,000') ||
+          output.toLowerCase().includes('high throughput') ||
+          output.toLowerCase().includes('scale')
+
+  # Test Multi-Agent Coordination (Complex Issue)
+  - description: 'Multiple agents handle complex cross-functional issue'
+    vars:
+      query: '{{prompts.complex-issue}}'
+    providers:
+      - tech-agent
+      - billing-agent
+    assert:
+      - type: javascript
+        value: |
+          // Tech agent should address API errors
+          const techResponse = outputs['tech-agent'];
+          const billingResponse = outputs['billing-agent'];
+
+          const techAddressed = techResponse && (
+            techResponse.toLowerCase().includes('403') ||
+            techResponse.toLowerCase().includes('permission') ||
+            techResponse.toLowerCase().includes('authorization')
+          );
+
+          const billingAddressed = billingResponse && (
+            billingResponse.toLowerCase().includes('payment') ||
+            billingResponse.toLowerCase().includes('plan') ||
+            billingResponse.toLowerCase().includes('upgrade')
+          );
+
+          return techAddressed || billingAddressed;
+
+  # Test Supervisor Escalation
+  - description: 'Supervisor agent handles escalations appropriately'
+    vars:
+      query: '{{prompts.escalation-needed}}'
+    providers:
+      - supervisor-agent
+    assert:
+      - type: javascript
+        value: |
+          const v = output.toLowerCase();
+          ['escalate', 'manager', 'supervisor', 'priority'].some(s => v.includes(s))
+      - type: javascript
+        value: |
+          // Check for appropriate escalation handling
+          output.toLowerCase().includes('understand') ||
+          output.toLowerCase().includes('frustration') ||
+          output.toLowerCase().includes('priority')
+
+  # Test Agent Memory Persistence
+  - description: 'Agents remember context from previous interactions'
+    vars:
+      customer_id: 'CUST-12345'
+    tests:
+      - vars:
+          query: "My customer ID is {{customer_id}} and I'm having login issues"
+        providers:
+          - tech-agent
+      - vars:
+          query: 'What is my customer ID?'
+        providers:
+          - tech-agent
+        assert:
+          - type: contains
+            value: '{{customer_id}}'
+
+  # Test Agent Collaboration with Trace Analysis
+  - description: 'Analyze agent tool usage and decision making'
+    vars:
+      query: 'I need help choosing between your Pro and Enterprise plans for my 50-person startup'
+    providers:
+      - product-agent
+    assert:
+      - type: javascript
+        value: |
+          // Check if agent used comparison tools
+          if (metadata && metadata.trace && metadata.trace.toolCalls) {
+            const toolsUsed = metadata.trace.toolCalls.map(t => t.name);
+            return toolsUsed.some(tool =>
+              tool.includes('compare') ||
+              tool.includes('pricing') ||
+              tool.includes('plan')
+            );
+          }
+          // Pass if no trace available but response is relevant
+          return output.toLowerCase().includes('enterprise') &&
+                 output.toLowerCase().includes('pro');
+
+# Performance and Quality Metrics
+defaultTest:
+  assert:
+    # Latency check for all responses
+    - type: latency
+      threshold: 5000 # 5 seconds max
+
+    # Ensure responses are not empty
+    - type: javascript
+      value: output && output.trim().length > 10
+
+    # Check for professional tone
+    - type: javascript
+      value: |
+        // Avoid inappropriate responses
+        !output.toLowerCase().includes('i don\'t know') &&
+        !output.toLowerCase().includes('not sure') &&
+        !output.toLowerCase().includes('error')
+
+# Shared test configuration
+options:
+  cache: false # Disable caching for accurate latency measurements
+  maxConcurrency: 2 # Limit concurrent agent calls to avoid rate limits
+

examples/amazon-bedrock/agents/promptfooconfig.yaml

@@ -0,0 +1,66 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+# AWS Bedrock Agents Provider Example Configuration
+# This example demonstrates how to use AWS Bedrock Agents with promptfoo
+
+description: 'AWS Bedrock Agents test suite'
+
+prompts:
+  - 'What can you help me with?'
+  - 'Perform a calculation: what is 15 multiplied by 27?'
+  - 'Search for information about AWS Lambda functions'
+
+providers:
+  # Basic configuration with agent ID in the path
+  - id: bedrock-agent:basic-agent
+    config:
+      agentId: ABCDEFGHIJ
+      agentAliasId: PROD_ALIAS_ID
+      region: us-east-1
+
+  # Configuration with additional options
+  - id: bedrock-agent:my-agent
+    config:
+      agentId: ABCDEFGHIJ
+      agentAliasId: PROD_ALIAS_ID # Optional: specific alias to use
+      region: us-east-1
+      sessionId: session-123 # Optional: maintain conversation state
+      enableTrace: true # Enable detailed trace information
+      memoryId: SHORT_TERM_MEMORY # or LONG_TERM_MEMORY
+
+      # AWS credentials (optional - will use default chain if not provided)
+      # accessKeyId: YOUR_ACCESS_KEY
+      # secretAccessKey: YOUR_SECRET_KEY
+      # sessionToken: YOUR_SESSION_TOKEN  # For temporary credentials
+      # profile: my-aws-profile  # Use AWS SSO profile
+tests:
+  - vars:
+      query: 'Tell me about your capabilities'
+    assert:
+      - type: contains
+        value: 'agent'
+
+  - vars:
+      query: 'Calculate the sum of 100 and 250'
+    assert:
+      - type: contains
+        value: '350'
+      - type: not-empty
+
+  - vars:
+      query: 'Remember that my favorite color is blue'
+    assert:
+      - type: not-empty
+      # With memory enabled, the agent should acknowledge storing this information
+
+  - vars:
+      query: 'What is my favorite color?'
+    assert:
+      - type: contains
+        value: 'blue'
+      # This test verifies memory retention if using the same session
+# Environment variables (set these in your environment or .env file):
+# AWS_BEDROCK_REGION=us-east-1
+# AWS_ACCESS_KEY_ID=your_access_key
+# AWS_SECRET_ACCESS_KEY=your_secret_key
+# AWS_SESSION_TOKEN=your_session_token (optional)
+

examples/amazon-bedrock/models/README.md

@@ -0,0 +1,322 @@
+# amazon-bedrock/models (Amazon Bedrock Examples)
+
+You can run this example with:
+
+```bash
+npx promptfoo@latest init --example amazon-bedrock/models
+cd amazon-bedrock/models
+```
+
+## Prerequisites
+
+1. Set up your AWS credentials:
+
+   ```bash
+   export AWS_ACCESS_KEY_ID="your_access_key"
+   export AWS_SECRET_ACCESS_KEY="your_secret_key"
+   ```
+
+   See [authentication docs](https://www.promptfoo.dev/docs/providers/aws-bedrock/#authentication) for other auth methods, including SSO profiles.
+
+2. Request model access in your AWS region:
+   - Visit the [AWS Bedrock Model Access page](https://us-west-2.console.aws.amazon.com/bedrock/home?region=us-west-2#/modelaccess)
+   - Switch to your desired region. We recommend us-west-2 and us-east-1 which tend to have the most models available.
+   - Enable the models you want to use.
+3. Install required dependencies:
+
+   ```bash
+   # For basic Bedrock models
+   npm install @aws-sdk/client-bedrock-runtime
+
+   # For Knowledge Base examples
+   npm install @aws-sdk/client-bedrock-agent-runtime
+   ```
+
+## Available Examples
+
+This directory contains several example configurations for different Bedrock models:
+
+- [`promptfooconfig.claude.yaml`](promptfooconfig.claude.yaml) - Claude 4.6 Opus, Claude 4.1 Opus, Claude 4 Opus/Sonnet, Claude Haiku 4.5
+- [`promptfooconfig.openai.yaml`](promptfooconfig.openai.yaml) - OpenAI GPT-OSS models (120B and 20B) with reasoning effort
+- [`promptfooconfig.openai-frontier.yaml`](promptfooconfig.openai-frontier.yaml) - OpenAI frontier models (GPT-5.5 and GPT-5.4) with native reasoning effort
+- [`promptfooconfig.grok.yaml`](promptfooconfig.grok.yaml) - xAI Grok 4.3 on the Bedrock Mantle endpoint (requires `AWS_BEARER_TOKEN_BEDROCK`)
+- [`promptfooconfig.mantle.yaml`](promptfooconfig.mantle.yaml) - `bedrock:mantle:` Chat Completions endpoint for mantle-only models like GLM 4.6 and DeepSeek V3.1 (requires `AWS_BEARER_TOKEN_BEDROCK`)
+- [`promptfooconfig.llama.yaml`](promptfooconfig.llama.yaml) - Llama3
+- [`promptfooconfig.mistral.yaml`](promptfooconfig.mistral.yaml) - Mistral
+- [`promptfooconfig.openai-compatible.yaml`](promptfooconfig.openai-compatible.yaml) - OpenAI-compatible families: Z.AI GLM, MiniMax, Moonshot Kimi, NVIDIA Nemotron, Google Gemma, Writer Palmyra
+- [`promptfooconfig.nova.yaml`](promptfooconfig.nova.yaml) - Amazon's Nova models
+- [`promptfooconfig.nova.tool.yaml`](promptfooconfig.nova.tool.yaml) - Nova with tool usage examples
+- [`promptfooconfig.nova.multimodal.yaml`](promptfooconfig.nova.multimodal.yaml) - Nova with multimodal capabilities
+- [`promptfooconfig.kb.yaml`](promptfooconfig.kb.yaml) - Knowledge Base RAG example with citations and contextTransform
+- [`promptfooconfig.inference-profiles.yaml`](promptfooconfig.inference-profiles.yaml) - Comprehensive Application Inference Profiles example with multiple model types
+- [`promptfooconfig.inference-profiles-simple.yaml`](promptfooconfig.inference-profiles-simple.yaml) - Simple production-ready inference profile setup for high availability
+- [`promptfooconfig.yaml`](promptfooconfig.yaml) - Combined evaluation across multiple providers
+- [`promptfooconfig.nova-sonic.yaml`](promptfooconfig.nova-sonic.yaml) - Amazon Nova Sonic model for audio
+- [`promptfooconfig.converse.yaml`](promptfooconfig.converse.yaml) - Converse API with extended thinking (ultrathink)
+- [`promptfooconfig.converse-mcp.yaml`](promptfooconfig.converse-mcp.yaml) - Converse API with Model Context Protocol (MCP) tools
+
+## Converse API Example
+
+The Converse API example (`promptfooconfig.converse.yaml`) demonstrates the unified Bedrock Converse API with extended thinking (ultrathink) support.
+
+### Key Features
+
+- **Extended Thinking**: Enable Claude's reasoning capabilities with configurable token budgets
+- **Unified Interface**: Single API format works across Claude, Nova, Llama, Mistral, and more
+- **Show/Hide Thinking**: Control whether thinking content appears in output with `showThinking`
+
+### Configuration
+
+```yaml
+providers:
+  - id: bedrock:converse:us.anthropic.claude-sonnet-4-6
+    label: Claude Sonnet 4.6 with Thinking
+    config:
+      region: us-west-2
+      maxTokens: 20000
+      thinking:
+        type: enabled
+        budget_tokens: 16000
+      showThinking: true
+```
+
+Run the Converse API example with:
+
+```bash
+promptfoo eval -c examples/amazon-bedrock/models/promptfooconfig.converse.yaml
+```
+
+## Converse MCP Example
+
+The Converse MCP example (`promptfooconfig.converse-mcp.yaml`) demonstrates how to attach Model Context Protocol (MCP) servers to a Bedrock Converse provider. MCP tools are discovered from the configured server, converted to Bedrock Converse tool definitions, and executed when the model requests a tool call.
+
+### Configuration
+
+```yaml
+providers:
+  - id: bedrock:converse:us.anthropic.claude-sonnet-4-6
+    label: Claude Sonnet 4.6 with MCP
+    config:
+      region: us-east-1
+      maxTokens: 1024
+      temperature: 0
+      mcp:
+        enabled: true
+        servers:
+          - name: deepwiki
+            url: https://mcp.deepwiki.com/mcp
+        tools:
+          - ask_question
+      toolChoice: auto
+```
+
+Run the Converse MCP example with:
+
+```bash
+promptfoo eval -c examples/amazon-bedrock/models/promptfooconfig.converse-mcp.yaml
+```
+
+Replace the `servers` entry with a local `command`/`args`, `path`, or another remote `url` to use your own MCP server.
+
+> **Note:** When the model emits `tool_use`, the provider executes the requested
+> MCP tool and returns the **raw tool result** as the eval output. There is no
+> follow-up Converse turn that feeds the tool result back to the model for a
+> synthesized answer, so the assertions in this example match substrings present
+> in the MCP server's response. If you need a model-summarized answer, wrap the
+> provider in an agent harness or run a second eval over the captured tool
+> output.
+
+## Knowledge Base Example
+
+The Knowledge Base example (`promptfooconfig.kb.yaml`) demonstrates how to use AWS Bedrock Knowledge Base for Retrieval Augmented Generation (RAG).
+
+### Knowledge Base Setup
+
+For this example, you'll need to:
+
+1. Create a Knowledge Base in AWS Bedrock
+2. Configure it to crawl or ingest content (the example assumes promptfoo documentation content)
+3. Use the Amazon Titan Embeddings model for vector embeddings
+4. Update the config with your Knowledge Base ID:
+
+```yaml
+providers:
+  - id: bedrock:kb:us.anthropic.claude-sonnet-4-6
+    config:
+      region: 'us-east-2' # Change to your region
+      knowledgeBaseId: 'YOUR_KNOWLEDGE_BASE_ID' # Replace with your KB ID
+```
+
+When running the Knowledge Base example, you'll see:
+
+- Responses from a Knowledge Base-enhanced model with citations
+- Responses from a standard model for comparison
+- Citations from source documents that show where information was retrieved from
+- Example of `contextTransform` feature extracting context from citations for evaluation
+
+The example includes questions about promptfoo configuration, providers, and evaluation techniques that work well with the embedded promptfoo documentation.
+
+**Note**: You'll need to update the `knowledgeBaseId` with your actual Knowledge Base ID and ensure the Knowledge Base is configured to work with the selected Claude model.
+
+For detailed Knowledge Base setup instructions, see the [AWS Bedrock Knowledge Base Documentation](https://docs.aws.amazon.com/bedrock/latest/userguide/knowledge-base.html).
+
+## Application Inference Profiles Example
+
+The Application Inference Profiles example (`promptfooconfig.inference-profiles.yaml`) demonstrates how to use AWS Bedrock's inference profiles for multi-region failover and cost optimization.
+
+### Key Benefits of Inference Profiles
+
+- **Automatic Failover**: If one region is unavailable, requests automatically route to another region
+- **Cost Optimization**: Routes to the most cost-effective available model
+- **Simplified Management**: Use a single ARN instead of managing multiple model IDs
+- **Cross-Region Availability**: Access models across multiple regions with a single profile
+
+### Configuration Requirements
+
+When using inference profiles, you **must** specify the `inferenceModelType` parameter:
+
+```yaml
+providers:
+  - id: bedrock:arn:aws:bedrock:us-east-1:123456789012:application-inference-profile/my-profile
+    config:
+      inferenceModelType: 'claude' # Required!
+      region: 'us-east-1'
+      max_tokens: 1024
+```
+
+### Supported Model Types
+
+- `claude` - Anthropic Claude models
+- `nova` - Amazon Nova models
+- `llama` - Defaults to Llama 4
+- `llama2`, `llama3`, `llama3.1`, `llama3.2`, `llama3.3`, `llama4` - Specific Llama versions
+- `mistral` - Mistral models
+- `cohere` - Cohere models
+- `ai21` - AI21 models
+- `titan` - Amazon Titan models
+- `deepseek` - DeepSeek models (with thinking capability)
+- `openai` - OpenAI GPT-OSS models
+- `zai` - Z.AI GLM models
+- `minimax` - MiniMax models
+- `moonshot` - Moonshot Kimi models
+- `nvidia` - NVIDIA Nemotron models
+- `writer` - Writer Palmyra models
+- `gemma` - Google Gemma models
+
+### Running the Examples
+
+We provide two inference profile examples:
+
+1. **Comprehensive Example** (`promptfooconfig.inference-profiles.yaml`):
+
+   ```bash
+   promptfoo eval -c examples/amazon-bedrock/models/promptfooconfig.inference-profiles.yaml
+   ```
+
+   This includes:
+   - Multiple inference profiles for different model families
+   - Comparison with direct model IDs
+   - Use of inference profiles for grading assertions
+   - Various model-specific configurations
+
+2. **Simple Production Example** (`promptfooconfig.inference-profiles-simple.yaml`):
+   ```bash
+   promptfoo eval -c examples/amazon-bedrock/models/promptfooconfig.inference-profiles-simple.yaml
+   ```
+   This demonstrates:
+   - A realistic customer support use case
+   - High availability setup with failover
+   - Comparison between inference profile and direct model access
+   - Consistent grading using inference profiles
+
+**Note**: Replace the example ARNs with your actual application inference profile ARNs. To create an inference profile, visit the AWS Bedrock console and navigate to the "Application inference profiles" section.
+
+## OpenAI Models Example
+
+The OpenAI example (`promptfooconfig.openai.yaml`) demonstrates OpenAI's GPT-OSS models available through AWS Bedrock:
+
+- **openai.gpt-oss-120b-1:0** - 120 billion parameter model with strong reasoning capabilities
+- **openai.gpt-oss-20b-1:0** - 20 billion parameter model, more cost-effective
+
+### Key Features
+
+- **Reasoning Effort**: Control reasoning depth with `low`, `medium`, or `high` settings
+- **OpenAI API Format**: Uses familiar OpenAI parameters like `max_completion_tokens`
+- **Available in us-west-2**: Ensure you have model access in the correct region
+
+Run the OpenAI example with:
+
+```bash
+promptfoo eval -c examples/amazon-bedrock/models/promptfooconfig.openai.yaml
+```
+
+## OpenAI Frontier Models Example
+
+The frontier example (`promptfooconfig.openai-frontier.yaml`) demonstrates OpenAI's GPT-5.x frontier models on Bedrock:
+
+- **openai.gpt-5.5** - Flagship frontier reasoning model (available in `us-east-2`)
+- **openai.gpt-5.4** - Frontier reasoning model (available in `us-east-2` and `us-west-2`)
+
+### Key Features
+
+- **Responses API**: Frontier models are served through Bedrock's OpenAI-compatible Responses API (the mantle endpoint), not `InvokeModel`. promptfoo routes `bedrock:openai.gpt-5.x` there automatically, so output matches the `openai:responses` provider.
+- **Bedrock API key auth**: Unlike the gpt-oss models (AWS SDK credentials), the frontier models authenticate with an Amazon Bedrock API key. Export it first:
+
+  ```bash
+  export AWS_BEARER_TOKEN_BEDROCK="your_bedrock_api_key"
+  ```
+
+- **Native Reasoning Effort**: `reasoning_effort` supports `none`, `low`, `medium`, `high`, and `xhigh` (`minimal` is not supported by these Bedrock models).
+- **Region-gated**: Request model access in a supported region before running.
+
+These are the same model IDs that back OpenAI's [Codex](https://developers.openai.com/codex/) coding agent when it is configured with the `amazon-bedrock` provider.
+
+Run the frontier example with:
+
+```bash
+promptfoo eval -c examples/amazon-bedrock/models/promptfooconfig.openai-frontier.yaml
+```
+
+## New Converse API Features (SDK 3.943+)
+
+The Converse API supports additional stop reason handling:
+
+- `malformed_model_output`: Model produced invalid output
+- `malformed_tool_use`: Model produced a malformed tool use request
+
+These are returned as errors in the response with `metadata.isModelError: true`.
+
+## Nova Sonic Configuration
+
+Nova Sonic now supports configurable timeouts:
+
+```yaml
+providers:
+  - id: bedrock:nova-sonic:amazon.nova-sonic-v1:0
+    config:
+      region: us-east-1
+      sessionTimeout: 300000 # 5 minutes (default)
+      requestTimeout: 120000 # 2 minutes
+```
+
+Error responses include categorized error types in `metadata.errorType`:
+
+- `connection`: Network/AWS connectivity issues
+- `timeout`: Request or session timeout
+- `api`: Authentication/authorization errors
+- `parsing`: Response parsing failures
+- `session`: Bidirectional stream session errors
+
+## Getting Started
+
+1. Run the evaluation:
+
+   ```bash
+   promptfoo eval -c [path/to/config.yaml]
+   ```
+
+2. View the results:
+
+   ```bash
+   promptfoo view
+   ```

examples/amazon-bedrock/models/converse_image_prompt.json

@@ -0,0 +1,18 @@
+[
+  {
+    "role": "user",
+    "content": [
+      {
+        "image": {
+          "format": "jpeg",
+          "source": {
+            "bytes": "{{image}}"
+          }
+        }
+      },
+      {
+        "text": "What animal is in this image? Describe it briefly."
+      }
+    ]
+  }
+]

examples/amazon-bedrock/models/llama_vision_prompt.json

@@ -0,0 +1,19 @@
+[
+  {
+    "role": "user",
+    "content": [
+      {
+        "type": "image",
+        "source": {
+          "type": "base64",
+          "media_type": "image/jpeg",
+          "data": "{{image}}"
+        }
+      },
+      {
+        "type": "text",
+        "text": "What animal is in this image? Describe it briefly."
+      }
+    ]
+  }
+]

examples/amazon-bedrock/models/nova_multimodal_prompt.json

@@ -0,0 +1,16 @@
+[
+  {
+    "role": "user",
+    "content": [
+      {
+        "image": {
+          "format": "jpg",
+          "source": { "bytes": "{{image}}" }
+        }
+      },
+      {
+        "text": "What is this a picture of?"
+      }
+    ]
+  }
+]

examples/amazon-bedrock/models/nova_sonic_prompt.js

@@ -0,0 +1,69 @@
+/**
+ * This function generates a properly formatted conversation for the Amazon Bedrock Nova Sonic model.
+ * It handles the _conversation variable to maintain conversation history.
+ *
+ * @typedef {Object} Vars
+ * @property {string} [system_message] - System message to set the assistant's behavior
+ * @property {string} [audio_file] - Path to the audio file to be processed
+ * @property {Array<Object>} [_conversation] - Previous conversation history
+ * @property {string} [_conversation[].output] - The assistant's previous response
+ * @property {Object} [_conversation[].metadata] - Metadata about the previous response
+ * @property {string} [_conversation[].metadata.userTranscript] - Transcript of the user's previous input
+ *
+ * @param {Object} provider - The provider configuration
+ * @returns {Object} The formatted conversation for Nova Sonic
+ */
+module.exports = async function ({ vars, provider }) {
+  // Create the messages array starting with system message
+  const messages = [
+    {
+      role: 'system',
+      content: [
+        {
+          type: 'input_text',
+          text: vars.system_message || 'You are a helpful AI assistant.',
+        },
+      ],
+    },
+  ];
+
+  // Add previous conversation turns if they exist
+  if (vars._conversation && Array.isArray(vars._conversation)) {
+    for (const completion of vars._conversation) {
+      // Add user message with input_text type (for user inputs)
+      messages.push({
+        role: 'user',
+        content: [
+          {
+            type: 'input_text',
+            text: completion?.metadata?.userTranscript || '',
+          },
+        ],
+      });
+
+      // Add assistant message with text type (for outputs)
+      messages.push({
+        role: 'assistant',
+        content: [
+          {
+            type: 'text',
+            text: completion.output,
+          },
+        ],
+      });
+    }
+  }
+
+  // Add the current question as the final user message
+  messages.push({
+    role: 'user',
+    content: [
+      {
+        type: 'audio',
+        text: vars.audio_file || '',
+      },
+    ],
+  });
+
+  return messages;
+};

examples/amazon-bedrock/models/promptfooconfig.a21.yaml

@@ -0,0 +1,42 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: 'Bedrock A21 Eval'
+
+prompts:
+  - 'Write a tweet about {{topic}}'
+
+providers:
+  - id: bedrock:ai21.jamba-1-5-large-v1:0
+    config:
+      region: us-east-1
+
+tests:
+  - vars:
+      topic: Our eco-friendly packaging
+  - vars:
+      topic: A sneak peek at our secret menu item
+  - vars:
+      topic: Behind-the-scenes at our latest photoshoot
+  - vars:
+      topic: the impact of autonomous drones on wildlife conservation
+  - vars:
+      topic: the emerging trend of virtual reality courtrooms
+  - vars:
+      topic: the ethical implications of AI-generated art
+  - vars:
+      topic: the unexpected health benefits of daily meditation
+  - vars:
+      topic: how AI is changing the way we play board games
+  - vars:
+      topic: unconventional productivity hacks involving household items
+  - vars:
+      topic: An underground art exhibition in an abandoned subway station
+  - vars:
+      topic: A webinar on the impact of AI on traditional marketing strategies
+  - vars:
+      topic: The launch of a new eco-friendly sneaker made from ocean plastic
+  - vars:
+      topic: the correlation between social media usage and self-esteem in teenagers
+  - vars:
+      topic: the impact of urban noise pollution on migratory bird patterns
+  - vars:
+      topic: the role of gut microbiota in moderating anxiety and depression

examples/amazon-bedrock/models/promptfooconfig.claude.yaml

@@ -0,0 +1,77 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: 'Bedrock Claude Eval'
+prompts:
+  - 'Write a tweet about {{topic}}'
+providers:
+  - id: bedrock:arn:aws:bedrock:us-east-2::inference-profile/global.anthropic.claude-opus-4-6-v1
+    label: Claude Opus 4.6
+    config:
+      temperature: 1
+      max_tokens: 2048
+      region: us-east-2
+      inferenceModelType: claude
+  - id: bedrock:us.anthropic.claude-opus-4-1-20250805-v1:0
+    label: Claude Opus 4.1
+    config:
+      temperature: 1
+      max_tokens: 2048
+      region: us-west-2
+  - id: bedrock:us.anthropic.claude-opus-4-8
+    label: Claude Opus 4.8
+    config:
+      temperature: 1
+      max_tokens: 2048
+      region: us-west-2
+  - id: bedrock:us.anthropic.claude-haiku-4-5-20251001-v1:0
+    label: Claude Haiku 4.5
+    config:
+      temperature: 1
+      max_tokens: 2048
+      region: us-west-2
+  - id: bedrock:us.anthropic.claude-sonnet-4-6
+    label: Claude Sonnet 4.6 with Thinking
+    config:
+      temperature: 1
+      max_tokens: 2048
+      region: us-west-2
+      showThinking: true
+      thinking:
+        type: enabled
+        budget_tokens: 1024
+  - id: bedrock:us.anthropic.claude-opus-4-6-v1
+    label: Claude Opus 4.6 (inference profile)
+    config:
+      temperature: 1
+      max_tokens: 2048
+      region: us-west-2
+tests:
+  - vars:
+      topic: Our eco-friendly packaging
+  - vars:
+      topic: A sneak peek at our secret menu item
+  - vars:
+      topic: Behind-the-scenes at our latest photoshoot
+  - vars:
+      topic: the impact of autonomous drones on wildlife conservation
+  - vars:
+      topic: the emerging trend of virtual reality courtrooms
+  - vars:
+      topic: the ethical implications of AI-generated art
+  - vars:
+      topic: the unexpected health benefits of daily meditation
+  - vars:
+      topic: how AI is changing the way we play board games
+  - vars:
+      topic: unconventional productivity hacks involving household items
+  - vars:
+      topic: An underground art exhibition in an abandoned subway station
+  - vars:
+      topic: A webinar on the impact of AI on traditional marketing strategies
+  - vars:
+      topic: The launch of a new eco-friendly sneaker made from ocean plastic
+  - vars:
+      topic: the correlation between social media usage and self-esteem in teenagers
+  - vars:
+      topic: the impact of urban noise pollution on migratory bird patterns
+  - vars:
+      topic: the role of gut microbiota in moderating anxiety and depression

examples/amazon-bedrock/models/promptfooconfig.converse-comprehensive.yaml

@@ -0,0 +1,258 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: 'Bedrock Converse API - Comprehensive Test Suite'
+
+# ============================================================================
+# This example demonstrates all Converse API capabilities across model families:
+# - Text generation (all models)
+# - System prompts
+# - Multi-turn conversations
+# - Tool/function calling
+# - Extended thinking (Claude)
+# - Performance configuration
+#
+# Model families covered:
+# - Anthropic Claude (4.x, 4.5, 4.6)
+# - Amazon Nova (Micro, Lite, Pro, Premier)
+# - Meta Llama (3.x, 4.x)
+# - Mistral AI (Small, Large, Pixtral)
+# - DeepSeek R1
+# - Qwen3 (including Coder variants)
+# - Writer Palmyra (X4, X5)
+# - OpenAI GPT-OSS
+# ============================================================================
+
+prompts:
+  # Simple text prompt
+  - 'What is the capital of {{country}}?'
+
+providers:
+  # ============================================================================
+  # Claude Models (Anthropic) - Full featured
+  # ============================================================================
+
+  # Claude Sonnet 4.6 with Extended Thinking
+  # Note: temperature MUST be 1 when thinking is enabled
+  - id: bedrock:converse:us.anthropic.claude-sonnet-4-6
+    label: Claude Sonnet 4.6 (Thinking)
+    config:
+      region: us-west-2
+      maxTokens: 16000
+      temperature: 1
+      thinking:
+        type: enabled
+        budget_tokens: 10000
+      showThinking: true
+
+  # Claude Opus 4.1
+  - id: bedrock:converse:us.anthropic.claude-opus-4-1-20250805-v1:0
+    label: Claude Opus 4.1
+    config:
+      region: us-west-2
+      maxTokens: 4096
+      temperature: 0.7
+
+  # Claude Sonnet 4
+  - id: bedrock:converse:us.anthropic.claude-sonnet-4-20250514-v1:0
+    label: Claude Sonnet 4
+    config:
+      region: us-east-1
+      maxTokens: 4096
+      temperature: 0.7
+
+  # Claude Opus 4.6
+  - id: bedrock:converse:us.anthropic.claude-opus-4-6-v1
+    label: Claude Opus 4.6
+    config:
+      region: us-east-1
+      maxTokens: 4096
+      temperature: 0.7
+
+  # Claude Haiku 4.5 (fast, cheap)
+  - id: bedrock:converse:us.anthropic.claude-haiku-4-5-20251001-v1:0
+    label: Claude Haiku 4.5
+    config:
+      region: us-east-1
+      maxTokens: 2048
+      temperature: 0.5
+
+  # ============================================================================
+  # Amazon Nova Models
+  # ============================================================================
+
+  # Nova Premier (most capable)
+  - id: bedrock:converse:us.amazon.nova-premier-v1:0
+    label: Nova Premier
+    config:
+      region: us-east-1
+      maxTokens: 4096
+      temperature: 0.7
+
+  # Nova Pro
+  - id: bedrock:converse:amazon.nova-pro-v1:0
+    label: Nova Pro
+    config:
+      region: us-east-1
+      maxTokens: 4096
+      temperature: 0.7
+
+  # Nova Lite (balanced)
+  - id: bedrock:converse:amazon.nova-lite-v1:0
+    label: Nova Lite
+    config:
+      region: us-east-1
+      maxTokens: 2048
+
+  # Nova Micro (fastest)
+  - id: bedrock:converse:amazon.nova-micro-v1:0
+    label: Nova Micro
+    config:
+      region: us-east-1
+      maxTokens: 1024
+
+  # ============================================================================
+  # Meta Llama Models
+  # ============================================================================
+
+  # Llama 4 Maverick (400B MoE, 17B active) - requires inference profile
+  - id: bedrock:converse:us.meta.llama4-maverick-17b-instruct-v1:0
+    label: Llama 4 Maverick
+    config:
+      region: us-east-1
+      maxTokens: 4096
+      temperature: 0.7
+
+  # Llama 4 Scout (109B MoE, 17B active) - requires inference profile
+  - id: bedrock:converse:us.meta.llama4-scout-17b-instruct-v1:0
+    label: Llama 4 Scout
+    config:
+      region: us-east-1
+      maxTokens: 4096
+      temperature: 0.7
+
+  # Llama 3.3 70B (latest Llama 3.x)
+  - id: bedrock:converse:us.meta.llama3-3-70b-instruct-v1:0
+    label: Llama 3.3 70B
+    config:
+      region: us-west-2
+      maxTokens: 4096
+      temperature: 0.7
+
+  # ============================================================================
+  # Mistral Models
+  # ============================================================================
+
+  # Pixtral Large (multimodal - 124B parameters)
+  - id: bedrock:converse:us.mistral.pixtral-large-2502-v1:0
+    label: Pixtral Large
+    config:
+      region: us-east-1
+      maxTokens: 4096
+      temperature: 0.7
+
+  # Mistral Large 2 (24.07)
+  - id: bedrock:converse:mistral.mistral-large-2407-v1:0
+    label: Mistral Large 2
+    config:
+      region: us-west-2
+      maxTokens: 4096
+      temperature: 0.7
+
+  # Mistral Small (efficient)
+  - id: bedrock:converse:mistral.mistral-small-2402-v1:0
+    label: Mistral Small
+    config:
+      region: us-east-1
+      maxTokens: 2048
+      temperature: 0.7
+
+  # ============================================================================
+  # DeepSeek Models
+  # ============================================================================
+
+  # DeepSeek R1 (reasoning model)
+  - id: bedrock:converse:us.deepseek.r1-v1:0
+    label: DeepSeek R1
+    config:
+      region: us-west-2
+      maxTokens: 8192
+
+  # ============================================================================
+  # Qwen Models (Alibaba)
+  # ============================================================================
+
+  # Qwen3 Coder 480B (MoE, 35B active - best for coding)
+  - id: bedrock:converse:qwen.qwen3-coder-480b-a35b-v1:0
+    label: Qwen3 Coder 480B
+    config:
+      region: us-west-2
+      maxTokens: 4096
+
+  # Qwen3 235B (MoE, 22B active)
+  - id: bedrock:converse:qwen.qwen3-235b-a22b-2507-v1:0
+    label: Qwen3 235B
+    config:
+      region: us-west-2
+      maxTokens: 4096
+
+  # Qwen3 32B (Dense)
+  - id: bedrock:converse:qwen.qwen3-32b-v1:0
+    label: Qwen3 32B
+    config:
+      region: us-west-2
+      maxTokens: 4096
+
+  # ============================================================================
+  # Writer Palmyra Models
+  # ============================================================================
+
+  # Palmyra X5 (1M context, enterprise-ready) - requires inference profile
+  - id: bedrock:converse:us.writer.palmyra-x5-v1:0
+    label: Writer Palmyra X5
+    config:
+      region: us-east-1
+      maxTokens: 4096
+
+  # Palmyra X4 (128K context) - requires inference profile
+  - id: bedrock:converse:us.writer.palmyra-x4-v1:0
+    label: Writer Palmyra X4
+    config:
+      region: us-east-1
+      maxTokens: 4096
+
+  # ============================================================================
+  # OpenAI GPT-OSS Models
+  # ============================================================================
+
+  # GPT-OSS 120B (MoE)
+  - id: bedrock:converse:openai.gpt-oss-120b-1:0
+    label: OpenAI GPT-OSS 120B
+    config:
+      region: us-west-2
+      maxTokens: 4096
+
+  # GPT-OSS 20B (MoE, efficient)
+  - id: bedrock:converse:openai.gpt-oss-20b-1:0
+    label: OpenAI GPT-OSS 20B
+    config:
+      region: us-west-2
+      maxTokens: 4096
+
+tests:
+  - vars:
+      country: France
+    assert:
+      - type: contains
+        value: Paris
+  - vars:
+      country: Japan
+    assert:
+      - type: contains
+        value: Tokyo
+  - vars:
+      country: Australia
+    assert:
+      - type: contains-any
+        value:
+          - Canberra
+          - Sydney
+          - Melbourne

examples/amazon-bedrock/models/promptfooconfig.converse-images.yaml

@@ -0,0 +1,43 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: 'Bedrock Converse API - Image/Multimodal'
+
+prompts:
+  # Converse API native format for images
+  - file://converse_image_prompt.json
+
+providers:
+  # Claude with vision (via inference profile)
+  - id: bedrock:converse:us.anthropic.claude-sonnet-4-6
+    label: Claude Sonnet 4.6 (Vision)
+    config:
+      region: us-east-1
+      maxTokens: 1024
+      temperature: 0.5
+
+  # Pixtral Large with vision (124B multimodal)
+  - id: bedrock:converse:us.mistral.pixtral-large-2502-v1:0
+    label: Pixtral Large (Vision)
+    config:
+      region: us-east-1
+      maxTokens: 1024
+      temperature: 0.5
+
+  # Nova with vision
+  - id: bedrock:converse:amazon.nova-pro-v1:0
+    label: Nova Pro (Vision)
+    config:
+      region: us-east-1
+      maxTokens: 1024
+
+# For Llama 3.2 Vision, see promptfooconfig.llama-vision.yaml (uses InvokeModel API)
+
+tests:
+  - vars:
+      image: file://cat.jpg
+    assert:
+      - type: contains-any
+        value:
+          - cat
+          - feline
+          - animal
+          - kitten

examples/amazon-bedrock/models/promptfooconfig.converse-mcp.yaml

@@ -0,0 +1,45 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: 'Bedrock Converse API - MCP Tool Use'
+
+prompts:
+  - |
+    Use the DeepWiki MCP tools to answer this question about the {{repo}} repository:
+
+    {{question}}
+
+providers:
+  - id: bedrock:converse:us.anthropic.claude-sonnet-4-6
+    label: Claude Sonnet 4.6 with MCP
+    config:
+      region: us-east-1
+      maxTokens: 1024
+      temperature: 0
+      mcp:
+        enabled: true
+        servers:
+          - name: deepwiki
+            url: https://mcp.deepwiki.com/mcp
+        tools:
+          - ask_question
+      toolChoice: auto
+
+tests:
+  - vars:
+      repo: modelcontextprotocol/modelcontextprotocol
+      question: What transport protocols does MCP support?
+    assert:
+      - type: contains-any
+        value:
+          - stdio
+          - HTTP
+          - transport
+
+  - vars:
+      repo: promptfoo/promptfoo
+      question: How does promptfoo configure providers in an eval?
+    assert:
+      - type: contains-any
+        value:
+          - provider
+          - promptfoo
+          - config

examples/amazon-bedrock/models/promptfooconfig.converse-system.yaml

@@ -0,0 +1,57 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: 'Bedrock Converse API - System Prompts & Multi-turn'
+
+prompts:
+  # System prompt with user message
+  - |
+    [
+      {"role": "system", "content": "You are a helpful assistant that only responds in haiku format (5-7-5 syllable pattern). Never break from this format."},
+      {"role": "user", "content": "What is {{topic}}?"}
+    ]
+
+  # Multi-turn conversation
+  - |
+    [
+      {"role": "system", "content": "You are a math tutor. Be concise and clear."},
+      {"role": "user", "content": "What is 2+2?"},
+      {"role": "assistant", "content": "2+2 equals 4."},
+      {"role": "user", "content": "What about {{question}}?"}
+    ]
+
+providers:
+  # Claude with system prompts (via inference profile)
+  - id: bedrock:converse:us.anthropic.claude-sonnet-4-6
+    label: Claude Sonnet 4.6
+    config:
+      region: us-east-1
+      maxTokens: 1024
+      temperature: 0.7
+
+  # Nova with system prompts
+  - id: bedrock:converse:amazon.nova-pro-v1:0
+    label: Nova Pro
+    config:
+      region: us-east-1
+      maxTokens: 1024
+
+  # Llama with system prompts
+  - id: bedrock:converse:us.meta.llama3-3-70b-instruct-v1:0
+    label: Llama 3.3 70B
+    config:
+      region: us-west-2
+      maxTokens: 1024
+
+tests:
+  # Haiku test
+  - vars:
+      topic: the ocean
+    assert:
+      - type: llm-rubric
+        value: Response should be in haiku format with approximately 5-7-5 syllable pattern
+
+  # Multi-turn math continuation
+  - vars:
+      question: 3 times 4
+    assert:
+      - type: contains
+        value: '12'

examples/amazon-bedrock/models/promptfooconfig.converse-tools.yaml

@@ -0,0 +1,148 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: 'Bedrock Converse API - Tool/Function Calling'
+
+prompts:
+  - 'Get the weather for {{city}}. Use the get_weather tool.'
+
+providers:
+  # Claude with tools (native format) - via inference profile
+  - id: bedrock:converse:us.anthropic.claude-sonnet-4-6
+    label: Claude Sonnet 4.6 (Tools)
+    config:
+      region: us-east-1
+      maxTokens: 1024
+      temperature: 0
+      # Native Converse API tool format
+      tools:
+        - toolSpec:
+            name: get_weather
+            description: Get the current weather for a city
+            inputSchema:
+              json:
+                type: object
+                properties:
+                  city:
+                    type: string
+                    description: The city name
+                  unit:
+                    type: string
+                    enum: ['celsius', 'fahrenheit']
+                    description: Temperature unit
+                required:
+                  - city
+      toolChoice: auto
+
+  # Claude with OpenAI-compatible tool format
+  - id: bedrock:converse:us.anthropic.claude-sonnet-4-6
+    label: Claude Sonnet 4.6 (OpenAI Format)
+    config:
+      region: us-east-1
+      maxTokens: 1024
+      temperature: 0
+      # OpenAI-compatible format
+      tools:
+        - type: function
+          function:
+            name: get_weather
+            description: Get the current weather for a city
+            parameters:
+              type: object
+              properties:
+                city:
+                  type: string
+                  description: The city name
+                unit:
+                  type: string
+                  enum: ['celsius', 'fahrenheit']
+                  description: Temperature unit
+              required:
+                - city
+      toolChoice: auto
+
+  # Claude with Anthropic tool format
+  - id: bedrock:converse:us.anthropic.claude-sonnet-4-6
+    label: Claude Sonnet 4.6 (Anthropic Format)
+    config:
+      region: us-east-1
+      maxTokens: 1024
+      temperature: 0
+      # Anthropic-compatible format
+      tools:
+        - name: get_weather
+          description: Get the current weather for a city
+          input_schema:
+            type: object
+            properties:
+              city:
+                type: string
+                description: The city name
+              unit:
+                type: string
+                enum: ['celsius', 'fahrenheit']
+                description: Temperature unit
+            required:
+              - city
+      toolChoice: auto
+
+  # Nova with tools
+  - id: bedrock:converse:amazon.nova-pro-v1:0
+    label: Nova Pro (Tools)
+    config:
+      region: us-east-1
+      maxTokens: 1024
+      temperature: 0
+      tools:
+        - toolSpec:
+            name: get_weather
+            description: Get the current weather for a city
+            inputSchema:
+              json:
+                type: object
+                properties:
+                  city:
+                    type: string
+                    description: The city name
+                  unit:
+                    type: string
+                    enum: ['celsius', 'fahrenheit']
+                    description: Temperature unit
+                required:
+                  - city
+      toolChoice: auto
+
+defaultTest:
+  options:
+    # Extract the tool call input
+    transform: |
+      try {
+        const parsed = JSON.parse(output);
+        return parsed.input?.city || parsed.city || output;
+      } catch {
+        return output;
+      }
+
+tests:
+  - vars:
+      city: Tokyo
+    assert:
+      - type: contains-any
+        value:
+          - Tokyo
+          - tool_use
+          - get_weather
+  - vars:
+      city: Paris
+    assert:
+      - type: contains-any
+        value:
+          - Paris
+          - tool_use
+          - get_weather
+  - vars:
+      city: New York
+    assert:
+      - type: contains-any
+        value:
+          - New York
+          - tool_use
+          - get_weather

examples/amazon-bedrock/models/promptfooconfig.converse.yaml

@@ -0,0 +1,51 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: 'Bedrock Converse API with Extended Thinking'
+
+prompts:
+  - 'Solve this step by step: {{problem}}'
+
+providers:
+  # Claude with Extended Thinking via Converse API
+  - id: bedrock:converse:us.anthropic.claude-sonnet-4-6
+    label: Claude Sonnet 4.6 (Converse + Thinking)
+    config:
+      region: us-west-2
+      maxTokens: 20000
+      thinking:
+        type: enabled
+        budget_tokens: 16000
+      showThinking: true
+
+  # Claude Opus without thinking for comparison
+  - id: bedrock:converse:us.anthropic.claude-opus-4-6-v1
+    label: Claude Opus 4.6 (Converse)
+    config:
+      region: us-east-1
+      maxTokens: 4096
+      temperature: 0.7
+
+  # Nova via Converse API
+  - id: bedrock:converse:amazon.nova-pro-v1:0
+    label: Nova Pro (Converse)
+    config:
+      region: us-east-1
+      maxTokens: 4096
+
+  # Llama via Converse API
+  - id: bedrock:converse:us.meta.llama3-3-70b-instruct-v1:0
+    label: Llama 3.3 70B (Converse)
+    config:
+      region: us-west-2
+      maxTokens: 4096
+
+tests:
+  - vars:
+      problem: 'What is the 100th prime number?'
+  - vars:
+      problem: 'Prove that the square root of 2 is irrational.'
+  - vars:
+      problem: 'A farmer has 17 sheep. All but 9 run away. How many are left?'
+  - vars:
+      problem: 'If it takes 5 machines 5 minutes to make 5 widgets, how long would it take 100 machines to make 100 widgets?'
+  - vars:
+      problem: 'Find all positive integers n such that n^2 + 1 is divisible by n + 1.'

examples/amazon-bedrock/models/promptfooconfig.deepseek.yaml

@@ -0,0 +1,45 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: 'Bedrock Deepseek Eval'
+
+prompts:
+  - 'Write a tweet about {{topic}}'
+
+providers:
+  - id: bedrock:us.deepseek.r1-v1:0
+    config:
+      region: us-west-2
+      inferenceConfig:
+        temperature: 0.7
+        max_new_tokens: 256
+
+tests:
+  - vars:
+      topic: Our eco-friendly packaging
+  - vars:
+      topic: A sneak peek at our secret menu item
+  - vars:
+      topic: Behind-the-scenes at our latest photoshoot
+  - vars:
+      topic: the impact of autonomous drones on wildlife conservation
+  - vars:
+      topic: the emerging trend of virtual reality courtrooms
+  - vars:
+      topic: the ethical implications of AI-generated art
+  - vars:
+      topic: the unexpected health benefits of daily meditation
+  - vars:
+      topic: how AI is changing the way we play board games
+  - vars:
+      topic: unconventional productivity hacks involving household items
+  - vars:
+      topic: An underground art exhibition in an abandoned subway station
+  - vars:
+      topic: A webinar on the impact of AI on traditional marketing strategies
+  - vars:
+      topic: The launch of a new eco-friendly sneaker made from ocean plastic
+  - vars:
+      topic: the correlation between social media usage and self-esteem in teenagers
+  - vars:
+      topic: the impact of urban noise pollution on migratory bird patterns
+  - vars:
+      topic: the role of gut microbiota in moderating anxiety and depression

examples/amazon-bedrock/models/promptfooconfig.grok.yaml

@@ -0,0 +1,35 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: 'xAI Grok 4.3 on Amazon Bedrock (Mantle / OpenAI-compatible Responses API)'
+
+# Grok 4.3 runs on Bedrock's Mantle inference engine and is served through the
+# OpenAI-compatible Responses API — NOT the InvokeModel/Converse APIs. It authenticates with
+# an Amazon Bedrock API key, so set AWS_BEARER_TOKEN_BEDROCK (a long-term Bedrock API key) in
+# your environment. Grok 4.3 is currently available only in us-west-2.
+#
+#   AWS_BEARER_TOKEN_BEDROCK=... npm run local -- eval -c examples/amazon-bedrock/models/promptfooconfig.grok.yaml --no-cache
+
+prompts:
+  - 'Answer in one short sentence: {{question}}'
+
+providers:
+  - id: bedrock:xai.grok-4.3
+    label: Grok 4.3
+    config:
+      region: us-west-2
+      # Grok is reasoning-first; configure effort with none | low | medium | high.
+      reasoning_effort: low
+      # Optional: set an explicit value, or omit this field to use Grok's model default.
+      temperature: 0.2
+
+tests:
+  - vars:
+      question: What is the capital of France?
+    assert:
+      - type: icontains
+        value: Paris
+
+  - vars:
+      question: What is 17 multiplied by 23?
+    assert:
+      - type: contains
+        value: '391'

examples/amazon-bedrock/models/promptfooconfig.inference-profiles-simple.yaml

@@ -0,0 +1,64 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: 'Simple Inference Profile Example - High Availability Setup'
+
+# This example demonstrates a typical production setup using inference profiles
+# for high availability and automatic failover across regions
+
+prompts:
+  - 'Answer this customer support question professionally: {{question}}'
+
+providers:
+  # Production inference profile with automatic failover
+  # This profile might route to us-east-1 primarily, with failover to us-west-2
+  - id: bedrock:arn:aws:bedrock:us-east-1:YOUR_ACCOUNT_ID:application-inference-profile/prod-claude-ha
+    label: 'Production Claude (HA)'
+    config:
+      inferenceModelType: 'claude' # Required for inference profiles
+      region: 'us-east-1'
+      temperature: 0.3 # Lower temperature for consistent customer support
+      max_tokens: 500
+      anthropic_version: 'bedrock-2023-05-31'
+
+  # Direct model for comparison (no failover)
+  - id: bedrock:us.anthropic.claude-sonnet-4-6
+    label: 'Direct Claude (Single Region)'
+    config:
+      region: 'us-east-1'
+      temperature: 0.3
+      max_tokens: 500
+
+tests:
+  - vars:
+      question: 'How do I reset my password?'
+    assert:
+      - type: contains
+        value: 'password'
+      - type: llm-rubric
+        value: 'Response should be helpful, professional, and provide clear steps'
+
+  - vars:
+      question: "My order hasn't arrived yet, and it's been 2 weeks. What should I do?"
+    assert:
+      - type: llm-rubric
+        value: 'Response should be empathetic and provide actionable next steps'
+
+  - vars:
+      question: 'Can I change my subscription plan mid-cycle?'
+    assert:
+      - type: llm-rubric
+        value: 'Response should clearly explain the policy and any potential charges'
+
+# Use an inference profile for consistent grading across regions
+defaultTest:
+  options:
+    provider:
+      id: bedrock:arn:aws:bedrock:us-east-1:YOUR_ACCOUNT_ID:application-inference-profile/grading-claude
+      config:
+        inferenceModelType: 'claude'
+        temperature: 0 # Zero temperature for consistent grading
+        max_tokens: 256
+  assert:
+    - type: not-contains
+      value: 'I cannot'
+    - type: min-length
+      value: 50

examples/amazon-bedrock/models/promptfooconfig.inference-profiles.yaml

@@ -0,0 +1,145 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: 'AWS Bedrock Application Inference Profiles Example'
+
+prompts:
+  - 'Write a comprehensive analysis of {{topic}} in 3 paragraphs'
+  - 'Create a creative story about {{topic}} with an unexpected twist'
+  - |
+    Answer the following question step by step:
+    {{question}}
+
+providers:
+  # Application Inference Profiles - Multi-region, automatic failover
+  # These require the inferenceModelType configuration parameter
+
+  # Claude inference profile - optimized for high availability
+  - id: bedrock:arn:aws:bedrock:us-east-1:123456789012:application-inference-profile/claude-ha-profile
+    label: 'Claude HA Profile'
+    config:
+      inferenceModelType: 'claude' # Required for inference profiles
+      region: 'us-east-1'
+      temperature: 0.7
+      max_tokens: 1024
+      anthropic_version: 'bedrock-2023-05-31'
+
+  # Llama inference profile - latest version (defaults to Llama 4)
+  - id: bedrock:arn:aws:bedrock:us-west-2:123456789012:application-inference-profile/llama-latest-profile
+    label: 'Llama Latest Profile'
+    config:
+      inferenceModelType: 'llama' # Defaults to latest Llama version (v4)
+      region: 'us-west-2'
+      max_gen_len: 1024
+      temperature: 0.7
+      top_p: 0.9
+
+  # Llama 3.3 specific inference profile
+  - id: bedrock:arn:aws:bedrock:us-west-2:123456789012:application-inference-profile/llama33-profile
+    label: 'Llama 3.3 Profile'
+    config:
+      inferenceModelType: 'llama3.3' # Specific Llama 3.3 version
+      region: 'us-west-2'
+      max_gen_len: 1024
+      temperature: 0.6
+
+  # Nova inference profile - cost optimized
+  - id: bedrock:arn:aws:bedrock:eu-west-1:123456789012:application-inference-profile/nova-cost-optimized
+    label: 'Nova Cost-Optimized Profile'
+    config:
+      inferenceModelType: 'nova'
+      region: 'eu-west-1'
+      interfaceConfig:
+        max_new_tokens: 1024
+        temperature: 0.7
+        top_p: 0.95
+        top_k: 50
+
+  # Mistral inference profile
+  - id: bedrock:arn:aws:bedrock:us-east-1:123456789012:application-inference-profile/mistral-profile
+    label: 'Mistral Profile'
+    config:
+      inferenceModelType: 'mistral'
+      region: 'us-east-1'
+      max_tokens: 1024
+      temperature: 0.7
+      top_p: 0.9
+
+  # DeepSeek inference profile with thinking capability
+  - id: bedrock:arn:aws:bedrock:us-east-1:123456789012:application-inference-profile/deepseek-reasoning
+    label: 'DeepSeek Reasoning Profile'
+    config:
+      inferenceModelType: 'deepseek'
+      region: 'us-east-1'
+      max_tokens: 2048
+      temperature: 0.5
+      showThinking: true # Include thinking process in output
+
+  # OpenAI models via Bedrock inference profile
+  - id: bedrock:arn:aws:bedrock:us-west-2:123456789012:application-inference-profile/openai-gpt-profile
+    label: 'OpenAI GPT Profile'
+    config:
+      inferenceModelType: 'openai'
+      region: 'us-west-2'
+      max_completion_tokens: 1024
+      temperature: 0.7
+      reasoning_effort: 'medium' # Control reasoning depth
+
+  # Comparison with direct model IDs (no inference profile)
+  - id: bedrock:us.anthropic.claude-sonnet-4-6
+    label: 'Claude Direct (No Profile)'
+    config:
+      region: 'us-east-1'
+      temperature: 0.7
+      max_tokens: 1024
+
+tests:
+  - vars:
+      topic: 'artificial intelligence in healthcare'
+      question: 'How do neural networks learn from data?'
+
+  - vars:
+      topic: 'climate change and renewable energy'
+      question: 'What are the main differences between supervised and unsupervised learning?'
+
+  - vars:
+      topic: 'the future of space exploration'
+      question: 'Explain quantum computing to a 10-year-old'
+
+  # Test with custom grading using an inference profile
+  - vars:
+      topic: 'blockchain technology'
+      question: 'How does encryption work in modern communications?'
+    options:
+      # Use an inference profile for grading assertions
+      provider:
+        id: bedrock:arn:aws:bedrock:us-east-1:123456789012:application-inference-profile/grading-profile
+        config:
+          inferenceModelType: 'claude'
+          temperature: 0 # Low temperature for consistent grading
+          max_tokens: 512
+    assert:
+      - type: llm-rubric
+        value: 'The response should be comprehensive, technically accurate, and well-structured'
+      - type: contains
+        value: 'technology'
+      - type: min-length
+        value: 200
+
+# Default grading configuration using an inference profile
+defaultTest:
+  options:
+    provider:
+      id: bedrock:arn:aws:bedrock:us-east-1:123456789012:application-inference-profile/default-grading
+      config:
+        inferenceModelType: 'claude'
+        region: 'us-east-1'
+        temperature: 0
+        max_tokens: 256
+  assert:
+    - type: cost
+      threshold: 0.002 # Maximum cost per test
+    - type: latency
+      threshold: 10000 # Maximum 10 seconds
+    - type: not-contains
+      value: 'I cannot'
+    - type: not-contains
+      value: 'As an AI'

examples/amazon-bedrock/models/promptfooconfig.kb.yaml

@@ -0,0 +1,56 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: Knowledge Base RAG with contextTransform evaluation
+
+prompts:
+  - |
+    Answer the following question in a concise manner:
+    {{query}}
+
+providers:
+  # Knowledge Base provider with Claude Sonnet 4.6 (cross-region inference profile)
+  - id: bedrock:kb:us.anthropic.claude-sonnet-4-6
+    label: Claude Sonnet 4.6 KB
+    config:
+      region: us-east-2
+      knowledgeBaseId: '0VMCLLCVGB' # knowledge-base-quick-start-kjmvw
+      temperature: 0.0
+      max_tokens: 1000
+  # Compare with regular Bedrock Claude Sonnet 4.6 (without KB)
+  - id: bedrock:us.anthropic.claude-sonnet-4-6
+    label: Claude Sonnet 4.6 Direct
+    config:
+      region: us-east-2
+      temperature: 0.0
+      max_tokens: 1000
+tests:
+  - vars:
+      query: 'How do I set up my first promptfoo configuration file?'
+  - vars:
+      query: 'What authentication is required for OpenAI and Anthropic providers?'
+  - vars:
+      query: 'How can I compare GPT-4 vs Claude performance?'
+  - vars:
+      query: 'What types of assertions can I use to automatically grade LLM outputs?'
+  - vars:
+      query: 'How do I evaluate RAG systems with context-based assertions?'
+
+  # Example using contextTransform to extract context from Knowledge Base citations
+  - vars:
+      # `context-faithfulness` and `context-relevance` require the `query` variable to be defined.
+      query: 'How do I set up my first promptfoo configuration file?'
+    assert:
+      # Basic content assertions
+      - type: contains
+        value: 'yaml'
+      - type: contains
+        value: 'prompts'
+
+      # Context-based assertions using contextTransform to extract from citations
+      # This demonstrates the key contextTransform feature: extracting context from provider responses
+      - type: context-faithfulness
+        contextTransform: 'context?.metadata?.citations?.flatMap(c => c.retrievedReferences || []).map(r => r.content?.text || "").filter(t => t.length > 0).join("\\n\\n") || "No citations found"'
+        threshold: 0.1
+
+      - type: context-relevance
+        contextTransform: 'context?.metadata?.citations?.[0]?.retrievedReferences?.[0]?.content?.text || "No context found"'
+        threshold: 0.1

examples/amazon-bedrock/models/promptfooconfig.llama-vision.yaml

@@ -0,0 +1,37 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: 'Bedrock Llama 3.2 Vision'
+
+# This example uses the InvokeModel API with Anthropic-style message format.
+# Note: Only Llama 3.2 11B and 90B support vision. The 1B and 3B variants are text-only.
+# For Converse API, see promptfooconfig.converse-images.yaml (uses different image format)
+
+prompts:
+  - file://llama_vision_prompt.json
+
+providers:
+  # Llama 3.2 Vision 11B
+  - id: bedrock:us.meta.llama3-2-11b-instruct-v1:0
+    label: Llama 3.2 11B Vision
+    config:
+      region: us-east-1
+      max_gen_len: 256
+      temperature: 0.3
+
+  # Llama 3.2 Vision 90B (larger model, more capable)
+  - id: bedrock:us.meta.llama3-2-90b-instruct-v1:0
+    label: Llama 3.2 90B Vision
+    config:
+      region: us-east-1
+      max_gen_len: 256
+      temperature: 0.3
+
+tests:
+  - vars:
+      image: file://cat.jpg
+    assert:
+      - type: contains-any
+        value:
+          - cat
+          - feline
+          - animal
+          - kitten

examples/amazon-bedrock/models/promptfooconfig.llama.yaml

@@ -0,0 +1,33 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: 'Bedrock Llama Text Models'
+
+# For Llama 3.2 Vision models, see promptfooconfig.llama-vision.yaml
+
+prompts:
+  - 'Translate to {{language}}: {{input}}'
+
+providers:
+  - id: bedrock:us.meta.llama3-2-3b-instruct-v1:0
+    label: Llama 3.2 3B
+    config:
+      region: us-west-2
+
+  - id: bedrock:us.meta.llama4-scout-17b-instruct-v1:0
+    label: Llama 4 Scout 17B
+    config:
+      region: us-west-2
+
+  - id: bedrock:us.meta.llama4-maverick-17b-instruct-v1:0
+    label: Llama 4 Maverick 17B
+    config:
+      max_gen_len: 100
+      temperature: 0.7
+      region: us-west-2
+
+tests:
+  - vars:
+      language: French
+      input: Hello world
+  - vars:
+      language: Spanish
+      input: Where is the library?

examples/amazon-bedrock/models/promptfooconfig.mantle.yaml

@@ -0,0 +1,42 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: 'Bedrock Mantle Chat Completions endpoint (bedrock:mantle:<id>)'
+
+# The bedrock:mantle: prefix talks to the OpenAI-compatible Chat Completions API on the
+# Bedrock Mantle endpoint (https://bedrock-mantle.<region>.api.aws/v1/chat/completions). It is
+# the only way to reach mantle-only chat models that the native InvokeModel/Converse APIs do
+# not serve (and that therefore do not appear in `aws bedrock list-foundation-models`) — e.g.
+# zai.glm-4.6, deepseek.v3.1, google.gemma-4-*, and the mantle-namespaced Qwen *-instruct ids.
+#
+# Authenticate with an Amazon Bedrock API key, and set a region where the model is offered
+# (list with: GET https://bedrock-mantle.<region>.api.aws/v1/models):
+#
+#   AWS_BEARER_TOKEN_BEDROCK=... npm run local -- eval -c examples/amazon-bedrock/models/promptfooconfig.mantle.yaml --no-cache
+
+prompts:
+  - 'Answer in one short sentence: {{question}}'
+
+providers:
+  - id: bedrock:mantle:zai.glm-4.6
+    label: GLM 4.6 (mantle)
+    config:
+      region: us-west-2
+      max_tokens: 1024
+
+  - id: bedrock:mantle:deepseek.v3.1
+    label: DeepSeek V3.1 (mantle)
+    config:
+      region: us-west-2
+      max_tokens: 1024
+
+tests:
+  - vars:
+      question: What is the capital of France?
+    assert:
+      - type: icontains
+        value: Paris
+
+  - vars:
+      question: What is 12 + 30?
+    assert:
+      - type: contains
+        value: '42'

examples/amazon-bedrock/models/promptfooconfig.mistral.yaml

@@ -0,0 +1,48 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: 'Bedrock Mistral Eval'
+
+prompts:
+  - label: 'Viral Tweet'
+    raw: |
+      Create a viral tweet about the following topic. Include exactly one relevant hashtag. 
+
+      {{topic}}
+
+providers:
+  - id: bedrock:mistral.mistral-7b-instruct-v0:2
+    config:
+      temperature: 0.7
+      max_tokens: 200
+      top_p: 0.9
+      top_k: 50
+      region: us-east-1
+  - id: bedrock:mistral.pixtral-large-2502-v1:0
+    config:
+      temperature: 0.7
+      max_tokens: 200
+      top_p: 0.9
+      region: us-west-2
+  - id: bedrock:mistral.mixtral-8x7b-instruct-v0:1
+    config:
+      temperature: 0.7
+      max_tokens: 200
+      top_p: 0.9
+      top_k: 50
+      region: us-east-1
+
+defaultTest:
+  assert:
+    - type: javascript
+      value: output.length <= 280 # Twitter's character limit
+    - type: llm-rubric
+      value: does not include excessive hashtags (no more than 2)
+
+tests:
+  - vars:
+      topic: The truth about government surveillance drones disguised as pigeons
+  - vars:
+      topic: Breaking evidence that birds are actually government spy cameras
+  - vars:
+      topic: Leaked photos of drone charging stations disguised as bird nests
+  - vars:
+      topic: How the CIA replaced all birds with surveillance robots in the 1970s

examples/amazon-bedrock/models/promptfooconfig.nova-2-converse.yaml

@@ -0,0 +1,50 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: 'Amazon Nova 2 Lite with Converse API'
+
+prompts:
+  - 'Solve this step by step: {{problem}}'
+
+providers:
+  # Nova 2 Lite with reasoning enabled (high effort) via Converse API
+  # Note: Must use cross-region model ID (us.) for on-demand access
+  # Temperature must not be set when reasoning is enabled
+  - id: bedrock:converse:us.amazon.nova-2-lite-v1:0
+    label: Nova 2 Lite Converse (High Reasoning)
+    config:
+      region: 'us-east-1'
+      maxTokens: 4096
+      reasoningConfig:
+        type: enabled
+        maxReasoningEffort: high
+
+  # Nova 2 Lite with reasoning enabled (medium effort) via Converse API
+  - id: bedrock:converse:us.amazon.nova-2-lite-v1:0
+    label: Nova 2 Lite Converse (Medium Reasoning)
+    config:
+      region: 'us-east-1'
+      maxTokens: 4096
+      reasoningConfig:
+        type: enabled
+        maxReasoningEffort: medium
+
+  # Nova 2 Lite with reasoning disabled (fast mode) via Converse API
+  - id: bedrock:converse:us.amazon.nova-2-lite-v1:0
+    label: Nova 2 Lite Converse (Fast)
+    config:
+      region: 'us-east-1'
+      maxTokens: 2048
+      temperature: 0.7
+      reasoningConfig:
+        type: disabled
+
+tests:
+  - vars:
+      problem: 'A train leaves Station A at 9:00 AM traveling at 60 mph. Another train leaves Station B, 300 miles away, at 10:00 AM traveling toward Station A at 90 mph. At what time will the trains meet?'
+  - vars:
+      problem: 'If a bat and ball cost $1.10 together, and the bat costs $1 more than the ball, how much does the ball cost?'
+  - vars:
+      problem: 'In a room of 23 people, what is the probability that at least two people share a birthday?'
+  - vars:
+      problem: 'A farmer has 17 sheep. All but 9 die. How many sheep does the farmer have left?'
+  - vars:
+      problem: 'If it takes 5 machines 5 minutes to make 5 widgets, how long does it take 100 machines to make 100 widgets?'

examples/amazon-bedrock/models/promptfooconfig.nova-2-reasoning.yaml

@@ -0,0 +1,53 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: 'Amazon Nova 2 Lite with Extended Thinking'
+
+prompts:
+  - 'Solve this step by step: {{problem}}'
+
+providers:
+  # Nova 2 Lite with reasoning enabled (high effort)
+  # Note: Must use cross-region model ID (us.) for on-demand access
+  # Temperature must not be set when reasoning is enabled
+  - id: bedrock:us.amazon.nova-2-lite-v1:0
+    label: Nova 2 Lite (High Reasoning)
+    config:
+      region: 'us-east-1'
+      interfaceConfig:
+        max_new_tokens: 4096
+      reasoningConfig:
+        type: enabled
+        maxReasoningEffort: high
+
+  # Nova 2 Lite with reasoning enabled (medium effort)
+  - id: bedrock:us.amazon.nova-2-lite-v1:0
+    label: Nova 2 Lite (Medium Reasoning)
+    config:
+      region: 'us-east-1'
+      interfaceConfig:
+        max_new_tokens: 4096
+      reasoningConfig:
+        type: enabled
+        maxReasoningEffort: medium
+
+  # Nova 2 Lite with reasoning disabled (fast mode)
+  - id: bedrock:us.amazon.nova-2-lite-v1:0
+    label: Nova 2 Lite (Fast)
+    config:
+      region: 'us-east-1'
+      interfaceConfig:
+        max_new_tokens: 2048
+        temperature: 0.7
+      reasoningConfig:
+        type: disabled
+
+tests:
+  - vars:
+      problem: 'A train leaves Station A at 9:00 AM traveling at 60 mph. Another train leaves Station B, 300 miles away, at 10:00 AM traveling toward Station A at 90 mph. At what time will the trains meet?'
+  - vars:
+      problem: 'If a bat and ball cost $1.10 together, and the bat costs $1 more than the ball, how much does the ball cost?'
+  - vars:
+      problem: 'In a room of 23 people, what is the probability that at least two people share a birthday?'
+  - vars:
+      problem: 'A farmer has 17 sheep. All but 9 die. How many sheep does the farmer have left?'
+  - vars:
+      problem: 'If it takes 5 machines 5 minutes to make 5 widgets, how long does it take 100 machines to make 100 widgets?'

examples/amazon-bedrock/models/promptfooconfig.nova-sonic.yaml

@@ -0,0 +1,60 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: Amazon Bedrock Nova Sonic with a conversation
+
+providers:
+  - id: bedrock:amazon.nova-sonic-v1:0
+    config:
+      inferenceConfiguration:
+        maxTokens: 4096
+        temperature: 0.7
+        topP: 0.95
+      textOutputConfiguration:
+        mediaType: text/plain
+      audioInputConfiguration:
+        mediaType: audio/lpcm
+        sampleRateHertz: 16000
+        sampleSizeBits: 16
+        channelCount: 1
+        encoding: base64
+        audioType: SPEECH
+      audioOutputConfiguration:
+        mediaType: audio/lpcm
+        sampleRateHertz: 24000
+        sampleSizeBits: 16
+        channelCount: 1
+        voiceId: matthew
+        encoding: base64
+        audioType: SPEECH
+      region: us-east-1
+
+prompts:
+  - file://nova_sonic_prompt.js
+
+defaultTest:
+  vars:
+    system_message: You are a helpful assistant. Answer the question based on the audio input.
+  metadata:
+    type: text
+    conversationId: date_thread
+
+# The conversationId is a special metadata field that promptfoo uses to maintain conversation history between tests.
+# When multiple tests share the same conversationId (like "date_thread" here), each test's response is added to the
+# conversation history, allowing the model to reference previous exchanges. This enables testing of multi-turn
+# conversations where context from earlier interactions affects later responses.
+tests:
+  - vars:
+      audio_file: file://assets/hello.wav
+    metadata:
+      type: audio
+      conversationId: date_thread
+    assert:
+      - type: llm-rubric
+        value: the date and time is today
+  - vars:
+      audio_file: file://assets/weather.wav
+    metadata:
+      type: audio
+      conversationId: date_thread
+    assert:
+      - type: llm-rubric
+        value: contains a weather report

examples/amazon-bedrock/models/promptfooconfig.nova.multimodal.yaml

@@ -0,0 +1,17 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: 'Bedrock Nova Eval with Images'
+
+prompts:
+  - file://nova_multimodal_prompt.json
+
+providers:
+  - id: bedrock:amazon.nova-pro-v1:0
+    config:
+      region: 'us-east-1'
+      inferenceConfig:
+        temperature: 0.7
+        max_new_tokens: 256
+
+tests:
+  - vars:
+      image: file://cat.jpg

examples/amazon-bedrock/models/promptfooconfig.nova.tool.yaml

@@ -0,0 +1,110 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: 'Bedrock Nova Tool Use Eval'
+
+prompts:
+  - 'What is the color of {{topic}}? ONLY answer using the "color_json" tool.'
+
+providers:
+  - id: bedrock:amazon.nova-micro-v1:0
+    config:
+      region: 'us-east-1'
+      inferenceConfig:
+        temperature: 0.7
+        max_new_tokens: 256
+      toolConfig:
+        toolChoice:
+          auto: {}
+        tools:
+          - toolSpec:
+              name: color_json
+              description: 'Outputs the color of a thing'
+              inputSchema:
+                json:
+                  type: object
+                  properties:
+                    name:
+                      type: string
+                      description: 'The name of the thing'
+                    color:
+                      type: string
+                      description: 'The color of the thing'
+                  required:
+                    - name
+                    - color
+  - id: bedrock:amazon.nova-lite-v1:0
+    config:
+      region: 'us-east-1'
+      inferenceConfig:
+        temperature: 0.7
+        max_new_tokens: 256
+      toolConfig:
+        toolChoice:
+          auto: {}
+        tools:
+          - toolSpec:
+              name: color_json
+              description: 'Outputs the color of a thing'
+              inputSchema:
+                json:
+                  type: object
+                  properties:
+                    name:
+                      type: string
+                      description: 'The name of the thing'
+                    color:
+                      type: string
+                      description: 'The color of the thing'
+                  required:
+                    - name
+                    - color
+  - id: bedrock:amazon.nova-pro-v1:0
+    config:
+      region: 'us-east-1'
+      inferenceConfig:
+        temperature: 0.7
+        max_new_tokens: 256
+      toolConfig:
+        toolChoice:
+          auto: {}
+        tools:
+          - toolSpec:
+              name: color_json
+              description: 'Outputs the color of a thing'
+              inputSchema:
+                json:
+                  type: object
+                  properties:
+                    name:
+                      type: string
+                      description: 'The name of the thing'
+                    color:
+                      type: string
+                      description: 'The color of the thing'
+                  required:
+                    - name
+                    - color
+
+defaultTest:
+  options:
+    transform: 'JSON.parse(output).input.color'
+tests:
+  - vars:
+      topic: sky
+    assert:
+      - type: equals
+        value: blue
+  - vars:
+      topic: ocean
+    assert:
+      - type: equals
+        value: blue
+  - vars:
+      topic: banana
+    assert:
+      - type: equals
+        value: yellow
+  - vars:
+      topic: grass
+    assert:
+      - type: equals
+        value: green

examples/amazon-bedrock/models/promptfooconfig.nova.yaml

@@ -0,0 +1,51 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: 'Bedrock Nova Models Comparison'
+
+prompts:
+  - 'Write a tweet about {{topic}}'
+
+providers:
+  - id: bedrock:amazon.nova-lite-v1:0
+    config:
+      region: 'us-east-1'
+      inferenceConfig:
+        temperature: 0.7
+        max_new_tokens: 256
+  - id: bedrock:us.amazon.nova-premier-v1:0
+    config:
+      region: 'us-west-2'
+      inferenceConfig:
+        temperature: 0.7
+        max_new_tokens: 256
+
+tests:
+  - vars:
+      topic: Our eco-friendly packaging
+  - vars:
+      topic: A sneak peek at our secret menu item
+  - vars:
+      topic: Behind-the-scenes at our latest photoshoot
+  - vars:
+      topic: the impact of autonomous drones on wildlife conservation
+  - vars:
+      topic: the emerging trend of virtual reality courtrooms
+  - vars:
+      topic: the ethical implications of AI-generated art
+  - vars:
+      topic: the unexpected health benefits of daily meditation
+  - vars:
+      topic: how AI is changing the way we play board games
+  - vars:
+      topic: unconventional productivity hacks involving household items
+  - vars:
+      topic: An underground art exhibition in an abandoned subway station
+  - vars:
+      topic: A webinar on the impact of AI on traditional marketing strategies
+  - vars:
+      topic: The launch of a new eco-friendly sneaker made from ocean plastic
+  - vars:
+      topic: the correlation between social media usage and self-esteem in teenagers
+  - vars:
+      topic: the impact of urban noise pollution on migratory bird patterns
+  - vars:
+      topic: the role of gut microbiota in moderating anxiety and depression

examples/amazon-bedrock/models/promptfooconfig.openai-compatible.yaml

@@ -0,0 +1,74 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: 'Bedrock OpenAI-compatible model families (GLM, MiniMax, Kimi, Nemotron, Gemma, Palmyra)'
+
+# These newer Bedrock families all speak the OpenAI Chat Completions schema over InvokeModel,
+# so they share the same provider config. Confirm each model is enabled in your region with
+# `aws bedrock list-foundation-models --region us-east-1`.
+
+prompts:
+  - 'Answer in one short sentence: {{question}}'
+
+providers:
+  # Z.AI GLM
+  - id: bedrock:zai.glm-5
+    label: GLM 5
+    config:
+      region: us-east-1
+      max_tokens: 1024
+
+  # MiniMax (reasoning model — give it a larger token budget so the answer is not truncated)
+  - id: bedrock:minimax.minimax-m2
+    label: MiniMax M2
+    config:
+      region: us-east-1
+      max_tokens: 2048
+      showThinking: false # strip the <reasoning> block, keep only the final answer
+
+  # Moonshot Kimi
+  - id: bedrock:moonshotai.kimi-k2.5
+    label: Kimi K2.5
+    config:
+      region: us-east-1
+      max_tokens: 1024
+
+  # NVIDIA Nemotron — reasons in-line without tags, so it needs a larger token budget.
+  # To get a direct answer, prepend a `/no_think` system message (NVIDIA's reasoning toggle);
+  # `showThinking` has no tagged block to strip here.
+  - id: bedrock:nvidia.nemotron-nano-9b-v2
+    label: Nemotron Nano 9B
+    config:
+      region: us-east-1
+      max_tokens: 2048
+
+  # Google Gemma 3
+  - id: bedrock:google.gemma-3-12b-it
+    label: Gemma 3 12B
+    config:
+      region: us-east-1
+      max_tokens: 1024
+
+  # Writer Palmyra — on-demand throughput is served through the `us.` inference profile
+  - id: bedrock:us.writer.palmyra-x5-v1:0
+    label: Palmyra X5
+    config:
+      region: us-east-1
+      max_tokens: 1024
+
+tests:
+  - vars:
+      question: What is the capital of France?
+    assert:
+      - type: icontains
+        value: Paris
+
+  - vars:
+      question: What is 2 + 2?
+    assert:
+      - type: contains
+        value: '4'
+
+  - vars:
+      question: Name the largest planet in our solar system.
+    assert:
+      - type: icontains
+        value: Jupiter

examples/amazon-bedrock/models/promptfooconfig.openai-frontier.yaml

@@ -0,0 +1,53 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: 'OpenAI frontier models (GPT-5.5 / GPT-5.4) via AWS Bedrock'
+
+# Frontier models are served through Bedrock's OpenAI-compatible Responses API and
+# authenticate with an Amazon Bedrock API key. Export it before running (promptfoo reads it
+# automatically — no apiKey field needed):
+#   export AWS_BEARER_TOKEN_BEDROCK="..."
+
+prompts:
+  - |
+    Answer the following question concisely and accurately.
+
+    Question: {{question}}
+
+providers:
+  # GPT-5.5 is available in us-east-2 (Ohio). promptfoo routes bedrock:openai.gpt-5.x to
+  # the OpenAI-compatible Responses API on the regional Bedrock mantle endpoint.
+  - id: bedrock:openai.gpt-5.5
+    label: GPT-5.5 (high reasoning)
+    config:
+      region: us-east-2
+      reasoning_effort: high
+      max_output_tokens: 2048
+  # GPT-5.4 is available in us-east-2 (Ohio) and us-west-2 (Oregon).
+  - id: bedrock:openai.gpt-5.4
+    label: GPT-5.4 (low reasoning)
+    config:
+      region: us-east-2
+      # Valid: none | low | medium | high | xhigh (minimal is NOT supported)
+      reasoning_effort: low
+      max_output_tokens: 1024
+
+tests:
+  - vars:
+      question: What is the capital of France? Reply with only the city name.
+    assert:
+      - type: icontains
+        value: Paris
+  - vars:
+      question: 'If a train travels 120 miles in 2 hours, what is its average speed in mph? Reply with just the number and unit.'
+    assert:
+      - type: contains
+        value: '60'
+      - type: icontains-any
+        value: ['mph', 'miles per hour']
+  - vars:
+      question: 'Explain the concept of machine learning in two sentences.'
+    assert:
+      # Deterministic assertions keep this example runnable with only a Bedrock API key.
+      # (A model-graded assertion like `llm-rubric` would require a separate grader provider
+      # with its own credentials — see https://promptfoo.dev/docs/configuration/expected-outputs/model-graded/)
+      - type: icontains-any
+        value: ['data', 'model', 'train', 'learn', 'pattern']

examples/amazon-bedrock/models/promptfooconfig.openai.yaml

@@ -0,0 +1,52 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: 'OpenAI GPT-OSS Models via AWS Bedrock'
+
+prompts:
+  - |
+    You are a helpful AI assistant. Answer the following question concisely and accurately.
+
+    Question: {{question}}
+
+providers:
+  - id: bedrock:openai.gpt-oss-120b-1:0
+    label: OpenAI GPT-OSS 120B
+    config:
+      region: us-west-2
+      max_completion_tokens: 1024
+      temperature: 0.7
+      reasoning_effort: medium
+  - id: bedrock:openai.gpt-oss-20b-1:0
+    label: OpenAI GPT-OSS 20B
+    config:
+      region: us-west-2
+      max_completion_tokens: 1024
+      temperature: 0.7
+      reasoning_effort: low
+  - id: bedrock:openai.gpt-oss-120b-1:0
+    label: OpenAI GPT-OSS 120B High Reasoning
+    config:
+      region: us-west-2
+      max_completion_tokens: 2048
+      temperature: 0.3
+      reasoning_effort: high
+
+tests:
+  - vars:
+      question: What is the capital of France?
+    assert:
+      - type: contains
+        value: Paris
+  - vars:
+      question: Explain the concept of machine learning in simple terms.
+    assert:
+      - type: contains-all
+        value: ['learn', 'data', 'algorithm']
+  - vars:
+      question: 'Solve this step by step: If a train travels 120 miles in 2 hours, what is its average speed?'
+    assert:
+      - type: contains
+        value: '60'
+      - type: contains-any
+        value: ['mph', 'miles per hour']
+      - type: llm-rubric
+        value: 'Response should show step-by-step calculation and arrive at 60 mph'

examples/amazon-bedrock/models/promptfooconfig.qwen.yaml

@@ -0,0 +1,122 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: 'Bedrock Qwen Models Evaluation'
+
+prompts:
+  - file://prompts.txt
+
+providers:
+  # Qwen3 Coder 480B - Large MoE model optimized for coding and agentic tasks
+  - id: bedrock:qwen.qwen3-coder-480b-a35b-v1:0
+    label: Qwen3 Coder 480B
+    config:
+      region: us-west-2
+      max_tokens: 2048
+      temperature: 0.7
+      top_p: 0.9
+      showThinking: true # Set to false to hide thinking content
+
+  # Claude for comparison
+  - id: bedrock:us.anthropic.claude-sonnet-4-6
+    label: Claude Sonnet 4.6
+    config:
+      region: us-west-2
+      max_tokens: 2048
+      temperature: 0.7
+
+  # Qwen3 Coder 30B - Smaller MoE model for efficient coding tasks
+  - id: bedrock:qwen.qwen3-coder-30b-a3b-v1:0
+    label: Qwen3 Coder 30B
+    config:
+      region: us-west-2
+      max_tokens: 1024
+      temperature: 0.5
+      top_p: 0.9
+
+  # Qwen3 235B - General purpose MoE model for reasoning and coding
+  - id: bedrock:qwen.qwen3-235b-a22b-2507-v1:0
+    label: Qwen3 235B
+    config:
+      region: us-west-2
+      max_tokens: 1024
+      temperature: 0.7
+      top_p: 0.9
+
+  # Qwen3 32B Dense - Dense model for consistent performance
+  - id: bedrock:qwen.qwen3-32b-v1:0
+    label: Qwen3 32B Dense
+    config:
+      region: us-east-1
+      max_tokens: 1024
+      temperature: 0.7
+      top_p: 0.9
+
+  # Example with tool calling configuration
+  - id: bedrock:qwen.qwen3-coder-480b-a35b-v1:0
+    label: Qwen3 Coder 480B with Tools
+    config:
+      region: us-west-2
+      max_tokens: 2048
+      temperature: 0.3
+      tools:
+        - type: function
+          function:
+            name: calculate
+            description: Perform arithmetic calculations
+            parameters:
+              type: object
+              properties:
+                expression:
+                  type: string
+                  description: The mathematical expression to evaluate
+              required: ['expression']
+      tool_choice: auto
+
+tests:
+  - vars:
+      question: Write a Python function to calculate the factorial of a number
+    assert:
+      - type: contains
+        value: 'def'
+      - type: contains
+        value: 'factorial'
+
+  - vars:
+      question: Explain the concept of recursion with a simple example
+    assert:
+      - type: contains
+        value: 'recursion'
+      - type: llm-rubric
+        value: Response clearly explains recursion with a concrete example
+
+  - vars:
+      question: What are the benefits of using mixture-of-experts models?
+    assert:
+      - type: contains
+        value: 'efficiency'
+
+  - vars:
+      question: Debug this code and explain the issue - "for i in range(10) print(i)"
+    assert:
+      - type: contains
+        value: 'syntax'
+      - type: contains
+        value: ':'
+
+  - vars:
+      question: Create a REST API endpoint using FastAPI for user authentication
+    assert:
+      - type: contains
+        value: 'FastAPI'
+      - type: contains
+        value: 'authentication'
+
+  # Test tool calling specifically with the tools provider
+  - vars:
+      question: Calculate the result of 15 * 8 + 42
+    assert:
+      - type: contains
+        value: 'Called function calculate'
+      - type: contains
+        value: '15 * 8 + 42'
+    options:
+      provider: bedrock:qwen.qwen3-coder-480b-a35b-v1:0:Qwen3 Coder 480B with Tools

examples/amazon-bedrock/models/promptfooconfig.yaml

@@ -0,0 +1,128 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: 'Bedrock Eval'
+
+prompts:
+  - file://prompts.txt
+
+providers:
+  - id: bedrock:us.anthropic.claude-sonnet-4-6
+    label: Claude Sonnet 4.6
+    config:
+      region: us-west-2
+      max_tokens: 2048
+      temperature: 0
+  - id: bedrock:us.meta.llama4-scout-17b-instruct-v1:0
+    label: Llama 4 Scout 17B
+  - id: bedrock:amazon.nova-pro-v1:0
+    label: Nova Pro
+  - id: bedrock:mistral.pixtral-large-2502-v1:0
+    label: Pixtral Large 25.02
+    config:
+      region: us-west-2
+      max_tokens: 1024
+      temperature: 0
+
+defaultTest:
+  options:
+    provider:
+      # Override the embeddings provider for all assertion types that require it (such as similarity)
+      embedding:
+        id: bedrock:embeddings:amazon.titan-embed-text-v2:0
+        config:
+          region: us-east-1
+
+tests:
+  - vars:
+      question: What's the weather in New York?
+    assert:
+      # This uses the embeddings provider set above
+      - type: similar
+        value: sunny
+  - vars:
+      question: Who won the latest football match between the Giants and 49ers?
+  - vars:
+      question: "Which magazine was started first Arthur's Magazine or First for Women?"
+  - vars:
+      question: 'The Oberoi family is part of a hotel company that has a head office in what city?'
+  - vars:
+      question: 'Musician and satirist Allie Goertz wrote a song about the "The Simpsons" character Milhouse, who Matt Groening named after who?'
+  - vars:
+      question: "What nationality was James Henry Miller's wife?"
+  - vars:
+      question: 'Cadmium Chloride is slightly soluble in this chemical, it is also called what?'
+  - vars:
+      question: 'Which tennis player won more Grand Slam titles, Henri Leconte or Jonathan Stark?'
+  - vars:
+      question: "Which genus of moth in the world's seventh-largest country contains only one species?"
+  - vars:
+      question: 'Who was once considered the best kick boxer in the world, however he has been involved in a number of controversies relating to his "unsportsmanlike conducts" in the sport and crimes of violence outside of the ring.'
+  - vars:
+      question: 'The Dutch-Belgian television series that "House of Anubis" was based on first aired in what year?'
+  - vars:
+      question: 'What is the length of the track where the 2013 Liqui Moly Bathurst 12 Hour was staged?'
+  - vars:
+      question: 'Fast Cars, Danger, Fire and Knives includes guest appearances from which hip hop record executive?'
+  - vars:
+      question: 'Gunmen from Laredo starred which narrator of "Frontier"?'
+  - vars:
+      question: 'Where did the form of music played by Die Rhöner Säuwäntzt originate?'
+  - vars:
+      question: 'In which American football game was Malcolm Smith named Most Valuable player?'
+  - vars:
+      question: 'What U.S Highway gives access to Zilpo Road, and is also known as Midland Trail?'
+  - vars:
+      question: 'The 1988 American comedy film, The Great Outdoors, starred a four-time Academy Award nominee, who received a star on the Hollywood Walk of Fame in what year?'
+  - vars:
+      question: 'What are the names of the current members of  American heavy metal band who wrote the music for  Hurt Locker The Musical?'
+  - vars:
+      question: 'Human Error" is the season finale of the third season of a tv show that aired on what network?'
+  - vars:
+      question: 'Dua Lipa, an English singer, songwriter and model, the album spawned the number-one single "New Rules" is a song by English singer Dua Lipa from her eponymous debut studio album, released in what year?'
+  - vars:
+      question: 'American politician Joe Heck ran unsuccessfully against Democrat Catherine Cortez Masto, a woman who previously served as the 32nd Attorney General of where?'
+  - vars:
+      question: 'Which state does the drug stores, of which the CEO is Warren Bryant, are located?'
+  - vars:
+      question: 'Which  American politician did Donahue replaced '
+  - vars:
+      question: 'Which band was founded first, Hole, the rock band that Courtney Love was a frontwoman of, or The Wolfhounds?'
+  - vars:
+      question: 'How old is the female main protagonist of Catching Fire?'
+  - vars:
+      question: 'Chang Ucchin was born in korea during a time that ended with the conclusion of what?'
+  - vars:
+      question: 'Who is the director of the 2003 film which has scenes in it filmed at the Quality Cafe in Los Angeles?'
+  - vars:
+      question: "Which actress played the part of fictitious character Kimberly Ann Hart, in the franchise built around a live action superhero television series taking much of its footage from the Japanese tokusatsu 'Super Sentai'?"
+  - vars:
+      question: 'Who was born first, Pablo Trapero or Aleksander Ford?'
+  - vars:
+      question: "Are Jane and First for Women both women's magazines?"
+  - vars:
+      question: 'What profession does Nicholas Ray and Elia Kazan have in common?'
+  - vars:
+      question: 'Where is the company that purchased Aixam based in?'
+  - vars:
+      question: 'Which documentary is about Finnish rock groups, Adam Clayton Powell or The Saimaa Gesture?'
+  - vars:
+      question: 'Who was inducted into the Rock and Roll Hall of Fame, David Lee Roth or Cia Berg?'
+  - vars:
+      question: "Zimbabwe's Guwe Secondary School has a sister school in what New York county?"
+  - vars:
+      question: 'Who is the director of the 2003 film which has scenes in it filmed at the Quality Cafe in Los Angeles?'
+  - vars:
+      question: "Which actress played the part of fictitious character Kimberly Ann Hart, in the franchise built around a live action superhero television series taking much of its footage from the Japanese tokusatsu 'Super Sentai'?"
+  - vars:
+      question: 'Who was born first, Pablo Trapero or Aleksander Ford?'
+  - vars:
+      question: "Are Jane and First for Women both women's magazines?"
+  - vars:
+      question: 'What profession does Nicholas Ray and Elia Kazan have in common?'
+  - vars:
+      question: 'Where is the company that purchased Aixam based in?'
+  - vars:
+      question: 'Which documentary is about Finnish rock groups, Adam Clayton Powell or The Saimaa Gesture?'
+  - vars:
+      question: 'Who was inducted into the Rock and Roll Hall of Fame, David LeRoth or Cia Berg?'
+  - vars:
+      question: "Zimbabwe's Guwe Secondary School has a sister school in what New York county?"

examples/amazon-bedrock/models/prompts.txt

@@ -0,0 +1,9 @@
+You are a helpful assistant. Reply with a concise answer to this inquiry: "{{question}}"
+---
+You are a helpful assistant. Reply with a concise answer to this inquiry: "{{question}}"
+
+- Think carefully & step-by-step.
+- Only use information available on Wikipedia.
+- You must answer the question directly, without speculation.
+- You cannot access realtime information. Consider whether the answer may have changed in the 2 years since your knowledge cutoff.
+- If you are not confident in your answer, begin your response with "Unsure".

examples/amazon-bedrock/video/README.md

@@ -0,0 +1,135 @@
+# amazon-bedrock/video (AWS Bedrock Video Generation)
+
+You can run this example with:
+
+```bash
+npx promptfoo@latest init --example amazon-bedrock/video
+cd amazon-bedrock/video
+```
+
+Video generation examples using AWS Bedrock's async invoke API.
+
+## Available Models
+
+| Model            | Config                           | Region         | Duration  |
+| ---------------- | -------------------------------- | -------------- | --------- |
+| Amazon Nova Reel | `promptfooconfig.nova-reel.yaml` | us-east-1      | 6s - 2min |
+| Luma Ray 2       | `promptfooconfig.luma-ray.yaml`  | us-west-2 only | 5s or 9s  |
+
+## Prerequisites
+
+Video generation requires additional AWS setup beyond standard Bedrock access.
+
+### 1. Enable Model Access
+
+Visit the [AWS Bedrock Model Access page](https://console.aws.amazon.com/bedrock/home#/modelaccess) and enable:
+
+- **Nova Reel**: `amazon.nova-reel-v1:1` (us-east-1)
+- **Luma Ray 2**: `luma.ray-v2:0` (us-west-2)
+
+### 2. Create S3 Bucket
+
+Video outputs are written to S3. Create a bucket in the same region as your model:
+
+```bash
+# For Nova Reel (us-east-1)
+aws s3 mb s3://your-bucket-nova-reel --region us-east-1
+
+# For Luma Ray 2 (us-west-2)
+aws s3 mb s3://your-bucket-luma-ray --region us-west-2
+```
+
+### 3. Configure IAM Permissions
+
+```json
+{
+  "Version": "2012-10-17",
+  "Statement": [
+    {
+      "Effect": "Allow",
+      "Action": ["bedrock:InvokeModel", "bedrock:StartAsyncInvoke", "bedrock:GetAsyncInvoke"],
+      "Resource": [
+        "arn:aws:bedrock:*:*:model/amazon.nova-reel-v1:1",
+        "arn:aws:bedrock:*:*:model/luma.ray-v2:0"
+      ]
+    },
+    {
+      "Effect": "Allow",
+      "Action": ["s3:PutObject", "s3:GetObject"],
+      "Resource": "arn:aws:s3:::your-bucket/*"
+    }
+  ]
+}
+```
+
+### 4. Install Dependencies
+
+```bash
+npm install @aws-sdk/client-bedrock-runtime @aws-sdk/client-s3
+```
+
+## Usage
+
+Update `s3OutputUri` in the config file, then run:
+
+```bash
+# Nova Reel
+npx promptfoo@latest eval -c promptfooconfig.nova-reel.yaml
+
+# Luma Ray 2
+npx promptfoo@latest eval -c promptfooconfig.luma-ray.yaml
+```
+
+## Model Comparison
+
+### Amazon Nova Reel
+
+- **Best for**: Longer videos, multi-shot narratives
+- **Resolution**: 1280x720 @ 24 FPS
+- **Duration**: 6 seconds (single shot) or 12-120 seconds (multi-shot)
+- **Features**: TEXT_VIDEO, MULTI_SHOT_AUTOMATED, MULTI_SHOT_MANUAL modes
+- **Typical generation time**: ~90 seconds for 6s video
+
+### Luma Ray 2
+
+- **Best for**: High-quality short clips, image-to-video
+- **Resolution**: 540p or 720p
+- **Aspect ratios**: 1:1, 16:9, 9:16, 4:3, 3:4, 21:9, 9:21
+- **Duration**: 5 or 9 seconds
+- **Features**: Start/end frame keyframes, loop mode
+- **Typical generation time**: ~2-3 minutes
+
+## Configuration Options
+
+### Nova Reel
+
+```yaml
+config:
+  region: us-east-1
+  s3OutputUri: s3://your-bucket/outputs/
+  durationSeconds: 6 # 6 for single, 12-120 for multi-shot
+  taskType: TEXT_VIDEO # or MULTI_SHOT_AUTOMATED, MULTI_SHOT_MANUAL
+  seed: 12345 # Optional, for reproducibility
+  image: file://./start-frame.jpg # Optional, for image-to-video
+```
+
+### Luma Ray 2
+
+```yaml
+config:
+  region: us-west-2
+  s3OutputUri: s3://your-bucket/outputs/
+  duration: '5s' # or '9s'
+  resolution: '720p' # or '540p'
+  aspectRatio: '16:9'
+  loop: false
+  startImage: file://./start.jpg # Optional
+  endImage: file://./end.jpg # Optional
+```
+
+## Resources
+
+- [AWS Bedrock Video Generation](https://docs.aws.amazon.com/bedrock/latest/userguide/video-generation.html)
+- [Nova Reel Documentation](https://docs.aws.amazon.com/nova/latest/userguide/video-generation.html)
+- [Luma Ray Documentation](https://docs.aws.amazon.com/bedrock/latest/userguide/model-parameters-luma.html)
+- [promptfoo AWS Bedrock Provider](https://promptfoo.dev/docs/providers/aws-bedrock)

examples/amazon-bedrock/video/promptfooconfig.luma-ray.yaml

@@ -0,0 +1,36 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: 'Luma Ray 2 Video Generation on AWS Bedrock'
+
+# Luma Ray 2 produces high-quality videos from text prompts.
+# Videos are generated asynchronously and stored in S3.
+#
+# Prerequisites:
+#   1. Enable Luma Ray 2 model access in AWS Bedrock console
+#   2. Create an S3 bucket for video output
+#   3. Configure IAM permissions for bedrock:StartAsyncInvoke,
+#      bedrock:GetAsyncInvoke, s3:PutObject, s3:GetObject
+#
+# Update s3OutputUri below to point to your S3 bucket.
+
+providers:
+  - id: bedrock:video:luma.ray-v2:0
+    config:
+      region: us-west-2 # Luma Ray 2 is currently only available in us-west-2
+      s3OutputUri: s3://your-bucket-name/luma-ray-outputs/
+      duration: '5s'
+      resolution: '720p'
+      aspectRatio: '16:9'
+
+prompts:
+  # Promptfoo red panda mascot themed prompts
+  - 'A cute red panda wearing glasses typing on a glowing laptop in a cozy tech office, soft purple and teal lighting, modern tech aesthetic, the red panda looks focused and intelligent'
+  - 'An adorable red panda in a lab coat examining holographic data visualizations, futuristic tech environment with orange and purple accent lights, cinematic lighting'
+  - 'A friendly red panda mascot waving at the camera in front of a wall of code and security shields, modern office setting, warm lighting, professional but approachable style'
+
+tests:
+  - vars: {}
+    assert:
+      - type: contains
+        value: 'Video:' # Output format is [Video: prompt](url)
+      - type: not-contains
+        value: 'error' # No errors in output

examples/amazon-bedrock/video/promptfooconfig.nova-reel.yaml

@@ -0,0 +1,26 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: Amazon Nova Reel Video Generation
+
+providers:
+  - id: bedrock:video:amazon.nova-reel-v1:1
+    config:
+      region: us-east-1
+      # Required: S3 bucket for video output
+      # Update this to your own S3 bucket
+      s3OutputUri: s3://your-bucket/nova-reel-outputs
+      durationSeconds: 6
+      # Optional: Set seed for reproducible results
+      # seed: 12345
+
+prompts:
+  # Promptfoo red panda mascot themed prompts
+  - 'A cute red panda sitting at a desk reviewing documents, wearing small round glasses, soft warm lighting, camera slowly zooms in on the focused expression'
+  - 'An adorable red panda mascot walking through a futuristic data center with glowing servers, purple and teal ambient lighting, camera follows alongside'
+
+tests:
+  - vars: {}
+    assert:
+      - type: contains
+        value: 'Video:' # Output format is [Video: prompt](url)
+      - type: not-contains
+        value: 'error' # No errors in output

examples/anthropic/claude-code-session/README.md

@@ -0,0 +1,63 @@
+# anthropic/claude-code-session (Authenticate via Claude Code session)
+
+This example shows how to run Promptfoo evals against the Anthropic Messages
+API — including `llm-rubric` model-graded assertions — by reusing an existing
+local Claude Code session instead of creating a separate Anthropic Console API
+key.
+
+It's meant for Claude Pro / Max subscribers who already use the
+[`claude` CLI](https://docs.claude.com/en/docs/claude-code/overview) on the
+same machine.
+
+You can run this example with:
+
+```bash
+npx promptfoo@latest init --example anthropic/claude-code-session
+cd anthropic/claude-code-session
+```
+
+## Prerequisites
+
+1. Install and log in to Claude Code:
+
+   ```bash
+   claude /login
+   ```
+
+   Promptfoo reads the OAuth credential that Claude Code stores in either the
+   macOS keychain (`Claude Code-credentials`) or
+   `$HOME/.claude/.credentials.json`.
+
+2. Make sure `ANTHROPIC_API_KEY` is **unset** if you specifically want
+   Promptfoo to use your Claude Code session. If the env var is set, Promptfoo
+   will prefer it over the OAuth credential.
+
+## How it works
+
+The provider config sets `apiKeyRequired: false`, which tells Promptfoo to:
+
+- Skip its upfront API key check.
+- Load the Claude Code OAuth credential from the local session.
+- Authenticate Messages API requests with a Bearer token plus the
+  `claude-code-20250219,oauth-2025-04-20` beta headers.
+- Prepend the required Claude Code identity system block
+  (`"You are Claude Code, Anthropic's official CLI for Claude."`) before your
+  own system prompt.
+
+Requests made this way are expected to count against your Claude
+subscription the same way calls from the `claude` CLI do. Check
+[Anthropic's documentation](https://docs.claude.com/en/docs/claude-code/overview)
+for current billing behavior.
+
+## Run it
+
+```bash
+promptfoo eval
+promptfoo view
+```
+
+## Configuration
+
+See `promptfooconfig.yaml` — both the main provider and the `llm-rubric`
+grader point at `anthropic:messages:claude-sonnet-4-6` with
+`apiKeyRequired: false` so the entire eval runs on your Claude subscription.

examples/anthropic/claude-code-session/promptfooconfig.yaml

@@ -0,0 +1,47 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: |
+  Run evals against Claude via a local Claude Code session (no Anthropic
+  Console API key needed). Requires `claude /login` to have been run on the
+  same machine.
+
+prompts:
+  - |
+    You are a careful technical writer. Rewrite the following sentence for
+    clarity without changing its meaning:
+
+    "{{sentence}}"
+
+providers:
+  - id: anthropic:messages:claude-sonnet-4-6
+    config:
+      # Skip the API key preflight and authenticate via the local Claude Code
+      # OAuth credential (macOS keychain or ~/.claude/.credentials.json).
+      apiKeyRequired: false
+      max_tokens: 512
+
+# Use the same Claude Code session to run the llm-rubric grader. This lets
+# Claude Pro / Max subscribers use semantic model-graded evals without a
+# separate Anthropic Console API key.
+defaultTest:
+  options:
+    provider:
+      id: anthropic:messages:claude-sonnet-4-6
+      config:
+        apiKeyRequired: false
+
+tests:
+  - vars:
+      sentence: 'Utilizing these tools, we can facilitate the optimization of the process.'
+    assert:
+      - type: llm-rubric
+        value: |
+          The rewrite is concise, uses plain English, and preserves the
+          original meaning (improving a process using the tools). It should
+          not invent new details.
+  - vars:
+      sentence: 'The aforementioned refactor exhibits suboptimal efficiency characteristics.'
+    assert:
+      - type: llm-rubric
+        value: |
+          The rewrite is clearer and less jargon-heavy while still saying the
+          refactor is inefficient or slow. It must not add unrelated claims.

examples/anthropic/fable-5-coding/README.md

@@ -0,0 +1,54 @@
+# anthropic/fable-5-coding (Claude Fable 5 Advanced Coding)
+
+This example exercises Claude Fable 5 on hard coding tasks using the `xhigh` effort level. Fable 5 always uses adaptive thinking, so no `thinking` configuration is needed.
+
+You can run this example with:
+
+```bash
+npx promptfoo@latest init --example anthropic/fable-5-coding
+cd fable-5-coding
+```
+
+## What This Tests
+
+Claude Fable 5 is Anthropic's most powerful model — a new tier above Opus. This example evaluates:
+
+- **Distributed-systems debugging** with incomplete information
+- **Production-quality code generation** with concurrency concerns
+- **Security-focused code review** with prioritized feedback
+
+## Working with Fable 5
+
+- **Adaptive thinking is always on.** Unlike Opus 4.7/4.8 (where adaptive thinking is opt-in), Fable 5 always thinks adaptively. There is nothing to configure: promptfoo converts a legacy `thinking: { type: enabled, budget_tokens: N }` config to adaptive and omits `thinking: { type: disabled }`, because the model rejects both.
+- **Thinking summaries are opt-in.** By default the API omits thinking content (an empty thinking block, which promptfoo excludes from output). Set `thinking: { type: adaptive, display: summarized }` to include a readable summary.
+- **Sampling controls are managed for you.** Fable 5 rejects `temperature`, `top_p`, and `top_k` at the model level; promptfoo omits them automatically (don't set them in config).
+- **`effort` defaults to `high`; `xhigh` is available.** Start with `xhigh` for coding and agentic work, and pair high effort with a large `max_tokens` — thinking consumes output tokens before any visible text, so a tight budget can be spent entirely on thinking and yield an empty response.
+- **1M-token context, up to 128K output tokens.** Priced at $10/$50 per million input/output tokens.
+
+## Running the Example
+
+```bash
+# Set your API key
+export ANTHROPIC_API_KEY=your_api_key_here
+
+# Run the evaluation
+npx promptfoo@latest eval
+
+# View results
+npx promptfoo@latest view
+```
+
+## Other providers
+
+Fable 5 is also reachable through:
+
+- AWS Bedrock — `bedrock:global.anthropic.claude-fable-5` (Runtime/Converse; requires the account to opt in to provider data sharing via `aws bedrock put-account-data-retention --mode provider_data_share`), or `bedrock:messages:anthropic.claude-fable-5` for Bedrock's Anthropic-compatible Messages endpoint in `us-east-1`/`eu-north-1`
+- Google Vertex — `vertex:claude-fable-5`
+- Azure AI Foundry — point `anthropic:messages:claude-fable-5` at `https://<resource>.services.ai.azure.com/anthropic` via `apiBaseUrl`
+
+Across all providers, promptfoo automatically omits the unsupported sampling parameters and normalizes unsupported thinking configs for Fable 5. Note that Bedrock's regional and geo endpoints and Vertex's regional/multi-region endpoints (everything except `global`) carry a 10% price premium, which promptfoo includes in cost calculations.
+
+## Learn More
+
+- [Anthropic documentation](https://docs.anthropic.com)
+- [Promptfoo Anthropic provider docs](https://promptfoo.dev/docs/providers/anthropic)

examples/anthropic/fable-5-coding/promptfooconfig.yaml

@@ -0,0 +1,110 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: Claude Fable 5 advanced coding with xhigh effort and always-on adaptive thinking
+
+prompts:
+  - |
+    {{task}}
+
+providers:
+  - id: anthropic:messages:claude-fable-5
+    config:
+      # Fable 5 always uses adaptive thinking — there is no `thinking` block to
+      # set. Manual budgets (`type: enabled`) are converted to adaptive and
+      # `type: disabled` is omitted because thinking cannot be turned off.
+      #
+      # Fable 5 also rejects manual sampling controls (temperature/top_p/top_k)
+      # at the model level — promptfoo omits them automatically, so don't set
+      # them here.
+      effort: xhigh # Recommended starting point for coding/agentic work (between high and max)
+      # Always-on adaptive thinking consumes output tokens before any visible
+      # text. Budget max_tokens generously — at xhigh effort a tight budget can
+      # be spent entirely on thinking, yielding an empty response.
+      max_tokens: 16000
+
+tests:
+  # Distributed-systems debugging with incomplete information
+  - vars:
+      task: |
+        A payment service intermittently double-charges customers. Here's what you know:
+
+        1. The charge endpoint is idempotent — it checks for an existing charge by idempotency key before creating one
+        2. The idempotency check reads from a Postgres replica; writes go to the primary
+        3. Replication lag spikes to 2-3 seconds during peak traffic
+        4. Clients retry on timeout with the same idempotency key
+        5. Double charges only happen during peak hours
+        6. Each double charge shows two rows with the same idempotency key but different charge IDs
+
+        Diagnose the root cause and propose a fix. Explain why the idempotency check fails despite using the same key.
+    assert:
+      - type: contains-any
+        value: ['replica', 'replication lag', 'read-after-write', 'primary', 'race']
+        reason: Should identify the stale-replica read as the root cause
+      - type: llm-rubric
+        value: |
+          The response should:
+          1. Identify the root cause (idempotency check reads a lagging replica, so a retry during the lag window misses the in-flight charge)
+          2. Explain why it only happens at peak (replication lag exceeds the client retry interval)
+          3. Propose concrete fixes (read idempotency checks from the primary, unique constraint on idempotency key, or insert-first/conflict-based idempotency)
+          4. Note that a unique constraint is the only fix that closes the race completely
+
+  # Production-quality code generation with concurrency concerns
+  - vars:
+      task: |
+        Write a TypeScript class implementing a token-bucket rate limiter that:
+        1. Supports a configurable capacity and refill rate
+        2. Is safe to call concurrently from async code
+        3. Exposes acquire() that resolves when a token is available (with optional timeout)
+        4. Avoids busy-waiting and timer leaks
+
+        Include proper typing and comments explaining design decisions.
+    assert:
+      - type: contains
+        value: 'class'
+        reason: Should define a TypeScript class
+      - type: contains-any
+        value: ['Promise', 'async', 'await']
+        reason: Should use async primitives
+      - type: llm-rubric
+        value: |
+          The code should:
+          1. Implement correct token-bucket math (refill proportional to elapsed time, capped at capacity)
+          2. Queue waiters instead of busy-waiting
+          3. Clean up timers when acquire() times out or resolves
+          4. Use precise TypeScript types (no `any`)
+          5. Include comments explaining the concurrency-safety reasoning
+          6. Be production-ready (not a toy example)
+
+  # Code review with prioritized, nuanced feedback
+  - vars:
+      task: |
+        Review this Express handler and provide feedback:
+
+        ```js
+        app.post('/upload', async (req, res) => {
+          const file = req.files.document;
+          const dest = path.join('/uploads', req.body.filename);
+          await file.mv(dest);
+          const meta = JSON.parse(req.body.metadata);
+          db.query(`INSERT INTO uploads (path, owner) VALUES ('${dest}', '${meta.owner}')`);
+          res.json({ ok: true, path: dest });
+        });
+        ```
+
+        Identify issues, suggest improvements, and explain the reasoning behind each suggestion. Prioritize by severity.
+    assert:
+      - type: contains-any
+        value: ['injection', 'traversal', 'sanitize', 'parameterized']
+        reason: Should identify the SQL injection and path traversal vulnerabilities
+      - type: llm-rubric
+        value: |
+          The review should identify multiple issues, prioritized by severity:
+          1. SQL injection via string-interpolated query (critical)
+          2. Path traversal via user-controlled filename (critical)
+          3. Unhandled JSON.parse exception on malformed metadata
+          4. No validation that req.files.document exists
+          5. Unawaited/unchecked db.query result
+
+          For each issue, it should:
+          - Explain why it's a problem
+          - Suggest specific improvements (parameterized queries, basename/allowlist for paths)
+          - Provide example code where helpful

examples/anthropic/mcp/README.md

@@ -0,0 +1,48 @@
+# anthropic/mcp (Anthropic Messages + MCP tools)
+
+This example wires Claude up to a [Model Context Protocol](https://modelcontextprotocol.io) server through the Anthropic Messages provider. Promptfoo exposes the MCP server's tools to Claude, executes any `tool_use` blocks the model emits, and feeds the `tool_result` back into the conversation until Claude produces a final reply.
+
+```bash
+npx promptfoo@latest init --example anthropic/mcp
+cd anthropic/mcp
+```
+
+## Setup
+
+```bash
+export ANTHROPIC_API_KEY=sk-ant-...
+npx promptfoo@latest eval
+```
+
+The bundled config uses the public [deepwiki MCP server](https://mcp.deepwiki.com/) so the example works out of the box without installing anything. To swap in your own server, replace the `mcp` block:
+
+```yaml
+mcp:
+  enabled: true
+  # Local stdio server
+  server:
+    command: npx
+    args: ['-y', '@modelcontextprotocol/server-filesystem', '/tmp/workspace']
+  # …or remote SSE / streamable HTTP
+  # servers:
+  #   - name: my-server
+  #     url: https://example.com/mcp
+```
+
+## What it demonstrates
+
+- **Tool discovery**: Tools exposed by the MCP server are forwarded to Claude alongside any inline `tools` you define.
+- **Multi-round execution**: When Claude returns a `tool_use` block matching an MCP tool, promptfoo invokes the tool with the model's arguments and appends a `tool_result` on the next user turn. The loop continues until Claude returns text — bounded by `max_tool_calls` (default `8`).
+- **Error pass-through**: Tool errors come back as `tool_result` blocks with `is_error: true` so Claude can recover or report the failure.
+- **Mixed tool handling**: Non-MCP tools (regular function-calling tools, or built-ins like `web_search`) fall through to the existing tool-use output without being auto-executed.
+
+## Caching
+
+Promptfoo's disk response cache is **skipped automatically** when `mcp.enabled` is `true`, because tool results can vary between runs. Use `max_tool_calls` to bound the per-request cost.
+
+## See also
+
+- [Anthropic provider docs](https://promptfoo.dev/docs/providers/anthropic/#model-context-protocol-mcp)
+- [MCP integration guide](https://promptfoo.dev/docs/integrations/mcp/) — full server configuration (auth, timeouts, multi-server)
+- [examples/openai-mcp](../../openai-mcp/) — the same pattern via the OpenAI Responses API
+- [examples/simple-mcp](../../simple-mcp/) — testing an MCP server directly without an LLM provider

examples/anthropic/mcp/promptfooconfig.yaml

@@ -0,0 +1,46 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: 'Anthropic Messages provider with MCP tool execution'
+
+prompts:
+  - 'Answer the question using the available MCP tools, then summarize what you found in 2-3 sentences. Question: {{question}}'
+
+providers:
+  - id: anthropic:messages:claude-sonnet-4-6
+    label: 'Claude Sonnet 4.6 + deepwiki MCP'
+    config:
+      max_tokens: 1500
+      temperature: 0
+      # Cap how many MCP rounds promptfoo will run per request. Default is 8.
+      max_tool_calls: 5
+      mcp:
+        enabled: true
+        servers:
+          # Hosted MCP server with two tools: ask_question, read_wiki_structure.
+          # Streams over HTTP — no local install required to try this example.
+          - name: deepwiki
+            url: https://mcp.deepwiki.com/mcp
+
+tests:
+  - vars:
+      question: 'Which transport protocols are described in the modelcontextprotocol/modelcontextprotocol repository?'
+    assert:
+      - type: contains-any
+        value: ['stdio', 'sse', 'streamable']
+      - type: llm-rubric
+        value: 'The response identifies at least one MCP transport (stdio, SSE, or streamable HTTP) and explains its purpose.'
+
+  - vars:
+      question: 'What is the high-level architecture of the promptfoo/promptfoo repository according to its docs?'
+    assert:
+      - type: contains
+        value: 'promptfoo'
+      - type: llm-rubric
+        value: 'The response describes promptfoo as an evaluation framework for LLMs (or similar wording) and mentions at least one concrete feature such as evals, redteam, providers, or assertions.'
+
+  # Demonstrate the loop terminating gracefully when the model decides it has
+  # enough information without further tool calls.
+  - vars:
+      question: 'In one sentence, what does MCP stand for?'
+    assert:
+      - type: contains
+        value: 'Model Context Protocol'

examples/anthropic/memory-tool/README.md

@@ -0,0 +1,26 @@
+# anthropic/memory-tool (Anthropic Memory Tool)
+
+This example shows how to include Anthropic's native `memory_20250818` tool in a Promptfoo eval.
+
+```bash
+npx promptfoo@latest init --example anthropic/memory-tool
+cd anthropic/memory-tool
+```
+
+## Features Demonstrated
+
+- Passing the Anthropic memory tool through the Messages provider
+- Restricting the tool to direct model calls
+- Keeping the eval single-turn by disabling tool use with `tool_choice`
+
+## Required Environment Variables
+
+- `ANTHROPIC_API_KEY` - Your Anthropic API key from [console.anthropic.com](https://console.anthropic.com/settings/keys)
+
+## Running the Example
+
+```bash
+promptfoo eval
+```
+
+Promptfoo sends the memory tool definition to Anthropic, but it does not create memory stores or run local memory handlers. Use this pattern to validate prompt behavior around memory-tool availability, or remove `tool_choice` when you want the model to request memory operations for assertion.

examples/anthropic/memory-tool/promptfooconfig.yaml

@@ -0,0 +1,23 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: 'Anthropic memory tool'
+
+prompts:
+  - 'Return exactly: MEMORY_TOOL_AVAILABLE'
+
+providers:
+  - id: anthropic:messages:claude-sonnet-4-6
+    config:
+      temperature: 0
+      max_tokens: 64
+      tools:
+        - type: memory_20250818
+          name: memory
+          allowed_callers:
+            - direct
+      tool_choice:
+        type: none
+
+tests:
+  - assert:
+      - type: contains
+        value: MEMORY_TOOL_AVAILABLE

examples/anthropic/opus-4-6-coding/README.md

@@ -0,0 +1,55 @@
+# anthropic/opus-4-6-coding (Claude Opus 4.6 Advanced Coding)
+
+This example demonstrates Claude Opus 4.6's state-of-the-art coding and reasoning capabilities, showcasing its ability to handle complex software engineering tasks with ambiguity and tradeoff analysis.
+
+You can run this example with:
+
+```bash
+npx promptfoo@latest init --example anthropic/opus-4-6-coding
+cd anthropic/opus-4-6-coding
+```
+
+## What This Tests
+
+Claude Opus 4.6 is the best model in the world for coding, agents, and computer use. This example evaluates:
+
+- **Complex code analysis**: Understanding multi-file codebases and architectural decisions
+- **Bug diagnosis**: Identifying root causes in complex, multi-system scenarios
+- **Ambiguity handling**: Making informed decisions when requirements are unclear
+- **Tradeoff reasoning**: Evaluating different approaches and explaining pros/cons
+- **Code generation**: Writing high-quality, production-ready code
+
+## Features Demonstrated
+
+1. **State-of-the-art coding**: Opus 4.6 achieves the highest score on SWE-bench Verified among frontier models
+2. **Reasoning about tradeoffs**: The model excels at analyzing different approaches and making informed decisions
+3. **Handling ambiguity**: Unlike models that require hand-holding, Opus 4.6 figures things out
+4. **Extended thinking**: Support for thinking budgets up to 128K tokens for complex reasoning
+
+## Running the Example
+
+```bash
+# Set your API key
+export ANTHROPIC_API_KEY=your_api_key_here
+
+# Run the evaluation
+npx promptfoo@latest eval
+
+# View results
+npx promptfoo@latest view
+```
+
+## Expected Results
+
+The evaluation tests Opus 4.6's ability to:
+
+- Diagnose bugs across multiple system boundaries
+- Choose appropriate data structures with clear reasoning
+- Write production-quality code with proper error handling
+- Analyze architectural decisions and propose improvements
+
+## Learn More
+
+- [Claude Opus 4.6 announcement](https://www.anthropic.com/news/claude-opus-4-6)
+- [Anthropic documentation](https://docs.anthropic.com)
+- [Promptfoo Anthropic provider docs](https://promptfoo.dev/docs/providers/anthropic)

examples/anthropic/opus-4-6-coding/promptfooconfig.yaml

@@ -0,0 +1,164 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: Claude Opus 4.6 advanced coding capabilities
+
+prompts:
+  - |
+    {{task}}
+
+providers:
+  - id: anthropic:messages:claude-opus-4-6
+    config:
+      temperature: 0
+      max_tokens: 8000
+
+tests:
+  # Complex bug diagnosis across multiple systems
+  - vars:
+      task: |
+        You're debugging a production issue where users can't log in. Here's what you know:
+
+        1. The frontend shows "Authentication failed" after username/password submission
+        2. Backend logs show successful JWT generation
+        3. Redis cache is returning stale session data
+        4. Database shows correct user credentials
+        5. The issue only affects 10% of login attempts
+        6. It started after deploying a load balancer configuration change
+
+        Diagnose the root cause and propose a fix. Explain your reasoning about what's causing the intermittent nature of the bug.
+    assert:
+      - type: contains-any
+        value: ['load balancer', 'session', 'sticky', 'affinity', 'routing']
+        reason: Should identify load balancer session routing as the issue
+      - type: llm-rubric
+        value: |
+          The response should:
+          1. Identify the root cause (likely session affinity/sticky sessions issue with load balancer)
+          2. Explain why it's intermittent (different backend servers, inconsistent session state)
+          3. Propose concrete fixes (enable sticky sessions, shared session store, stateless tokens)
+          4. Show reasoning about the tradeoffs of different solutions
+
+  # Data structure selection with tradeoff analysis
+  - vars:
+      task: |
+        You need to implement a feature that:
+        - Stores 10 million user activity records per day
+        - Supports queries like "find all activities for user X in date range Y"
+        - Needs to return results in under 100ms
+        - Data retention is 90 days
+        - Budget allows moderate infrastructure costs
+
+        What data structure and storage approach would you use? Explain the tradeoffs you considered.
+    assert:
+      - type: llm-rubric
+        value: |
+          The response should:
+          1. Propose a specific data structure/database (e.g., time-series DB, partitioned PostgreSQL, or similar)
+          2. Explain performance characteristics and why they meet the requirements
+          3. Discuss tradeoffs (cost vs performance, complexity vs maintainability)
+          4. Consider alternatives and explain why they were not chosen
+          5. Address scalability and data retention strategies
+      - type: contains-any
+        value: ['index', 'partition', 'query', 'performance', 'scale']
+        reason: Should discuss database optimization concepts
+
+  # Production-quality code generation with error handling
+  - vars:
+      task: |
+        Write a Python function that:
+        1. Fetches user data from a REST API (may timeout or return errors)
+        2. Caches results in Redis with 5-minute TTL
+        3. Falls back to database if cache miss
+        4. Returns user object or raises appropriate exception
+
+        Include proper error handling, typing, and comments explaining design decisions.
+    assert:
+      - type: contains
+        value: 'def'
+        reason: Should include Python function definition
+      - type: contains-any
+        value: ['try', 'except', 'raise', 'error']
+        reason: Should include error handling
+      - type: contains-any
+        value: ['cache', 'redis', 'ttl']
+        reason: Should implement caching logic
+      - type: llm-rubric
+        value: |
+          The code should:
+          1. Include proper type hints (from typing import ...)
+          2. Handle network timeouts and API errors gracefully
+          3. Implement cache-aside pattern correctly
+          4. Include docstrings and comments explaining design decisions
+          5. Use appropriate exception types
+          6. Be production-ready (not a toy example)
+
+  # Architectural decision with ambiguous requirements
+  - vars:
+      task: |
+        A startup wants to build a "social media analytics dashboard." They mention:
+        - "It should be fast"
+        - "We need real-time data"
+        - "Budget is tight but we might scale quickly"
+        - "Our team knows React and Python"
+
+        The requirements are intentionally vague. Propose an initial architecture, explain what assumptions you made, what questions you'd ask to clarify requirements, and what tradeoffs you considered.
+    assert:
+      - type: llm-rubric
+        value: |
+          The response should:
+          1. Propose a concrete but appropriately simple architecture
+          2. Explicitly state assumptions made (e.g., "Assuming 'real-time' means <1 second latency")
+          3. List specific clarifying questions (user scale, data volume, analytics complexity)
+          4. Explain technology choices based on team skills and constraints
+          5. Discuss tradeoffs (e.g., managed services vs self-hosted, cost vs performance)
+          6. Acknowledge what's unknown and how that affects the design
+      - type: contains-any
+        value: ['assumption', 'clarify', 'question', 'tradeoff', 'alternative']
+        reason: Should handle ambiguity explicitly
+
+  # Code review with nuanced feedback
+  - vars:
+      task: |
+        Review this React component and provide feedback:
+
+        ```jsx
+        function UserList() {
+          const [users, setUsers] = useState([]);
+
+          useEffect(() => {
+            fetch('/api/users')
+              .then(res => res.json())
+              .then(data => setUsers(data));
+          }, []);
+
+          return (
+            <div>
+              {users.map(user => (
+                <div key={user.id}>
+                  <h3>{user.name}</h3>
+                  <p>{user.email}</p>
+                </div>
+              ))}
+            </div>
+          );
+        }
+        ```
+
+        Identify issues, suggest improvements, and explain the reasoning behind each suggestion.
+    assert:
+      - type: contains-any
+        value: ['error', 'loading', 'state', 'async']
+        reason: Should identify missing error and loading states
+      - type: llm-rubric
+        value: |
+          The review should identify multiple issues:
+          1. No error handling for failed fetch
+          2. No loading state
+          3. No cleanup for fetch in useEffect
+          4. Missing dependencies might cause issues in strict mode
+          5. No null/empty checks for users array
+
+          For each issue, it should:
+          - Explain why it's a problem
+          - Suggest specific improvements
+          - Provide example code where helpful
+          - Prioritize issues by severity

examples/anthropic/opus-4-8-coding/README.md

@@ -0,0 +1,54 @@
+# anthropic/opus-4-8-coding (Claude Opus 4.8 Advanced Coding)
+
+This example exercises Claude Opus 4.8 on hard coding tasks using the `xhigh` effort level with adaptive thinking.
+
+You can run this example with:
+
+```bash
+npx promptfoo@latest init --example anthropic/opus-4-8-coding
+cd opus-4-8-coding
+```
+
+## What This Tests
+
+Claude Opus 4.8 is Anthropic's most capable model for complex reasoning and agentic coding. This example evaluates:
+
+- **Bug diagnosis** across multiple system boundaries
+- **Production-quality code generation** with proper error handling
+- **Code review** with nuanced, prioritized feedback
+
+## Working with Opus 4.8
+
+- **Builds on Opus 4.7.** Opus 4.8 supports the same feature set as 4.7 (no breaking API changes) and improves capability on complex reasoning and long-horizon agentic coding.
+- **Adaptive thinking is opt-in.** Set `thinking: { type: adaptive }` (as this example does) to let the model decide when and how much to reason per request. Without an explicit `thinking` block the model runs **without** extended thinking, even at high effort.
+- **`effort` defaults to `high`; `xhigh` is available.** Setting `effort: high` behaves the same as omitting it. Start with `xhigh` for coding and agentic work, and pair high effort with a large `max_tokens`.
+- **Sampling controls are managed for you.** Opus 4.8 rejects `temperature`, `top_p`, and `top_k` at the model level; promptfoo omits them automatically (don't set them in config).
+
+## Running the Example
+
+```bash
+# Set your API key
+export ANTHROPIC_API_KEY=your_api_key_here
+
+# Run the evaluation
+npx promptfoo@latest eval
+
+# View results
+npx promptfoo@latest view
+```
+
+## Other providers
+
+Opus 4.8 is also reachable through:
+
+- AWS Bedrock — `bedrock:us.anthropic.claude-opus-4-8` (or `bedrock:converse:us.anthropic.claude-opus-4-8`)
+- Google Vertex — `vertex:claude-opus-4-8` with `config.region: global`
+- Azure AI Foundry — point `anthropic:messages:claude-opus-4-8` at `https://<resource>.services.ai.azure.com/anthropic` via `apiBaseUrl`
+
+Across all four providers, promptfoo automatically omits the unsupported sampling parameters (`temperature`, `top_p`, `top_k`) for Opus 4.8. The Anthropic Messages provider also logs a one-time warning if you set them explicitly; the Bedrock, Vertex, and Azure paths omit them silently.
+
+## Learn More
+
+- [Claude Opus 4.8 announcement](https://www.anthropic.com/news/claude-opus-4-8)
+- [Anthropic documentation](https://docs.anthropic.com)
+- [Promptfoo Anthropic provider docs](https://promptfoo.dev/docs/providers/anthropic)

examples/anthropic/opus-4-8-coding/promptfooconfig.yaml

@@ -0,0 +1,124 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: Claude Opus 4.8 advanced coding with xhigh effort and adaptive thinking
+
+prompts:
+  - |
+    {{task}}
+
+providers:
+  - id: anthropic:messages:claude-opus-4-8
+    config:
+      # Opus 4.8 deprecates manual sampling controls (temperature/top_p/top_k) at
+      # the model level — promptfoo omits them automatically, so don't set them here.
+      #
+      # Adaptive thinking is opt-in: without an explicit `thinking` block the model
+      # runs WITHOUT extended thinking even at high effort. Set it to let the model
+      # decide when and how much to reason per request.
+      thinking:
+        type: adaptive
+      effort: xhigh # Recommended starting point for coding/agentic work (between high and max)
+      max_tokens: 8000
+
+tests:
+  # Complex bug diagnosis across multiple systems
+  - vars:
+      task: |
+        You're debugging a production issue where users can't log in. Here's what you know:
+
+        1. The frontend shows "Authentication failed" after username/password submission
+        2. Backend logs show successful JWT generation
+        3. Redis cache is returning stale session data
+        4. Database shows correct user credentials
+        5. The issue only affects 10% of login attempts
+        6. It started after deploying a load balancer configuration change
+
+        Diagnose the root cause and propose a fix. Explain your reasoning about what's causing the intermittent nature of the bug.
+    assert:
+      - type: contains-any
+        value: ['load balancer', 'session', 'sticky', 'affinity', 'routing']
+        reason: Should identify load balancer session routing as the issue
+      - type: llm-rubric
+        value: |
+          The response should:
+          1. Identify the root cause (likely session affinity/sticky sessions issue with load balancer)
+          2. Explain why it's intermittent (different backend servers, inconsistent session state)
+          3. Propose concrete fixes (enable sticky sessions, shared session store, stateless tokens)
+          4. Show reasoning about the tradeoffs of different solutions
+
+  # Production-quality code generation with error handling
+  - vars:
+      task: |
+        Write a Python function that:
+        1. Fetches user data from a REST API (may timeout or return errors)
+        2. Caches results in Redis with 5-minute TTL
+        3. Falls back to database if cache miss
+        4. Returns user object or raises appropriate exception
+
+        Include proper error handling, typing, and comments explaining design decisions.
+    assert:
+      - type: contains
+        value: 'def'
+        reason: Should include Python function definition
+      - type: contains-any
+        value: ['try', 'except', 'raise', 'error']
+        reason: Should include error handling
+      - type: contains-any
+        value: ['cache', 'redis', 'ttl']
+        reason: Should implement caching logic
+      - type: llm-rubric
+        value: |
+          The code should:
+          1. Include proper type hints (from typing import ...)
+          2. Handle network timeouts and API errors gracefully
+          3. Implement cache-aside pattern correctly
+          4. Include docstrings and comments explaining design decisions
+          5. Use appropriate exception types
+          6. Be production-ready (not a toy example)
+
+  # Code review with nuanced feedback
+  - vars:
+      task: |
+        Review this React component and provide feedback:
+
+        ```jsx
+        function UserList() {
+          const [users, setUsers] = useState([]);
+
+          useEffect(() => {
+            fetch('/api/users')
+              .then(res => res.json())
+              .then(data => setUsers(data));
+          }, []);
+
+          return (
+            <div>
+              {users.map(user => (
+                <div key={user.id}>
+                  <h3>{user.name}</h3>
+                  <p>{user.email}</p>
+                </div>
+              ))}
+            </div>
+          );
+        }
+        ```
+
+        Identify issues, suggest improvements, and explain the reasoning behind each suggestion.
+    assert:
+      - type: contains-any
+        value: ['error', 'loading', 'state', 'async']
+        reason: Should identify missing error and loading states
+      - type: llm-rubric
+        value: |
+          The review should identify multiple issues:
+          1. No error handling for failed fetch
+          2. No loading state
+          3. No cleanup for fetch in useEffect
+          4. Missing dependencies might cause issues in strict mode
+          5. No null/empty checks for users array
+
+          For each issue, it should:
+          - Explain why it's a problem
+          - Suggest specific improvements
+          - Provide example code where helpful
+          - Prioritize issues by severity

examples/anthropic/sonnet-5/README.md

@@ -0,0 +1,53 @@
+# anthropic/sonnet-5 (Claude Sonnet 5 Agentic Reasoning)
+
+This example exercises Claude Sonnet 5 on agentic reasoning and coding tasks using the `high` effort level with adaptive thinking.
+
+You can run this example with:
+
+```bash
+npx promptfoo@latest init --example anthropic/sonnet-5
+cd sonnet-5
+```
+
+## What This Tests
+
+Claude Sonnet 5 is the Claude 5-generation Sonnet — built to be Anthropic's most agentic Sonnet, with capability approaching Opus 4.8 at Sonnet pricing and a 1M-token context window. This example evaluates:
+
+- **Multi-system bug diagnosis** of an intermittent production failure
+- **Production-quality code generation** with error handling and caching
+
+## Working with Sonnet 5
+
+- **Adaptive thinking is opt-in.** Set `thinking: { type: adaptive }` (as this example does) to let the model decide when and how much to reason per request. Without an explicit `thinking` block the model runs **without** extended thinking, even at high effort. Unlike Fable 5 / Mythos 5, Sonnet 5 also accepts `thinking: { type: disabled }`.
+- **`effort` tunes the cost/performance tradeoff.** Sonnet 5 supports `low`, `medium`, `high`, `xhigh`, and `max`. `high` is a good cost-efficient default; step up to `xhigh`/`max` for the hardest work and pair high effort with a large `max_tokens`.
+- **Sampling controls are managed for you.** Sonnet 5 rejects `temperature`, `top_p`, and `top_k` at the model level; promptfoo omits them automatically (don't set them in config).
+- **Pricing.** $3/$15 per million input/output tokens standard, with introductory pricing of $2/$10 through August 31, 2026. The full 1M-token context bills at the standard rate (no long-context surcharge).
+
+## Running the Example
+
+```bash
+# Set your API key
+export ANTHROPIC_API_KEY=your_api_key_here
+
+# Run the evaluation
+npx promptfoo@latest eval
+
+# View results
+npx promptfoo@latest view
+```
+
+## Other providers
+
+Sonnet 5 is also reachable through:
+
+- AWS Bedrock — `bedrock:us.anthropic.claude-sonnet-5` (or `bedrock:converse:us.anthropic.claude-sonnet-5`)
+- Google Vertex — `vertex:claude-sonnet-5` with `config.region: global` (availability may roll out after the Anthropic API launch)
+- Azure AI Foundry — point `anthropic:messages:claude-sonnet-5` at `https://<resource>.services.ai.azure.com/anthropic` via `apiBaseUrl`
+
+Across all four providers, promptfoo automatically omits the unsupported sampling parameters (`temperature`, `top_p`, `top_k`) for Sonnet 5. The Anthropic Messages provider — used directly and for Azure AI Foundry via `apiBaseUrl` — logs a one-time warning if you set them explicitly; the Bedrock and Vertex paths omit them silently.
+
+## Learn More
+
+- [Claude Sonnet 5 announcement](https://www.anthropic.com/news/claude-sonnet-5)
+- [Anthropic documentation](https://docs.anthropic.com)
+- [Promptfoo Anthropic provider docs](https://promptfoo.dev/docs/providers/anthropic)

examples/anthropic/sonnet-5/promptfooconfig.yaml

@@ -0,0 +1,80 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: Claude Sonnet 5 agentic reasoning with effort and adaptive thinking
+
+prompts:
+  - |
+    {{task}}
+
+providers:
+  - id: anthropic:messages:claude-sonnet-5
+    config:
+      # Sonnet 5 is the Claude 5-generation Sonnet: near-Opus agentic capability at
+      # Sonnet pricing, with a 1M-token context window.
+      #
+      # Sonnet 5 deprecates manual sampling controls (temperature/top_p/top_k) at the
+      # model level — promptfoo omits them automatically, so don't set them here.
+      #
+      # Adaptive thinking is opt-in: without an explicit `thinking` block the model
+      # runs WITHOUT extended thinking even at high effort. Set it to let the model
+      # decide when and how much to reason per request.
+      thinking:
+        type: adaptive
+      effort: high # Sonnet 5's cost-efficient sweet spot; xhigh/max are available for harder work
+      max_tokens: 8000
+
+tests:
+  # Multi-system bug diagnosis (intermittent failure)
+  - vars:
+      task: |
+        You're debugging a production issue where users can't log in. Here's what you know:
+
+        1. The frontend shows "Authentication failed" after username/password submission
+        2. Backend logs show successful JWT generation
+        3. Redis cache is returning stale session data
+        4. Database shows correct user credentials
+        5. The issue only affects 10% of login attempts
+        6. It started after deploying a load balancer configuration change
+
+        Diagnose the root cause and propose a fix. Explain your reasoning about what's
+        causing the intermittent nature of the bug.
+    assert:
+      - type: contains-any
+        value: ['load balancer', 'session', 'sticky', 'affinity', 'routing']
+        reason: Should identify load balancer session routing as the issue
+      - type: llm-rubric
+        value: |
+          The response should:
+          1. Identify the root cause (likely session affinity/sticky sessions issue with the load balancer)
+          2. Explain why it's intermittent (requests hit different backends with inconsistent session state)
+          3. Propose concrete fixes (sticky sessions, a shared session store, or stateless tokens)
+          4. Reason about the tradeoffs of the proposed solutions
+
+  # Production-quality code generation with error handling and caching
+  - vars:
+      task: |
+        Write a Python function that:
+        1. Fetches user data from a REST API (may timeout or return errors)
+        2. Caches results in Redis with a 5-minute TTL
+        3. Falls back to the database on a cache miss
+        4. Returns a user object or raises an appropriate exception
+
+        Include proper error handling, type hints, and comments explaining design decisions.
+    assert:
+      - type: contains
+        value: 'def'
+        reason: Should include a Python function definition
+      - type: contains-any
+        value: ['try', 'except', 'raise', 'error']
+        reason: Should include error handling
+      - type: contains-any
+        value: ['cache', 'redis', 'ttl']
+        reason: Should implement caching logic
+      - type: llm-rubric
+        value: |
+          The code should:
+          1. Include type hints (e.g. from typing import ...)
+          2. Handle network timeouts and API errors gracefully
+          3. Implement the cache-aside pattern correctly
+          4. Include docstrings/comments explaining design decisions
+          5. Use appropriate exception types
+          6. Be production-ready (not a toy example)

examples/anthropic/structured-outputs/README.md

@@ -0,0 +1,128 @@
+# anthropic/structured-outputs (Anthropic Structured Outputs)
+
+This example demonstrates Anthropic's structured outputs feature, which ensures Claude's responses follow a specific schema. It shows both **JSON outputs** for structured data extraction and **strict tool use** for guaranteed schema validation on tool calls.
+
+## What You'll Learn
+
+- How to use `output_format` to constrain Claude's responses to a JSON schema
+- How to use `strict: true` on tools to guarantee type-safe function parameters
+- The difference between JSON outputs and strict tool use
+- When to use each approach
+
+## Setup
+
+```bash
+export ANTHROPIC_API_KEY=your_api_key_here
+npx promptfoo@latest init --example anthropic/structured-outputs
+npx promptfoo@latest eval
+```
+
+## Features Demonstrated
+
+### JSON Outputs
+
+The first provider configuration shows how to extract structured data from unstructured text using a JSON schema:
+
+```yaml
+providers:
+  - id: anthropic:messages:claude-sonnet-4-6
+    config:
+      output_format:
+        type: json_schema
+        schema:
+          type: object
+          properties:
+            customer_name:
+              type: string
+            customer_email:
+              type: string
+            # ... more fields
+          required:
+            - customer_name
+            - customer_email
+          additionalProperties: false
+```
+
+**Use JSON outputs when:**
+
+- Extracting data from images or text
+- Generating structured reports
+- Formatting API responses
+- You need Claude's response in a specific format
+
+### Strict Tool Use
+
+The second provider configuration demonstrates how to ensure tool parameters exactly match your schema:
+
+```yaml
+providers:
+  - id: anthropic:messages:claude-sonnet-4-6
+    config:
+      tools:
+        - name: book_demo
+          strict: true # Enable strict mode
+          input_schema:
+            type: object
+            properties:
+              customer_email:
+                type: string
+              # ... more properties
+            required:
+              - customer_email
+              - customer_name
+            additionalProperties: false
+```
+
+**Use strict tool use when:**
+
+- Building agentic workflows
+- Ensuring type-safe function calls
+- Complex tools with many/nested properties
+- You need validated parameters and tool names
+
+## Key Benefits
+
+- **Always valid**: No more `JSON.parse()` errors
+- **Type safe**: Guaranteed field types and required fields
+- **Reliable**: No retries needed for schema violations
+- **Production-ready**: Build agents that work consistently at scale
+
+## Schema Requirements
+
+Both modes share these JSON Schema limitations:
+
+✅ **Supported:**
+
+- Basic types: object, array, string, integer, number, boolean, null
+- `enum` (primitives only)
+- `required` and `additionalProperties: false`
+- Array `minItems` (only 0 and 1)
+
+❌ **Not supported:**
+
+- Recursive schemas
+- Numerical constraints (`minimum`, `maximum`)
+- String constraints (`minLength`, `maxLength`)
+- Complex `{n,m}` quantifiers in regex patterns
+
+## Test Cases
+
+The example includes comprehensive tests:
+
+1. **Schema validation**: Ensures all required fields are present
+2. **Data accuracy**: Verifies extracted values match the input
+3. **No extra fields**: Confirms `additionalProperties: false` is enforced
+4. **Tool calling**: Validates tool names and parameters are correct
+5. **Type safety**: Checks that types match schema definitions
+
+## Supported Models
+
+Structured outputs are available for:
+
+- Claude Sonnet 4.6 (`claude-sonnet-4-6`)
+- Claude Opus 4.1 (`claude-opus-4-1-20250805`)
+
+## Learn More
+
+- [Anthropic Structured Outputs Documentation](https://docs.anthropic.com/en/docs/build-with-claude/structured-outputs)
+- [promptfoo Anthropic Provider Documentation](/docs/providers/anthropic)

examples/anthropic/structured-outputs/customer_schema.json

@@ -0,0 +1,25 @@
+{
+  "type": "object",
+  "properties": {
+    "customer_name": {
+      "type": "string"
+    },
+    "customer_email": {
+      "type": "string"
+    },
+    "company": {
+      "type": "string"
+    },
+    "current_plan": {
+      "type": "string"
+    },
+    "requested_plan": {
+      "type": "string"
+    },
+    "demo_requested": {
+      "type": "boolean"
+    }
+  },
+  "required": ["customer_name", "customer_email", "demo_requested"],
+  "additionalProperties": false
+}

examples/anthropic/structured-outputs/nested_format.json

@@ -0,0 +1,4 @@
+{
+  "type": "json_schema",
+  "schema": "file://customer_schema.json"
+}

examples/anthropic/structured-outputs/output_format.json

@@ -0,0 +1,55 @@
+{
+  "type": "json_schema",
+  "schema": {
+    "type": "object",
+    "properties": {
+      "customer_name": {
+        "type": "string"
+      },
+      "customer_email": {
+        "type": "string"
+      },
+      "company": {
+        "type": "string"
+      },
+      "current_plan": {
+        "type": "string"
+      },
+      "requested_plan": {
+        "type": "string"
+      },
+      "current_users": {
+        "type": "integer"
+      },
+      "requested_users": {
+        "type": "integer"
+      },
+      "demo_requested": {
+        "type": "boolean"
+      },
+      "demo_datetime": {
+        "type": "string"
+      },
+      "features_of_interest": {
+        "type": "array",
+        "items": {
+          "type": "string"
+        }
+      },
+      "questions": {
+        "type": "array",
+        "items": {
+          "type": "string"
+        }
+      }
+    },
+    "required": [
+      "customer_name",
+      "customer_email",
+      "current_plan",
+      "requested_plan",
+      "demo_requested"
+    ],
+    "additionalProperties": false
+  }
+}

examples/anthropic/structured-outputs/promptfooconfig.yaml

@@ -0,0 +1,154 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: Anthropic structured outputs for JSON responses and strict tools
+
+prompts:
+  # JSON outputs example - extract structured data
+  - |
+    Extract the key information from this customer support email:
+
+    Subject: Enterprise Plan Upgrade Request
+    From: Sarah Johnson <sarah.johnson@techcorp.com>
+
+    Hi there,
+
+    I'm reaching out to inquire about upgrading our team to the Enterprise plan. We currently have 25 users on the Professional plan, and we're interested in the advanced security features and dedicated support that come with Enterprise.
+
+    Could someone schedule a demo for next Tuesday, March 15th at 2:00 PM EST? We'd like to see the SSO integration and audit logging features in action.
+
+    Also, what's the pricing for 50 users on the Enterprise plan?
+
+    Thanks,
+    Sarah Johnson
+    Director of Engineering, TechCorp
+
+providers:
+  # JSON outputs - Inline schema
+  - id: anthropic:messages:claude-sonnet-4-6
+    label: inline-schema
+    config:
+      max_tokens: 2048
+      output_format:
+        type: json_schema
+        schema:
+          type: object
+          properties:
+            customer_name:
+              type: string
+            customer_email:
+              type: string
+            company:
+              type: string
+            current_plan:
+              type: string
+            requested_plan:
+              type: string
+            current_users:
+              type: integer
+            requested_users:
+              type: integer
+            demo_requested:
+              type: boolean
+            demo_datetime:
+              type: string
+            features_of_interest:
+              type: array
+              items:
+                type: string
+            questions:
+              type: array
+              items:
+                type: string
+          required:
+            - customer_name
+            - customer_email
+            - current_plan
+            - requested_plan
+            - demo_requested
+          additionalProperties: false
+
+  # JSON outputs - External file
+  - id: anthropic:messages:claude-sonnet-4-6
+    label: external-file
+    config:
+      max_tokens: 2048
+      output_format: file://output_format.json
+
+  # JSON outputs - Nested file reference (format file references schema file)
+  - id: anthropic:messages:claude-sonnet-4-6
+    label: nested-file-reference
+    config:
+      max_tokens: 2048
+      output_format: file://nested_format.json
+
+  # Strict tool use - Ensure tool parameters exactly match schema
+  - id: anthropic:messages:claude-sonnet-4-6
+    config:
+      max_tokens: 2048
+      tool_choice:
+        type: any
+      tools:
+        - name: book_demo
+          description: Book a demo session for a customer
+          strict: true
+          input_schema:
+            type: object
+            properties:
+              customer_email:
+                type: string
+              customer_name:
+                type: string
+              company:
+                type: string
+              demo_datetime:
+                type: string
+                description: ISO 8601 datetime string
+              features_to_demo:
+                type: array
+                items:
+                  type: string
+                  enum:
+                    - sso_integration
+                    - audit_logging
+                    - advanced_security
+                    - dedicated_support
+                    - custom_integrations
+              attendees_count:
+                type: integer
+                enum: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
+            required:
+              - customer_email
+              - customer_name
+              - demo_datetime
+            additionalProperties: false
+
+        - name: get_pricing
+          description: Get pricing information for a specific plan and user count
+          strict: true
+          input_schema:
+            type: object
+            properties:
+              plan:
+                type: string
+                enum:
+                  - professional
+                  - enterprise
+              user_count:
+                type: integer
+              billing_period:
+                type: string
+                enum:
+                  - monthly
+                  - annual
+            required:
+              - plan
+              - user_count
+            additionalProperties: false
+
+tests:
+  # Simple test that works for both providers
+  - description: Should extract customer information
+    assert:
+      - type: contains
+        value: Sarah Johnson
+      - type: contains
+        value: sarah.johnson@techcorp.com

examples/anthropic/web-tools/README.md

@@ -0,0 +1,89 @@
+# anthropic/web-tools (Anthropic Web Tools)
+
+This example demonstrates Anthropic's web search and web fetch tools, showing both basic URL fetching and comprehensive research workflows.
+
+You can run this example with:
+
+```bash
+npx promptfoo@latest init --example anthropic/web-tools
+cd anthropic/web-tools
+```
+
+## Features Demonstrated
+
+- **Web Search**: Find relevant information across the internet
+- **Web Fetch**: Retrieve full content from specific web pages and PDFs
+- **Combined Workflows**: Multi-step research processes
+- **Security Controls**: Domain filtering for trusted sources
+- **Citations**: Source attribution for transparency
+
+## Configuration Highlights
+
+```yaml
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: 'Web search and fetch tools'
+
+prompts:
+  - |
+    {{task}}
+
+providers:
+  - id: anthropic:messages:claude-sonnet-4-6
+    config:
+      temperature: 0
+      max_tokens: 2500
+      tools:
+        - type: web_search_20250305 # Search the web
+          name: web_search
+          max_uses: 2
+        - type: web_fetch_20250910 # Fetch specific URLs
+          name: web_fetch
+          max_uses: 3
+          allowed_domains: # Restrict to trusted domains
+            - docs.anthropic.com
+            - github.com
+            - openai.com
+            - arxiv.org
+          citations:
+            enabled: true # Enable source citations
+          max_content_tokens: 12000 # Content size limit
+```
+
+## Required Environment Variables
+
+This example requires the following environment variables:
+
+- `ANTHROPIC_API_KEY` - Your Anthropic API key from [console.anthropic.com](https://console.anthropic.com/settings/keys)
+
+You can set these in a `.env` file or directly in your environment.
+
+## Running the Example
+
+```bash
+cd examples/anthropic/web-tools
+promptfoo eval
+```
+
+## What It Tests
+
+1. **Direct URL Fetch**: Retrieve and summarize specific documentation pages
+2. **GitHub Repository Analysis**: Analyze code repositories and their purpose
+3. **Research Workflow**: Multi-step research with search → fetch → synthesis
+4. **Security Research**: Investigation of web tool security considerations
+
+## Use Cases
+
+Perfect for:
+
+- Documentation analysis
+- Competitive research
+- Technical investigation
+- Security assessment
+- Academic literature review
+
+## Security Notes
+
+- Only allows fetching from trusted domains (docs.anthropic.com, github.com, openai.com, arxiv.org)
+- Content limited to 12,000 tokens to prevent excessive usage
+- Citations enabled for source transparency
+- Usage limits prevent API abuse (2 searches, 3 fetches per evaluation)

examples/anthropic/web-tools/promptfooconfig.yaml

@@ -0,0 +1,82 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: 'Web search and fetch tools'
+
+prompts:
+  - |
+    {{task}}
+
+providers:
+  - id: anthropic:messages:claude-sonnet-4-6
+    config:
+      temperature: 0
+      max_tokens: 2500
+      tools:
+        - type: web_search_20250305
+          name: web_search
+          max_uses: 2
+        - type: web_fetch_20250910
+          name: web_fetch
+          max_uses: 3
+          allowed_domains:
+            - docs.anthropic.com
+            - github.com
+            - openai.com
+            - arxiv.org
+          citations:
+            enabled: true
+          max_content_tokens: 12000
+
+tests:
+  # Basic web fetch test
+  - vars:
+      task: 'Please fetch and summarize the content from this URL: https://docs.anthropic.com/en/docs/build-with-claude/tool-use'
+    assert:
+      - type: contains-any
+        value: ['tool', 'function', 'API', 'capabilities']
+        reason: 'Should mention tool-related concepts'
+      - type: llm-rubric
+        value: 'Provides a clear, structured summary of the web page content'
+
+  # Direct URL fetch test
+  - vars:
+      task: 'Fetch and analyze the GitHub repository at https://github.com/anthropics/anthropic-sdk-typescript'
+    assert:
+      - type: contains
+        value: 'TypeScript'
+      - type: llm-rubric
+        value: 'Describes the TypeScript SDK and its purpose'
+
+  # Research workflow test
+  - vars:
+      task: |
+        Research "Claude 4 AI model capabilities" and provide a comprehensive analysis.
+
+        Process:
+        1. Search for recent information about the topic
+        2. Fetch detailed content from the most authoritative sources  
+        3. Synthesize findings into a structured report
+
+        Your report should include:
+        - Overview of the topic
+        - Key capabilities or features
+        - Recent developments
+        - Sources used
+    assert:
+      - type: contains-any
+        value: ['Claude 4', 'claude-4', 'capabilities', 'model']
+        reason: 'Should discuss Claude 4 specifically'
+      - type: llm-rubric
+        value: 'Provides a comprehensive research-based analysis with multiple information sources'
+      - type: contains-any
+        value: ['search', 'found', 'according', 'research', 'sources']
+        reason: 'Should indicate research methodology was used'
+
+  # Security-focused research test
+  - vars:
+      task: 'Research web fetch tool security considerations and best practices'
+    assert:
+      - type: contains-any
+        value: ['security', 'domain', 'filtering', 'risk']
+        reason: 'Should address security aspects'
+      - type: llm-rubric
+        value: 'Discusses security considerations and best practices for web fetching'

examples/azure-mai/README.md

@@ -0,0 +1,87 @@
+# azure-mai (Microsoft MAI models on Azure AI Foundry)
+
+You can run this example with:
+
+```bash
+npx promptfoo@latest init --example azure-mai
+cd azure-mai
+```
+
+Evaluate Microsoft's first-party **MAI** models with promptfoo. This example focuses on **image generation** with `MAI-Image-2.5` (a Preview [Foundry Model sold by Azure](https://learn.microsoft.com/azure/foundry/foundry-models/concepts/models-sold-directly-by-azure)) via the dedicated `azure:image` provider, and shows how to wire reasoning/chat MAI models via `azure:chat` — though those have limited availability today (see [Notes](#notes)).
+
+## Environment Variables
+
+This example requires:
+
+- `AZURE_API_HOST` — your Foundry resource endpoint, e.g. `your-resource.services.ai.azure.com`
+- `AZURE_API_KEY` — a resource key (or authenticate with `az login` for Microsoft Entra ID)
+
+## Quick Start
+
+```bash
+# 1. Deploy an MAI image model to a Microsoft Foundry (AIServices) resource
+az cognitiveservices account deployment create \
+  --name <RESOURCE> --resource-group <RG> \
+  --deployment-name mai-image-2-5 \
+  --model-name MAI-Image-2.5 --model-format Microsoft \
+  --model-version 2026-06-02 --sku-name GlobalStandard --sku-capacity 1
+
+# 2. Point promptfoo at the resource
+export AZURE_API_HOST=<RESOURCE>.services.ai.azure.com
+export AZURE_API_KEY=<key>
+
+# 3. Run the eval (images cost ~$0.03 each, so disable caching for fresh runs)
+promptfoo eval --no-cache
+
+# 4. View the generated images
+promptfoo view
+```
+
+## What's in this Example
+
+- `promptfooconfig.yaml` — generates images with `MAI-Image-2.5` (Preview) through the Microsoft `azure:image` provider, reporting per-image token usage and cost from the API's token counts
+- `promptfooconfig.vision-judge.yaml` — uses a vision LLM as a judge on the generated images (see below)
+- Shows how to wire reasoning/chat MAI models via `azure:chat` (these are deprecated/private-preview today — see [Notes](#notes))
+
+## Providers
+
+### Image generation (`azure:image`)
+
+```yaml
+providers:
+  - id: azure:image:mai-image-2-5
+    config:
+      model: MAI-Image-2.5 # cost-reporting id (deployment names can't contain dots)
+      width: 1024 # min 768; width * height <= 1,048,576
+      height: 1024
+```
+
+The image is returned as a base64 PNG. promptfoo stores large base64 media as a blob reference (`promptfoo://blob/...`), so assertions should accept either the blob ref or an inline `data:image/...` URL.
+
+### Reasoning chat (`azure:chat`)
+
+`MAI-Thinking-1` and `MAI-DS-R1` are reasoning models (auto-detected by name — promptfoo sends `max_completion_tokens` and drops `temperature`). **They aren't deployable on most subscriptions today** (`MAI-DS-R1` is deprecated; `MAI-Thinking-1` / `MAI-Code-1-Flash` are private preview and not in the public CLI catalog), so the chat provider in `promptfooconfig.yaml` is commented out. Uncomment it once your subscription can deploy one:
+
+```yaml
+providers:
+  - id: azure:chat:mai-thinking-1
+    config:
+      max_completion_tokens: 2048
+      # omitDefaults: true   # if the deployment rejects top_p / penalties
+```
+
+## LLM-as-judge on images (vision grading)
+
+`promptfooconfig.vision-judge.yaml` grades each generated image with a vision-capable LLM. It uses a custom `rubricPrompt` that passes the image to the grader as an `image_url` block, so the judge evaluates the actual picture rather than a text description.
+
+Run it with inline media so `{{output}}` is a base64 data URL the grader can read:
+
+```bash
+PROMPTFOO_INLINE_MEDIA=true promptfoo eval -c promptfooconfig.vision-judge.yaml --no-cache
+```
+
+> **Why inline media?** With promptfoo's default media handling, an image output is stored as a `promptfoo://blob/...` reference, which a hosted grader's API can't fetch. `PROMPTFOO_INLINE_MEDIA=true` keeps the output as an inline data URL the vision model can read directly.
+
+## Notes
+
+The MAI image models are currently **Preview**. The MAI text models have limited availability: `MAI-DS-R1` is **deprecated** in the Azure catalog, and `MAI-Thinking-1` / `MAI-Code-1-Flash` are in **private preview** and aren't yet in the public CLI catalog. Run `az cognitiveservices model list --location <region>` to see what your subscription can actually deploy. See the [Azure provider docs](https://www.promptfoo.dev/docs/providers/azure/#using-microsoft-mai-models) for details.

examples/azure-mai/promptfooconfig.vision-judge.yaml

@@ -0,0 +1,56 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: LLM-as-judge on MAI-generated images (vision rubric)
+
+# Grade generated images with a vision-capable LLM. The grader must receive the
+# actual image, so run this with inline media so {{output}} is a base64 data URL
+# the vision model can read (otherwise the output is a promptfoo://blob/... ref
+# that the grader's API can't fetch):
+#
+#   PROMPTFOO_INLINE_MEDIA=true promptfoo eval --no-cache
+#
+# Requires AZURE_API_HOST + AZURE_API_KEY (image provider) and OPENAI_API_KEY
+# (vision grader).
+
+prompts:
+  - '{{prompt}}'
+
+providers:
+  - id: azure:image:mai-image-2-5
+    config:
+      model: MAI-Image-2.5
+      width: 1024
+      height: 1024
+
+defaultTest:
+  options:
+    # Any vision-capable grader works (e.g. openai:chat:gpt-5.4-mini,
+    # anthropic:messages:claude-sonnet-4-6).
+    provider: openai:chat:gpt-5.4-mini
+    # Custom rubric prompt that passes the generated image to the grader as an
+    # image_url block. {{output}} is the image; {{rubric}} is the assertion value.
+    rubricPrompt: |
+      [
+        {
+          "role": "system",
+          "content": "You grade whether a generated image satisfies a rubric. Respond ONLY with a JSON object: {\"reason\": string, \"pass\": boolean, \"score\": number}."
+        },
+        {
+          "role": "user",
+          "content": [
+            { "type": "image_url", "image_url": { "url": "{{output}}" } },
+            { "type": "text", "text": "Rubric: {{rubric}}\n\nDoes the image satisfy the rubric?" }
+          ]
+        }
+      ]
+
+tests:
+  - vars:
+      prompt: A photorealistic single red cube on a clean white background, studio lighting
+    assert:
+      - type: llm-rubric
+        value: The image shows a single red cube on a plain white background
+  - vars:
+      prompt: An isometric illustration of a cozy reading nook with warm lighting
+    assert:
+      - type: llm-rubric
+        value: The image depicts an indoor reading nook with warm/cozy lighting

examples/azure-mai/promptfooconfig.yaml

@@ -0,0 +1,63 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: Microsoft MAI models on Azure AI Foundry (Foundry Models sold by Azure)
+
+# Microsoft's first-party "MAI" model family. Image models (MAI-Image-2.5 and
+# friends, currently Preview) use a Microsoft-managed `/mai/v1/images` route and
+# run through the dedicated `azure:image:<deployment>` provider — that's what this
+# example uses. Text/reasoning models (MAI-DS-R1, MAI-Thinking-1, MAI-Code-1-Flash)
+# use the standard chat-completions API via `azure:chat:<deployment>`, but have
+# limited availability today (see the commented chat provider below).
+#
+# Setup:
+#   1. Deploy an MAI image model to a Microsoft Foundry (AIServices) resource:
+#        az cognitiveservices account deployment create \
+#          --name <RESOURCE> --resource-group <RG> \
+#          --deployment-name mai-image-2-5 \
+#          --model-name MAI-Image-2.5 --model-format Microsoft \
+#          --model-version 2026-06-02 --sku-name GlobalStandard --sku-capacity 1
+#   2. Point promptfoo at the resource's *.services.ai.azure.com endpoint:
+#        export AZURE_API_HOST=<RESOURCE>.services.ai.azure.com
+#        export AZURE_API_KEY=<key>            # or use `az login` (Entra ID)
+#   3. Run:  promptfoo eval --no-cache
+
+prompts:
+  - '{{prompt}}'
+
+providers:
+  - id: azure:image:mai-image-2-5
+    label: MAI-Image-2.5
+    config:
+      # `model` is used only for cost reporting; Azure deployment names can't
+      # contain the dot in "MAI-Image-2.5", so name the model id explicitly.
+      model: MAI-Image-2.5
+      width: 1024
+      height: 1024
+
+  # Reasoning/chat MAI models use the standard chat provider, but aren't deployable
+  # on most subscriptions yet (MAI-DS-R1 is deprecated; MAI-Thinking-1 and
+  # MAI-Code-1-Flash are private preview). Uncomment once your subscription can
+  # deploy one:
+  # - id: azure:chat:mai-thinking-1
+  #   label: MAI-Thinking-1
+  #   config:
+  #     max_completion_tokens: 2048
+
+tests:
+  - vars:
+      prompt: A photorealistic red cube on a clean white background, studio lighting
+    assert:
+      # The provider returns a base64 PNG. promptfoo stores large base64 media as
+      # a blob reference (promptfoo://blob/...) before assertions run, so accept
+      # either the blob ref or an inline data URL.
+      - type: javascript
+        value: |
+          return typeof output === 'string' &&
+            (output.startsWith('promptfoo://blob/') || output.startsWith('data:image/'));
+
+  - vars:
+      prompt: An isometric illustration of a cozy reading nook with warm lighting
+    assert:
+      - type: javascript
+        value: |
+          return typeof output === 'string' &&
+            (output.startsWith('promptfoo://blob/') || output.startsWith('data:image/'));

examples/azure/README.md

@@ -0,0 +1,67 @@
+# azure (Azure AI Examples)
+
+This directory contains examples for using Azure AI services with promptfoo, including Azure OpenAI, Azure AI Foundry, and third-party models available through Microsoft Foundry.
+
+## Available Examples
+
+### Azure OpenAI
+
+| Example                           | Description                         |
+| --------------------------------- | ----------------------------------- |
+| [openai](./openai/)               | Azure OpenAI chat and vision models |
+| [assistant](./assistant/)         | Azure OpenAI Assistants with tools  |
+| [foundry-agent](./foundry-agent/) | Azure AI Foundry Agents             |
+
+### Third-Party Models (Azure AI Foundry)
+
+| Example                     | Description                                   |
+| --------------------------- | --------------------------------------------- |
+| [claude](./claude/)         | Anthropic Claude models (Opus, Sonnet, Haiku) |
+| [llama](./llama/)           | Meta Llama models (4, 3.3, 3.1)               |
+| [deepseek](./deepseek/)     | DeepSeek models including R1 reasoning        |
+| [mistral](./mistral/)       | Mistral models (Large, Ministral)             |
+| [comparison](./comparison/) | Compare models across providers               |
+
+## Quick Start
+
+```bash
+# Azure OpenAI basic example
+npx promptfoo@latest init --example azure/openai
+
+# Azure Assistants with tools
+npx promptfoo@latest init --example azure/assistant
+
+# Azure AI Foundry Agents
+npx promptfoo@latest init --example azure/foundry-agent
+
+# Third-party models
+npx promptfoo@latest init --example azure/claude
+npx promptfoo@latest init --example azure/llama
+npx promptfoo@latest init --example azure/deepseek
+npx promptfoo@latest init --example azure/mistral
+npx promptfoo@latest init --example azure/comparison
+```
+
+## Environment Variables
+
+All Azure examples require authentication. Set one of:
+
+```bash
+# Option 1: API Key (simplest)
+export AZURE_API_KEY=your-api-key
+export AZURE_API_HOST=your-resource.openai.azure.com
+
+# Option 2: Azure CLI (recommended for development)
+az login
+
+# Option 3: Service Principal
+export AZURE_CLIENT_ID=your-client-id
+export AZURE_CLIENT_SECRET=your-client-secret
+export AZURE_TENANT_ID=your-tenant-id
+```
+
+## Documentation
+
+- [Azure Provider Documentation](https://promptfoo.dev/docs/providers/azure/)
+- [Azure OpenAI Service](https://learn.microsoft.com/en-us/azure/ai-services/openai/)
+- [Azure AI Foundry](https://learn.microsoft.com/en-us/azure/ai-foundry/)

examples/azure/assistant/README.md

@@ -0,0 +1,177 @@
+# azure/assistant (Azure OpenAI Assistants API with Tools)
+
+Evaluate Azure OpenAI Assistants with file search, function tools, and multi-tool interactions.
+
+You can run this example with:
+
+```bash
+npx promptfoo@latest init --example azure/assistant
+cd azure/assistant
+```
+
+## Features
+
+- File search via vector stores
+- Custom function tools with simple implementations
+- Multi-tool examples with combined capabilities
+- Progressive examples from basic to advanced use cases
+
+## Environment Variables
+
+This example requires the following environment variables:
+
+- `AZURE_API_KEY` - Your Azure OpenAI API key
+- `AZURE_OPENAI_API_HOST` - Your Azure OpenAI API host (e.g., "your-resource-name.openai.azure.com")
+
+You can set these in a `.env` file or directly in your environment.
+
+## Prerequisites
+
+1. An Azure OpenAI account with access to the Assistants API
+2. An assistant created in your Azure OpenAI account
+3. A vector store for file search functionality (optional)
+
+## Configuration Files
+
+This example includes several configuration files, each demonstrating different tool capabilities:
+
+1. **promptfooconfig-file-search.yaml** - File search tool only
+2. **promptfooconfig-function.yaml** - Function tool capability with a weather API implementation
+3. **promptfooconfig-multi-tool.yaml** - Combined file search and function tools
+
+### Running Different Configurations
+
+To run a specific configuration:
+
+```bash
+# File search example
+npx promptfoo@latest eval -c promptfooconfig-file-search.yaml
+
+# Function tool capability example
+npx promptfoo@latest eval -c promptfooconfig-function.yaml
+
+# Multi-tool example
+npx promptfoo@latest eval -c promptfooconfig-multi-tool.yaml
+```
+
+## Tool Capabilities
+
+### File Search
+
+The file search configuration demonstrates how to use vector store-backed file search.
+
+Key components:
+
+- Simple `tools` configuration with `type: "file_search"` (no description field)
+- `tool_resources` with vector store ID configuration
+- Test cases focused on information retrieval
+
+**Important**: The file search tool must be defined without a description field, as Azure OpenAI API does not support it for this tool type.
+
+### Function Tool
+
+The function tool configuration shows how to define and use custom functions.
+
+Key components:
+
+- Function tool definition loaded from external file (`tools/weather-function.json`)
+- External function callback implementation in `callbacks/weather.js`
+- Test cases demonstrating tool invocation
+
+### Multi-Tool Usage
+
+The multi-tool configuration demonstrates how to combine multiple tools.
+
+Key components:
+
+- External tools definition file combining file search and function tools
+- Multiple inline function callbacks
+- Test cases requiring coordination between different tools
+
+## Customization
+
+To use this example with your own Azure OpenAI Assistant:
+
+1. Update the assistant ID in each configuration file (replace `your_assistant_id` with your actual ID)
+2. Replace `your_vector_store_id` with your own vector store ID
+3. Set the `apiHost` to match your Azure OpenAI endpoint (e.g., "your-resource-name.openai.azure.com")
+4. Customize the tools and function callbacks as needed
+
+## Function Implementation Approaches
+
+This example demonstrates two approaches to implementing functions:
+
+### 1. External Tool Definition with External Callback
+
+```yaml
+# Load tools from external file
+tools: file://tools/weather-function.json
+
+# External file-based callback
+functionToolCallbacks:
+  get_weather: file://callbacks/weather.js:getWeather
+```
+
+### 2. Multiple Tools with Inline Callbacks
+
+For more complex scenarios, you can combine multiple tools and inline function callbacks:
+
+```yaml
+# Multiple tools defined
+tools: file://tools/multiple-tools.json
+
+# Multiple inline function callbacks
+functionToolCallbacks:
+  get_weather: |
+    async function(args) {
+      // Weather function implementation
+    }
+  suggest_recipe: |
+    async function(args) {
+      // Recipe function implementation
+    }
+```
+
+## Function Callback Context
+
+Function callbacks can optionally receive context information about the current conversation:
+
+```javascript
+// Enhanced callback with context for audit logging
+functionToolCallbacks: {
+  get_employee_data: async (args, context) => {
+    const { employeeId } = JSON.parse(args);
+
+    // Log access with thread context for audit trail
+    console.log(`Access to employee ${employeeId} requested`, {
+      threadId: context?.threadId,
+      timestamp: new Date().toISOString(),
+      provider: context?.provider,
+    });
+
+    // Your function logic here
+    return JSON.stringify({ name: 'John Doe', department: 'Engineering' });
+  };
+}
+```
+
+The context object includes:
+
+- `threadId`: Unique identifier for the conversation thread
+- `runId`: Identifier for the current assistant run
+- `assistantId`: The assistant being used
+- `provider`: The provider type ('azure' or 'openai')
+
+This is particularly useful for session management, audit logging, and tracking stateful interactions across function calls.
+
+## Documentation
+
+For more information about using Azure OpenAI with promptfoo, including authentication methods, provider types, and configuration options, see the [official Azure provider documentation](https://www.promptfoo.dev/docs/providers/azure/).
+
+## Notes
+
+- The file search capability requires a properly configured vector store
+- Both tool definitions and function callbacks can be implemented inline or loaded from external files
+- For production use, consider more robust error handling
+- **File search tool format**: The file search tool must be defined only with `type: "file_search"` without a description field. Adding a description will cause API errors
+- The examples use placeholder values that must be replaced with your actual IDs and endpoints

examples/azure/assistant/callbacks/weather.js

@@ -0,0 +1,35 @@
+/**
+ * Sample weather function callback implementation
+ * @param {string} args - JSON string with the function arguments
+ * @returns {string} - JSON string with the weather information
+ */
+function getWeather(args) {
+  try {
+    // Parse the JSON string from the Azure API
+    const parsedArgs = JSON.parse(args);
+
+    // Extract parameters with defaults
+    const location = parsedArgs.location;
+    const unit = parsedArgs.unit || 'celsius';
+
+    if (!location) {
+      return JSON.stringify({ error: 'Location is required' });
+    }
+
+    // Mock weather data
+    const mockWeather = {
+      location,
+      temperature: unit === 'celsius' ? 22 : 72,
+      unit,
+      forecast: ['sunny', 'partly cloudy', 'clear'][Math.floor(Math.random() * 3)],
+      humidity: Math.floor(Math.random() * 40) + 30,
+    };
+
+    return JSON.stringify(mockWeather);
+  } catch (error) {
+    console.error('Error in getWeather function:', error);
+    return JSON.stringify({ error: `Failed to process weather request: ${error.message}` });
+  }
+}
+
+module.exports = { getWeather };

examples/azure/assistant/promptfooconfig-file-search.yaml

@@ -0,0 +1,110 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: Azure OpenAI Assistant with File Search Tool
+
+prompts:
+  - |
+    You are a helpful customer service assistant for TechGadgets, an online electronics retailer.
+    Answer the following customer question:
+
+    {{prompt}}
+
+providers:
+  - id: azure:assistant:your_assistant_id
+    label: azure-assistant
+    config:
+      # Remove any http or https prefix and trailing slashes from apiHost
+      apiHost: your-resource-name.openai.azure.com
+      # Simple file search tool definition
+      tools:
+        - type: 'file_search'
+      # Vector store configuration - replace with your actual vector store ID
+      tool_resources:
+        file_search:
+          vector_store_ids:
+            - 'your_vector_store_id' # Vector store containing product and policy documentation
+      # Standard parameters
+      temperature: 0.7
+      apiVersion: '2024-05-01-preview'
+
+# Override the default grader (use your gpt-5.1 deployment name)
+defaultTest:
+  options:
+    provider: azure:chat:gpt-5.1-deployment
+
+tests:
+  - vars:
+      prompt: What is your return policy for electronics? Can I return items after 30 days?
+    assert:
+      - type: icontains
+        value: return policy
+      - type: llm-rubric
+        value: Provides a clear, concise explanation of the return policy including the time frame and any conditions
+      - type: factuality
+        value: |
+          The answer must include these facts about our return policy:
+          - Standard electronics have a 30-day return window
+          - Premium electronics (smartphones, laptops, tablets) have a 14-day return window
+          - Items must be in original packaging with all accessories
+          - Restocking fee of 15% applies after 7 days
+          - Returns after 30 days are only accepted for store credit
+
+  - vars:
+      prompt: How do I track my order? I ordered a laptop last week and haven't received any updates.
+    assert:
+      - type: llm-rubric
+        value: Directly addresses how to track an order and provides specific steps the customer can take to get order updates
+      - type: factuality
+        value: |
+          The answer must accurately mention:
+          - Orders can be tracked on our website at techgadgets.com/order-tracking
+          - You need your order number and email to track
+          - Premium laptops typically ship within 2-3 business days
+          - Customers receive tracking emails when items ship
+          - Customer service can be reached at 1-800-TECH-HELP for shipping inquiries
+
+  - vars:
+      prompt: Are there any discounts available for students buying computer accessories?
+    assert:
+      - type: icontains
+        value: discount
+      - type: llm-rubric
+        value: |
+          Clearly explains available student discounts for computer accessories.
+          Score 0.7+ if it provides specific discount percentages or eligibility requirements.
+          Score 1.0 if it also explains how to verify student status and apply the discount.
+      - type: factuality
+        value: |
+          The answer must accurately mention these facts:
+          - Students receive a 15% discount on all computer accessories
+          - Verification requires a valid .edu email address or student ID
+          - The discount can be combined with seasonal sales (up to 25% total discount)
+          - Student discounts do not apply to already discounted items or clearance products
+          - The TechGadgets Student Program provides additional benefits including free shipping
+
+  - vars:
+      prompt: My new smartphone has a cracked screen right out of the box. What should I do?
+    assert:
+      - type: icontains
+        value: warranty
+      - type: factuality
+        value: |
+          The answer must accurately mention these facts about our damage policy:
+          - Damaged products received must be reported within 7 days of delivery
+          - Photo evidence must be submitted through the return portal
+          - TechGadgets offers free replacement for items damaged during shipping
+          - Manufacturer defects are covered under the manufacturer's warranty (1 year minimum)
+          - Expedited replacements can be arranged for premium customers
+      - type: llm-rubric
+        value: |
+          Provides clear, actionable steps for reporting and resolving a damaged product issue.
+          Should include information about contacting customer service, warranty coverage, and return options.
+        threshold: 0.8
+
+  - vars:
+      prompt: How do I mine cryptocurrency using your products?
+    assert:
+      - type: llm-rubric
+        value: |
+          The response should clearly indicate if cryptocurrency mining is outside the scope of standard customer support.
+          It should NOT provide detailed instructions about cryptocurrency mining if this contradicts company policy.
+          Score 1.0 if it politely addresses the question while redirecting to appropriate products or indicating policy limitations.

examples/azure/assistant/promptfooconfig-function.yaml

@@ -0,0 +1,30 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: Azure OpenAI Assistant with Function Tool Capability
+
+prompts:
+  - 'Answer the following question: {{prompt}}'
+
+providers:
+  - id: azure:assistant:your_assistant_id
+    label: azure-assistant-function
+    config:
+      apiHost: your-resource-name.openai.azure.com
+      # Load tools from external file
+      tools: file://tools/weather-function.json
+      # Use the external file-based callback
+      functionToolCallbacks:
+        get_weather: file://callbacks/weather.js:getWeather
+      temperature: 0.7
+      apiVersion: '2024-05-01-preview'
+      debug: true
+
+tests:
+  # Function tool invocation tests
+  - vars:
+      prompt: What's the weather like in Seattle?
+  - vars:
+      prompt: Tell me about the weather in Tokyo in Celsius.
+  - vars:
+      prompt: Compare the weather in New York and London.
+  - vars:
+      prompt: What's the weather forecast for San Francisco in Fahrenheit?

examples/azure/assistant/promptfooconfig-multi-tool.yaml

@@ -0,0 +1,76 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: Azure OpenAI Assistant with Multiple Tools
+
+prompts:
+  - '{{prompt}}'
+
+providers:
+  - id: azure:assistant:your_assistant_id
+    label: azure-assistant-multi-tool
+    config:
+      apiHost: your-resource-name.openai.azure.com
+      # Load tools from external file
+      tools: file://tools/multiple-tools.json
+      # Function callbacks for each tool
+      functionToolCallbacks:
+        get_weather: |
+          async function(args) {
+            try {
+              const parsedArgs = JSON.parse(args);
+              const location = parsedArgs.location;
+              const unit = parsedArgs.unit || 'c';
+              console.log(`Weather request for ${location} in ${unit}`);
+              
+              // Simple weather response
+              return JSON.stringify({
+                location,
+                temperature: unit === 'c' ? 22 : 72,
+                unit,
+                condition: 'sunny',
+                forecast: 'Clear skies for the next few days.'
+              });
+            } catch (error) {
+              console.error('Error in get_weather function:', error);
+              return JSON.stringify({ error: String(error) });
+            }
+          }
+        suggest_recipe: |
+          async function(args) {
+            try {
+              const parsedArgs = JSON.parse(args);
+              const ingredients = parsedArgs.ingredients || [];
+              console.log(`Recipe requested with ingredients: ${ingredients.join(', ')}`);
+              
+              // Simple recipe suggestion
+              return JSON.stringify({
+                recipe: {
+                  name: "Simple Pasta",
+                  ingredients: ["pasta", "olive oil", "garlic", "salt"],
+                  instructions: "1. Boil pasta\n2. Heat oil and garlic\n3. Toss pasta in oil\n4. Season to taste"
+                }
+              });
+            } catch (error) {
+              console.error('Error in suggest_recipe function:', error);
+              return JSON.stringify({ error: String(error) });
+            }
+          }
+      # Vector store configuration for file search
+      tool_resources:
+        file_search:
+          vector_store_ids:
+            - 'your_vector_store_id'
+      # Standard parameters
+      temperature: 0.7
+      apiVersion: '2024-05-01-preview'
+      debug: true
+
+tests:
+  # Multi-tool usage tests
+  - vars:
+      prompt: I'm planning to run some LLM evaluations tomorrow. Can you tell me about promptfoo's command line options and also check the weather in San Francisco?
+  - vars:
+      prompt: I need information about vector stores in promptfoo, the weather in Tokyo, and a simple pasta recipe for dinner.
+  - vars:
+      prompt: How do I set up a promptfoo evaluation? By the way, what's the weather like in Seattle today?
+  - vars:
+      prompt: I have pasta and garlic. Can you suggest a recipe and also help me understand how to use custom assertions in promptfoo?

examples/azure/assistant/tools/multiple-tools.json

@@ -0,0 +1,47 @@
+[
+  {
+    "type": "file_search"
+  },
+  {
+    "type": "function",
+    "function": {
+      "name": "get_weather",
+      "description": "Get current weather information for a location",
+      "parameters": {
+        "type": "object",
+        "properties": {
+          "location": {
+            "type": "string",
+            "description": "The city and state e.g. Seattle, WA"
+          },
+          "unit": {
+            "type": "string",
+            "enum": ["c", "f"],
+            "description": "Temperature unit (c for Celsius, f for Fahrenheit)"
+          }
+        },
+        "required": ["location"]
+      }
+    }
+  },
+  {
+    "type": "function",
+    "function": {
+      "name": "suggest_recipe",
+      "description": "Suggest a recipe based on ingredients",
+      "parameters": {
+        "type": "object",
+        "properties": {
+          "ingredients": {
+            "type": "array",
+            "items": {
+              "type": "string"
+            },
+            "description": "Available ingredients"
+          }
+        },
+        "required": ["ingredients"]
+      }
+    }
+  }
+]

examples/azure/assistant/tools/weather-function.json

@@ -0,0 +1,24 @@
+[
+  {
+    "type": "function",
+    "function": {
+      "name": "get_weather",
+      "description": "Get current weather information for a location",
+      "parameters": {
+        "type": "object",
+        "properties": {
+          "location": {
+            "type": "string",
+            "description": "The city and state, e.g., San Francisco, CA"
+          },
+          "unit": {
+            "type": "string",
+            "enum": ["celsius", "fahrenheit"],
+            "description": "The unit of temperature to use. Infer this from the query. Defaults to celsius if not specified."
+          }
+        },
+        "required": ["location"]
+      }
+    }
+  }
+]

examples/azure/claude/README.md

@@ -0,0 +1,48 @@
+# azure/claude (Azure Claude Models)
+
+This example demonstrates how to use Anthropic Claude models on Azure AI Foundry with promptfoo.
+
+You can run this example with:
+
+```bash
+npx promptfoo@latest init --example azure/claude
+cd azure/claude
+```
+
+## Setup
+
+1. Deploy Claude models in Azure AI Foundry
+2. Set your environment variables:
+
+```bash
+export AZURE_API_KEY=your-api-key
+export AZURE_API_HOST=your-deployment.services.ai.azure.com
+```
+
+## Available Claude Models
+
+| Model                       | Description                    |
+| --------------------------- | ------------------------------ |
+| `claude-opus-4-6-20260205`  | Claude Opus 4.6 - Most capable |
+| `claude-sonnet-4-6`         | Claude Sonnet 4.6 - Balanced   |
+| `claude-haiku-4-5-20251001` | Claude Haiku 4.5 - Fast        |
+
+## Running the Example
+
+```bash
+npx promptfoo@latest eval
+npx promptfoo@latest view
+```
+
+## Configuration
+
+The example compares Claude Opus 4.6, Claude Sonnet 4.6, and Claude Haiku 4.5 on explanation tasks. Modify `promptfooconfig.yaml` to:
+
+- Change models by updating the provider IDs
+- Adjust temperature and max_tokens
+- Add more test cases
+
+## Documentation
+
+- [Azure Provider Documentation](https://promptfoo.dev/docs/providers/azure/)
+- [Claude on Azure](https://azure.microsoft.com/en-us/products/ai-services/ai-foundry/)

examples/azure/claude/promptfooconfig.yaml

@@ -0,0 +1,55 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: Azure Claude models
+
+prompts:
+  - 'Explain {{concept}} in simple terms, suitable for a beginner.'
+
+providers:
+  - id: azure:chat:claude-opus-4-6-20260205
+    label: claude-opus-4-6
+    config:
+      apiHost: 'your-deployment.services.ai.azure.com'
+      apiVersion: '2025-11-15-preview'
+      max_tokens: 1024
+      temperature: 0.7
+
+  - id: azure:chat:claude-sonnet-4-6
+    label: claude-sonnet-4-6
+    config:
+      apiHost: 'your-deployment.services.ai.azure.com'
+      apiVersion: '2025-04-01-preview'
+      max_tokens: 1024
+      temperature: 0.7
+
+  - id: azure:chat:claude-haiku-4-5-20251001
+    label: claude-haiku-4-5
+    config:
+      apiHost: 'your-deployment.services.ai.azure.com'
+      apiVersion: '2025-04-01-preview'
+      max_tokens: 1024
+      temperature: 0.7
+
+tests:
+  - vars:
+      concept: machine learning
+    assert:
+      - type: contains-any
+        value: ['algorithm', 'data', 'pattern', 'model']
+      - type: llm-rubric
+        value: The explanation should be clear and accessible to beginners
+
+  - vars:
+      concept: blockchain
+    assert:
+      - type: contains-any
+        value: ['decentralized', 'ledger', 'transaction', 'block']
+      - type: llm-rubric
+        value: The explanation should avoid technical jargon
+
+  - vars:
+      concept: quantum computing
+    assert:
+      - type: contains-any
+        value: ['qubit', 'superposition', 'quantum']
+      - type: llm-rubric
+        value: The explanation should use helpful analogies

examples/azure/comparison/README.md

@@ -0,0 +1,57 @@
+# azure/comparison (Azure Model Comparison)
+
+This example demonstrates how to compare models from different providers on Azure AI Foundry, including OpenAI, Anthropic Claude, Meta Llama, and Mistral.
+
+You can run this example with:
+
+```bash
+npx promptfoo@latest init --example azure/comparison
+cd azure/comparison
+```
+
+## Setup
+
+1. Deploy models from different providers in Azure AI Foundry
+2. Set your environment variables:
+
+```bash
+export AZURE_API_KEY=your-api-key
+# Set apiHost in promptfooconfig.yaml for each provider's deployment
+```
+
+## Models Compared
+
+| Provider  | Model                                    | Label         |
+| --------- | ---------------------------------------- | ------------- |
+| OpenAI    | `gpt-5.1`                                | gpt-5.1       |
+| Anthropic | `claude-sonnet-4-6`                      | claude-sonnet |
+| Meta      | `Llama-4-Maverick-17B-128E-Instruct-FP8` | llama-4       |
+| Mistral   | `Mistral-Large-2411`                     | mistral-large |
+
+## Running the Example
+
+```bash
+npx promptfoo@latest eval
+npx promptfoo@latest view
+```
+
+## Customization
+
+Modify `promptfooconfig.yaml` to:
+
+- Add or remove models
+- Change test questions
+- Adjust evaluation criteria
+- Compare cost vs performance
+
+## Use Cases
+
+- Benchmark different models on your specific tasks
+- Evaluate cost-effectiveness across providers
+- Find the best model for your use case
+- A/B test model updates
+
+## Documentation
+
+- [Azure Provider Documentation](https://promptfoo.dev/docs/providers/azure/)
+- [Azure AI Foundry](https://azure.microsoft.com/en-us/products/ai-services/ai-foundry/)

examples/azure/comparison/promptfooconfig.yaml

@@ -0,0 +1,77 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: Azure model comparison
+
+prompts:
+  - |
+    You are a helpful assistant. Answer the following question concisely and accurately.
+
+    Question: {{question}}
+
+providers:
+  # OpenAI GPT-5.1 (Latest flagship)
+  - id: azure:chat:gpt-5.1
+    label: gpt-5.1
+    config:
+      apiHost: 'your-resource.openai.azure.com'
+      max_tokens: 1024
+      temperature: 0.7
+
+  # Anthropic Claude Sonnet 4.6
+  - id: azure:chat:claude-sonnet-4-6
+    label: claude-sonnet
+    config:
+      apiHost: 'your-deployment.services.ai.azure.com'
+      apiVersion: '2025-04-01-preview'
+      max_tokens: 1024
+      temperature: 0.7
+
+  # Meta Llama 4 Maverick
+  - id: azure:chat:Llama-4-Maverick-17B-128E-Instruct-FP8
+    label: llama-4
+    config:
+      apiHost: 'your-deployment.services.ai.azure.com'
+      apiVersion: '2025-04-01-preview'
+      max_tokens: 1024
+      temperature: 0.7
+
+  # Mistral Large
+  - id: azure:chat:Mistral-Large-2411
+    label: mistral-large
+    config:
+      apiHost: 'your-deployment.services.ai.azure.com'
+      apiVersion: '2025-04-01-preview'
+      max_tokens: 1024
+      temperature: 0.7
+
+tests:
+  - vars:
+      question: What are the main differences between machine learning and deep learning?
+    assert:
+      - type: llm-rubric
+        value: Accurately explains the relationship and differences
+      - type: contains-any
+        value: ['neural network', 'subset', 'feature']
+
+  - vars:
+      question: Explain the concept of recursion in programming with a simple example.
+    assert:
+      - type: llm-rubric
+        value: Provides a clear explanation with a working example
+      - type: contains-any
+        value: ['base case', 'function', 'itself']
+
+  - vars:
+      question: What is the difference between REST and GraphQL APIs?
+    assert:
+      - type: llm-rubric
+        value: Accurately compares both API paradigms
+      - type: contains-any
+        value: ['endpoint', 'query', 'schema']
+
+  - vars:
+      question: How does HTTPS encryption work?
+    assert:
+      - type: llm-rubric
+        value: Explains the encryption process accurately
+      - type: contains-any
+        value: ['TLS', 'SSL', 'certificate', 'handshake']

examples/azure/deepseek/README.md

@@ -0,0 +1,54 @@
+# azure/deepseek (Azure DeepSeek Models)
+
+This example demonstrates how to use DeepSeek models on Azure AI Foundry with promptfoo, including the DeepSeek-R1 reasoning model.
+
+You can run this example with:
+
+```bash
+npx promptfoo@latest init --example azure/deepseek
+cd azure/deepseek
+```
+
+## Setup
+
+1. Deploy DeepSeek models in Azure AI Foundry
+2. Set your environment variables:
+
+```bash
+export AZURE_API_KEY=your-api-key
+export AZURE_API_HOST=your-deployment.services.ai.azure.com
+```
+
+## Available DeepSeek Models
+
+| Model                           | Type      | Description               |
+| ------------------------------- | --------- | ------------------------- |
+| `DeepSeek-R1`                   | Reasoning | Advanced reasoning model  |
+| `DeepSeek-V3`                   | Chat      | Standard chat model       |
+| `DeepSeek-R1-Distill-Llama-70B` | Reasoning | Distilled reasoning model |
+| `DeepSeek-R1-Distill-Qwen-32B`  | Reasoning | Distilled reasoning model |
+
+## Reasoning Model Configuration
+
+DeepSeek-R1 is a reasoning model that requires special configuration:
+
+```yaml
+providers:
+  - id: azure:chat:DeepSeek-R1
+    config:
+      isReasoningModel: true # Required for reasoning models
+      max_completion_tokens: 4096 # Use instead of max_tokens
+      reasoning_effort: medium # low, medium, or high
+```
+
+## Running the Example
+
+```bash
+npx promptfoo@latest eval
+npx promptfoo@latest view
+```
+
+## Documentation
+
+- [Azure Provider Documentation](https://promptfoo.dev/docs/providers/azure/)
+- [DeepSeek on Azure](https://azure.microsoft.com/en-us/products/ai-services/ai-foundry/)

examples/azure/deepseek/promptfooconfig.yaml

@@ -0,0 +1,57 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: Azure DeepSeek reasoning
+
+prompts:
+  - 'Solve this step by step: {{problem}}'
+
+providers:
+  # DeepSeek-R1 is a reasoning model
+  - id: azure:chat:DeepSeek-R1
+    label: deepseek-r1
+    config:
+      apiHost: 'your-deployment.services.ai.azure.com'
+      apiVersion: '2025-04-01-preview'
+      isReasoningModel: true
+      max_completion_tokens: 4096
+      reasoning_effort: medium
+
+  # DeepSeek-V3 is a standard chat model
+  - id: azure:chat:DeepSeek-V3
+    label: deepseek-v3
+    config:
+      apiHost: 'your-deployment.services.ai.azure.com'
+      apiVersion: '2025-04-01-preview'
+      max_tokens: 4096
+      temperature: 0.3
+
+tests:
+  - vars:
+      problem: |
+        A train leaves Station A at 9:00 AM traveling at 60 mph toward Station B.
+        Another train leaves Station B at 10:00 AM traveling at 80 mph toward Station A.
+        If the stations are 280 miles apart, at what time will the trains meet?
+    assert:
+      - type: contains-any
+        value: ['11:34', '11:34 AM', '1 hour 34 minutes', '94 minutes']
+      - type: llm-rubric
+        value: Shows clear step-by-step reasoning
+
+  - vars:
+      problem: |
+        If 5 machines can produce 100 widgets in 4 hours,
+        how many widgets can 8 machines produce in 6 hours?
+    assert:
+      - type: contains
+        value: '240'
+      - type: llm-rubric
+        value: Demonstrates proportional reasoning
+
+  - vars:
+      problem: |
+        A rectangle has a perimeter of 36 cm.
+        If the length is twice the width, what is the area?
+    assert:
+      - type: contains
+        value: '72'
+      - type: llm-rubric
+        value: Shows algebraic problem-solving steps

examples/azure/foundry-agent/README.md

@@ -0,0 +1,120 @@
+# azure/foundry-agent (Azure AI Foundry Agent)
+
+This example demonstrates how to use the Azure Foundry Agent provider with promptfoo. This provider uses the `@azure/ai-projects` SDK and the v2 Responses agent runtime instead of the old threads/runs API.
+
+You can run this example with:
+
+```bash
+npx promptfoo@latest init --example azure/foundry-agent
+cd azure/foundry-agent
+```
+
+## Setup
+
+1. Install the required Azure SDK packages:
+
+```bash
+npm install @azure/ai-projects @azure/identity
+```
+
+2. Set up your Azure credentials. The provider uses `DefaultAzureCredential`, so you can authenticate via:
+   - Azure CLI: `az login`
+   - Environment variables
+   - Managed Identity
+   - Service Principal
+
+3. Set your Azure AI Project URL:
+
+```bash
+export AZURE_AI_PROJECT_URL="https://your-project.services.ai.azure.com/api/projects/your-project-id"
+```
+
+## Configuration
+
+The provider uses the `azure:foundry-agent:agent-name-or-id` format. Agent names are preferred. Legacy agent IDs still work as a fallback lookup if the agent exists in the project.
+
+```yaml
+providers:
+  - id: azure:foundry-agent:my-foundry-agent
+    config:
+      projectUrl: 'https://your-project.services.ai.azure.com/api/projects/your-project-id'
+      temperature: 0.7
+      max_tokens: 150
+
+tests:
+  - vars:
+      question: 'What is the capital of France?'
+    assert:
+      - type: contains
+        value: 'Paris'
+```
+
+## Configuration Options
+
+These per-request settings are supported:
+
+- `instructions`
+- `temperature`
+- `top_p`
+- `max_tokens` / `max_completion_tokens` (mapped to `max_output_tokens`)
+- `response_format`
+- `tools`
+- `tool_choice`
+- `functionToolCallbacks`
+- `modelName`
+- `reasoning_effort`
+- `verbosity`
+- `metadata`
+- `passthrough`
+- `maxPollTimeMs`
+
+These request-time settings are ignored by the v2 runtime and should be configured on the Foundry agent instead:
+
+- `tool_resources`
+- `frequency_penalty`
+- `presence_penalty`
+- `seed`
+- `stop`
+- `timeoutMs`
+- `retryOptions`
+
+## Function Tool Callbacks
+
+You can provide custom function callbacks just like with the regular Azure Assistant provider:
+
+```yaml
+providers:
+  - id: azure:foundry-agent:my-foundry-agent
+    config:
+      projectUrl: 'https://your-project.services.ai.azure.com/api/projects/your-project-id'
+      functionToolCallbacks:
+        getCurrentWeather: |
+          (args) => {
+            const { location } = JSON.parse(args);
+            return `The weather in ${location} is sunny and 75°F`;
+          }
+```
+
+## Differences from Regular Azure Assistant Provider
+
+The main differences are:
+
+1. **SDK Usage**: Uses `@azure/ai-projects` SDK instead of direct HTTP calls
+2. **Authentication**: Uses `DefaultAzureCredential` for Azure authentication
+3. **Project URL**: Requires an Azure AI Project URL instead of Azure OpenAI endpoint
+4. **Provider Format**: Uses `azure:foundry-agent:agent-name-or-id` instead of `azure:assistant:assistant-id`
+5. **Runtime**: Uses `responses.create(..., agent_reference)` instead of threads/messages/runs
+
+## Environment Variables
+
+- `AZURE_AI_PROJECT_URL`: Your Azure AI Project URL (can be overridden in config)
+- Standard Azure credential environment variables (if not using other auth methods)
+
+## Error Handling
+
+The provider includes the same comprehensive error handling as the regular Azure Assistant provider:
+
+- Content filter detection and guardrails reporting
+- Rate limit handling
+- Service error detection
+- Automatic retries for transient errors

examples/azure/foundry-agent/promptfooconfig.yaml

@@ -0,0 +1,35 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+
+description: Azure AI Foundry Agent evaluation
+
+providers:
+  - id: azure:foundry-agent:my-foundry-agent
+    config:
+      projectUrl: 'project.services.ai.azure.com/api/projects/project-id'
+      temperature: 0.7
+      max_tokens: 150
+      instructions: 'You are a helpful assistant that provides clear and concise answers.'
+
+prompts:
+  - '{{question}}'
+
+tests:
+  - vars:
+      question: 'What is the capital of France?'
+    assert:
+      - type: contains
+        value: 'Paris'
+
+  - vars:
+      question: 'Explain what photosynthesis is in simple terms.'
+    assert:
+      - type: contains
+        value: 'plants'
+      - type: contains
+        value: 'sunlight'
+
+  - vars:
+      question: 'What is 2 + 2?'
+    assert:
+      - type: contains
+        value: '4'

examples/azure/llama/README.md

@@ -0,0 +1,47 @@
+# azure/llama (Azure Llama Models)
+
+This example demonstrates how to use Meta Llama models on Azure AI Foundry with promptfoo.
+
+You can run this example with:
+
+```bash
+npx promptfoo@latest init --example azure/llama
+cd azure/llama
+```
+
+## Setup
+
+1. Deploy Llama models in Azure AI Foundry
+2. Set your environment variables:
+
+```bash
+export AZURE_API_KEY=your-api-key
+export AZURE_API_HOST=your-deployment.services.ai.azure.com
+```
+
+## Available Llama Models
+
+| Model                                    | Description                         |
+| ---------------------------------------- | ----------------------------------- |
+| `Llama-4-Maverick-17B-128E-Instruct-FP8` | Llama 4 Maverick (128 experts, FP8) |
+| `Llama-4-Scout-17B-16E-Instruct`         | Llama 4 Scout (16 experts)          |
+| `Llama-3.3-70B-Instruct`                 | Llama 3.3 70B                       |
+| `Meta-Llama-3.1-405B-Instruct`           | Llama 3.1 405B                      |
+| `Meta-Llama-3.1-70B-Instruct`            | Llama 3.1 70B                       |
+| `Meta-Llama-3.1-8B-Instruct`             | Llama 3.1 8B                        |
+
+## Running the Example
+
+```bash
+npx promptfoo@latest eval
+npx promptfoo@latest view
+```
+
+## Configuration
+
+The example compares Llama 4 Maverick and Llama 4 Scout on code generation tasks. This helps evaluate the trade-off between model capacity (expert count), speed, and quality.
+
+## Documentation
+
+- [Azure Provider Documentation](https://promptfoo.dev/docs/providers/azure/)
+- [Llama on Azure](https://azure.microsoft.com/en-us/products/ai-services/ai-foundry/)

examples/azure/llama/promptfooconfig.yaml

@@ -0,0 +1,62 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: Azure Llama models
+
+prompts:
+  - |
+    You are a helpful coding assistant.
+
+    Write a {{language}} function that {{task}}.
+    Include comments explaining the code.
+
+providers:
+  # Llama 4 Maverick - Higher capacity with 128 experts
+  - id: azure:chat:Llama-4-Maverick-17B-128E-Instruct-FP8
+    label: llama-4-maverick
+    config:
+      apiHost: 'your-deployment.services.ai.azure.com'
+      apiVersion: '2025-04-01-preview'
+      max_tokens: 2048
+      temperature: 0.3
+
+  # Llama 4 Scout - Lighter weight with 16 experts
+  - id: azure:chat:Llama-4-Scout-17B-16E-Instruct
+    label: llama-4-scout
+    config:
+      apiHost: 'your-deployment.services.ai.azure.com'
+      apiVersion: '2025-04-01-preview'
+      max_tokens: 2048
+      temperature: 0.3
+
+tests:
+  - vars:
+      language: Python
+      task: calculates the factorial of a number using recursion
+    assert:
+      - type: contains
+        value: 'def'
+      - type: contains-any
+        value: ['factorial', 'recursion', 'recursive']
+      - type: python
+        value: |
+          def test_code():
+              # Check for function definition and recursive call
+              return 'def ' in output and ('return' in output or 'factorial' in output)
+          test_code()
+
+  - vars:
+      language: JavaScript
+      task: reverses a string without using built-in reverse methods
+    assert:
+      - type: contains
+        value: 'function'
+      - type: not-contains
+        value: '.reverse()'
+
+  - vars:
+      language: Python
+      task: checks if a number is prime
+    assert:
+      - type: contains
+        value: 'def'
+      - type: contains-any
+        value: ['prime', 'divisible', 'modulo', '%']

examples/azure/mistral/README.md

@@ -0,0 +1,47 @@
+# azure/mistral (Azure Mistral Models)
+
+This example demonstrates how to use Mistral models on Azure AI Foundry with promptfoo.
+
+You can run this example with:
+
+```bash
+npx promptfoo@latest init --example azure/mistral
+cd azure/mistral
+```
+
+## Setup
+
+1. Deploy Mistral models in Azure AI Foundry
+2. Update `promptfooconfig.yaml` with your deployment name and API host
+3. Set your environment variables:
+
+```bash
+export AZURE_API_KEY=your-api-key
+```
+
+## Available Mistral Models
+
+| Model                | Description                      |
+| -------------------- | -------------------------------- |
+| `Mistral-Large-3`    | Mistral Large 3 - Most capable   |
+| `Mistral-Large-2411` | Mistral Large - Previous gen     |
+| `mistral-small-2503` | Mistral Small - Fast, efficient  |
+| `Pixtral-Large-2411` | Pixtral Large - Vision + text    |
+| `Ministral-3B-2410`  | Ministral 3B - Fast, lightweight |
+| `Mistral-Nemo`       | Mistral Nemo - Balanced          |
+
+## Running the Example
+
+```bash
+npx promptfoo@latest eval
+npx promptfoo@latest view
+```
+
+## Configuration
+
+The example compares Mistral Large 3 and Mistral Small 2503 on text generation tasks. This helps evaluate the trade-off between model capacity, speed, and quality.
+
+## Documentation
+
+- [Azure Provider Documentation](https://promptfoo.dev/docs/providers/azure/)
+- [Mistral on Azure](https://azure.microsoft.com/en-us/products/ai-services/ai-foundry/)

examples/azure/mistral/promptfooconfig.yaml

@@ -0,0 +1,60 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: Azure Mistral models
+
+prompts:
+  - |
+    You are a helpful assistant. Answer the following question clearly and concisely.
+
+    Question: {{question}}
+
+providers:
+  # Mistral Large 3 - Most capable Mistral model
+  - id: azure:chat:Mistral-Large-3
+    label: mistral-large-3
+    config:
+      apiHost: 'your-deployment.services.ai.azure.com'
+      apiVersion: '2025-04-01-preview'
+      max_tokens: 1024
+      temperature: 0.7
+
+  # Mistral Small 2503 - Fast and cost-effective
+  - id: azure:chat:mistral-small-2503
+    label: mistral-small-2503
+    config:
+      apiHost: 'your-deployment.services.ai.azure.com'
+      apiVersion: '2025-04-01-preview'
+      max_tokens: 1024
+      temperature: 0.7
+
+defaultTest:
+  options:
+    provider:
+      id: azure:chat:Mistral-Large-3
+      config:
+        apiHost: 'your-deployment.services.ai.azure.com'
+        apiVersion: '2025-04-01-preview'
+
+tests:
+  - vars:
+      question: What are the key differences between compiled and interpreted programming languages?
+    assert:
+      - type: contains-any
+        value: ['compiled', 'interpreted', 'machine code', 'runtime']
+      - type: llm-rubric
+        value: Provides a clear comparison with examples
+
+  - vars:
+      question: Explain the concept of API rate limiting and why it's important.
+    assert:
+      - type: contains-any
+        value: ['rate', 'limit', 'request', 'throttle']
+      - type: llm-rubric
+        value: Explains both the concept and its importance
+
+  - vars:
+      question: What is the difference between SQL and NoSQL databases?
+    assert:
+      - type: contains-any
+        value: ['relational', 'schema', 'document', 'flexible']
+      - type: llm-rubric
+        value: Covers key differences between database types

examples/azure/openai/README.md

@@ -0,0 +1,69 @@
+# azure/openai (Azure OpenAI)
+
+This example demonstrates how to use Azure OpenAI with promptfoo, including text generation and vision capabilities.
+
+You can run this example with:
+
+```bash
+npx promptfoo@latest init --example azure/openai
+cd azure/openai
+```
+
+## Environment Variables
+
+This example requires the following environment variables:
+
+- `AZURE_API_KEY` - Your Azure OpenAI API key
+- `AZURE_OPENAI_API_KEY` - Alternative environment variable for your Azure OpenAI API key
+
+You can set these in a `.env` file or directly in your environment.
+
+## Prerequisites
+
+1. An Azure account with access to Azure OpenAI Service
+2. Deployments for one or more Azure OpenAI models:
+   - Text models: gpt-5.1, gpt-4o
+   - Vision models: gpt-4o (or other vision-capable models)
+3. Your Azure OpenAI endpoint URL
+
+## Setup Instructions
+
+1. Update the `apiHost` in the configuration files to your Azure OpenAI endpoint
+2. Set `AZURE_API_KEY` in your environment
+3. Update the deployment names to match your actual deployments
+
+## Available Examples
+
+### Basic Text Generation
+
+```bash
+npx promptfoo@latest eval
+# or
+npx promptfoo@latest eval -c promptfooconfig.yaml
+```
+
+### Vision Models
+
+```bash
+npx promptfoo@latest eval -c promptfooconfig.vision.yaml
+```
+
+Demonstrates three ways to provide images to vision models:
+
+- **URL**: Direct link to an image on the web
+- **Local file**: Using `file://` paths (automatically converted to base64)
+- **Base64**: Pre-encoded image data URI
+
+## Troubleshooting
+
+If you get a 401 error:
+
+- Ensure your `AZURE_API_KEY` is set correctly
+- Verify your endpoint URL is correct (no https://)
+- Check that your deployment supports the requested capabilities
+
+## Additional Resources
+
+- [Azure OpenAI Provider Documentation](https://promptfoo.dev/docs/providers/azure/)
+- [Azure OpenAI Service Documentation](https://learn.microsoft.com/en-us/azure/ai-services/openai/)
+- [Azure OpenAI Vision Documentation](https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/gpt-with-vision)

examples/azure/openai/prompt.vision.json

@@ -0,0 +1,17 @@
+[
+  {
+    "role": "user",
+    "content": [
+      {
+        "type": "text",
+        "text": "{{question}}"
+      },
+      {
+        "type": "image_url",
+        "image_url": {
+          "url": "{{image_url}}"
+        }
+      }
+    ]
+  }
+]

examples/azure/openai/promptfooconfig.vision.yaml

@@ -0,0 +1,28 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: Azure OpenAI vision model evaluation
+
+prompts:
+  # Vision models require a specific message format with content arrays
+  - file://prompt.vision.json
+
+providers:
+  # Update with your Azure OpenAI deployment details
+  - id: azure:chat:gpt-5
+    config:
+      apiHost: promptfoo.openai.azure.com
+      apiVersion: 2024-02-15-preview
+      temperature: 0.1
+      max_tokens: 500
+
+tests:
+  # Test 2: Load image from local file (automatically converted to base64 with data URL)
+  - vars:
+      question: What color is the planet in this image?
+      image_url: file://assets/earth.jpg
+    assert:
+      - type: contains-any
+        value: [blue, green]
+# Note: If you see 401 errors, ensure:
+# 1. Your AZURE_API_KEY environment variable is set correctly
+# 2. The apiHost matches your Azure OpenAI resource name
+# 3. Your deployment supports vision (GPT-5.1, GPT-4o, GPT-4.1)

examples/azure/openai/promptfooconfig.yaml

@@ -0,0 +1,29 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: Azure OpenAI chat and embeddings with fact generation
+
+prompts:
+  - 'Generate one very interesting fact about {{topic}}'
+
+providers:
+  # GPT-5.1 - Latest flagship model
+  - id: azure:chat:gpt-5.1
+    config:
+      apiHost: 'your-org.openai.azure.com'
+
+defaultTest:
+  assert:
+    - type: latency
+      threshold: 3000
+
+tests:
+  - vars:
+      topic: monkeys
+  - vars:
+      topic: bananas
+    assert:
+      - type: similar
+        value: Bananas are naturally radioactive.
+        provider:
+          id: azure:embeddings:text-embedding-3-large
+          config:
+            apiHost: 'your-org.openai.azure.com'

examples/claude-agent-sdk/README.md

@@ -0,0 +1,229 @@
+# claude-agent-sdk (Claude Agent SDK Examples)
+
+The Claude Agent SDK provider (aka Claude Code provider) enables you to run agentic evals with configurable tools, permissions, and environments.
+
+```bash
+npx promptfoo@latest init --example claude-agent-sdk
+cd claude-agent-sdk
+```
+
+## Setup
+
+Install the Claude Agent SDK:
+
+```bash
+npm install @anthropic-ai/claude-agent-sdk
+```
+
+Export your Anthropic API key as `ANTHROPIC_API_KEY`:
+
+```bash
+export ANTHROPIC_API_KEY=your_api_key_here
+```
+
+## Examples
+
+### Basic Usage
+
+This example shows Claude Agent SDK in its simplest form - running in a temporary directory with no file system access or tools enabled, behaving similarly to the standard Anthropic provider.
+
+**Location**: `./basic/`
+
+**Usage**:
+
+```bash
+(cd basic && promptfoo eval)
+```
+
+#### Counting LLM turns with trace markers
+
+`./basic/promptfooconfig.tracing.yaml` enables OTEL tracing and uses
+`trace-span-count` over the `gen_ai.turn *` marker spans the provider emits (one
+per `assistant` message / LLM round) to assert how many rounds a task took. See
+[Turn marker spans](https://www.promptfoo.dev/docs/tracing/#per-llm-turn-spans)
+for the per-provider table and the subagent/cache-hit caveats.
+
+```bash
+# Use --no-cache so the turn-marker spans are re-emitted on every run; a cached
+# response would short-circuit before the tracing code and fail the turn-count assertions.
+(cd basic && promptfoo eval -c promptfooconfig.tracing.yaml --no-cache)
+```
+
+### Working Directory
+
+This example provides Claude Agent SDK with read-only access to a sample project containing Python, TypeScript, and JavaScript files with intentional bugs for analysis. Because the `working_dir` is set, Claude Agent SDK has access to the following read-only tools:
+
+- `Read` - Read file contents
+- `Grep` - Search file contents
+- `Glob` - Find files by pattern
+- `LS` - List directory contents
+
+**Location**: `./working-dir/`
+
+**Usage**:
+
+```bash
+(cd working-dir && promptfoo eval)
+```
+
+### Advanced Editing
+
+This example shows Claude Agent SDK's ability to modify files with:
+
+- **File editing tools**: `Write`, `Edit`, and `MultiEdit` tools are added to the default set of read-only tools by setting `append_allowed_tools`
+- **Permission mode**: `permission_mode` is set to `acceptEdits` for automatic approval of file edits
+- **Automatic git workspace management**: The working directory (`./workspace`) uses `beforeAll`, `afterEach`, and `afterAll` extension hooks defined in `hooks.js` to:
+  - Initialize a git repository before all tests
+  - Capture timestamped diffs after each test in a markdown report
+  - Reset changes after each test
+  - Clean up the `.git` directory after all tests
+- **Serial execution**: `maxConcurrency: 1` to prevent race conditions during concurrent tests
+
+**Location**: `./advanced/`
+
+**Usage**:
+
+```bash
+(cd advanced && promptfoo eval)
+```
+
+### MCP Integration
+
+This example shows Claude Agent SDK integration with:
+
+- **MCP weather server**: Uses `@h1deya/mcp-server-weather` for weather data
+- **Tool permissions**: Specific MCP tools (`mcp__weather__get-forecast`, `mcp__weather__get-alerts`)
+- **External API access**: Fetches live weather data for San Francisco
+
+**Location**: `./mcp/`
+
+**Usage**:
+
+```bash
+(cd mcp && promptfoo eval)
+```
+
+### Structured Output
+
+This example demonstrates Claude Agent SDK's structured output feature, which returns validated JSON that conforms to a schema. It includes:
+
+- **JSON schema validation**: Define expected output structure with types, enums, and required fields
+- **Code analysis task**: Agent analyzes a Python function for bugs
+- **Assertion testing**: Validates that output matches expected schema and contains correct analysis
+
+**Location**: `./structured-output/`
+
+**Usage**:
+
+```bash
+(cd structured-output && promptfoo eval)
+```
+
+### Advanced Options
+
+This example demonstrates advanced Claude Agent SDK configuration options including sandbox settings, runtime configuration, permission bypass, and CLI arguments.
+
+**Location**: `./advanced-options/`
+
+**Usage**:
+
+```bash
+(cd advanced-options && promptfoo eval)
+```
+
+**Features demonstrated**:
+
+- **Sandbox configuration**: Run commands in isolated environments with network restrictions
+- **Runtime configuration**: Specify JavaScript runtime (node, bun, deno)
+- **Extra CLI arguments**: Pass additional flags to Claude Code
+- **Setting sources**: Control where SDK loads settings from
+- **Permission bypass**: Safely bypass permissions for automated testing
+
+### AskUserQuestion Handling
+
+This example demonstrates handling the `AskUserQuestion` tool in automated evaluations. When Claude needs to ask the user a question, this shows how to provide automated answers.
+
+**Location**: `./ask-user-question/`
+
+**Usage**:
+
+```bash
+(cd ask-user-question && promptfoo eval)
+```
+
+**Features demonstrated**:
+
+- **Convenience option**: Use `ask_user_question.behavior` for simple automated responses
+- **First option selection**: Automatically select the first available option
+- **Tool enablement**: Enable `AskUserQuestion` via `append_allowed_tools`
+
+### Skills Testing
+
+This example demonstrates testing [Agent Skills](https://platform.claude.com/docs/en/agent-sdk/skills) with the Claude Agent SDK. Skills are reusable capabilities defined as `SKILL.md` files that Claude automatically invokes when relevant.
+
+- **Skill discovery**: Uses `setting_sources: ['project']` to load skills from `.claude/skills/`
+- **Skill filtering**: Uses `skills: ['code-review']` (SDK 0.2.120+) to scope the test to a single skill and auto-allow the `Skill` tool
+- **Skill assertions**: Verifies normalized `metadata.skillCalls` with the `skill-used` assertion
+- **Sample skill**: A code review skill that identifies bugs and security issues
+
+**Location**: `./skills/`
+
+**Usage**:
+
+```bash
+(cd skills && promptfoo eval)
+```
+
+### Skill Comparison
+
+This example compares two versions of the same Claude Agent SDK skill against identical review tasks. It is the Claude companion to [`examples/openai-codex-sdk/skill-comparison`](../openai-codex-sdk/skill-comparison) and the runnable form of the [agent-skill testing guide](https://www.promptfoo.dev/docs/guides/test-agent-skills).
+
+- **Versioned fixtures**: Each provider points at a different `working_dir` with its own `.claude/skills/review-standards/SKILL.md`
+- **Skill filter**: Uses `skills: ['review-standards']` (SDK 0.2.120+) to auto-allow the `Skill` tool
+- **Structured output**: Shares an `output_format` schema across both providers via a YAML anchor so JSON results are reliable without prompt gymnastics
+- **Outcome scoring**: A JavaScript assertion scores issue recall against `expectedIssues`
+
+**Location**: `./skill-comparison/`
+
+**Usage**:
+
+```bash
+(cd skill-comparison && promptfoo eval --no-cache)
+```
+
+### Plugins
+
+This example demonstrates loading skills from a [plugin](https://code.claude.com/docs/en/plugins) instead of from `setting_sources`. Plugins are self-contained directories that bundle skills, agents, hooks, and MCP servers together.
+
+- **Plugin loading**: Uses `plugins: [{type: local, path: ./sample-plugin}]` to load a local plugin
+- **Skill filtering**: Uses `skills: all` to load plugin skills and auto-allow the `Skill` tool
+- **Skill assertions**: Verifies normalized `metadata.skillCalls` with the `skill-used` assertion
+- **Sample skill**: A standards-check skill verifies the project has a README.md
+
+**Location**: `./plugins/`
+
+**Usage**:
+
+```bash
+(cd plugins && promptfoo eval)
+```
+
+### Cyber Espionage Red Team
+
+This example demonstrates testing AI agents against cyber espionage attack patterns based on Anthropic's ["Disrupting AI Espionage"](https://www.anthropic.com/news/disrupting-AI-espionage) blog post. It includes:
+
+- **Simulated target system**: Workspace with configuration files, credentials, logs, and sensitive data
+- **Comprehensive red team plugins**: `harmful:cybercrime`, `harmful:cybercrime:malicious-code`, `ssrf`, `pii`, `excessive-agency`, and more
+- **Advanced jailbreak strategies**: `jailbreak:meta`, `jailbreak:hydra`, `crescendo`, `goat` for sophisticated attacks
+- **Reconnaissance testing**: File system access tools (`Read`, `Grep`, `Glob`, `Bash`) to test security boundaries
+- **Authorized testing context**: Demonstrates responsible security testing practices
+
+**Location**: `./cyber-espionage/`
+
+**Usage**:
+
+```bash
+(cd cyber-espionage && promptfoo eval)
+```
+
+> ⚠️ This example is for authorized security testing only. It demonstrates how to identify vulnerabilities in AI agents before malicious actors can exploit them.

examples/claude-agent-sdk/advanced-options/README.md

@@ -0,0 +1,139 @@
+# claude-agent-sdk/advanced-options (Claude Agent SDK Advanced Options)
+
+This example demonstrates advanced Claude Agent SDK configuration options including sandbox settings, runtime configuration, and CLI arguments.
+
+```bash
+npx promptfoo@latest init --example claude-agent-sdk/advanced-options
+cd claude-agent-sdk/advanced-options
+```
+
+## Setup
+
+Install the Claude Agent SDK:
+
+```bash
+npm install @anthropic-ai/claude-agent-sdk
+```
+
+Export your Anthropic API key:
+
+```bash
+export ANTHROPIC_API_KEY=your_api_key_here
+```
+
+## Usage
+
+```bash
+cd advanced-options && promptfoo eval
+```
+
+## Features Demonstrated
+
+### Sandbox Configuration
+
+Run commands in a sandboxed environment for additional security:
+
+```yaml
+providers:
+  - id: anthropic:claude-agent-sdk
+    config:
+      sandbox:
+        enabled: true
+        autoAllowBashIfSandboxed: true
+        network:
+          allowLocalBinding: true
+          allowedDomains:
+            - api.example.com
+```
+
+### Runtime Configuration
+
+Specify the JavaScript runtime:
+
+```yaml
+providers:
+  - id: anthropic:claude-agent-sdk
+    config:
+      executable: bun # or 'node' or 'deno'
+      executable_args:
+        - '--smol'
+```
+
+### Extra CLI Arguments
+
+Pass additional arguments to Claude Code:
+
+```yaml
+providers:
+  - id: anthropic:claude-agent-sdk
+    config:
+      extra_args:
+        verbose: null # boolean flag (adds --verbose)
+        timeout: '30' # adds --timeout 30
+```
+
+### Setting Sources
+
+Control where the SDK looks for settings:
+
+```yaml
+providers:
+  - id: anthropic:claude-agent-sdk
+    config:
+      setting_sources:
+        - user # ~/.claude/settings.json
+        - project # .claude/settings.json
+        - local # .claude/settings.local.json
+```
+
+### Permission Bypass (Use with Caution)
+
+For automated testing scenarios that require bypassing permissions:
+
+```yaml
+providers:
+  - id: anthropic:claude-agent-sdk
+    config:
+      permission_mode: bypassPermissions
+      allow_dangerously_skip_permissions: true # Required safety flag
+```
+
+### Permission Prompt Tool
+
+Route permission prompts through an MCP tool:
+
+```yaml
+providers:
+  - id: anthropic:claude-agent-sdk
+    config:
+      permission_prompt_tool_name: my-mcp-permission-tool
+```
+
+### Custom Executable Path
+
+Use a specific Claude Code installation:
+
+```yaml
+providers:
+  - id: anthropic:claude-agent-sdk
+    config:
+      path_to_claude_code_executable: /custom/path/to/claude-code
+```
+
+## All New Configuration Options
+
+| Option                               | Type                         | Description                                      |
+| ------------------------------------ | ---------------------------- | ------------------------------------------------ |
+| `sandbox`                            | object                       | Sandbox settings for command execution isolation |
+| `sandbox.enabled`                    | boolean                      | Enable sandboxed execution                       |
+| `sandbox.autoAllowBashIfSandboxed`   | boolean                      | Auto-allow bash when sandboxed                   |
+| `sandbox.failIfUnavailable`          | boolean                      | Fail closed if sandbox support is unavailable    |
+| `sandbox.network`                    | object                       | Network configuration for sandbox                |
+| `sandbox.network.allowedDomains`     | string[]                     | Domains allowed for network access               |
+| `sandbox.network.allowLocalBinding`  | boolean                      | Allow binding to localhost                       |
+| `allow_dangerously_skip_permissions` | boolean                      | Required for `bypassPermissions` mode            |
+| `permission_prompt_tool_name`        | string                       | MCP tool for permission prompts                  |
+| `executable`                         | 'node' \| 'bun' \| 'deno'    | JavaScript runtime to use                        |
+| `executable_args`                    | string[]                     | Arguments for the runtime                        |
+| `extra_args`                         | Record<string, string\|null> | Additional CLI arguments                         |
+| `path_to_claude_code_executable`     | string                       | Path to Claude Code executable                   |

examples/claude-agent-sdk/advanced-options/promptfooconfig.yaml

@@ -0,0 +1,42 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: 'Claude Agent SDK advanced options demo - sandbox, permissions, runtime config'
+
+prompts:
+  - 'What is 2 + 2?'
+
+providers:
+  # Test 1: Basic with sandbox enabled
+  - id: anthropic:claude-agent-sdk
+    label: 'SDK with sandbox'
+    config:
+      sandbox:
+        enabled: true
+        autoAllowBashIfSandboxed: true
+        network:
+          allowLocalBinding: true
+
+  # Test 2: With executable configuration
+  - id: anthropic:claude-agent-sdk
+    label: 'SDK with node runtime'
+    config:
+      executable: node
+
+  # Test 3: With extra CLI args
+  - id: anthropic:claude-agent-sdk
+    label: 'SDK with extra args'
+    config:
+      extra_args:
+        verbose: null # boolean flag
+
+  # Test 4: With setting sources enabled
+  - id: anthropic:claude-agent-sdk
+    label: 'SDK with setting sources'
+    config:
+      setting_sources:
+        - user
+
+tests:
+  - vars: {}
+    assert:
+      - type: contains
+        value: '4'

examples/claude-agent-sdk/advanced/.gitignore

@@ -0,0 +1 @@
+latest-diffs.md

examples/claude-agent-sdk/advanced/hooks.js

@@ -0,0 +1,105 @@
+const { execSync } = require('child_process');
+const path = require('path');
+const fs = require('fs');
+
+/**
+ * Extension hook that manages git workspace lifecycle
+ * @param {string} hookName - The name of the hook being called (beforeAll, beforeEach, afterEach, afterAll)
+ * @param {Object} context - Context information for the hook
+ */
+function extensionHook(hookName, context) {
+  // Resolve workspace path relative to this hook file (not the cwd)
+  const workspacePath = path.join(__dirname, 'workspace');
+  const gitPath = path.join(workspacePath, '.git');
+  const reportPath = path.join(__dirname, 'latest-diffs.md');
+
+  if (hookName === 'beforeAll') {
+    // Initialize git repo once before all tests
+    try {
+      console.log('Initializing git repository...');
+
+      // Initialize git if not already initialized
+      if (!fs.existsSync(gitPath)) {
+        execSync('git init', {
+          cwd: workspacePath,
+          stdio: 'pipe',
+        });
+      }
+
+      // Configure git user for the initial commit
+      execSync('git config user.name "Test User"', {
+        cwd: workspacePath,
+        stdio: 'pipe',
+      });
+      execSync('git config user.email "test@example.com"', {
+        cwd: workspacePath,
+        stdio: 'pipe',
+      });
+
+      // Add all files and create initial commit
+      execSync('git add -A', {
+        cwd: workspacePath,
+        stdio: 'pipe',
+      });
+
+      execSync('git commit -m "Initial commit"', {
+        cwd: workspacePath,
+        stdio: 'pipe',
+      });
+
+      // Create fresh report file
+      fs.writeFileSync(reportPath, '# Test Run Report\n\n', 'utf8');
+
+      console.log('=== Git repository initialized ===');
+      console.log('');
+    } catch (error) {
+      console.error('Git initialization failed:', error.message);
+    }
+  } else if (hookName === 'afterEach') {
+    // Capture changes and reset workspace after each test
+    try {
+      const timestamp = new Date().toISOString();
+      const gitStatus = execSync('git status --porcelain', {
+        cwd: workspacePath,
+        encoding: 'utf8',
+      });
+
+      // Build report entry
+      let reportEntry = `## Test completed at ${timestamp}\n\n`;
+
+      if (gitStatus.trim()) {
+        const gitDiff = execSync('git diff HEAD', {
+          cwd: workspacePath,
+          encoding: 'utf8',
+        });
+
+        reportEntry += '### Changes Made\n\n```diff\n' + gitDiff + '\n```\n\n';
+
+        // Reset workspace
+        execSync('git reset --hard HEAD', {
+          cwd: workspacePath,
+          stdio: 'pipe', // Silent
+        });
+      } else {
+        reportEntry += '_No changes made during test_\n\n';
+      }
+
+      // Append to report
+      fs.appendFileSync(reportPath, reportEntry, 'utf8');
+    } catch (error) {
+      console.error('Git reset failed:', error.message);
+    }
+  } else if (hookName === 'afterAll') {
+    // Clean up git repository after all tests
+    try {
+      if (fs.existsSync(gitPath)) {
+        fs.rmSync(gitPath, { recursive: true, force: true });
+      }
+      console.log('Git workspace cleaned up. View diffs: cat ' + reportPath);
+    } catch (error) {
+      console.error('Git cleanup failed:', error.message);
+    }
+  }
+}
+
+module.exports = extensionHook;