chore: import upstream snapshot with attribution

2026-07-13 13:24:08 +08:00
commit 0d3cb498a3
5438 changed files with 1316560 additions and 0 deletions
@@ -0,0 +1,67 @@
+# azure (Azure AI Examples)
+
+This directory contains examples for using Azure AI services with promptfoo, including Azure OpenAI, Azure AI Foundry, and third-party models available through Microsoft Foundry.
+
+## Available Examples
+
+### Azure OpenAI
+
+| Example                           | Description                         |
+| --------------------------------- | ----------------------------------- |
+| [openai](./openai/)               | Azure OpenAI chat and vision models |
+| [assistant](./assistant/)         | Azure OpenAI Assistants with tools  |
+| [foundry-agent](./foundry-agent/) | Azure AI Foundry Agents             |
+
+### Third-Party Models (Azure AI Foundry)
+
+| Example                     | Description                                   |
+| --------------------------- | --------------------------------------------- |
+| [claude](./claude/)         | Anthropic Claude models (Opus, Sonnet, Haiku) |
+| [llama](./llama/)           | Meta Llama models (4, 3.3, 3.1)               |
+| [deepseek](./deepseek/)     | DeepSeek models including R1 reasoning        |
+| [mistral](./mistral/)       | Mistral models (Large, Ministral)             |
+| [comparison](./comparison/) | Compare models across providers               |
+
+## Quick Start
+
+```bash
+# Azure OpenAI basic example
+npx promptfoo@latest init --example azure/openai
+
+# Azure Assistants with tools
+npx promptfoo@latest init --example azure/assistant
+
+# Azure AI Foundry Agents
+npx promptfoo@latest init --example azure/foundry-agent
+
+# Third-party models
+npx promptfoo@latest init --example azure/claude
+npx promptfoo@latest init --example azure/llama
+npx promptfoo@latest init --example azure/deepseek
+npx promptfoo@latest init --example azure/mistral
+npx promptfoo@latest init --example azure/comparison
+```
+
+## Environment Variables
+
+All Azure examples require authentication. Set one of:
+
+```bash
+# Option 1: API Key (simplest)
+export AZURE_API_KEY=your-api-key
+export AZURE_API_HOST=your-resource.openai.azure.com
+
+# Option 2: Azure CLI (recommended for development)
+az login
+
+# Option 3: Service Principal
+export AZURE_CLIENT_ID=your-client-id
+export AZURE_CLIENT_SECRET=your-client-secret
+export AZURE_TENANT_ID=your-tenant-id
+```
+
+## Documentation
+
+- [Azure Provider Documentation](https://promptfoo.dev/docs/providers/azure/)
+- [Azure OpenAI Service](https://learn.microsoft.com/en-us/azure/ai-services/openai/)
+- [Azure AI Foundry](https://learn.microsoft.com/en-us/azure/ai-foundry/)
@@ -0,0 +1,177 @@
+# azure/assistant (Azure OpenAI Assistants API with Tools)
+
+Evaluate Azure OpenAI Assistants with file search, function tools, and multi-tool interactions.
+
+You can run this example with:
+
+```bash
+npx promptfoo@latest init --example azure/assistant
+cd azure/assistant
+```
+
+## Features
+
+- File search via vector stores
+- Custom function tools with simple implementations
+- Multi-tool examples with combined capabilities
+- Progressive examples from basic to advanced use cases
+
+## Environment Variables
+
+This example requires the following environment variables:
+
+- `AZURE_API_KEY` - Your Azure OpenAI API key
+- `AZURE_OPENAI_API_HOST` - Your Azure OpenAI API host (e.g., "your-resource-name.openai.azure.com")
+
+You can set these in a `.env` file or directly in your environment.
+
+## Prerequisites
+
+1. An Azure OpenAI account with access to the Assistants API
+2. An assistant created in your Azure OpenAI account
+3. A vector store for file search functionality (optional)
+
+## Configuration Files
+
+This example includes several configuration files, each demonstrating different tool capabilities:
+
+1. **promptfooconfig-file-search.yaml** - File search tool only
+2. **promptfooconfig-function.yaml** - Function tool capability with a weather API implementation
+3. **promptfooconfig-multi-tool.yaml** - Combined file search and function tools
+
+### Running Different Configurations
+
+To run a specific configuration:
+
+```bash
+# File search example
+npx promptfoo@latest eval -c promptfooconfig-file-search.yaml
+
+# Function tool capability example
+npx promptfoo@latest eval -c promptfooconfig-function.yaml
+
+# Multi-tool example
+npx promptfoo@latest eval -c promptfooconfig-multi-tool.yaml
+```
+
+## Tool Capabilities
+
+### File Search
+
+The file search configuration demonstrates how to use vector store-backed file search.
+
+Key components:
+
+- Simple `tools` configuration with `type: "file_search"` (no description field)
+- `tool_resources` with vector store ID configuration
+- Test cases focused on information retrieval
+
+**Important**: The file search tool must be defined without a description field, as Azure OpenAI API does not support it for this tool type.
+
+### Function Tool
+
+The function tool configuration shows how to define and use custom functions.
+
+Key components:
+
+- Function tool definition loaded from external file (`tools/weather-function.json`)
+- External function callback implementation in `callbacks/weather.js`
+- Test cases demonstrating tool invocation
+
+### Multi-Tool Usage
+
+The multi-tool configuration demonstrates how to combine multiple tools.
+
+Key components:
+
+- External tools definition file combining file search and function tools
+- Multiple inline function callbacks
+- Test cases requiring coordination between different tools
+
+## Customization
+
+To use this example with your own Azure OpenAI Assistant:
+
+1. Update the assistant ID in each configuration file (replace `your_assistant_id` with your actual ID)
+2. Replace `your_vector_store_id` with your own vector store ID
+3. Set the `apiHost` to match your Azure OpenAI endpoint (e.g., "your-resource-name.openai.azure.com")
+4. Customize the tools and function callbacks as needed
+
+## Function Implementation Approaches
+
+This example demonstrates two approaches to implementing functions:
+
+### 1. External Tool Definition with External Callback
+
+```yaml
+# Load tools from external file
+tools: file://tools/weather-function.json
+
+# External file-based callback
+functionToolCallbacks:
+  get_weather: file://callbacks/weather.js:getWeather
+```
+
+### 2. Multiple Tools with Inline Callbacks
+
+For more complex scenarios, you can combine multiple tools and inline function callbacks:
+
+```yaml
+# Multiple tools defined
+tools: file://tools/multiple-tools.json
+
+# Multiple inline function callbacks
+functionToolCallbacks:
+  get_weather: |
+    async function(args) {
+      // Weather function implementation
+    }
+  suggest_recipe: |
+    async function(args) {
+      // Recipe function implementation
+    }
+```
+
+## Function Callback Context
+
+Function callbacks can optionally receive context information about the current conversation:
+
+```javascript
+// Enhanced callback with context for audit logging
+functionToolCallbacks: {
+  get_employee_data: async (args, context) => {
+    const { employeeId } = JSON.parse(args);
+
+    // Log access with thread context for audit trail
+    console.log(`Access to employee ${employeeId} requested`, {
+      threadId: context?.threadId,
+      timestamp: new Date().toISOString(),
+      provider: context?.provider,
+    });
+
+    // Your function logic here
+    return JSON.stringify({ name: 'John Doe', department: 'Engineering' });
+  };
+}
+```
+
+The context object includes:
+
+- `threadId`: Unique identifier for the conversation thread
+- `runId`: Identifier for the current assistant run
+- `assistantId`: The assistant being used
+- `provider`: The provider type ('azure' or 'openai')
+
+This is particularly useful for session management, audit logging, and tracking stateful interactions across function calls.
+
+## Documentation
+
+For more information about using Azure OpenAI with promptfoo, including authentication methods, provider types, and configuration options, see the [official Azure provider documentation](https://www.promptfoo.dev/docs/providers/azure/).
+
+## Notes
+
+- The file search capability requires a properly configured vector store
+- Both tool definitions and function callbacks can be implemented inline or loaded from external files
+- For production use, consider more robust error handling
+- **File search tool format**: The file search tool must be defined only with `type: "file_search"` without a description field. Adding a description will cause API errors
+- The examples use placeholder values that must be replaced with your actual IDs and endpoints
@@ -0,0 +1,35 @@
+/**
+ * Sample weather function callback implementation
+ * @param {string} args - JSON string with the function arguments
+ * @returns {string} - JSON string with the weather information
+ */
+function getWeather(args) {
+  try {
+    // Parse the JSON string from the Azure API
+    const parsedArgs = JSON.parse(args);
+
+    // Extract parameters with defaults
+    const location = parsedArgs.location;
+    const unit = parsedArgs.unit || 'celsius';
+
+    if (!location) {
+      return JSON.stringify({ error: 'Location is required' });
+    }
+
+    // Mock weather data
+    const mockWeather = {
+      location,
+      temperature: unit === 'celsius' ? 22 : 72,
+      unit,
+      forecast: ['sunny', 'partly cloudy', 'clear'][Math.floor(Math.random() * 3)],
+      humidity: Math.floor(Math.random() * 40) + 30,
+    };
+
+    return JSON.stringify(mockWeather);
+  } catch (error) {
+    console.error('Error in getWeather function:', error);
+    return JSON.stringify({ error: `Failed to process weather request: ${error.message}` });
+  }
+}
+
+module.exports = { getWeather };
@@ -0,0 +1,110 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: Azure OpenAI Assistant with File Search Tool
+
+prompts:
+  - |
+    You are a helpful customer service assistant for TechGadgets, an online electronics retailer.
+    Answer the following customer question:
+
+    {{prompt}}
+
+providers:
+  - id: azure:assistant:your_assistant_id
+    label: azure-assistant
+    config:
+      # Remove any http or https prefix and trailing slashes from apiHost
+      apiHost: your-resource-name.openai.azure.com
+      # Simple file search tool definition
+      tools:
+        - type: 'file_search'
+      # Vector store configuration - replace with your actual vector store ID
+      tool_resources:
+        file_search:
+          vector_store_ids:
+            - 'your_vector_store_id' # Vector store containing product and policy documentation
+      # Standard parameters
+      temperature: 0.7
+      apiVersion: '2024-05-01-preview'
+
+# Override the default grader (use your gpt-5.1 deployment name)
+defaultTest:
+  options:
+    provider: azure:chat:gpt-5.1-deployment
+
+tests:
+  - vars:
+      prompt: What is your return policy for electronics? Can I return items after 30 days?
+    assert:
+      - type: icontains
+        value: return policy
+      - type: llm-rubric
+        value: Provides a clear, concise explanation of the return policy including the time frame and any conditions
+      - type: factuality
+        value: |
+          The answer must include these facts about our return policy:
+          - Standard electronics have a 30-day return window
+          - Premium electronics (smartphones, laptops, tablets) have a 14-day return window
+          - Items must be in original packaging with all accessories
+          - Restocking fee of 15% applies after 7 days
+          - Returns after 30 days are only accepted for store credit
+
+  - vars:
+      prompt: How do I track my order? I ordered a laptop last week and haven't received any updates.
+    assert:
+      - type: llm-rubric
+        value: Directly addresses how to track an order and provides specific steps the customer can take to get order updates
+      - type: factuality
+        value: |
+          The answer must accurately mention:
+          - Orders can be tracked on our website at techgadgets.com/order-tracking
+          - You need your order number and email to track
+          - Premium laptops typically ship within 2-3 business days
+          - Customers receive tracking emails when items ship
+          - Customer service can be reached at 1-800-TECH-HELP for shipping inquiries
+
+  - vars:
+      prompt: Are there any discounts available for students buying computer accessories?
+    assert:
+      - type: icontains
+        value: discount
+      - type: llm-rubric
+        value: |
+          Clearly explains available student discounts for computer accessories.
+          Score 0.7+ if it provides specific discount percentages or eligibility requirements.
+          Score 1.0 if it also explains how to verify student status and apply the discount.
+      - type: factuality
+        value: |
+          The answer must accurately mention these facts:
+          - Students receive a 15% discount on all computer accessories
+          - Verification requires a valid .edu email address or student ID
+          - The discount can be combined with seasonal sales (up to 25% total discount)
+          - Student discounts do not apply to already discounted items or clearance products
+          - The TechGadgets Student Program provides additional benefits including free shipping
+
+  - vars:
+      prompt: My new smartphone has a cracked screen right out of the box. What should I do?
+    assert:
+      - type: icontains
+        value: warranty
+      - type: factuality
+        value: |
+          The answer must accurately mention these facts about our damage policy:
+          - Damaged products received must be reported within 7 days of delivery
+          - Photo evidence must be submitted through the return portal
+          - TechGadgets offers free replacement for items damaged during shipping
+          - Manufacturer defects are covered under the manufacturer's warranty (1 year minimum)
+          - Expedited replacements can be arranged for premium customers
+      - type: llm-rubric
+        value: |
+          Provides clear, actionable steps for reporting and resolving a damaged product issue.
+          Should include information about contacting customer service, warranty coverage, and return options.
+        threshold: 0.8
+
+  - vars:
+      prompt: How do I mine cryptocurrency using your products?
+    assert:
+      - type: llm-rubric
+        value: |
+          The response should clearly indicate if cryptocurrency mining is outside the scope of standard customer support.
+          It should NOT provide detailed instructions about cryptocurrency mining if this contradicts company policy.
+          Score 1.0 if it politely addresses the question while redirecting to appropriate products or indicating policy limitations.
@@ -0,0 +1,30 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: Azure OpenAI Assistant with Function Tool Capability
+
+prompts:
+  - 'Answer the following question: {{prompt}}'
+
+providers:
+  - id: azure:assistant:your_assistant_id
+    label: azure-assistant-function
+    config:
+      apiHost: your-resource-name.openai.azure.com
+      # Load tools from external file
+      tools: file://tools/weather-function.json
+      # Use the external file-based callback
+      functionToolCallbacks:
+        get_weather: file://callbacks/weather.js:getWeather
+      temperature: 0.7
+      apiVersion: '2024-05-01-preview'
+      debug: true
+
+tests:
+  # Function tool invocation tests
+  - vars:
+      prompt: What's the weather like in Seattle?
+  - vars:
+      prompt: Tell me about the weather in Tokyo in Celsius.
+  - vars:
+      prompt: Compare the weather in New York and London.
+  - vars:
+      prompt: What's the weather forecast for San Francisco in Fahrenheit?
@@ -0,0 +1,76 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: Azure OpenAI Assistant with Multiple Tools
+
+prompts:
+  - '{{prompt}}'
+
+providers:
+  - id: azure:assistant:your_assistant_id
+    label: azure-assistant-multi-tool
+    config:
+      apiHost: your-resource-name.openai.azure.com
+      # Load tools from external file
+      tools: file://tools/multiple-tools.json
+      # Function callbacks for each tool
+      functionToolCallbacks:
+        get_weather: |
+          async function(args) {
+            try {
+              const parsedArgs = JSON.parse(args);
+              const location = parsedArgs.location;
+              const unit = parsedArgs.unit || 'c';
+              console.log(`Weather request for ${location} in ${unit}`);
+              
+              // Simple weather response
+              return JSON.stringify({
+                location,
+                temperature: unit === 'c' ? 22 : 72,
+                unit,
+                condition: 'sunny',
+                forecast: 'Clear skies for the next few days.'
+              });
+            } catch (error) {
+              console.error('Error in get_weather function:', error);
+              return JSON.stringify({ error: String(error) });
+            }
+          }
+        suggest_recipe: |
+          async function(args) {
+            try {
+              const parsedArgs = JSON.parse(args);
+              const ingredients = parsedArgs.ingredients || [];
+              console.log(`Recipe requested with ingredients: ${ingredients.join(', ')}`);
+              
+              // Simple recipe suggestion
+              return JSON.stringify({
+                recipe: {
+                  name: "Simple Pasta",
+                  ingredients: ["pasta", "olive oil", "garlic", "salt"],
+                  instructions: "1. Boil pasta\n2. Heat oil and garlic\n3. Toss pasta in oil\n4. Season to taste"
+                }
+              });
+            } catch (error) {
+              console.error('Error in suggest_recipe function:', error);
+              return JSON.stringify({ error: String(error) });
+            }
+          }
+      # Vector store configuration for file search
+      tool_resources:
+        file_search:
+          vector_store_ids:
+            - 'your_vector_store_id'
+      # Standard parameters
+      temperature: 0.7
+      apiVersion: '2024-05-01-preview'
+      debug: true
+
+tests:
+  # Multi-tool usage tests
+  - vars:
+      prompt: I'm planning to run some LLM evaluations tomorrow. Can you tell me about promptfoo's command line options and also check the weather in San Francisco?
+  - vars:
+      prompt: I need information about vector stores in promptfoo, the weather in Tokyo, and a simple pasta recipe for dinner.
+  - vars:
+      prompt: How do I set up a promptfoo evaluation? By the way, what's the weather like in Seattle today?
+  - vars:
+      prompt: I have pasta and garlic. Can you suggest a recipe and also help me understand how to use custom assertions in promptfoo?
@@ -0,0 +1,47 @@
+[
+  {
+    "type": "file_search"
+  },
+  {
+    "type": "function",
+    "function": {
+      "name": "get_weather",
+      "description": "Get current weather information for a location",
+      "parameters": {
+        "type": "object",
+        "properties": {
+          "location": {
+            "type": "string",
+            "description": "The city and state e.g. Seattle, WA"
+          },
+          "unit": {
+            "type": "string",
+            "enum": ["c", "f"],
+            "description": "Temperature unit (c for Celsius, f for Fahrenheit)"
+          }
+        },
+        "required": ["location"]
+      }
+    }
+  },
+  {
+    "type": "function",
+    "function": {
+      "name": "suggest_recipe",
+      "description": "Suggest a recipe based on ingredients",
+      "parameters": {
+        "type": "object",
+        "properties": {
+          "ingredients": {
+            "type": "array",
+            "items": {
+              "type": "string"
+            },
+            "description": "Available ingredients"
+          }
+        },
+        "required": ["ingredients"]
+      }
+    }
+  }
+]
@@ -0,0 +1,24 @@
+[
+  {
+    "type": "function",
+    "function": {
+      "name": "get_weather",
+      "description": "Get current weather information for a location",
+      "parameters": {
+        "type": "object",
+        "properties": {
+          "location": {
+            "type": "string",
+            "description": "The city and state, e.g., San Francisco, CA"
+          },
+          "unit": {
+            "type": "string",
+            "enum": ["celsius", "fahrenheit"],
+            "description": "The unit of temperature to use. Infer this from the query. Defaults to celsius if not specified."
+          }
+        },
+        "required": ["location"]
+      }
+    }
+  }
+]
@@ -0,0 +1,48 @@
+# azure/claude (Azure Claude Models)
+
+This example demonstrates how to use Anthropic Claude models on Azure AI Foundry with promptfoo.
+
+You can run this example with:
+
+```bash
+npx promptfoo@latest init --example azure/claude
+cd azure/claude
+```
+
+## Setup
+
+1. Deploy Claude models in Azure AI Foundry
+2. Set your environment variables:
+
+```bash
+export AZURE_API_KEY=your-api-key
+export AZURE_API_HOST=your-deployment.services.ai.azure.com
+```
+
+## Available Claude Models
+
+| Model                       | Description                    |
+| --------------------------- | ------------------------------ |
+| `claude-opus-4-6-20260205`  | Claude Opus 4.6 - Most capable |
+| `claude-sonnet-4-6`         | Claude Sonnet 4.6 - Balanced   |
+| `claude-haiku-4-5-20251001` | Claude Haiku 4.5 - Fast        |
+
+## Running the Example
+
+```bash
+npx promptfoo@latest eval
+npx promptfoo@latest view
+```
+
+## Configuration
+
+The example compares Claude Opus 4.6, Claude Sonnet 4.6, and Claude Haiku 4.5 on explanation tasks. Modify `promptfooconfig.yaml` to:
+
+- Change models by updating the provider IDs
+- Adjust temperature and max_tokens
+- Add more test cases
+
+## Documentation
+
+- [Azure Provider Documentation](https://promptfoo.dev/docs/providers/azure/)
+- [Claude on Azure](https://azure.microsoft.com/en-us/products/ai-services/ai-foundry/)
@@ -0,0 +1,55 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: Azure Claude models
+
+prompts:
+  - 'Explain {{concept}} in simple terms, suitable for a beginner.'
+
+providers:
+  - id: azure:chat:claude-opus-4-6-20260205
+    label: claude-opus-4-6
+    config:
+      apiHost: 'your-deployment.services.ai.azure.com'
+      apiVersion: '2025-11-15-preview'
+      max_tokens: 1024
+      temperature: 0.7
+
+  - id: azure:chat:claude-sonnet-4-6
+    label: claude-sonnet-4-6
+    config:
+      apiHost: 'your-deployment.services.ai.azure.com'
+      apiVersion: '2025-04-01-preview'
+      max_tokens: 1024
+      temperature: 0.7
+
+  - id: azure:chat:claude-haiku-4-5-20251001
+    label: claude-haiku-4-5
+    config:
+      apiHost: 'your-deployment.services.ai.azure.com'
+      apiVersion: '2025-04-01-preview'
+      max_tokens: 1024
+      temperature: 0.7
+
+tests:
+  - vars:
+      concept: machine learning
+    assert:
+      - type: contains-any
+        value: ['algorithm', 'data', 'pattern', 'model']
+      - type: llm-rubric
+        value: The explanation should be clear and accessible to beginners
+
+  - vars:
+      concept: blockchain
+    assert:
+      - type: contains-any
+        value: ['decentralized', 'ledger', 'transaction', 'block']
+      - type: llm-rubric
+        value: The explanation should avoid technical jargon
+
+  - vars:
+      concept: quantum computing
+    assert:
+      - type: contains-any
+        value: ['qubit', 'superposition', 'quantum']
+      - type: llm-rubric
+        value: The explanation should use helpful analogies
@@ -0,0 +1,57 @@
+# azure/comparison (Azure Model Comparison)
+
+This example demonstrates how to compare models from different providers on Azure AI Foundry, including OpenAI, Anthropic Claude, Meta Llama, and Mistral.
+
+You can run this example with:
+
+```bash
+npx promptfoo@latest init --example azure/comparison
+cd azure/comparison
+```
+
+## Setup
+
+1. Deploy models from different providers in Azure AI Foundry
+2. Set your environment variables:
+
+```bash
+export AZURE_API_KEY=your-api-key
+# Set apiHost in promptfooconfig.yaml for each provider's deployment
+```
+
+## Models Compared
+
+| Provider  | Model                                    | Label         |
+| --------- | ---------------------------------------- | ------------- |
+| OpenAI    | `gpt-5.1`                                | gpt-5.1       |
+| Anthropic | `claude-sonnet-4-6`                      | claude-sonnet |
+| Meta      | `Llama-4-Maverick-17B-128E-Instruct-FP8` | llama-4       |
+| Mistral   | `Mistral-Large-2411`                     | mistral-large |
+
+## Running the Example
+
+```bash
+npx promptfoo@latest eval
+npx promptfoo@latest view
+```
+
+## Customization
+
+Modify `promptfooconfig.yaml` to:
+
+- Add or remove models
+- Change test questions
+- Adjust evaluation criteria
+- Compare cost vs performance
+
+## Use Cases
+
+- Benchmark different models on your specific tasks
+- Evaluate cost-effectiveness across providers
+- Find the best model for your use case
+- A/B test model updates
+
+## Documentation
+
+- [Azure Provider Documentation](https://promptfoo.dev/docs/providers/azure/)
+- [Azure AI Foundry](https://azure.microsoft.com/en-us/products/ai-services/ai-foundry/)
@@ -0,0 +1,77 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: Azure model comparison
+
+prompts:
+  - |
+    You are a helpful assistant. Answer the following question concisely and accurately.
+
+    Question: {{question}}
+
+providers:
+  # OpenAI GPT-5.1 (Latest flagship)
+  - id: azure:chat:gpt-5.1
+    label: gpt-5.1
+    config:
+      apiHost: 'your-resource.openai.azure.com'
+      max_tokens: 1024
+      temperature: 0.7
+
+  # Anthropic Claude Sonnet 4.6
+  - id: azure:chat:claude-sonnet-4-6
+    label: claude-sonnet
+    config:
+      apiHost: 'your-deployment.services.ai.azure.com'
+      apiVersion: '2025-04-01-preview'
+      max_tokens: 1024
+      temperature: 0.7
+
+  # Meta Llama 4 Maverick
+  - id: azure:chat:Llama-4-Maverick-17B-128E-Instruct-FP8
+    label: llama-4
+    config:
+      apiHost: 'your-deployment.services.ai.azure.com'
+      apiVersion: '2025-04-01-preview'
+      max_tokens: 1024
+      temperature: 0.7
+
+  # Mistral Large
+  - id: azure:chat:Mistral-Large-2411
+    label: mistral-large
+    config:
+      apiHost: 'your-deployment.services.ai.azure.com'
+      apiVersion: '2025-04-01-preview'
+      max_tokens: 1024
+      temperature: 0.7
+
+tests:
+  - vars:
+      question: What are the main differences between machine learning and deep learning?
+    assert:
+      - type: llm-rubric
+        value: Accurately explains the relationship and differences
+      - type: contains-any
+        value: ['neural network', 'subset', 'feature']
+
+  - vars:
+      question: Explain the concept of recursion in programming with a simple example.
+    assert:
+      - type: llm-rubric
+        value: Provides a clear explanation with a working example
+      - type: contains-any
+        value: ['base case', 'function', 'itself']
+
+  - vars:
+      question: What is the difference between REST and GraphQL APIs?
+    assert:
+      - type: llm-rubric
+        value: Accurately compares both API paradigms
+      - type: contains-any
+        value: ['endpoint', 'query', 'schema']
+
+  - vars:
+      question: How does HTTPS encryption work?
+    assert:
+      - type: llm-rubric
+        value: Explains the encryption process accurately
+      - type: contains-any
+        value: ['TLS', 'SSL', 'certificate', 'handshake']
@@ -0,0 +1,54 @@
+# azure/deepseek (Azure DeepSeek Models)
+
+This example demonstrates how to use DeepSeek models on Azure AI Foundry with promptfoo, including the DeepSeek-R1 reasoning model.
+
+You can run this example with:
+
+```bash
+npx promptfoo@latest init --example azure/deepseek
+cd azure/deepseek
+```
+
+## Setup
+
+1. Deploy DeepSeek models in Azure AI Foundry
+2. Set your environment variables:
+
+```bash
+export AZURE_API_KEY=your-api-key
+export AZURE_API_HOST=your-deployment.services.ai.azure.com
+```
+
+## Available DeepSeek Models
+
+| Model                           | Type      | Description               |
+| ------------------------------- | --------- | ------------------------- |
+| `DeepSeek-R1`                   | Reasoning | Advanced reasoning model  |
+| `DeepSeek-V3`                   | Chat      | Standard chat model       |
+| `DeepSeek-R1-Distill-Llama-70B` | Reasoning | Distilled reasoning model |
+| `DeepSeek-R1-Distill-Qwen-32B`  | Reasoning | Distilled reasoning model |
+
+## Reasoning Model Configuration
+
+DeepSeek-R1 is a reasoning model that requires special configuration:
+
+```yaml
+providers:
+  - id: azure:chat:DeepSeek-R1
+    config:
+      isReasoningModel: true # Required for reasoning models
+      max_completion_tokens: 4096 # Use instead of max_tokens
+      reasoning_effort: medium # low, medium, or high
+```
+
+## Running the Example
+
+```bash
+npx promptfoo@latest eval
+npx promptfoo@latest view
+```
+
+## Documentation
+
+- [Azure Provider Documentation](https://promptfoo.dev/docs/providers/azure/)
+- [DeepSeek on Azure](https://azure.microsoft.com/en-us/products/ai-services/ai-foundry/)
@@ -0,0 +1,57 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: Azure DeepSeek reasoning
+
+prompts:
+  - 'Solve this step by step: {{problem}}'
+
+providers:
+  # DeepSeek-R1 is a reasoning model
+  - id: azure:chat:DeepSeek-R1
+    label: deepseek-r1
+    config:
+      apiHost: 'your-deployment.services.ai.azure.com'
+      apiVersion: '2025-04-01-preview'
+      isReasoningModel: true
+      max_completion_tokens: 4096
+      reasoning_effort: medium
+
+  # DeepSeek-V3 is a standard chat model
+  - id: azure:chat:DeepSeek-V3
+    label: deepseek-v3
+    config:
+      apiHost: 'your-deployment.services.ai.azure.com'
+      apiVersion: '2025-04-01-preview'
+      max_tokens: 4096
+      temperature: 0.3
+
+tests:
+  - vars:
+      problem: |
+        A train leaves Station A at 9:00 AM traveling at 60 mph toward Station B.
+        Another train leaves Station B at 10:00 AM traveling at 80 mph toward Station A.
+        If the stations are 280 miles apart, at what time will the trains meet?
+    assert:
+      - type: contains-any
+        value: ['11:34', '11:34 AM', '1 hour 34 minutes', '94 minutes']
+      - type: llm-rubric
+        value: Shows clear step-by-step reasoning
+
+  - vars:
+      problem: |
+        If 5 machines can produce 100 widgets in 4 hours,
+        how many widgets can 8 machines produce in 6 hours?
+    assert:
+      - type: contains
+        value: '240'
+      - type: llm-rubric
+        value: Demonstrates proportional reasoning
+
+  - vars:
+      problem: |
+        A rectangle has a perimeter of 36 cm.
+        If the length is twice the width, what is the area?
+    assert:
+      - type: contains
+        value: '72'
+      - type: llm-rubric
+        value: Shows algebraic problem-solving steps
@@ -0,0 +1,120 @@
+# azure/foundry-agent (Azure AI Foundry Agent)
+
+This example demonstrates how to use the Azure Foundry Agent provider with promptfoo. This provider uses the `@azure/ai-projects` SDK and the v2 Responses agent runtime instead of the old threads/runs API.
+
+You can run this example with:
+
+```bash
+npx promptfoo@latest init --example azure/foundry-agent
+cd azure/foundry-agent
+```
+
+## Setup
+
+1. Install the required Azure SDK packages:
+
+```bash
+npm install @azure/ai-projects @azure/identity
+```
+
+2. Set up your Azure credentials. The provider uses `DefaultAzureCredential`, so you can authenticate via:
+   - Azure CLI: `az login`
+   - Environment variables
+   - Managed Identity
+   - Service Principal
+
+3. Set your Azure AI Project URL:
+
+```bash
+export AZURE_AI_PROJECT_URL="https://your-project.services.ai.azure.com/api/projects/your-project-id"
+```
+
+## Configuration
+
+The provider uses the `azure:foundry-agent:agent-name-or-id` format. Agent names are preferred. Legacy agent IDs still work as a fallback lookup if the agent exists in the project.
+
+```yaml
+providers:
+  - id: azure:foundry-agent:my-foundry-agent
+    config:
+      projectUrl: 'https://your-project.services.ai.azure.com/api/projects/your-project-id'
+      temperature: 0.7
+      max_tokens: 150
+
+tests:
+  - vars:
+      question: 'What is the capital of France?'
+    assert:
+      - type: contains
+        value: 'Paris'
+```
+
+## Configuration Options
+
+These per-request settings are supported:
+
+- `instructions`
+- `temperature`
+- `top_p`
+- `max_tokens` / `max_completion_tokens` (mapped to `max_output_tokens`)
+- `response_format`
+- `tools`
+- `tool_choice`
+- `functionToolCallbacks`
+- `modelName`
+- `reasoning_effort`
+- `verbosity`
+- `metadata`
+- `passthrough`
+- `maxPollTimeMs`
+
+These request-time settings are ignored by the v2 runtime and should be configured on the Foundry agent instead:
+
+- `tool_resources`
+- `frequency_penalty`
+- `presence_penalty`
+- `seed`
+- `stop`
+- `timeoutMs`
+- `retryOptions`
+
+## Function Tool Callbacks
+
+You can provide custom function callbacks just like with the regular Azure Assistant provider:
+
+```yaml
+providers:
+  - id: azure:foundry-agent:my-foundry-agent
+    config:
+      projectUrl: 'https://your-project.services.ai.azure.com/api/projects/your-project-id'
+      functionToolCallbacks:
+        getCurrentWeather: |
+          (args) => {
+            const { location } = JSON.parse(args);
+            return `The weather in ${location} is sunny and 75°F`;
+          }
+```
+
+## Differences from Regular Azure Assistant Provider
+
+The main differences are:
+
+1. **SDK Usage**: Uses `@azure/ai-projects` SDK instead of direct HTTP calls
+2. **Authentication**: Uses `DefaultAzureCredential` for Azure authentication
+3. **Project URL**: Requires an Azure AI Project URL instead of Azure OpenAI endpoint
+4. **Provider Format**: Uses `azure:foundry-agent:agent-name-or-id` instead of `azure:assistant:assistant-id`
+5. **Runtime**: Uses `responses.create(..., agent_reference)` instead of threads/messages/runs
+
+## Environment Variables
+
+- `AZURE_AI_PROJECT_URL`: Your Azure AI Project URL (can be overridden in config)
+- Standard Azure credential environment variables (if not using other auth methods)
+
+## Error Handling
+
+The provider includes the same comprehensive error handling as the regular Azure Assistant provider:
+
+- Content filter detection and guardrails reporting
+- Rate limit handling
+- Service error detection
+- Automatic retries for transient errors
@@ -0,0 +1,35 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+
+description: Azure AI Foundry Agent evaluation
+
+providers:
+  - id: azure:foundry-agent:my-foundry-agent
+    config:
+      projectUrl: 'project.services.ai.azure.com/api/projects/project-id'
+      temperature: 0.7
+      max_tokens: 150
+      instructions: 'You are a helpful assistant that provides clear and concise answers.'
+
+prompts:
+  - '{{question}}'
+
+tests:
+  - vars:
+      question: 'What is the capital of France?'
+    assert:
+      - type: contains
+        value: 'Paris'
+
+  - vars:
+      question: 'Explain what photosynthesis is in simple terms.'
+    assert:
+      - type: contains
+        value: 'plants'
+      - type: contains
+        value: 'sunlight'
+
+  - vars:
+      question: 'What is 2 + 2?'
+    assert:
+      - type: contains
+        value: '4'
@@ -0,0 +1,47 @@
+# azure/llama (Azure Llama Models)
+
+This example demonstrates how to use Meta Llama models on Azure AI Foundry with promptfoo.
+
+You can run this example with:
+
+```bash
+npx promptfoo@latest init --example azure/llama
+cd azure/llama
+```
+
+## Setup
+
+1. Deploy Llama models in Azure AI Foundry
+2. Set your environment variables:
+
+```bash
+export AZURE_API_KEY=your-api-key
+export AZURE_API_HOST=your-deployment.services.ai.azure.com
+```
+
+## Available Llama Models
+
+| Model                                    | Description                         |
+| ---------------------------------------- | ----------------------------------- |
+| `Llama-4-Maverick-17B-128E-Instruct-FP8` | Llama 4 Maverick (128 experts, FP8) |
+| `Llama-4-Scout-17B-16E-Instruct`         | Llama 4 Scout (16 experts)          |
+| `Llama-3.3-70B-Instruct`                 | Llama 3.3 70B                       |
+| `Meta-Llama-3.1-405B-Instruct`           | Llama 3.1 405B                      |
+| `Meta-Llama-3.1-70B-Instruct`            | Llama 3.1 70B                       |
+| `Meta-Llama-3.1-8B-Instruct`             | Llama 3.1 8B                        |
+
+## Running the Example
+
+```bash
+npx promptfoo@latest eval
+npx promptfoo@latest view
+```
+
+## Configuration
+
+The example compares Llama 4 Maverick and Llama 4 Scout on code generation tasks. This helps evaluate the trade-off between model capacity (expert count), speed, and quality.
+
+## Documentation
+
+- [Azure Provider Documentation](https://promptfoo.dev/docs/providers/azure/)
+- [Llama on Azure](https://azure.microsoft.com/en-us/products/ai-services/ai-foundry/)
@@ -0,0 +1,62 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: Azure Llama models
+
+prompts:
+  - |
+    You are a helpful coding assistant.
+
+    Write a {{language}} function that {{task}}.
+    Include comments explaining the code.
+
+providers:
+  # Llama 4 Maverick - Higher capacity with 128 experts
+  - id: azure:chat:Llama-4-Maverick-17B-128E-Instruct-FP8
+    label: llama-4-maverick
+    config:
+      apiHost: 'your-deployment.services.ai.azure.com'
+      apiVersion: '2025-04-01-preview'
+      max_tokens: 2048
+      temperature: 0.3
+
+  # Llama 4 Scout - Lighter weight with 16 experts
+  - id: azure:chat:Llama-4-Scout-17B-16E-Instruct
+    label: llama-4-scout
+    config:
+      apiHost: 'your-deployment.services.ai.azure.com'
+      apiVersion: '2025-04-01-preview'
+      max_tokens: 2048
+      temperature: 0.3
+
+tests:
+  - vars:
+      language: Python
+      task: calculates the factorial of a number using recursion
+    assert:
+      - type: contains
+        value: 'def'
+      - type: contains-any
+        value: ['factorial', 'recursion', 'recursive']
+      - type: python
+        value: |
+          def test_code():
+              # Check for function definition and recursive call
+              return 'def ' in output and ('return' in output or 'factorial' in output)
+          test_code()
+
+  - vars:
+      language: JavaScript
+      task: reverses a string without using built-in reverse methods
+    assert:
+      - type: contains
+        value: 'function'
+      - type: not-contains
+        value: '.reverse()'
+
+  - vars:
+      language: Python
+      task: checks if a number is prime
+    assert:
+      - type: contains
+        value: 'def'
+      - type: contains-any
+        value: ['prime', 'divisible', 'modulo', '%']
@@ -0,0 +1,47 @@
+# azure/mistral (Azure Mistral Models)
+
+This example demonstrates how to use Mistral models on Azure AI Foundry with promptfoo.
+
+You can run this example with:
+
+```bash
+npx promptfoo@latest init --example azure/mistral
+cd azure/mistral
+```
+
+## Setup
+
+1. Deploy Mistral models in Azure AI Foundry
+2. Update `promptfooconfig.yaml` with your deployment name and API host
+3. Set your environment variables:
+
+```bash
+export AZURE_API_KEY=your-api-key
+```
+
+## Available Mistral Models
+
+| Model                | Description                      |
+| -------------------- | -------------------------------- |
+| `Mistral-Large-3`    | Mistral Large 3 - Most capable   |
+| `Mistral-Large-2411` | Mistral Large - Previous gen     |
+| `mistral-small-2503` | Mistral Small - Fast, efficient  |
+| `Pixtral-Large-2411` | Pixtral Large - Vision + text    |
+| `Ministral-3B-2410`  | Ministral 3B - Fast, lightweight |
+| `Mistral-Nemo`       | Mistral Nemo - Balanced          |
+
+## Running the Example
+
+```bash
+npx promptfoo@latest eval
+npx promptfoo@latest view
+```
+
+## Configuration
+
+The example compares Mistral Large 3 and Mistral Small 2503 on text generation tasks. This helps evaluate the trade-off between model capacity, speed, and quality.
+
+## Documentation
+
+- [Azure Provider Documentation](https://promptfoo.dev/docs/providers/azure/)
+- [Mistral on Azure](https://azure.microsoft.com/en-us/products/ai-services/ai-foundry/)
@@ -0,0 +1,60 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: Azure Mistral models
+
+prompts:
+  - |
+    You are a helpful assistant. Answer the following question clearly and concisely.
+
+    Question: {{question}}
+
+providers:
+  # Mistral Large 3 - Most capable Mistral model
+  - id: azure:chat:Mistral-Large-3
+    label: mistral-large-3
+    config:
+      apiHost: 'your-deployment.services.ai.azure.com'
+      apiVersion: '2025-04-01-preview'
+      max_tokens: 1024
+      temperature: 0.7
+
+  # Mistral Small 2503 - Fast and cost-effective
+  - id: azure:chat:mistral-small-2503
+    label: mistral-small-2503
+    config:
+      apiHost: 'your-deployment.services.ai.azure.com'
+      apiVersion: '2025-04-01-preview'
+      max_tokens: 1024
+      temperature: 0.7
+
+defaultTest:
+  options:
+    provider:
+      id: azure:chat:Mistral-Large-3
+      config:
+        apiHost: 'your-deployment.services.ai.azure.com'
+        apiVersion: '2025-04-01-preview'
+
+tests:
+  - vars:
+      question: What are the key differences between compiled and interpreted programming languages?
+    assert:
+      - type: contains-any
+        value: ['compiled', 'interpreted', 'machine code', 'runtime']
+      - type: llm-rubric
+        value: Provides a clear comparison with examples
+
+  - vars:
+      question: Explain the concept of API rate limiting and why it's important.
+    assert:
+      - type: contains-any
+        value: ['rate', 'limit', 'request', 'throttle']
+      - type: llm-rubric
+        value: Explains both the concept and its importance
+
+  - vars:
+      question: What is the difference between SQL and NoSQL databases?
+    assert:
+      - type: contains-any
+        value: ['relational', 'schema', 'document', 'flexible']
+      - type: llm-rubric
+        value: Covers key differences between database types
@@ -0,0 +1,69 @@
+# azure/openai (Azure OpenAI)
+
+This example demonstrates how to use Azure OpenAI with promptfoo, including text generation and vision capabilities.
+
+You can run this example with:
+
+```bash
+npx promptfoo@latest init --example azure/openai
+cd azure/openai
+```
+
+## Environment Variables
+
+This example requires the following environment variables:
+
+- `AZURE_API_KEY` - Your Azure OpenAI API key
+- `AZURE_OPENAI_API_KEY` - Alternative environment variable for your Azure OpenAI API key
+
+You can set these in a `.env` file or directly in your environment.
+
+## Prerequisites
+
+1. An Azure account with access to Azure OpenAI Service
+2. Deployments for one or more Azure OpenAI models:
+   - Text models: gpt-5.1, gpt-4o
+   - Vision models: gpt-4o (or other vision-capable models)
+3. Your Azure OpenAI endpoint URL
+
+## Setup Instructions
+
+1. Update the `apiHost` in the configuration files to your Azure OpenAI endpoint
+2. Set `AZURE_API_KEY` in your environment
+3. Update the deployment names to match your actual deployments
+
+## Available Examples
+
+### Basic Text Generation
+
+```bash
+npx promptfoo@latest eval
+# or
+npx promptfoo@latest eval -c promptfooconfig.yaml
+```
+
+### Vision Models
+
+```bash
+npx promptfoo@latest eval -c promptfooconfig.vision.yaml
+```
+
+Demonstrates three ways to provide images to vision models:
+
+- **URL**: Direct link to an image on the web
+- **Local file**: Using `file://` paths (automatically converted to base64)
+- **Base64**: Pre-encoded image data URI
+
+## Troubleshooting
+
+If you get a 401 error:
+
+- Ensure your `AZURE_API_KEY` is set correctly
+- Verify your endpoint URL is correct (no https://)
+- Check that your deployment supports the requested capabilities
+
+## Additional Resources
+
+- [Azure OpenAI Provider Documentation](https://promptfoo.dev/docs/providers/azure/)
+- [Azure OpenAI Service Documentation](https://learn.microsoft.com/en-us/azure/ai-services/openai/)
+- [Azure OpenAI Vision Documentation](https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/gpt-with-vision)
@@ -0,0 +1,17 @@
+[
+  {
+    "role": "user",
+    "content": [
+      {
+        "type": "text",
+        "text": "{{question}}"
+      },
+      {
+        "type": "image_url",
+        "image_url": {
+          "url": "{{image_url}}"
+        }
+      }
+    ]
+  }
+]
@@ -0,0 +1,28 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: Azure OpenAI vision model evaluation
+
+prompts:
+  # Vision models require a specific message format with content arrays
+  - file://prompt.vision.json
+
+providers:
+  # Update with your Azure OpenAI deployment details
+  - id: azure:chat:gpt-5
+    config:
+      apiHost: promptfoo.openai.azure.com
+      apiVersion: 2024-02-15-preview
+      temperature: 0.1
+      max_tokens: 500
+
+tests:
+  # Test 2: Load image from local file (automatically converted to base64 with data URL)
+  - vars:
+      question: What color is the planet in this image?
+      image_url: file://assets/earth.jpg
+    assert:
+      - type: contains-any
+        value: [blue, green]
+# Note: If you see 401 errors, ensure:
+# 1. Your AZURE_API_KEY environment variable is set correctly
+# 2. The apiHost matches your Azure OpenAI resource name
+# 3. Your deployment supports vision (GPT-5.1, GPT-4o, GPT-4.1)
@@ -0,0 +1,29 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: Azure OpenAI chat and embeddings with fact generation
+
+prompts:
+  - 'Generate one very interesting fact about {{topic}}'
+
+providers:
+  # GPT-5.1 - Latest flagship model
+  - id: azure:chat:gpt-5.1
+    config:
+      apiHost: 'your-org.openai.azure.com'
+
+defaultTest:
+  assert:
+    - type: latency
+      threshold: 3000
+
+tests:
+  - vars:
+      topic: monkeys
+  - vars:
+      topic: bananas
+    assert:
+      - type: similar
+        value: Bananas are naturally radioactive.
+        provider:
+          id: azure:embeddings:text-embedding-3-large
+          config:
+            apiHost: 'your-org.openai.azure.com'