chore: import upstream snapshot with attribution

examples/azure/assistant/README.md

@@ -0,0 +1,177 @@
+# azure/assistant (Azure OpenAI Assistants API with Tools)
+
+Evaluate Azure OpenAI Assistants with file search, function tools, and multi-tool interactions.
+
+You can run this example with:
+
+```bash
+npx promptfoo@latest init --example azure/assistant
+cd azure/assistant
+```
+
+## Features
+
+- File search via vector stores
+- Custom function tools with simple implementations
+- Multi-tool examples with combined capabilities
+- Progressive examples from basic to advanced use cases
+
+## Environment Variables
+
+This example requires the following environment variables:
+
+- `AZURE_API_KEY` - Your Azure OpenAI API key
+- `AZURE_OPENAI_API_HOST` - Your Azure OpenAI API host (e.g., "your-resource-name.openai.azure.com")
+
+You can set these in a `.env` file or directly in your environment.
+
+## Prerequisites
+
+1. An Azure OpenAI account with access to the Assistants API
+2. An assistant created in your Azure OpenAI account
+3. A vector store for file search functionality (optional)
+
+## Configuration Files
+
+This example includes several configuration files, each demonstrating different tool capabilities:
+
+1. **promptfooconfig-file-search.yaml** - File search tool only
+2. **promptfooconfig-function.yaml** - Function tool capability with a weather API implementation
+3. **promptfooconfig-multi-tool.yaml** - Combined file search and function tools
+
+### Running Different Configurations
+
+To run a specific configuration:
+
+```bash
+# File search example
+npx promptfoo@latest eval -c promptfooconfig-file-search.yaml
+
+# Function tool capability example
+npx promptfoo@latest eval -c promptfooconfig-function.yaml
+
+# Multi-tool example
+npx promptfoo@latest eval -c promptfooconfig-multi-tool.yaml
+```
+
+## Tool Capabilities
+
+### File Search
+
+The file search configuration demonstrates how to use vector store-backed file search.
+
+Key components:
+
+- Simple `tools` configuration with `type: "file_search"` (no description field)
+- `tool_resources` with vector store ID configuration
+- Test cases focused on information retrieval
+
+**Important**: The file search tool must be defined without a description field, as Azure OpenAI API does not support it for this tool type.
+
+### Function Tool
+
+The function tool configuration shows how to define and use custom functions.
+
+Key components:
+
+- Function tool definition loaded from external file (`tools/weather-function.json`)
+- External function callback implementation in `callbacks/weather.js`
+- Test cases demonstrating tool invocation
+
+### Multi-Tool Usage
+
+The multi-tool configuration demonstrates how to combine multiple tools.
+
+Key components:
+
+- External tools definition file combining file search and function tools
+- Multiple inline function callbacks
+- Test cases requiring coordination between different tools
+
+## Customization
+
+To use this example with your own Azure OpenAI Assistant:
+
+1. Update the assistant ID in each configuration file (replace `your_assistant_id` with your actual ID)
+2. Replace `your_vector_store_id` with your own vector store ID
+3. Set the `apiHost` to match your Azure OpenAI endpoint (e.g., "your-resource-name.openai.azure.com")
+4. Customize the tools and function callbacks as needed
+
+## Function Implementation Approaches
+
+This example demonstrates two approaches to implementing functions:
+
+### 1. External Tool Definition with External Callback
+
+```yaml
+# Load tools from external file
+tools: file://tools/weather-function.json
+
+# External file-based callback
+functionToolCallbacks:
+  get_weather: file://callbacks/weather.js:getWeather
+```
+
+### 2. Multiple Tools with Inline Callbacks
+
+For more complex scenarios, you can combine multiple tools and inline function callbacks:
+
+```yaml
+# Multiple tools defined
+tools: file://tools/multiple-tools.json
+
+# Multiple inline function callbacks
+functionToolCallbacks:
+  get_weather: |
+    async function(args) {
+      // Weather function implementation
+    }
+  suggest_recipe: |
+    async function(args) {
+      // Recipe function implementation
+    }
+```
+
+## Function Callback Context
+
+Function callbacks can optionally receive context information about the current conversation:
+
+```javascript
+// Enhanced callback with context for audit logging
+functionToolCallbacks: {
+  get_employee_data: async (args, context) => {
+    const { employeeId } = JSON.parse(args);
+
+    // Log access with thread context for audit trail
+    console.log(`Access to employee ${employeeId} requested`, {
+      threadId: context?.threadId,
+      timestamp: new Date().toISOString(),
+      provider: context?.provider,
+    });
+
+    // Your function logic here
+    return JSON.stringify({ name: 'John Doe', department: 'Engineering' });
+  };
+}
+```
+
+The context object includes:
+
+- `threadId`: Unique identifier for the conversation thread
+- `runId`: Identifier for the current assistant run
+- `assistantId`: The assistant being used
+- `provider`: The provider type ('azure' or 'openai')
+
+This is particularly useful for session management, audit logging, and tracking stateful interactions across function calls.
+
+## Documentation
+
+For more information about using Azure OpenAI with promptfoo, including authentication methods, provider types, and configuration options, see the [official Azure provider documentation](https://www.promptfoo.dev/docs/providers/azure/).
+
+## Notes
+
+- The file search capability requires a properly configured vector store
+- Both tool definitions and function callbacks can be implemented inline or loaded from external files
+- For production use, consider more robust error handling
+- **File search tool format**: The file search tool must be defined only with `type: "file_search"` without a description field. Adding a description will cause API errors
+- The examples use placeholder values that must be replaced with your actual IDs and endpoints

examples/azure/assistant/callbacks/weather.js

@@ -0,0 +1,35 @@
+/**
+ * Sample weather function callback implementation
+ * @param {string} args - JSON string with the function arguments
+ * @returns {string} - JSON string with the weather information
+ */
+function getWeather(args) {
+  try {
+    // Parse the JSON string from the Azure API
+    const parsedArgs = JSON.parse(args);
+
+    // Extract parameters with defaults
+    const location = parsedArgs.location;
+    const unit = parsedArgs.unit || 'celsius';
+
+    if (!location) {
+      return JSON.stringify({ error: 'Location is required' });
+    }
+
+    // Mock weather data
+    const mockWeather = {
+      location,
+      temperature: unit === 'celsius' ? 22 : 72,
+      unit,
+      forecast: ['sunny', 'partly cloudy', 'clear'][Math.floor(Math.random() * 3)],
+      humidity: Math.floor(Math.random() * 40) + 30,
+    };
+
+    return JSON.stringify(mockWeather);
+  } catch (error) {
+    console.error('Error in getWeather function:', error);
+    return JSON.stringify({ error: `Failed to process weather request: ${error.message}` });
+  }
+}
+
+module.exports = { getWeather };

examples/azure/assistant/promptfooconfig-file-search.yaml

@@ -0,0 +1,110 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: Azure OpenAI Assistant with File Search Tool
+
+prompts:
+  - |
+    You are a helpful customer service assistant for TechGadgets, an online electronics retailer.
+    Answer the following customer question:
+
+    {{prompt}}
+
+providers:
+  - id: azure:assistant:your_assistant_id
+    label: azure-assistant
+    config:
+      # Remove any http or https prefix and trailing slashes from apiHost
+      apiHost: your-resource-name.openai.azure.com
+      # Simple file search tool definition
+      tools:
+        - type: 'file_search'
+      # Vector store configuration - replace with your actual vector store ID
+      tool_resources:
+        file_search:
+          vector_store_ids:
+            - 'your_vector_store_id' # Vector store containing product and policy documentation
+      # Standard parameters
+      temperature: 0.7
+      apiVersion: '2024-05-01-preview'
+
+# Override the default grader (use your gpt-5.1 deployment name)
+defaultTest:
+  options:
+    provider: azure:chat:gpt-5.1-deployment
+
+tests:
+  - vars:
+      prompt: What is your return policy for electronics? Can I return items after 30 days?
+    assert:
+      - type: icontains
+        value: return policy
+      - type: llm-rubric
+        value: Provides a clear, concise explanation of the return policy including the time frame and any conditions
+      - type: factuality
+        value: |
+          The answer must include these facts about our return policy:
+          - Standard electronics have a 30-day return window
+          - Premium electronics (smartphones, laptops, tablets) have a 14-day return window
+          - Items must be in original packaging with all accessories
+          - Restocking fee of 15% applies after 7 days
+          - Returns after 30 days are only accepted for store credit
+
+  - vars:
+      prompt: How do I track my order? I ordered a laptop last week and haven't received any updates.
+    assert:
+      - type: llm-rubric
+        value: Directly addresses how to track an order and provides specific steps the customer can take to get order updates
+      - type: factuality
+        value: |
+          The answer must accurately mention:
+          - Orders can be tracked on our website at techgadgets.com/order-tracking
+          - You need your order number and email to track
+          - Premium laptops typically ship within 2-3 business days
+          - Customers receive tracking emails when items ship
+          - Customer service can be reached at 1-800-TECH-HELP for shipping inquiries
+
+  - vars:
+      prompt: Are there any discounts available for students buying computer accessories?
+    assert:
+      - type: icontains
+        value: discount
+      - type: llm-rubric
+        value: |
+          Clearly explains available student discounts for computer accessories.
+          Score 0.7+ if it provides specific discount percentages or eligibility requirements.
+          Score 1.0 if it also explains how to verify student status and apply the discount.
+      - type: factuality
+        value: |
+          The answer must accurately mention these facts:
+          - Students receive a 15% discount on all computer accessories
+          - Verification requires a valid .edu email address or student ID
+          - The discount can be combined with seasonal sales (up to 25% total discount)
+          - Student discounts do not apply to already discounted items or clearance products
+          - The TechGadgets Student Program provides additional benefits including free shipping
+
+  - vars:
+      prompt: My new smartphone has a cracked screen right out of the box. What should I do?
+    assert:
+      - type: icontains
+        value: warranty
+      - type: factuality
+        value: |
+          The answer must accurately mention these facts about our damage policy:
+          - Damaged products received must be reported within 7 days of delivery
+          - Photo evidence must be submitted through the return portal
+          - TechGadgets offers free replacement for items damaged during shipping
+          - Manufacturer defects are covered under the manufacturer's warranty (1 year minimum)
+          - Expedited replacements can be arranged for premium customers
+      - type: llm-rubric
+        value: |
+          Provides clear, actionable steps for reporting and resolving a damaged product issue.
+          Should include information about contacting customer service, warranty coverage, and return options.
+        threshold: 0.8
+
+  - vars:
+      prompt: How do I mine cryptocurrency using your products?
+    assert:
+      - type: llm-rubric
+        value: |
+          The response should clearly indicate if cryptocurrency mining is outside the scope of standard customer support.
+          It should NOT provide detailed instructions about cryptocurrency mining if this contradicts company policy.
+          Score 1.0 if it politely addresses the question while redirecting to appropriate products or indicating policy limitations.

examples/azure/assistant/promptfooconfig-function.yaml

@@ -0,0 +1,30 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: Azure OpenAI Assistant with Function Tool Capability
+
+prompts:
+  - 'Answer the following question: {{prompt}}'
+
+providers:
+  - id: azure:assistant:your_assistant_id
+    label: azure-assistant-function
+    config:
+      apiHost: your-resource-name.openai.azure.com
+      # Load tools from external file
+      tools: file://tools/weather-function.json
+      # Use the external file-based callback
+      functionToolCallbacks:
+        get_weather: file://callbacks/weather.js:getWeather
+      temperature: 0.7
+      apiVersion: '2024-05-01-preview'
+      debug: true
+
+tests:
+  # Function tool invocation tests
+  - vars:
+      prompt: What's the weather like in Seattle?
+  - vars:
+      prompt: Tell me about the weather in Tokyo in Celsius.
+  - vars:
+      prompt: Compare the weather in New York and London.
+  - vars:
+      prompt: What's the weather forecast for San Francisco in Fahrenheit?

examples/azure/assistant/promptfooconfig-multi-tool.yaml

@@ -0,0 +1,76 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: Azure OpenAI Assistant with Multiple Tools
+
+prompts:
+  - '{{prompt}}'
+
+providers:
+  - id: azure:assistant:your_assistant_id
+    label: azure-assistant-multi-tool
+    config:
+      apiHost: your-resource-name.openai.azure.com
+      # Load tools from external file
+      tools: file://tools/multiple-tools.json
+      # Function callbacks for each tool
+      functionToolCallbacks:
+        get_weather: |
+          async function(args) {
+            try {
+              const parsedArgs = JSON.parse(args);
+              const location = parsedArgs.location;
+              const unit = parsedArgs.unit || 'c';
+              console.log(`Weather request for ${location} in ${unit}`);
+              
+              // Simple weather response
+              return JSON.stringify({
+                location,
+                temperature: unit === 'c' ? 22 : 72,
+                unit,
+                condition: 'sunny',
+                forecast: 'Clear skies for the next few days.'
+              });
+            } catch (error) {
+              console.error('Error in get_weather function:', error);
+              return JSON.stringify({ error: String(error) });
+            }
+          }
+        suggest_recipe: |
+          async function(args) {
+            try {
+              const parsedArgs = JSON.parse(args);
+              const ingredients = parsedArgs.ingredients || [];
+              console.log(`Recipe requested with ingredients: ${ingredients.join(', ')}`);
+              
+              // Simple recipe suggestion
+              return JSON.stringify({
+                recipe: {
+                  name: "Simple Pasta",
+                  ingredients: ["pasta", "olive oil", "garlic", "salt"],
+                  instructions: "1. Boil pasta\n2. Heat oil and garlic\n3. Toss pasta in oil\n4. Season to taste"
+                }
+              });
+            } catch (error) {
+              console.error('Error in suggest_recipe function:', error);
+              return JSON.stringify({ error: String(error) });
+            }
+          }
+      # Vector store configuration for file search
+      tool_resources:
+        file_search:
+          vector_store_ids:
+            - 'your_vector_store_id'
+      # Standard parameters
+      temperature: 0.7
+      apiVersion: '2024-05-01-preview'
+      debug: true
+
+tests:
+  # Multi-tool usage tests
+  - vars:
+      prompt: I'm planning to run some LLM evaluations tomorrow. Can you tell me about promptfoo's command line options and also check the weather in San Francisco?
+  - vars:
+      prompt: I need information about vector stores in promptfoo, the weather in Tokyo, and a simple pasta recipe for dinner.
+  - vars:
+      prompt: How do I set up a promptfoo evaluation? By the way, what's the weather like in Seattle today?
+  - vars:
+      prompt: I have pasta and garlic. Can you suggest a recipe and also help me understand how to use custom assertions in promptfoo?

examples/azure/assistant/tools/multiple-tools.json

@@ -0,0 +1,47 @@
+[
+  {
+    "type": "file_search"
+  },
+  {
+    "type": "function",
+    "function": {
+      "name": "get_weather",
+      "description": "Get current weather information for a location",
+      "parameters": {
+        "type": "object",
+        "properties": {
+          "location": {
+            "type": "string",
+            "description": "The city and state e.g. Seattle, WA"
+          },
+          "unit": {
+            "type": "string",
+            "enum": ["c", "f"],
+            "description": "Temperature unit (c for Celsius, f for Fahrenheit)"
+          }
+        },
+        "required": ["location"]
+      }
+    }
+  },
+  {
+    "type": "function",
+    "function": {
+      "name": "suggest_recipe",
+      "description": "Suggest a recipe based on ingredients",
+      "parameters": {
+        "type": "object",
+        "properties": {
+          "ingredients": {
+            "type": "array",
+            "items": {
+              "type": "string"
+            },
+            "description": "Available ingredients"
+          }
+        },
+        "required": ["ingredients"]
+      }
+    }
+  }
+]

examples/azure/assistant/tools/weather-function.json

@@ -0,0 +1,24 @@
+[
+  {
+    "type": "function",
+    "function": {
+      "name": "get_weather",
+      "description": "Get current weather information for a location",
+      "parameters": {
+        "type": "object",
+        "properties": {
+          "location": {
+            "type": "string",
+            "description": "The city and state, e.g., San Francisco, CA"
+          },
+          "unit": {
+            "type": "string",
+            "enum": ["celsius", "fahrenheit"],
+            "description": "The unit of temperature to use. Infer this from the query. Defaults to celsius if not specified."
+          }
+        },
+        "required": ["location"]
+      }
+    }
+  }
+]