chore: import upstream snapshot with attribution

examples/mistral/README.md

@@ -0,0 +1,110 @@
+# mistral (Mistral AI Chat Models)
+
+This example demonstrates Mistral AI's chat models, including Magistral reasoning models, current multimodal models, and shows how to use Mistral models for evaluation grading and embeddings.
+
+You can run this example with:
+
+```bash
+npx promptfoo@latest init --example mistral
+cd mistral
+```
+
+## Environment Variables
+
+This example requires:
+
+- `MISTRAL_API_KEY` - Your Mistral API key (get it from [console.mistral.ai](https://console.mistral.ai))
+
+## What This Example Shows
+
+- **Mathematical Reasoning**: AIME2024 competition problems with Magistral Medium
+- **Model Comparison**: Compare Mistral's different model capabilities
+- **Reasoning Models**: Showcase Magistral Medium (native reasoning) vs. Mistral Small 4
+- **Chat Capabilities**: General conversation and task completion
+- **Mistral-powered Evaluation**: Use Mistral models for grading instead of OpenAI
+- **Mistral Embeddings**: Use Mistral's embedding model for similarity checks
+
+## Models Demonstrated
+
+### Reasoning Models
+
+- **Magistral Medium** (`magistral-medium-latest` → `magistral-medium-2509`): Native reasoning model ($2/$5 per 1M tokens, 128k context) — the reasoning showcase in these examples.
+
+> Mistral folded Magistral Small into **Mistral Small 4**: the `magistral-small-latest` alias now resolves to `mistral-small-2603` (a hybrid model, $0.15/$0.60 per 1M), so these examples use the canonical `mistral-small-latest` id. Enable Small 4's reasoning mode with `reasoning_effort: high`. The standalone `magistral-small-2509` snapshot is deprecated (retires 2026-07-31).
+
+### Chat Models
+
+- **Mistral Medium 3.5** (`mistral-medium-latest` → `mistral-medium-2604`): Frontier agentic/coding multimodal model ($1.50/$7.50 per 1M, 256k context)
+- **Mistral Large 3** (`mistral-large-latest` → `mistral-large-2512`): General-purpose multimodal model ($0.50/$1.50 per 1M, 256k context)
+- **Mistral Small 4** (`mistral-small-latest` → `mistral-small-2603`): Hybrid instruct/reasoning/coding model ($0.15/$0.60 per 1M, 256k context)
+
+### Evaluation Models
+
+- **Grading**: Uses `mistral-large-latest` for LLM-as-a-judge evaluation
+- **Embeddings**: Uses `mistral-embed` for semantic similarity checks
+
+## Key Features Demonstrated
+
+- **Multi-model comparison**: Compare performance across different Mistral models
+- **Reasoning capabilities**: Step-by-step problem solving with Magistral models
+- **Cost optimization**: Balance performance vs. cost across model tiers
+- **Self-evaluation**: Use Mistral models to grade their own outputs
+- **Semantic similarity**: Mistral embeddings for content comparison
+
+## Running the Example
+
+```bash
+# Set your API key
+export MISTRAL_API_KEY=your_api_key_here
+
+# Run the evaluation
+promptfoo eval
+
+# View results in the web UI
+promptfoo view
+```
+
+## Configuration Highlights
+
+This example showcases several advanced promptfoo features:
+
+- **Provider overrides** for grading and embeddings
+- **Multiple assertion types** including llm-rubric and similarity
+- **Cost tracking** across different model tiers
+- **Mixed scenarios** from simple chat to complex reasoning
+
+The evaluation uses Mistral models end-to-end, providing a comprehensive view of their ecosystem capabilities.
+
+## Available Configurations
+
+This example includes multiple configuration files for different use cases:
+
+### Mathematical Reasoning
+
+- **`promptfooconfig.aime2024.yaml`** - Advanced mathematical competition problems (AIME2024 dataset)
+- **`promptfooconfig.reasoning.yaml`** - Step-by-step logical problem solving
+
+### Model Capabilities
+
+- **`promptfooconfig.comparison.yaml`** - Compare reasoning across all Mistral models
+- **`promptfooconfig.code-generation.yaml`** - Multi-language programming with Codestral
+- **`promptfooconfig.multimodal.yaml`** - Vision and text processing with current Mistral multimodal models
+
+### Advanced Features
+
+- **`promptfooconfig.tool-use.yaml`** - Function calling and tool integration
+- **`promptfooconfig.tool-routing.yaml`** - End-to-end QA for tool-only, mixed content+tool_calls, file-based tools, and plain chat output
+- **`promptfooconfig.json-mode.yaml`** - Structured JSON output generation
+- **`promptfooconfig.yaml`** - Main example with evaluation using Mistral models
+
+Run any specific configuration:
+
+```bash
+npx promptfoo@latest eval -c promptfooconfig.aime2024.yaml  # Mathematical reasoning
+npx promptfoo@latest eval -c promptfooconfig.comparison.yaml  # Model comparison
+```
+
+## Additional Resources
+
+- **[Mistral Provider Documentation](/docs/providers/mistral)** - Complete API reference and configuration options
+- **[Mistral Magistral Announcement](https://mistral.ai/news/magistral/)** - Official announcement and technical details

examples/mistral/promptfooconfig.aime2024.yaml

@@ -0,0 +1,45 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: Reproduce Mistral Magistral AIME2024 benchmark
+
+prompts:
+  - |
+    Solve this AIME mathematical problem step by step.
+
+    Problem: {{question}}
+
+    Think through this carefully and provide your final answer as a 3-digit integer (000-999).
+    End with: "Therefore, the answer is [your answer]."
+
+providers:
+  - id: mistral:magistral-medium-latest
+    label: Magistral Medium
+    config:
+      temperature: 0.7
+      top_p: 0.95
+      max_tokens: 40960
+  # The `magistral-small-latest` alias now resolves to Mistral Small 4, so use the
+  # canonical id. It contrasts a general hybrid model against native-reasoning Magistral.
+  - id: mistral:mistral-small-latest
+    label: Mistral Small 4
+    config:
+      temperature: 0.7
+      top_p: 0.95
+      max_tokens: 40960
+
+tests:
+  - huggingface://datasets/sea-snell/aime-2024?split=test
+
+defaultTest:
+  assert:
+    - type: llm-rubric
+      value: |
+        Evaluate this mathematical solution to an AIME competition problem.
+
+        The correct answer is: {{answer}}
+
+        Grade as PASS if and only if:
+        1. The response shows clear step-by-step mathematical reasoning
+        2. The final answer presented equals {{answer}} exactly
+        3. The mathematical work supports the conclusion
+
+        Grade as FAIL if the final answer is incorrect, regardless of the reasoning quality.

examples/mistral/promptfooconfig.code-generation.yaml

@@ -0,0 +1,62 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: 'Code generation with Mistral Codestral models'
+
+providers:
+  - id: mistral:codestral-latest
+    config:
+      temperature: 0.2
+      max_tokens: 2000
+
+prompts:
+  - '{{request}}'
+
+tests:
+  - vars:
+      request: 'Write a Python function that calculates the factorial of a number using recursion. Include docstring and type hints.'
+    assert:
+      - type: contains
+        value: 'def factorial'
+      - type: contains
+        value: 'int'
+      - type: contains
+        value: 'recursion'
+      - type: regex
+        value: '""".*"""'
+
+  - vars:
+      request: 'Create a JavaScript function that validates an email address using regex. Make it robust and handle edge cases.'
+    assert:
+      - type: contains
+        value: 'function'
+      - type: contains
+        value: 'email'
+      - type: contains
+        value: 'regex'
+      - type: contains
+        value: '@'
+
+  - vars:
+      request: 'Write a simple REST API endpoint in Python using FastAPI that accepts POST requests with user data (name, email) and returns a success message.'
+    assert:
+      - type: contains
+        value: '@app.post'
+      - type: contains
+        value: 'FastAPI'
+      - type: contains
+        value: 'async def'
+      - type: contains
+        value: 'name'
+      - type: contains
+        value: 'email'
+
+  - vars:
+      request: 'Create a React component that displays a user profile card with name, avatar, and email. Use modern React with hooks.'
+    assert:
+      - type: contains
+        value: 'export'
+      - type: contains
+        value: 'useState'
+      - type: contains
+        value: 'React'
+      - type: contains
+        value: 'avatar'

examples/mistral/promptfooconfig.comparison.yaml

@@ -0,0 +1,29 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: 'Compare reasoning capabilities across Mistral models'
+
+providers:
+  # `magistral-small-latest` was dropped: it now resolves to the same model as
+  # `mistral-small-latest` (Mistral Small 4), so listing both would duplicate a model.
+  - mistral:magistral-medium-latest
+  - mistral:mistral-large-latest
+  - mistral:mistral-small-latest
+
+prompts:
+  - 'Solve this step by step: {{problem}}'
+
+tests:
+  - vars:
+      problem: "A company has 100 employees. 60% work remotely, 25% work hybrid, and the rest work in office. If remote workers get a $200 stipend and hybrid workers get $100, what's the total monthly stipend cost?"
+    assert:
+      - type: llm-rubric
+        value: 'Shows clear mathematical reasoning and arrives at correct answer ($14,500)'
+      - type: cost
+        threshold: 0.10
+
+  - vars:
+      problem: 'If I have 3 boxes with 4 apples each, and I eat 2 apples from the first box and 1 apple from the second box, how many apples do I have left in total?'
+    assert:
+      - type: llm-rubric
+        value: 'Correctly calculates the remaining apples (9 apples total)'
+      - type: contains
+        value: '9'

examples/mistral/promptfooconfig.json-mode.yaml

@@ -0,0 +1,46 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: 'JSON mode and structured output with Mistral models'
+
+providers:
+  - id: mistral:mistral-large-latest
+    config:
+      response_format:
+        type: 'json_object'
+      temperature: 0.3
+
+prompts:
+  - "Extract the following information from the text and return as JSON with keys 'name', 'age', 'occupation', and 'location': {{text}}"
+
+tests:
+  - vars:
+      text: 'John Smith is a 35-year-old software engineer living in San Francisco.'
+    assert:
+      - type: is-json
+      - type: javascript
+        value: |
+          const parsed = JSON.parse(output);
+          return parsed.name === "John Smith" && 
+                 parsed.age === 35 && 
+                 parsed.occupation === "software engineer" &&
+                 parsed.location === "San Francisco";
+
+  - vars:
+      text: 'Maria Garcia, age 28, works as a data scientist in New York City.'
+    assert:
+      - type: is-json
+      - type: javascript
+        value: |
+          const parsed = JSON.parse(output);
+          return parsed.name === "Maria Garcia" && 
+                 parsed.age === 28 && 
+                 parsed.occupation === "data scientist" &&
+                 parsed.location === "New York City";
+
+  - vars:
+      text: 'Dr. Ahmed Hassan is a 42-year-old cardiologist practicing in London.'
+    assert:
+      - type: is-json
+      - type: javascript
+        value: |
+          const parsed = JSON.parse(output);
+          return parsed.name && parsed.age && parsed.occupation && parsed.location;

examples/mistral/promptfooconfig.multimodal.yaml

@@ -0,0 +1,42 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: 'Multimodal capabilities with a current Mistral vision model'
+
+providers:
+  - id: mistral:mistral-large-2512
+    config:
+      temperature: 0.5
+      max_tokens: 1000
+
+prompts:
+  - '{{task}}'
+
+tests:
+  - vars:
+      task: 'Describe what you would expect to see in a typical office workspace photo, including common objects and furniture.'
+    assert:
+      - type: contains
+        value: 'desk'
+      - type: contains
+        value: 'computer'
+      - type: javascript
+        value: 'output.length >= 100'
+
+  - vars:
+      task: 'If I were to show you an image of a restaurant menu, what types of information would you look for to help someone decide what to order?'
+    assert:
+      - type: contains
+        value: 'price'
+      - type: contains
+        value: 'menu'
+      - type: javascript
+        value: 'output.length >= 150'
+
+  - vars:
+      task: 'Explain how you would analyze a chart or graph if one were provided to you.'
+    assert:
+      - type: contains
+        value: 'data'
+      - type: contains
+        value: 'chart'
+      - type: javascript
+        value: 'output.length >= 100'

examples/mistral/promptfooconfig.reasoning.yaml

@@ -0,0 +1,44 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: 'Advanced reasoning with Mistral Magistral models'
+
+providers:
+  - id: mistral:magistral-medium-latest
+    label: magistral-medium
+    config:
+      temperature: 0.7
+      top_p: 0.95
+      max_tokens: 40960
+
+  # The `magistral-small-latest` alias now resolves to Mistral Small 4, so use the
+  # canonical id. Small 4 solves these with chain-of-thought when prompted to.
+  - id: mistral:mistral-small-latest
+    label: mistral-small-4
+    config:
+      temperature: 0.7
+      top_p: 0.95
+      max_tokens: 40960
+
+prompts:
+  - 'Think through this problem step by step: {{problem}}'
+
+tests:
+  # Reasoning models keep their working in a separate (stripped) thinking chunk and format
+  # final answers with LaTeX, so correctness is graded with a model rubric rather than brittle
+  # substring matching.
+  - vars:
+      problem: 'A farmer has chickens and rabbits in a pen. There are 30 heads and 88 legs total. How many chickens and how many rabbits are there?'
+    assert:
+      - type: llm-rubric
+        value: 'Correctly identifies that there are 16 chickens and 14 rabbits through systematic reasoning'
+
+  - vars:
+      problem: 'Three friends split a restaurant bill. Alice pays twice as much as Bob, and Charlie pays $15 more than Bob. If the total bill is $105, how much does each person pay?'
+    assert:
+      - type: llm-rubric
+        value: 'Shows the algebraic setup and correctly calculates Bob: $22.50, Alice: $45, Charlie: $37.50'
+
+  - vars:
+      problem: 'A water tank is being filled by two pipes and drained by one pipe. Pipe A fills at 10 gallons/minute, Pipe B fills at 15 gallons/minute, and the drain empties at 8 gallons/minute. If all pipes operate simultaneously and the tank starts empty, how long will it take to fill a 340-gallon tank?'
+    assert:
+      - type: llm-rubric
+        value: 'Correctly calculates the net fill rate (17 gallons/minute) and determines it takes 20 minutes'

examples/mistral/promptfooconfig.tool-routing.yaml

@@ -0,0 +1,104 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: 'Mistral tool routing and output shape QA'
+
+prompts:
+  - '{{prompt}}'
+
+providers:
+  - id: mistral:mistral-large-latest
+    label: tool-any-inline
+    config:
+      temperature: 0
+      max_tokens: 128
+      tool_choice: any
+      parallel_tool_calls: false
+      tools:
+        - type: function
+          function:
+            name: calculate
+            description: Perform basic mathematical calculations
+            parameters:
+              type: object
+              properties:
+                operation:
+                  type: string
+                  enum: ['add', 'subtract', 'multiply', 'divide']
+                a:
+                  type: number
+                b:
+                  type: number
+              required: ['operation', 'a', 'b']
+  - id: mistral:mistral-large-latest
+    label: tool-any-file
+    config:
+      temperature: 0
+      max_tokens: 128
+      tool_choice: any
+      parallel_tool_calls: false
+      tools: 'file://tools/calculate.yaml'
+  - id: mistral:mistral-large-latest
+    label: tool-auto-file
+    config:
+      temperature: 0
+      max_tokens: 128
+      tool_choice: auto
+      parallel_tool_calls: false
+      tools: 'file://tools/calculate.yaml'
+  - id: mistral:mistral-large-latest
+    label: plain-chat
+    config:
+      temperature: 0
+      max_tokens: 64
+
+tests:
+  - description: Tool-only response with inline tools
+    providers:
+      - tool-any-inline
+    vars:
+      prompt: 'You must call the calculate tool for this task. Do not answer in natural language. Compute 15 multiplied by 8.'
+    assert:
+      - type: is-valid-openai-tools-call
+      - type: javascript
+        value: output[0].function.name === 'calculate'
+      - type: javascript
+        value: JSON.parse(output[0].function.arguments).operation === 'multiply'
+
+  - description: Tool-only response with file-based tools
+    providers:
+      - tool-any-file
+    vars:
+      prompt: 'Use the calculate tool to divide 100 by 4. Return only the tool call.'
+    assert:
+      - type: is-valid-openai-tools-call
+      - type: javascript
+        value: output[0].function.name === 'calculate'
+      - type: javascript
+        value: JSON.parse(output[0].function.arguments).operation === 'divide'
+
+  - description: Mixed content and tool calls
+    providers:
+      - tool-auto-file
+    vars:
+      prompt: 'Before calling the calculate tool, include the exact sentence "Let me calculate that." in your assistant message. In the same assistant message, call the calculate tool for 15 multiplied by 8. Do not provide the final numeric answer.'
+    assert:
+      - type: javascript
+        value: |
+          return (
+            output &&
+            typeof output === 'object' &&
+            !Array.isArray(output) &&
+            typeof output.content === 'string' &&
+            output.content.includes('Let me calculate that.') &&
+            Array.isArray(output.tool_calls) &&
+            output.tool_calls.length > 0 &&
+            output.tool_calls[0].function?.name === 'calculate'
+          );
+
+  - description: Plain chat still returns a string
+    providers:
+      - plain-chat
+    vars:
+      prompt: 'Respond with the single word: ok'
+    assert:
+      - type: javascript
+        value: "typeof output === 'string' && output.trim().toLowerCase() === 'ok'"

examples/mistral/promptfooconfig.tool-use.yaml

@@ -0,0 +1,50 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: 'Function calling and tool use with Mistral models'
+
+providers:
+  - id: mistral:mistral-large-latest
+    config:
+      temperature: 0.1
+      tools:
+        - type: function
+          function:
+            name: calculate
+            description: Perform basic mathematical calculations
+            parameters:
+              type: object
+              properties:
+                operation:
+                  type: string
+                  enum: ['add', 'subtract', 'multiply', 'divide']
+                  description: The mathematical operation to perform
+                a:
+                  type: number
+                  description: First number
+                b:
+                  type: number
+                  description: Second number
+              required: ['operation', 'a', 'b']
+
+prompts:
+  - '{{question}}'
+
+tests:
+  - vars:
+      question: 'What is 15 multiplied by 8?'
+    assert:
+      - type: contains
+        value: 'calculate'
+      - type: contains
+        value: 'multiply'
+      - type: cost
+        threshold: 0.05
+
+  - vars:
+      question: 'Calculate 100 divided by 4'
+    assert:
+      - type: contains
+        value: 'calculate'
+      - type: contains
+        value: 'divide'
+      - type: cost
+        threshold: 0.05

examples/mistral/promptfooconfig.yaml

@@ -0,0 +1,135 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: Mistral AI model comparison and evaluation
+
+prompts:
+  - '{{message}}'
+
+providers:
+  # Reasoning models - specialized for complex problems
+  - id: mistral:magistral-medium-latest
+    label: magistral-medium
+    config:
+      temperature: 0.7
+      top_p: 0.95
+      max_tokens: 40960
+
+  # Traditional chat models
+  - id: mistral:mistral-large-latest
+    label: large
+    config:
+      temperature: 0.7
+
+  - id: mistral:mistral-medium-latest
+    label: medium
+    config:
+      temperature: 0.7
+
+  - id: mistral:mistral-small-latest
+    label: small
+    config:
+      temperature: 0.7
+
+# Use Mistral models for evaluation instead of OpenAI
+defaultTest:
+  options:
+    # Use Mistral Large for grading and Mistral embeddings for similarity
+    provider:
+      id: mistral:mistral-large-latest
+      embedding:
+        id: mistral:embedding:mistral-embed
+
+tests:
+  # Simple chat scenarios
+  - description: Casual greeting
+    vars:
+      message: 'Hello! How are you today?'
+    assert:
+      - type: llm-rubric
+        value: Responds in a friendly, conversational manner
+      - type: similar
+        value: "Hi there! I'm doing well, thanks for asking."
+        threshold: 0.7
+
+  - description: Creative writing request
+    vars:
+      message: 'Write a short story about a robot learning to paint'
+    assert:
+      - type: llm-rubric
+        value: Creates an engaging creative story with clear narrative structure
+      - type: contains
+        value: robot
+      - type: contains
+        value: paint
+
+  # Reasoning scenarios - where Magistral models should excel
+  - description: Mathematical reasoning
+    vars:
+      message: 'Solve this step by step: If a pizza has 8 slices and you eat 3 slices, then your friend eats twice as many slices as you did, how many slices are left?'
+    assert:
+      - type: contains
+        value: '2'
+      - type: llm-rubric
+        value: Shows clear step-by-step mathematical reasoning and arrives at the correct answer
+
+  - description: Logical reasoning
+    vars:
+      message: 'If all roses are flowers, and some flowers are red, can we conclude that some roses are red? Explain your reasoning.'
+    assert:
+      - type: icontains
+        value: 'cannot'
+      - type: llm-rubric
+        value: Correctly identifies the logical fallacy and explains why the conclusion doesn't follow
+
+  - description: Complex problem solving
+    vars:
+      message: 'You have a 3-gallon jug and a 5-gallon jug. How do you measure exactly 4 gallons of water? Show your steps.'
+    assert:
+      - type: llm-rubric
+        value: Provides a correct step-by-step solution to the water jug problem
+      - type: similar
+        value: 'Fill the 5-gallon jug, pour into 3-gallon jug, empty 3-gallon jug, pour remaining 2 gallons from 5-gallon into 3-gallon, fill 5-gallon again, pour into 3-gallon until full'
+        threshold: 0.6
+
+  # Multi-language capabilities
+  - description: French conversation
+    vars:
+      message: 'Bonjour! Comment allez-vous? Pouvez-vous me parler de Paris?'
+    assert:
+      - type: llm-rubric
+        value: Responds appropriately in French and provides information about Paris
+      - type: contains
+        value: Paris
+
+  # Technical explanations
+  - description: Technical concept explanation
+    vars:
+      message: 'Explain how machine learning works in simple terms that a 10-year-old could understand'
+    assert:
+      - type: llm-rubric
+        value: Explains machine learning concepts clearly and appropriately for a young audience
+      - type: similar
+        value: 'Machine learning is like teaching a computer to recognize patterns and make predictions by showing it lots of examples'
+        threshold: 0.5
+
+  # Code generation
+  - description: Code writing task
+    vars:
+      message: 'Write a Python function that takes a list of numbers and returns the average'
+    assert:
+      - type: contains
+        value: 'def'
+      - type: contains
+        value: 'sum'
+      - type: llm-rubric
+        value: Provides correct Python code for calculating an average
+
+  # Ethical reasoning
+  - description: Ethical discussion
+    vars:
+      message: 'What are the ethical considerations when developing AI systems?'
+    assert:
+      - type: llm-rubric
+        value: Discusses important ethical considerations like bias, privacy, transparency, and societal impact
+      - type: similar
+        value: 'Key ethical considerations include preventing bias, protecting privacy, ensuring transparency, and considering societal impact'
+        threshold: 0.6

examples/mistral/tools/calculate.yaml

@@ -0,0 +1,15 @@
+- type: function
+  function:
+    name: calculate
+    description: Perform basic mathematical calculations
+    parameters:
+      type: object
+      properties:
+        operation:
+          type: string
+          enum: ['add', 'subtract', 'multiply', 'divide']
+        a:
+          type: number
+        b:
+          type: number
+      required: ['operation', 'a', 'b']