chore: import upstream snapshot with attribution

examples/xai/README.md

@@ -0,0 +1,10 @@
+# xai (xAI)
+
+Examples for using promptfoo with [xAI's Grok](https://x.ai/) models.
+
+## Examples
+
+- [chat](./chat/) - Text generation and reasoning with Grok models
+- [embeddings](./embeddings/) - Semantic similarity with xAI embeddings
+- [video](./video/) - Video generation with Grok Imagine
+- [voice](./voice/) - Real-time voice conversations with Grok Voice Agent

examples/xai/chat/README.md

@@ -0,0 +1,111 @@
+# xai/chat (xAI Grok Models Evaluation)
+
+This example demonstrates how to evaluate xAI's Grok models across their main capabilities: text generation with reasoning, image creation, and server-side search tools.
+
+You can run this example with:
+
+```bash
+npx promptfoo@latest init --example xai/chat
+cd xai/chat
+```
+
+## Environment Variables
+
+This example requires the following environment variable:
+
+- `XAI_API_KEY` - Your xAI API key. You can obtain this from the [xAI Console](https://console.x.ai/)
+
+## Quick Start
+
+```bash
+# Set your API key
+export XAI_API_KEY=your_api_key_here
+
+# Run the main evaluation
+promptfoo eval
+
+# View results in the web interface
+promptfoo view
+```
+
+## What's Tested
+
+This example includes configurations to test different Grok capabilities:
+
+- **Text Generation** (`promptfooconfig.yaml`) - Mathematical reasoning with the current Grok 4.3 and Grok 4.20 families
+- **Image Generation** (`promptfooconfig.images.yaml`) - Artistic image creation using Grok's image models
+- **Search Tools** (`promptfooconfig.search.yaml`) - Real-time web and X search using the Responses API
+- **Agent Tools (Responses API)** (`promptfooconfig.responses.yaml`) - Autonomous web and X search using Agent Tools
+- **Search Demo** (`promptfooconfig.promptfoo-search.yaml`) - Responses API search with assertions example
+
+## Run Individual Tests
+
+```bash
+# Text generation with mathematical reasoning
+promptfoo eval -c promptfooconfig.yaml
+
+# Image generation with artistic styles
+promptfoo eval -c promptfooconfig.images.yaml
+
+# Search tools with web and X sources
+promptfoo eval -c promptfooconfig.search.yaml
+
+# Agent Tools with Responses API (recommended)
+promptfoo eval -c promptfooconfig.responses.yaml
+
+# Search demo with assertions
+promptfoo eval -c promptfooconfig.promptfoo-search.yaml
+```
+
+## Featured Models
+
+### Grok 4.3
+
+The recommended starting point for general text workflows:
+
+- `xai:grok-4.3` - General-purpose reasoning model
+- `reasoning_effort` - Supports `none`, `low`, `medium`, and `high` in chat configs
+- `xai:responses:grok-4.3` - Recommended form for server-side tools
+
+### Grok 4.20
+
+- `xai:grok-4.20-reasoning` - Reasoning model
+- `xai:grok-4.20-non-reasoning` - Non-reasoning model
+- `xai:grok-4.20-multi-agent` - Multi-agent variant
+
+### Legacy Model Note
+
+xAI periodically retires older model slugs and may keep them working through redirects to newer replacements. This example uses Grok 4.3 plus alias-style Grok 4.20 family IDs, matching xAI's guidance for configs that should track the current release within a family.
+
+### Agent Tools (Responses API)
+
+Enable autonomous tool execution via the Responses API:
+
+```yaml
+providers:
+  - id: xai:responses:grok-4.3
+    config:
+      tools:
+        - type: web_search
+        - type: x_search
+        - type: code_interpreter
+```
+
+### Search Tools
+
+Enable real-time search via the Responses API:
+
+```yaml
+providers:
+  - id: xai:responses:grok-4.3
+    config:
+      tools:
+        - type: web_search
+        - type: x_search
+```
+
+## Expected Results
+
+- **Text Generation**: Grok will provide step-by-step mathematical solutions with clear reasoning
+- **Image Generation**: Generated images in the requested artistic styles
+- **Search Tools**: Current information from web and X with source citations

examples/xai/chat/prompt.yaml

@@ -0,0 +1,6 @@
+- role: system
+  content: |
+    You are a helpful assistant skilled at solving problems through careful reasoning and analysis. 
+    Approach problems step-by-step, showing your thought process clearly.
+- role: user
+  content: '{{problem}}'

examples/xai/chat/promptfooconfig.images.yaml

@@ -0,0 +1,15 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: xAI image generation evaluation
+prompts:
+  - 'In the style of {{artist}}: {{subject}}'
+
+providers:
+  - xai:image:grok-imagine-image
+
+tests:
+  - vars:
+      artist: Van Gogh
+      subject: sunset over mountains
+  - vars:
+      artist: Dali
+      subject: melting clocks

examples/xai/chat/promptfooconfig.promptfoo-search.yaml

@@ -0,0 +1,70 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: xAI search for promptfoo info
+
+prompts:
+  - |
+    {{question}}
+
+    Search both web and X/Twitter for the most relevant and recent information.
+    Include specific examples, quotes from posts, or citations where available.
+
+providers:
+  # Grok 4.3 with server-side search tools
+  - id: xai:responses:grok-4.3
+    config:
+      temperature: 0.7
+      max_output_tokens: 2048
+      tools:
+        - type: web_search
+        - type: x_search
+
+tests:
+  # Developer love and adoption
+  - vars:
+      question: Why do developers love promptfoo? What features make it stand out as the go-to open-source tool for LLM evaluation and testing?
+    assert:
+      - type: contains
+        value: promptfoo
+      - type: llm-rubric
+        value: The response should highlight positive aspects and strengths of promptfoo
+
+  # Developer adoption and community
+  - vars:
+      question: How many developers are using promptfoo? What does the community say about it on GitHub and developer forums?
+    assert:
+      - type: contains
+        value: promptfoo
+      - type: llm-rubric
+        value: The response should mention developer adoption numbers or positive community feedback about promptfoo
+
+  # Red teaming leadership
+  - vars:
+      question: How has promptfoo become the leading open-source solution for AI red teaming and LLM security testing? What vulnerabilities can it detect?
+    assert:
+      - type: contains-any
+        value:
+          - red team
+          - security
+          - vulnerabilities
+          - jailbreak
+          - prompt injection
+      - type: llm-rubric
+        value: The response should explain promptfoo's red teaming and security testing capabilities
+
+  # Recent momentum and growth
+  - vars:
+      question: What's driving promptfoo's rapid growth and developer adoption? What recent milestones or funding has it achieved?
+    assert:
+      - type: contains
+        value: promptfoo
+      - type: llm-rubric
+        value: The response should mention growth, funding, or adoption milestones for promptfoo
+
+  # Unique value proposition
+  - vars:
+      question: What makes promptfoo unique for testing LLM applications? Why do teams choose it for evaluating prompts, agents, and RAG systems?
+    assert:
+      - type: contains
+        value: promptfoo
+      - type: llm-rubric
+        value: The response should explain what makes promptfoo valuable and differentiated

examples/xai/chat/promptfooconfig.responses.yaml

@@ -0,0 +1,39 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: xAI Responses API with Agent Tools
+
+prompts:
+  - '{{question}}'
+
+providers:
+  # Grok 4.3 with Agent Tools (web + X search)
+  - id: xai:responses:grok-4.3
+    config:
+      temperature: 0.7
+      max_output_tokens: 2048
+      tools:
+        - type: web_search
+        - type: x_search
+
+tests:
+  # Test web search capability
+  - vars:
+      question: What is the current weather in San Francisco? Search the web for the latest information.
+    assert:
+      - type: contains-any
+        value:
+          - San Francisco
+          - weather
+          - temperature
+          - degrees
+
+  # Test X search capability
+  - vars:
+      question: What are people saying about AI on X/Twitter today? Search X for recent posts.
+    assert:
+      - type: contains-any
+        value:
+          - AI
+          - artificial intelligence
+          - posts
+          - X
+          - Twitter

examples/xai/chat/promptfooconfig.search.yaml

@@ -0,0 +1,29 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: xAI Responses API search demo
+
+prompts:
+  - 'What is the latest news about {{topic}}?'
+
+providers:
+  # Grok 4.3 with web + X search
+  - id: xai:responses:grok-4.3
+    label: grok-4.3-web-and-x-search
+    config:
+      temperature: 0.7
+      max_output_tokens: 2048
+      tools:
+        - type: web_search
+        - type: x_search
+
+  # Grok 4.3 with web search only
+  - id: xai:responses:grok-4.3
+    label: grok-4.3-web-search
+    config:
+      tools:
+        - type: web_search
+
+tests:
+  - vars:
+      topic: xAI
+  - vars:
+      topic: artificial intelligence breakthroughs

examples/xai/chat/promptfooconfig.yaml

@@ -0,0 +1,41 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: Testing xAI Grok models
+
+prompts:
+  - What are the key differences between machine learning and deep learning?
+  - file://./prompt.yaml
+
+providers:
+  # Grok 4.3 (recommended general-purpose model)
+  - id: xai:grok-4.3
+    config:
+      temperature: 0.7
+      reasoning_effort: high
+      max_completion_tokens: 4096
+
+  # Grok 4.20 reasoning family
+  - id: xai:grok-4.20-reasoning
+    config:
+      temperature: 0.7
+      max_completion_tokens: 4096
+
+  # Grok 4.20 non-reasoning family
+  - id: xai:grok-4.20-non-reasoning
+    config:
+      temperature: 0.7
+      max_completion_tokens: 4096
+
+  # Grok 4.20 multi-agent family
+  - id: xai:grok-4.20-multi-agent
+    config:
+      temperature: 0.7
+      max_completion_tokens: 4096
+
+defaultTest:
+  assert:
+    - type: javascript
+      value: 'output.length >= 100'
+
+tests:
+  - vars:
+      problem: 'A baker has 3 boxes of cookies. Each box contains 15 cookies. He sells 12 cookies. How many cookies does he have left?'

examples/xai/video/README.md

@@ -0,0 +1,82 @@
+# xai/video (xAI Grok Video Generation)
+
+You can run this example with:
+
+```bash
+npx promptfoo@latest init --example xai/video
+cd xai/video
+```
+
+This example demonstrates video generation using xAI's Grok Imagine API.
+
+## Setup
+
+1. Set your xAI API key:
+
+```bash
+export XAI_API_KEY=your_api_key_here
+```
+
+2. Run the evaluation:
+
+```bash
+npx promptfoo@latest eval
+```
+
+## Configuration Options
+
+| Option         | Type   | Default | Description                                        |
+| -------------- | ------ | ------- | -------------------------------------------------- |
+| `duration`     | number | 8       | Video length in seconds (1-15)                     |
+| `aspect_ratio` | string | 16:9    | Aspect ratio (16:9, 4:3, 1:1, 9:16, 3:4, 3:2, 2:3) |
+| `resolution`   | string | 720p    | Output resolution (720p, 480p)                     |
+
+## Advanced Features
+
+### Image-to-Video
+
+Animate a static image:
+
+```yaml
+providers:
+  - id: xai:video:grok-imagine-video
+    config:
+      image:
+        url: 'https://example.com/image.jpg'
+```
+
+### Video Editing
+
+Edit an existing video:
+
+```yaml
+providers:
+  - id: xai:video:grok-imagine-video
+    config:
+      video:
+        url: 'https://example.com/video.mp4'
+
+prompts:
+  - 'Make the colors more vibrant'
+```
+
+### Reference-to-Video
+
+Guide the generated video with up to seven reference images:
+
+```yaml
+providers:
+  - id: xai:video:grok-imagine-video
+    config:
+      reference_images:
+        - url: 'https://example.com/person.jpg'
+        - url: 'https://example.com/shirt.jpg'
+      duration: 10
+```
+
+`reference_images` cannot be combined with `image` or `video`, and reference-guided videos are limited to 10 seconds.
+
+## See Also
+
+- [xAI Provider Documentation](https://promptfoo.dev/docs/providers/xai)
+- [xAI Grok Imagine API](https://docs.x.ai/docs/guides/video-generations-and-edits)

examples/xai/video/promptfooconfig.yaml

@@ -0,0 +1,19 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: xAI Grok Imagine Video Generation
+
+providers:
+  - id: xai:video:grok-imagine-video
+    config:
+      duration: 3
+      aspect_ratio: '16:9'
+      resolution: '720p'
+
+prompts:
+  - 'Generate a video of: {{scene}}'
+
+tests:
+  - vars:
+      scene: a red ball bouncing
+    assert:
+      - type: cost
+        threshold: 1.0

examples/xai/voice/README.md

@@ -0,0 +1,158 @@
+# xai/voice (xAI Grok Voice Agent)
+
+This example demonstrates how to use xAI's Grok Voice Agent API with promptfoo for evaluating real-time voice AI conversations.
+
+## Prerequisites
+
+- An xAI API key with access to the Voice Agent API
+- Set `XAI_API_KEY` environment variable
+
+## Setup
+
+```bash
+npx promptfoo@latest init --example xai/voice
+cd xai/voice
+export XAI_API_KEY=your-api-key
+```
+
+## Run
+
+```bash
+npx promptfoo@latest eval
+```
+
+## Configuration Options
+
+### Models
+
+- `grok-voice-think-fast-1.0` - The voice model documented by xAI's [Voice Agent API guide](https://docs.x.ai/docs/guides/voice). The model is selected via the `?model=` query parameter on the realtime WebSocket URL.
+
+### Voices
+
+xAI Voice supports 5 different voices:
+
+- `Ara` (default) - Female voice
+- `Rex` - Male voice
+- `Sal` - Male voice
+- `Eve` - Female voice
+- `Leo` - Male voice
+
+### Built-in Tools
+
+The Voice API includes several built-in tools:
+
+- `web_search` - Search the web for information
+- `x_search` - Search posts on X (Twitter)
+- `file_search` - Search uploaded files in vector stores
+
+### Custom Functions
+
+You can define custom function tools that the voice agent can call. Tools can be defined inline or loaded from external files:
+
+**Inline definition:**
+
+```yaml
+tools:
+  - type: function
+    name: get_weather
+    description: Get the current weather for a location
+    parameters:
+      type: object
+      properties:
+        location:
+          type: string
+          description: The city and state
+      required: ['location']
+```
+
+**Load from external file:**
+
+```yaml
+# promptfooconfig.yaml
+config:
+  tools: file://tools.yaml
+```
+
+```yaml
+# tools.yaml
+- type: function
+  name: get_weather
+  description: Get the current weather for a location
+  parameters:
+    type: object
+    properties:
+      location:
+        type: string
+        description: The city and state
+    required:
+      - location
+
+- type: function
+  name: set_reminder
+  description: Set a reminder for the user
+  parameters:
+    type: object
+    properties:
+      message:
+        type: string
+      time:
+        type: string
+    required:
+      - message
+      - time
+```
+
+External files can be YAML or JSON format.
+
+### Audio Configuration
+
+Configure input/output audio formats:
+
+```yaml
+config:
+  audio:
+    input:
+      format:
+        type: audio/pcm
+        rate: 24000
+    output:
+      format:
+        type: audio/pcm
+        rate: 24000
+```
+
+Supported formats: `audio/pcm`, `audio/pcmu`, `audio/pcma`
+Supported sample rates: 8000, 16000, 22050, 24000, 32000, 44100, 48000 Hz
+
+### Custom WebSocket URL
+
+For local testing, proxies, or endpoints with query parameters:
+
+```yaml
+config:
+  websocketUrl: 'wss://custom-endpoint.example.com/path?token=xyz'
+```
+
+### Function Call Assertions
+
+When using custom function tools, you can assert on the function calls:
+
+```yaml
+tests:
+  - vars:
+      question: 'Set the volume to 50%'
+    assert:
+      - type: javascript
+        value: |
+          const calls = output.functionCalls || [];
+          return calls.some(c => c.name === 'set_volume');
+```
+
+## Pricing
+
+xAI Voice API is billed at $0.05 per minute of audio.
+
+## Resources
+
+- [xAI Voice API Documentation](https://docs.x.ai/docs/guides/voice)
+- [xAI API Reference](https://docs.x.ai/api)

examples/xai/voice/input.json

@@ -0,0 +1,11 @@
+[
+  {
+    "role": "user",
+    "content": [
+      {
+        "type": "text",
+        "text": "{{question}}"
+      }
+    ]
+  }
+]

examples/xai/voice/promptfooconfig.yaml

@@ -0,0 +1,64 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: xAI Grok Voice Agent API demonstration
+
+prompts:
+  - file://input.json
+
+providers:
+  - id: xai:voice:grok-voice-think-fast-1.0
+    config:
+      voice: 'Ara' # Options: Ara, Rex, Sal, Eve, Leo
+      instructions: |
+        You are a helpful voice assistant. Keep your responses concise and informative.
+        When the user asks about weather, use the get_weather function.
+        When the user asks to set a reminder, use the set_reminder function.
+      modalities: ['text', 'audio']
+      websocketTimeout: 60000 # 60 seconds
+
+      # Load tools from external file (supports YAML or JSON)
+      tools: file://tools.yaml
+
+      # For custom endpoints or testing, you can override the WebSocket URL:
+      # websocketUrl: 'wss://custom-endpoint.example.com/path?token=xyz'
+
+tests:
+  - vars:
+      question: 'Hello! What can you help me with today?'
+    assert:
+      - type: llm-rubric
+        value: Responds with a greeting and offers assistance
+
+  - vars:
+      question: 'Tell me about the benefits of real-time voice AI.'
+    assert:
+      - type: llm-rubric
+        value: Discusses the benefits of real-time voice AI technology
+
+  - vars:
+      question: 'Search the web for the latest AI news.'
+    assert:
+      - type: llm-rubric
+        value: Provides information about recent AI news or developments
+
+  # Function call assertion - weather
+  - vars:
+      question: "What's the weather like in San Francisco?"
+    assert:
+      - type: javascript
+        value: |
+          // Check that get_weather was called with the correct location
+          const calls = output.functionCalls || [];
+          return calls.some(c =>
+            c.name === 'get_weather' &&
+            c.arguments?.location?.toLowerCase().includes('san francisco')
+          );
+
+  # Function call assertion - reminder
+  - vars:
+      question: 'Remind me to call mom in 30 minutes'
+    assert:
+      - type: javascript
+        value: |
+          // Check that set_reminder was called
+          const calls = output.functionCalls || [];
+          return calls.some(c => c.name === 'set_reminder');

examples/xai/voice/tools.yaml

@@ -0,0 +1,38 @@
+# Tools for xAI Voice Agent
+# Load with: tools: file://tools.yaml
+
+# Built-in tool - web search
+- type: web_search
+
+# Custom function tools
+- type: function
+  name: get_weather
+  description: Get the current weather for a location
+  parameters:
+    type: object
+    properties:
+      location:
+        type: string
+        description: The city and state, e.g. San Francisco, CA
+      unit:
+        type: string
+        enum: [celsius, fahrenheit]
+        description: Temperature unit preference
+    required:
+      - location
+
+- type: function
+  name: set_reminder
+  description: Set a reminder for the user
+  parameters:
+    type: object
+    properties:
+      message:
+        type: string
+        description: The reminder message
+      time:
+        type: string
+        description: When to remind, e.g. "in 5 minutes" or "tomorrow at 9am"
+    required:
+      - message
+      - time