chore: import upstream snapshot with attribution

examples/openai-responses/README.md

@@ -0,0 +1,157 @@
+# openai-responses (OpenAI Responses API Examples)
+
+This directory contains examples for testing OpenAI's Responses API with promptfoo.
+
+You can run this example with:
+
+```bash
+npx promptfoo@latest init --example openai-responses
+cd openai-responses
+```
+
+## Examples
+
+### Basic Responses API (`promptfooconfig.yaml`)
+
+Basic example showing how to use the Responses API with GPT-5.5, the GPT-5.4 family (`gpt-5.4-mini`, `gpt-5.4-nano`), and a GPT-4.1 comparison model.
+
+### External Response Format (`promptfooconfig.external-format.yaml`)
+
+Example demonstrating how to load `response_format` configuration from external files. This is useful for:
+
+- Reusing complex JSON schemas across multiple configurations
+- Managing large schemas in separate files for better organization
+- Version controlling schemas independently
+
+This example compares inline vs. external file approach:
+
+- **Inline**: JSON schema defined directly in the config
+- **External**: JSON schema loaded from `response_format.json` using `file://` syntax
+
+### Function Calling (`promptfooconfig.function-call.yaml`)
+
+Example demonstrating function calling capabilities with the Responses API.
+
+### Function Callbacks (`promptfooconfig.function-callback.yaml`)
+
+Example showing how to use function callbacks to execute functions locally instead of just returning the function call. This allows you to:
+
+- Execute custom logic when the model calls a function
+- Return the result directly to the test assertions
+- Test end-to-end workflows including function execution
+
+Key differences from regular function calling:
+
+- Uses `functionToolCallbacks` to define JavaScript functions
+- Functions are executed locally and results are returned
+- Perfect for testing tool-using AI agents
+
+### Reasoning Models (`promptfooconfig.reasoning.yaml`)
+
+Example showing how to use reasoning models (o1, o3, etc.) with specific configurations.
+
+### GPT-5.1 (`promptfooconfig.gpt-5.1.yaml`)
+
+Example demonstrating GPT-5.1's key features including:
+
+- **`none` reasoning mode**: No reasoning tokens for fastest responses
+- **Verbosity control**: Adjustable output length (`low`, `medium`, `high`)
+- **Reasoning effort levels**: Compare `none`, `medium`, and `high` reasoning modes
+- **Coding tasks**: Optimized for coding and problem-solving workflows
+
+### GPT-5.2 (`promptfooconfig.gpt-5.2.yaml`)
+
+Example comparing GPT-5.2 with different reasoning effort levels:
+
+- **none**: No reasoning tokens for fastest responses
+- **medium**: Balanced reasoning for most tasks
+- **high**: Maximum reasoning for complex problem-solving
+
+### GPT-5.5 (`promptfooconfig.gpt-5.5.yaml`)
+
+Example comparing GPT-5.5 standard and pro models with different Responses API reasoning settings.
+
+### GPT-5.6 (`promptfooconfig.gpt-5.6.yaml`)
+
+Example comparing the Sol, Terra, and Luna tiers. The `gpt-5.6` alias routes to Sol. All tiers support `max` reasoning; Codex `ultra` is available for Sol and Terra rather than as a Responses API reasoning value.
+
+### Image Processing (`promptfooconfig.image.yaml`)
+
+Example demonstrating image input capabilities with vision models.
+
+### Web Search (`promptfooconfig.web-search.yaml`)
+
+Example showing web search capabilities.
+
+### Prompt Caching (`promptfooconfig.prompt-cache.yaml`)
+
+Example combining `prompt_cache_key`, `prompt_cache_retention`, and included
+`web_search_call.results` payloads in a Responses request.
+
+### Codex Models (`promptfooconfig.codex.yaml`)
+
+Example using Codex models for code generation tasks.
+
+### MCP (Model Context Protocol) (`promptfooconfig.mcp.yaml`)
+
+Example demonstrating OpenAI's MCP integration with remote MCP servers. This example uses the DeepWiki MCP server to query GitHub repositories.
+
+#### MCP Features Demonstrated:
+
+- Remote MCP server integration
+- Tool filtering with `allowed_tools`
+- Approval settings configuration
+- Authentication headers (when needed)
+
+## Running the Examples
+
+To run any of these examples:
+
+```bash
+# Basic Responses API example
+npx promptfoo eval -c promptfooconfig.yaml
+
+# External response format example
+npx promptfoo eval -c promptfooconfig.external-format.yaml
+
+# MCP example
+npx promptfoo eval -c promptfooconfig.mcp.yaml
+
+# Function calling example
+npx promptfoo eval -c promptfooconfig.function-call.yaml
+
+# Function callbacks example
+npx promptfoo eval -c promptfooconfig.function-callback.yaml
+
+# Reasoning models example
+npx promptfoo eval -c promptfooconfig.reasoning.yaml
+
+# GPT-5.1 example
+npx promptfoo eval -c promptfooconfig.gpt-5.1.yaml
+
+# GPT-5.2 example
+npx promptfoo eval -c promptfooconfig.gpt-5.2.yaml
+
+# GPT-5.5 example
+npx promptfoo eval -c promptfooconfig.gpt-5.5.yaml
+
+# GPT-5.6 example
+npx promptfoo eval -c promptfooconfig.gpt-5.6.yaml
+
+# Prompt caching example
+npx promptfoo eval -c promptfooconfig.prompt-cache.yaml
+
+```
+
+## Prerequisites
+
+- OpenAI API key set in `OPENAI_API_KEY` environment variable
+- For MCP examples: Access to remote MCP servers (some may require authentication)
+
+## Notes
+
+- The MCP example uses the public DeepWiki MCP server which doesn't require authentication
+- For production use with MCP, carefully review the data being shared with third-party servers
+- Some MCP servers may require API keys or authentication tokens in the `headers` configuration
+- External file references support both JSON and YAML formats
+- External files are resolved relative to the config file location

examples/openai-responses/event_schema.json

@@ -0,0 +1,30 @@
+{
+  "type": "object",
+  "properties": {
+    "event_name": {
+      "type": "string",
+      "description": "The name of the event"
+    },
+    "date": {
+      "type": "string",
+      "description": "When the event takes place"
+    },
+    "location": {
+      "type": "string",
+      "description": "Where the event takes place"
+    },
+    "participants": {
+      "type": "array",
+      "items": {
+        "type": "string"
+      },
+      "description": "People attending the event"
+    },
+    "description": {
+      "type": "string",
+      "description": "Brief description of the event"
+    }
+  },
+  "required": ["event_name", "date", "location", "participants", "description"],
+  "additionalProperties": false
+}

examples/openai-responses/nested_response_format.json

@@ -0,0 +1,5 @@
+{
+  "type": "json_schema",
+  "name": "event_extraction",
+  "schema": "file://event_schema.json"
+}

examples/openai-responses/prompt.image.json

@@ -0,0 +1,16 @@
+[
+  {
+    "type": "message",
+    "role": "user",
+    "content": [
+      {
+        "type": "input_text",
+        "text": "Describe what you see in this image and tell me a short story inspired by it about {{topic}}."
+      },
+      {
+        "type": "input_image",
+        "image_url": "{{image_url}}"
+      }
+    ]
+  }
+]

examples/openai-responses/promptfooconfig.codex.yaml

@@ -0,0 +1,21 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: Codex
+
+prompts:
+  - |
+    Please write a function in Python that {{func_purpose}} and is named {{func_name}}.
+
+providers:
+  - id: openai:responses:gpt-5.3-codex
+    label: gpt-5.3-codex
+    config:
+      max_output_tokens: 1000
+      reasoning_effort: 'medium'
+
+tests:
+  - vars:
+      func_name: 'sum_of_numbers'
+      func_purpose: 'takes a list of numbers and returns the sum of the numbers'
+    assert:
+      - type: llm-rubric
+        value: 'Output should contain >=1 function(s) named {{func_name}} that take a list of numbers as input and return the sum of the numbers and are written in Python'

examples/openai-responses/promptfooconfig.external-format.yaml

@@ -0,0 +1,56 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: External response format file loading
+prompts:
+  - 'Extract information about the event: {{event}}'
+
+providers:
+  - id: openai:responses:gpt-4.1
+    label: inline-json-schema
+    config:
+      temperature: 0.7
+      instructions: 'You are a helpful AI assistant. Extract key details about the event.'
+      response_format:
+        type: 'json_schema'
+        name: 'event_information'
+        schema:
+          type: 'object'
+          properties:
+            event_name:
+              type: 'string'
+              description: 'The name of the event'
+            date:
+              type: 'string'
+              description: 'When the event takes place'
+            location:
+              type: 'string'
+              description: 'Where the event takes place'
+            participants:
+              type: 'array'
+              items:
+                type: 'string'
+              description: 'People attending the event'
+            description:
+              type: 'string'
+              description: 'Brief description of the event'
+          required: ['event_name', 'date', 'location', 'participants', 'description']
+          additionalProperties: false
+
+  - id: openai:responses:gpt-4.1
+    label: external-file-format
+    config:
+      temperature: 0.7
+      instructions: 'You are a helpful AI assistant. Extract key details about the event.'
+      response_format: file://response_format.json
+
+  - id: openai:responses:gpt-4.1
+    label: nested-file-reference
+    config:
+      temperature: 0.7
+      instructions: 'You are a helpful AI assistant. Extract key details about the event.'
+      response_format: file://nested_response_format.json
+
+tests:
+  - vars:
+      event: 'Alice and Bob are attending a JSON schema conference in New York on September 15th, 2025.'
+  - vars:
+      event: 'The annual music festival will take place in Central Park this weekend, featuring local artists and food vendors.'

examples/openai-responses/promptfooconfig.function-call.yaml

@@ -0,0 +1,81 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: Function Calling with OpenAI Responses API
+prompts:
+  - 'What is the weather like in {{location}}?'
+
+providers:
+  - id: openai:responses:o1-pro
+    label: o1-pro
+    config:
+      temperature: 0.7
+      max_output_tokens: 300
+      tool_choice: auto
+      tools:
+        - type: function
+          name: get_current_weather
+          description: 'Get the current weather in a given location'
+          parameters:
+            type: object
+            properties:
+              location:
+                type: string
+                description: 'The city and state, e.g. San Francisco, CA'
+              unit:
+                type: string
+                enum: ['celsius', 'fahrenheit']
+            required: ['location', 'unit']
+
+  - id: openai:responses:o3
+    label: o3
+    config:
+      reasoning_effort: 'medium'
+      max_output_tokens: 300
+      tool_choice: auto
+      tools:
+        - type: function
+          name: get_current_weather
+          description: 'Get the current weather in a given location'
+          parameters:
+            type: object
+            properties:
+              location:
+                type: string
+                description: 'The city and state, e.g. San Francisco, CA'
+              unit:
+                type: string
+                enum: ['celsius', 'fahrenheit']
+            required: ['location', 'unit']
+
+  - id: openai:responses:o4-mini
+    label: o4-mini
+    config:
+      reasoning_effort: 'low'
+      max_output_tokens: 300
+      tool_choice: auto
+      tools:
+        - type: function
+          name: get_current_weather
+          description: 'Get the current weather in a given location'
+          parameters:
+            type: object
+            properties:
+              location:
+                type: string
+                description: 'The city and state, e.g. San Francisco, CA'
+              unit:
+                type: string
+                enum: ['celsius', 'fahrenheit']
+            required: ['location', 'unit']
+
+tests:
+  - vars:
+      location: Boston, MA
+    assert:
+      - type: contains
+        value: 'get_current_weather'
+
+  - vars:
+      location: San Francisco, CA
+    assert:
+      - type: contains
+        value: 'get_current_weather'

examples/openai-responses/promptfooconfig.function-callback.yaml

@@ -0,0 +1,56 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: Function Calling with Callbacks in OpenAI Responses API
+prompts:
+  - 'Please add these numbers: {{a}} and {{b}}'
+  - 'Calculate the sum of {{a}} + {{b}}'
+  - 'What is {{a}} plus {{b}}?'
+
+providers:
+  - id: openai:responses:gpt-4o
+    config:
+      temperature: 0
+      max_output_tokens: 100
+      tool_choice: auto
+      tools:
+        - type: function
+          name: addNumbers
+          description: Add two numbers together
+          parameters:
+            type: object
+            properties:
+              a:
+                type: number
+                description: The first number to add
+              b:
+                type: number
+                description: The second number to add
+            required: ['a', 'b']
+      # Function callbacks execute locally instead of returning the raw function call
+      functionToolCallbacks:
+        addNumbers: |
+          async (args) => {
+            const { a, b } = JSON.parse(args);
+            return String(a + b);
+          }
+
+tests:
+  - vars:
+      a: 5
+      b: 6
+    assert:
+      - type: equals
+        value: '11'
+
+  - vars:
+      a: 123
+      b: 456
+    assert:
+      - type: equals
+        value: '579'
+
+  - vars:
+      a: -10
+      b: 20
+    assert:
+      - type: equals
+        value: '10'

examples/openai-responses/promptfooconfig.gpt-5.1.yaml

@@ -0,0 +1,54 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: GPT-5.1 Reasoning and Verbosity Comparison
+prompts:
+  - |
+    Please help me solve this coding problem:
+    {{coding_problem}}
+
+providers:
+  - id: openai:responses:gpt-5.1
+    label: gpt-5.1-none-low-verbosity
+    config:
+      reasoning:
+        effort: 'none' # Default - no reasoning tokens, fastest
+      verbosity: 'low' # Concise output
+      max_output_tokens: 1000
+
+  - id: openai:responses:gpt-5.1
+    label: gpt-5.1-medium-reasoning
+    config:
+      reasoning:
+        effort: 'medium' # Balanced reasoning
+      verbosity: 'medium' # Default verbosity
+      max_output_tokens: 2000
+
+  - id: openai:responses:gpt-5.1
+    label: gpt-5.1-high-reasoning
+    config:
+      reasoning:
+        effort: 'high' # Maximum reasoning for complex tasks
+      verbosity: 'high' # Detailed output
+      max_output_tokens: 4000
+
+tests:
+  - vars:
+      coding_problem: 'Write a Python function to find the longest palindromic substring in a given string. Optimize for time complexity.'
+    assert:
+      - type: javascript
+        value: output.toLowerCase().includes('palindrome')
+      - type: javascript
+        value: output.toLowerCase().includes('def ') || output.toLowerCase().includes('function')
+
+  - vars:
+      coding_problem: 'Implement a function to reverse a linked list in-place without using extra space. Return the new head.'
+    assert:
+      - type: contains
+        value: 'head'
+      - type: javascript
+        value: output.toLowerCase().includes('next') || output.toLowerCase().includes('pointer')
+
+  - vars:
+      coding_problem: 'Given a binary tree, write a function to find the maximum path sum. A path can start and end at any node in the tree.'
+    assert:
+      - type: llm-rubric
+        value: 'Solution correctly handles maximum path sum in a binary tree'

examples/openai-responses/promptfooconfig.gpt-5.2.yaml

@@ -0,0 +1,51 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: GPT-5.2 Reasoning Comparison
+prompts:
+  - |
+    Please help me solve this coding problem:
+    {{coding_problem}}
+
+providers:
+  - id: openai:responses:gpt-5.2
+    label: gpt-5.2-none-reasoning
+    config:
+      reasoning:
+        effort: 'none' # Fastest responses, no reasoning tokens
+      max_output_tokens: 2000
+
+  - id: openai:responses:gpt-5.2
+    label: gpt-5.2-medium-reasoning
+    config:
+      reasoning:
+        effort: 'medium' # Balanced reasoning
+      max_output_tokens: 2000
+
+  - id: openai:responses:gpt-5.2
+    label: gpt-5.2-high-reasoning
+    config:
+      reasoning:
+        effort: 'high' # Maximum reasoning for complex tasks
+      max_output_tokens: 4000
+
+tests:
+  - vars:
+      coding_problem: 'Write a Python function to find the longest palindromic substring in a given string. Optimize for time complexity.'
+    assert:
+      - type: javascript
+        value: output.toLowerCase().includes('palindrome') || output.toLowerCase().includes('def ')
+      - type: llm-rubric
+        value: 'Solution addresses finding palindromic substrings'
+
+  - vars:
+      coding_problem: 'Implement a function to reverse a linked list in-place without using extra space. Return the new head.'
+    assert:
+      - type: contains
+        value: 'head'
+      - type: javascript
+        value: output.toLowerCase().includes('next') || output.toLowerCase().includes('pointer')
+
+  - vars:
+      coding_problem: 'Given a binary tree, write a function to find the maximum path sum. A path can start and end at any node in the tree.'
+    assert:
+      - type: llm-rubric
+        value: 'Solution correctly handles maximum path sum in a binary tree'

examples/openai-responses/promptfooconfig.gpt-5.5.yaml

@@ -0,0 +1,50 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: GPT-5.5 Responses API reasoning comparison
+prompts:
+  - |
+    Solve this planning problem and explain your reasoning clearly:
+    {{problem}}
+
+providers:
+  - id: openai:responses:gpt-5.5
+    label: gpt-5.5-low-reasoning
+    config:
+      max_output_tokens: 2048
+      reasoning:
+        effort: 'low'
+      verbosity: 'medium'
+
+  - id: openai:responses:gpt-5.5
+    label: gpt-5.5-high-reasoning
+    config:
+      max_output_tokens: 4096
+      reasoning:
+        effort: 'high'
+      verbosity: 'medium'
+
+  - id: openai:responses:gpt-5.5-pro
+    label: gpt-5.5-pro-xhigh-reasoning
+    config:
+      max_output_tokens: 8192
+      reasoning:
+        effort: 'xhigh'
+      verbosity: 'medium'
+
+tests:
+  - vars:
+      problem: 'A team has 6 engineers, 3 designers, and 2 product managers. Build a two-week launch plan that sequences discovery, implementation, QA, and release readiness without overloading any one function.'
+    assert:
+      - type: icontains
+        value: discovery
+      - type: icontains
+        value: QA
+
+  - vars:
+      problem: 'A support queue has 120 tickets, 30% are billing issues, and 15% require engineering investigation. Propose a triage plan that minimizes customer wait time while preserving escalation quality.'
+    assert:
+      - type: icontains-any
+        value:
+          - triage
+          - prioritize
+      - type: icontains
+        value: escalation

examples/openai-responses/promptfooconfig.gpt-5.6.yaml

@@ -0,0 +1,39 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: GPT-5.6 Responses API comparison
+prompts:
+  - |
+    Solve this planning problem and explain the tradeoffs clearly:
+    {{problem}}
+
+providers:
+  - id: openai:responses:gpt-5.6-sol
+    label: gpt-5.6-sol-max-reasoning
+    config:
+      max_output_tokens: 8192
+      reasoning:
+        effort: 'max'
+
+  - id: openai:responses:gpt-5.6-terra
+    label: gpt-5.6-terra-medium-reasoning
+    config:
+      max_output_tokens: 4096
+      reasoning:
+        effort: 'medium'
+
+  - id: openai:responses:gpt-5.6-luna
+    label: gpt-5.6-luna-low-reasoning
+    config:
+      max_output_tokens: 2048
+      reasoning:
+        effort: 'low'
+
+tests:
+  - vars:
+      problem: 'Plan a two-week launch for a small product team, including discovery, implementation, QA, and release readiness.'
+    assert:
+      - type: icontains
+        value: QA
+      - type: icontains-any
+        value:
+          - launch
+          - release

examples/openai-responses/promptfooconfig.image.yaml

@@ -0,0 +1,15 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: Image Input with OpenAI Responses API
+prompts:
+  - file://prompt.image.json
+
+providers:
+  - id: openai:responses:gpt-4.1-mini
+    config:
+      temperature: 0.7
+      max_output_tokens: 300
+
+tests:
+  - vars:
+      topic: secret service agents
+      image_url: https://upload.wikimedia.org/wikipedia/commons/0/06/Commander_Biden_22_December_2021_%28cropped_to_face%29.jpg

examples/openai-responses/promptfooconfig.mcp.yaml

@@ -0,0 +1,52 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+
+description: OpenAI MCP integration example
+
+prompts:
+  - 'Can you search for information about {{topic}} in the {{repo}} repository?'
+
+providers:
+  - id: openai:responses:gpt-4.1-2025-04-14
+    config:
+      tools:
+        - type: mcp
+          server_label: deepwiki
+          server_url: https://mcp.deepwiki.com/mcp
+          require_approval: never
+          allowed_tools: ['ask_question']
+      max_output_tokens: 1000
+      temperature: 0.3
+  - id: openai:responses:gpt-5.4
+    config:
+      tools:
+        - type: mcp
+          server_label: deepwiki
+          server_url: https://mcp.deepwiki.com/mcp
+          require_approval: never
+          allowed_tools: ['ask_question']
+      max_output_tokens: 1000
+      temperature: 0.3
+
+tests:
+  - vars:
+      repo: modelcontextprotocol/modelcontextprotocol
+      topic: transport protocols
+    assert:
+      - type: contains
+        value: 'transport'
+      - type: contains
+        value: 'protocol'
+
+  - vars:
+      repo: facebook/react
+      topic: hooks
+    assert:
+      - type: contains
+        value: 'hook'
+
+  - vars:
+      repo: microsoft/typescript
+      topic: type system
+    assert:
+      - type: contains
+        value: 'type'

examples/openai-responses/promptfooconfig.prompt-cache.yaml

@@ -0,0 +1,26 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: Prompt caching and included web results with OpenAI Responses API
+prompts:
+  - |
+    Give me one concise update from the past week about {{topic}}.
+    Include a cited source.
+
+providers:
+  - id: openai:responses:gpt-5.5
+    config:
+      max_output_tokens: 300
+      prompt_cache_key: recent-topic-update
+      prompt_cache_retention: 24h
+      include:
+        - web_search_call.results
+      tools:
+        - type: web_search
+
+tests:
+  - vars:
+      topic: NASA missions
+    assert:
+      - type: icontains-any
+        value:
+          - NASA
+          - mission

examples/openai-responses/promptfooconfig.reasoning.yaml

@@ -0,0 +1,54 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: Mathematical Reasoning with OpenAI Responses API
+prompts:
+  - |
+    Please solve this math problem step-by-step, showing all your work and explaining your reasoning at each step:
+    {{math_problem}}
+
+providers:
+  - id: openai:responses:o3-mini
+    label: o3-mini-reasoning
+    config:
+      max_output_tokens: 1000
+      reasoning:
+        effort: 'high'
+      user: 'math-student-001'
+
+  - id: openai:responses:o3-pro
+    label: o3-pro-reasoning
+    config:
+      max_output_tokens: 2000
+      reasoning:
+        effort: 'high'
+
+  - id: openai:responses:o4-mini
+    label: o4-mini-reasoning
+    config:
+      max_output_tokens: 1000
+      reasoning:
+        effort: 'medium'
+  - id: openai:responses:gpt-5.4
+    label: gpt-5.4-none-reasoning
+    config:
+      max_output_tokens: 1000
+      reasoning:
+        effort: 'none'
+
+tests:
+  - vars:
+      math_problem: 'A bag contains 5 red marbles, 3 blue marbles, and 7 green marbles. Two marbles are drawn from the bag without replacement. What is the probability that both marbles are the same color? Express your answer as a fraction in its simplest form.'
+    assert:
+      - type: contains
+        value: '34/105'
+
+  - vars:
+      math_problem: 'Find all values of x that satisfy the equation: log₂(x+3) + log₂(x-1) = 3. Show all steps of your solution and verify your answers.'
+    assert:
+      - type: llm-rubric
+        value: 'Answer: -1 + 2√3'
+
+  - vars:
+      math_problem: 'In a right triangle, the hypotenuse is 13 cm, and one of the legs is 5 cm. A circle is inscribed in the triangle (touching all three sides). What is the radius of this inscribed circle? Show your complete solution with all geometric reasoning.'
+    assert:
+      - type: contains
+        value: '2 cm'

examples/openai-responses/promptfooconfig.web-search.yaml

@@ -0,0 +1,20 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: Web Search for Recent Positive News with OpenAI Responses API
+prompts:
+  - 'Find me something fun or positive that happened recently in the news about {{topic}}. I want to hear about developments or stories from the past week. Make sure to cite your sources.'
+
+providers:
+  - id: openai:responses:gpt-4o
+    config:
+      temperature: 0.7
+      max_output_tokens: 500
+      tools:
+        - type: web_search
+
+tests:
+  - vars:
+      topic: space exploration and NASA missions
+  - vars:
+      topic: artificial intelligence breakthroughs
+  - vars:
+      topic: environmental conservation success stories

examples/openai-responses/promptfooconfig.yaml

@@ -0,0 +1,123 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: OpenAI Responses API with Different Response Formats
+prompts:
+  - 'Extract information about the event: {{event}}'
+
+providers:
+  # GPT-5 models showcasing latest capabilities
+  - id: openai:responses:gpt-5.6
+    label: gpt-5.6-json-schema
+    config:
+      instructions: 'You are a helpful AI assistant. Extract key details about the event.'
+      response_format:
+        type: 'json_schema'
+        name: 'event_information'
+        schema:
+          type: 'object'
+          properties:
+            event_name:
+              type: 'string'
+              description: 'The name of the event'
+            date:
+              type: 'string'
+              description: 'When the event takes place'
+            location:
+              type: 'string'
+              description: 'Where the event takes place'
+            participants:
+              type: 'array'
+              items:
+                type: 'string'
+              description: 'People attending the event'
+          required: ['event_name', 'date', 'location', 'participants']
+          additionalProperties: false
+
+  - id: openai:responses:gpt-5.4-nano
+    label: gpt-5.4-nano-json-schema
+    config:
+      instructions: 'You are a helpful AI assistant. Extract key details about the event.'
+      response_format:
+        type: 'json_schema'
+        name: 'event_information'
+        schema:
+          type: 'object'
+          properties:
+            event_name:
+              type: 'string'
+              description: 'The name of the event'
+            date:
+              type: 'string'
+              description: 'When the event takes place'
+            location:
+              type: 'string'
+              description: 'Where the event takes place'
+            participants:
+              type: 'array'
+              items:
+                type: 'string'
+              description: 'People attending the event'
+          required: ['event_name', 'date', 'location', 'participants']
+          additionalProperties: false
+      store: true
+
+  - id: openai:responses:gpt-5.4-mini
+    label: gpt-5.4-mini-json-schema
+    config:
+      instructions: 'You are a helpful AI assistant. Extract key details about the event.'
+      response_format:
+        type: 'json_schema'
+        name: 'event_information'
+        schema:
+          type: 'object'
+          properties:
+            event_name:
+              type: 'string'
+              description: 'The name of the event'
+            date:
+              type: 'string'
+              description: 'When the event takes place'
+            location:
+              type: 'string'
+              description: 'Where the event takes place'
+            participants:
+              type: 'array'
+              items:
+                type: 'string'
+              description: 'People attending the event'
+          required: ['event_name', 'date', 'location', 'participants']
+          additionalProperties: false
+
+  # Legacy GPT-4.1 for comparison
+  - id: openai:responses:gpt-4.1
+    label: gpt-4.1-json-schema
+    config:
+      temperature: 0.7
+      instructions: 'You are a helpful AI assistant. Extract key details about the event.'
+      response_format:
+        type: 'json_schema'
+        name: 'event_information'
+        schema:
+          type: 'object'
+          properties:
+            event_name:
+              type: 'string'
+              description: 'The name of the event'
+            date:
+              type: 'string'
+              description: 'When the event takes place'
+            location:
+              type: 'string'
+              description: 'Where the event takes place'
+            participants:
+              type: 'array'
+              items:
+                type: 'string'
+              description: 'People attending the event'
+          required: ['event_name', 'date', 'location', 'participants']
+          additionalProperties: false
+
+tests:
+  - vars:
+      event: 'Alice and Bob are attending a JSON schema conference in New York on September 15th, 2025.'
+  - vars:
+      event: 'The annual music festival will take place in Central Park this weekend, featuring local artists and food vendors.'

examples/openai-responses/response_format.json

@@ -0,0 +1,34 @@
+{
+  "type": "json_schema",
+  "name": "event_extraction",
+  "schema": {
+    "type": "object",
+    "properties": {
+      "event_name": {
+        "type": "string",
+        "description": "The name of the event"
+      },
+      "date": {
+        "type": "string",
+        "description": "When the event takes place"
+      },
+      "location": {
+        "type": "string",
+        "description": "Where the event takes place"
+      },
+      "participants": {
+        "type": "array",
+        "items": {
+          "type": "string"
+        },
+        "description": "People attending the event"
+      },
+      "description": {
+        "type": "string",
+        "description": "Brief description of the event"
+      }
+    },
+    "required": ["event_name", "date", "location", "participants", "description"],
+    "additionalProperties": false
+  }
+}