chore: import upstream snapshot with attribution

2026-07-13 13:24:08 +08:00
commit 0d3cb498a3
5438 changed files with 1316560 additions and 0 deletions
@@ -0,0 +1,153 @@
+# openai-mcp (OpenAI MCP Integration)
+
+This example demonstrates how to use OpenAI's Model Context Protocol (MCP) integration with the Responses API in promptfoo.
+
+You can run this example with:
+
+```bash
+npx promptfoo@latest init --example openai-mcp
+cd openai-mcp
+```
+
+## What is MCP?
+
+Model Context Protocol (MCP) is an open protocol that standardizes how applications provide tools and context to LLMs. OpenAI's MCP integration allows models to use remote MCP servers to perform tasks like searching repositories, accessing APIs, and more.
+
+## Environment Variables
+
+This example requires the following environment variables:
+
+- `OPENAI_API_KEY` - Your OpenAI API key from the OpenAI platform
+- `STRIPE_API_KEY` - Your Stripe API key (for authenticated examples only)
+
+You can set these in a `.env` file or directly in your environment.
+
+## Running the Examples
+
+1. **Set up environment variables**:
+
+   ```bash
+   export OPENAI_API_KEY="your_openai_api_key"
+   export STRIPE_API_KEY="your_stripe_api_key"  # For authenticated example only
+   ```
+
+2. **Run individual examples**:
+
+   ```bash
+   # Basic MCP integration
+   npx promptfoo eval -c promptfooconfig.yaml
+
+   # Authenticated MCP servers
+   npx promptfoo eval -c promptfooconfig.authenticated.yaml
+
+   # Approval workflow examples
+   npx promptfoo eval -c promptfooconfig.approval.yaml
+   ```
+
+3. **View results**:
+   ```bash
+   npx promptfoo view
+   ```
+
+## Examples
+
+### Basic MCP Usage (`promptfooconfig.yaml`)
+
+Demonstrates basic MCP integration using the public DeepWiki MCP server to search GitHub repositories.
+
+### Authenticated MCP (`promptfooconfig.authenticated.yaml`)
+
+Shows how to use MCP servers that require authentication, using Stripe as an example.
+
+### Approval Workflows (`promptfooconfig.approval.yaml`)
+
+Demonstrates different approval settings for MCP tool usage.
+
+## MCP Configuration
+
+### Basic Configuration
+
+```yaml
+tools:
+  - type: mcp
+    server_label: deepwiki
+    server_url: https://mcp.deepwiki.com/mcp
+    require_approval: never
+```
+
+### With Authentication
+
+```yaml
+tools:
+  - type: mcp
+    server_label: stripe
+    server_url: https://mcp.stripe.com
+    headers:
+      Authorization: 'Bearer ${STRIPE_API_KEY}'
+    require_approval: never
+```
+
+### Tool Filtering
+
+```yaml
+tools:
+  - type: mcp
+    server_label: deepwiki
+    server_url: https://mcp.deepwiki.com/mcp
+    allowed_tools: ['ask_question', 'read_wiki_structure']
+    require_approval: never
+```
+
+### Selective Approval
+
+```yaml
+tools:
+  - type: mcp
+    server_label: deepwiki
+    server_url: https://mcp.deepwiki.com/mcp
+    require_approval:
+      never:
+        tool_names: ['ask_question']
+```
+
+## Assertion Patterns
+
+The examples demonstrate assertion patterns for validating MCP tool interactions:
+
+### Enhanced OpenAI Tools Validation
+
+```yaml
+assert:
+  - type: is-valid-openai-tools-call # Validates both function and MCP tools
+```
+
+### MCP-Specific Content Validation
+
+```yaml
+assert:
+  - type: contains
+    value: 'MCP Tool Result' # Verify MCP tools were used
+  - type: not-contains
+    value: 'MCP Tool Error' # Ensure no MCP errors occurred
+```
+
+### Multi-layered Validation
+
+```yaml
+assert:
+  - type: is-valid-openai-tools-call
+    weight: 0.4
+  - type: contains-any
+    value: ['expected', 'content']
+    weight: 0.3
+  - type: llm-rubric
+    value: 'Response quality criteria'
+    weight: 0.3
+```
+
+## Learn More
+
+- [OpenAI MCP Documentation](https://platform.openai.com/docs/guides/mcp)
+- [Model Context Protocol Specification](https://modelcontextprotocol.io/)
+- [Promptfoo OpenAI Provider Documentation](/docs/providers/openai)
+- [MCP Server Registry](https://github.com/modelcontextprotocol/servers)
@@ -0,0 +1,95 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+
+description: MCP approval workflow examples
+
+prompts:
+  - 'Search for information about {{query}} in the {{repo}} repository'
+  - 'What are the latest updates in {{repo}}?'
+
+providers:
+  # Provider with approval required (default behavior)
+  - id: openai:responses:gpt-4.1-2025-04-14
+    label: 'With Approval Required'
+    config:
+      tools:
+        - type: mcp
+          server_label: deepwiki
+          server_url: https://mcp.deepwiki.com/mcp
+          # require_approval is not specified, so approval is required by default
+      max_output_tokens: 1000
+      temperature: 0.2
+      instructions: 'You are a research assistant. When approval is required for MCP tools, explain what you would do if the tool were approved.'
+
+  # Provider with selective approval
+  - id: openai:responses:gpt-4.1-2025-04-14
+    label: 'Selective Approval'
+    config:
+      tools:
+        - type: mcp
+          server_label: deepwiki
+          server_url: https://mcp.deepwiki.com/mcp
+          require_approval:
+            never:
+              tool_names: ['read_wiki_structure']
+          # ask_question will still require approval, but read_wiki_structure won't
+      max_output_tokens: 1000
+      temperature: 0.2
+      instructions: 'You are a research assistant. Use the available tools to gather information.'
+
+  # Provider with no approval required
+  - id: openai:responses:gpt-4.1-2025-04-14
+    label: 'No Approval Required'
+    config:
+      tools:
+        - type: mcp
+          server_label: deepwiki
+          server_url: https://mcp.deepwiki.com/mcp
+          require_approval: never
+      max_output_tokens: 1000
+      temperature: 0.2
+      instructions: 'You are a research assistant. Use the available MCP tools to search for information.'
+
+tests:
+  - vars:
+      query: 'getting started guide'
+      repo: 'facebook/react'
+    assert:
+      # For approval-required provider, expect either approval request or explanation
+      - type: contains-any
+        value: ['MCP Tool Result', 'react', 'React', 'would search', 'would look']
+      # Should not have tool errors
+      - type: not-contains
+        value: 'MCP Tool Error'
+      - type: llm-rubric
+        value: 'The response provides information about React or explains what would be searched'
+
+  - vars:
+      query: 'API documentation'
+      repo: 'openai/openai-python'
+    assert:
+      # Test different approval behaviors across providers
+      - type: contains-any
+        value: ['API', 'MCP Tool Result', 'Python', 'client']
+      # Validate that at least one provider succeeded with MCP tools
+      - type: contains-any
+        value: ['MCP Tool Result', 'documentation', 'API client', 'library']
+      - type: llm-rubric
+        value: 'The response mentions API documentation or explains the search intent'
+
+  # Add a test specifically for approval workflow validation
+  - vars:
+      query: 'installation instructions'
+      repo: 'nodejs/node'
+    assert:
+      # Comprehensive approval workflow testing
+      - type: contains-any
+        value: ['MCP Tool Result', 'installation', 'Node.js', 'setup']
+        metric: mcp_interaction_detected
+      # Ensure no unexpected errors
+      - type: not-contains
+        value: 'MCP Tool Error'
+        weight: 0.4
+      # Quality check regardless of approval status
+      - type: llm-rubric
+        value: 'The response addresses installation, setup, or Node.js information'
+        weight: 0.6
@@ -0,0 +1,82 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+
+description: Authenticated MCP server integration
+
+prompts:
+  - 'Please {{action}} for customer {{customer_id}} using the Stripe API.'
+  - 'Can you help me {{action}} for {{customer_id}}? Use the available tools to get the information.'
+  - 'I need to {{action}} for customer ID {{customer_id}}. Please use the Stripe integration to fetch this data.'
+
+providers:
+  - id: openai:responses:gpt-4.1-2025-04-14
+    config:
+      tools:
+        - type: mcp
+          server_label: stripe
+          server_url: https://mcp.stripe.com
+          headers:
+            Authorization: 'Bearer {{ env.STRIPE_API_KEY }}'
+          require_approval: never
+          allowed_tools: ['create_payment_link', 'list_payment_methods']
+      max_output_tokens: 1000
+      temperature: 0.1
+      instructions: 'You are a payment processing assistant. Use the Stripe MCP tools to help with payment-related tasks. Always provide clear and accurate information about payment operations.'
+
+tests:
+  - vars:
+      customer_id: 'cus_test123'
+      action: 'retrieve customer information'
+    assert:
+      # Validate successful MCP tool execution with authentication
+      - type: is-valid-openai-tools-call
+        weight: 0.4
+      # Check for Stripe-specific content
+      - type: contains-any
+        value: ['customer', 'Stripe', 'payment', 'subscription']
+        weight: 0.3
+      # Ensure authentication worked (no auth errors)
+      - type: not-contains
+        value: 'authentication failed'
+        weight: 0.1
+      # Verify MCP tool was used successfully
+      - type: contains
+        value: 'MCP Tool Result'
+        weight: 0.1
+      # Quality validation
+      - type: llm-rubric
+        value: 'The response provides customer information or payment details'
+
+  - vars:
+      customer_id: 'cus_test456'
+      action: 'list payment methods'
+    assert:
+      # Comprehensive authentication and API validation
+      - type: is-valid-openai-tools-call
+        metric: stripe_api_success
+      # Check for payment-related content
+      - type: contains-any
+        value: ['payment', 'method', 'card', 'bank']
+      # Ensure no MCP errors (including auth errors)
+      - type: not-contains-any
+        value: ['MCP Tool Error', 'unauthorized', 'authentication failed']
+      # Validate API response quality
+      - type: llm-rubric
+        value: 'The response lists payment methods or explains payment options'
+
+  - vars:
+      customer_id: 'cus_test789'
+      action: 'check subscription status'
+    assert:
+      # Test authenticated MCP with subscription data
+      - type: is-valid-openai-tools-call
+      - type: contains-any
+        value: ['subscription', 'plan', 'billing', 'active', 'status']
+      # Verify successful API integration
+      - type: contains
+        value: 'MCP Tool Result'
+        metric: api_integration_success
+        weight: 0
+      - type: llm-rubric
+        value: 'The response provides subscription status or billing information'
+# Note: This example requires a valid STRIPE_API_KEY environment variable
+# For testing purposes, you may want to use a test API key from Stripe
@@ -0,0 +1,90 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+
+description: OpenAI MCP tool integration
+
+prompts:
+  - 'What are the transport protocols supported in the MCP specification for {{repo}}?'
+  - 'Can you search for information about {{topic}} in the {{repo}} repository and summarize the key points?'
+  - 'What are the main features of {{repo}}? Please provide a detailed overview.'
+
+providers:
+  - id: openai:responses:gpt-4.1-2025-04-14
+    config:
+      tools:
+        - type: mcp
+          server_label: deepwiki
+          server_url: https://mcp.deepwiki.com/mcp
+          require_approval: never
+          allowed_tools: ['ask_question', 'read_wiki_structure']
+      max_output_tokens: 1500
+      temperature: 0.3
+      instructions: 'You are a helpful research assistant. Use the available MCP tools to search for accurate information about repositories and provide comprehensive answers.'
+
+tests:
+  - vars:
+      repo: modelcontextprotocol/modelcontextprotocol
+      topic: transport protocols
+    assert:
+      # Validate MCP tool execution was successful
+      - type: is-valid-openai-tools-call
+        weight: 0.3
+      # Check for specific content in the response
+      - type: contains
+        value: 'transport'
+        weight: 0.2
+      - type: contains
+        value: 'protocol'
+        weight: 0.2
+      # Ensure MCP tool was actually used (check for tool result)
+      - type: contains
+        value: 'MCP Tool Result'
+        weight: 0.1
+      # Validate the quality of the response
+      - type: llm-rubric
+        value: 'The response mentions transport protocols or MCP specification details'
+        weight: 0.2
+
+  - vars:
+      repo: facebook/react
+      topic: hooks
+    assert:
+      # Comprehensive MCP validation
+      - type: is-valid-openai-tools-call
+      - type: contains
+        value: 'React'
+      # Verify MCP integration worked
+      - type: contains
+        value: 'MCP Tool Result'
+      - type: llm-rubric
+        value: 'The response explains React functionality or features'
+
+  - vars:
+      repo: microsoft/typescript
+      topic: type system
+    assert:
+      # Test both success and content validation
+      - type: is-valid-openai-tools-call
+      - type: contains-any
+        value: ['TypeScript', 'type']
+      # Ensure no MCP errors occurred
+      - type: not-contains
+        value: 'MCP Tool Error'
+      - type: llm-rubric
+        value: 'The response describes TypeScript features or type system'
+
+  - vars:
+      repo: openai/openai-python
+      topic: API client
+    assert:
+      # Multi-layered validation approach
+      - type: is-valid-openai-tools-call
+        metric: mcp_tool_success
+      - type: contains-any
+        value: ['API', 'client', 'Python']
+      # Check that MCP tools were discovered and used
+      - type: contains
+        value: 'MCP Tool Result'
+        metric: mcp_tool_used
+        weight: 0
+      - type: llm-rubric
+        value: 'The response describes the OpenAI Python client library or API features'