chore: import upstream snapshot with attribution

examples/openai-deep-research/README.md

@@ -0,0 +1,160 @@
+# openai-deep-research (OpenAI Deep Research Models)
+
+You can run this example with:
+
+```bash
+npx promptfoo@latest init --example openai-deep-research
+cd openai-deep-research
+```
+
+This example demonstrates OpenAI's deep research models with web search capabilities via the Responses API.
+
+## Important Notes
+
+⚠️ **Response Times**: Deep research models can take **2-10 minutes** to complete research tasks as they perform extensive web searches and reasoning.
+
+⚠️ **Token Usage**: These models use significant tokens for internal reasoning. Always set high `max_output_tokens` (50,000+) to avoid incomplete responses.
+
+⚠️ **Access**: Deep research models may require special access from OpenAI. Check your API access if you encounter persistent 429 errors.
+
+## Setup
+
+1. Set your OpenAI API key:
+
+```bash
+export OPENAI_API_KEY=your-key-here
+```
+
+2. Run the evaluation with appropriate timeout:
+
+```bash
+# Set a 10-minute timeout for deep research tasks
+export PROMPTFOO_EVAL_TIMEOUT_MS=600000
+promptfoo eval
+```
+
+For local development:
+
+```bash
+PROMPTFOO_EVAL_TIMEOUT_MS=600000 npm run local -- eval -c examples/openai-deep-research/promptfooconfig.yaml
+```
+
+## What's happening?
+
+This example:
+
+- Tests OpenAI's `o4-mini-deep-research` model with web search tools
+- Evaluates research capabilities on machine learning and space exploration topics
+- Uses the model's ability to automatically search the web for current information
+- Checks that responses contain relevant technical terminology
+- Demonstrates handling of web search results and citations
+
+The model automatically decides when to use web search to provide comprehensive, up-to-date answers.
+
+## Configuration Details
+
+```yaml
+providers:
+  - id: openai:responses:o4-mini-deep-research
+    config:
+      max_output_tokens: 50000 # Required for complete research responses
+      tools:
+        - type: web_search_preview # Required for deep research models
+      # Optional parameters:
+      # max_tool_calls: 50 # Control number of searches (default: unlimited)
+      # background: true # Use background mode for long-running tasks
+      # store: true # Store the conversation for 30 days
+```
+
+## Available Models
+
+- `o3-deep-research` - Most powerful deep research model ($10/1M input, $40/1M output)
+- `o3-deep-research-2025-06-26` - Snapshot version
+- `o4-mini-deep-research` - Faster, more affordable ($2/1M input, $8/1M output)
+- `o4-mini-deep-research-2025-06-26` - Snapshot version
+
+## Advanced Features
+
+### Background Mode (Recommended)
+
+For production use, run deep research tasks in background mode to avoid timeouts:
+
+```yaml
+providers:
+  - id: openai:responses:o4-mini-deep-research
+    config:
+      background: true
+      webhook_url: https://your-api.com/webhook # Optional: Get notified when complete
+```
+
+### Using Code Interpreter
+
+Deep research models can analyze data using code:
+
+```yaml
+providers:
+  - id: openai:responses:o4-mini-deep-research
+    config:
+      tools:
+        - type: web_search_preview
+        - type: code_interpreter
+          container:
+            type: auto
+```
+
+### MCP Server Integration
+
+Connect to private data sources using MCP servers:
+
+```yaml
+providers:
+  - id: openai:responses:o4-mini-deep-research
+    config:
+      tools:
+        - type: web_search_preview
+        - type: mcp
+          server_label: mycompany_mcp
+          server_url: https://mycompany.com/mcp
+          require_approval: never # Required for deep research
+```
+
+### Prompt Enhancement
+
+For better results, consider preprocessing user queries:
+
+1. **Clarification**: Use a faster model to gather context
+2. **Prompt rewriting**: Expand the query with specific requirements
+3. **Deep research**: Pass the enhanced prompt to the research model
+
+See the [OpenAI Deep Research Guide](https://platform.openai.com/docs/guides/deep-research) for detailed examples.
+
+## Response Format
+
+Deep research responses include:
+
+- **output_text**: The final research report with inline citations
+- **annotations**: Citation details with URLs and titles
+- **web_search_call**: Details of searches performed
+- **code_interpreter_call**: Any code analysis performed
+
+## Troubleshooting
+
+- **Timeouts**: Increase `PROMPTFOO_EVAL_TIMEOUT_MS` if evaluations time out
+- **Incomplete responses**: Increase `max_output_tokens` to 50,000 or higher
+- **429 errors**: May indicate rate limits or access restrictions
+- **Tool validation errors**: Ensure `web_search_preview` is configured
+
+## Best Practices
+
+1. **Always use high token limits**: Set `max_output_tokens: 50000` or higher
+2. **Handle long response times**: Use background mode or set high timeouts
+3. **Monitor costs**: These models use significant tokens for reasoning
+4. **Validate citations**: Check that returned URLs are accessible
+5. **Consider prompt enhancement**: Preprocess queries for better results
+
+## Learn More
+
+- [OpenAI Deep Research Guide](https://platform.openai.com/docs/guides/deep-research)
+- [Promptfoo Documentation](https://promptfoo.dev/docs)
+- [MCP Integration Guide](https://platform.openai.com/docs/mcp)
+- [Building a Deep Research Compatible MCP Server](mcp-server-example.md)

examples/openai-deep-research/example-config.yaml

@@ -0,0 +1,50 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: Comprehensive test of deep research features
+prompts:
+  - |
+    Research and provide a brief answer to: {{query}}
+    Limit your research to 2-3 sources for efficiency.
+
+providers:
+  # Test with limited searches
+  - id: openai:responses:o4-mini-deep-research
+    label: Quick Research (2 searches)
+    config:
+      max_output_tokens: 5000
+      max_tool_calls: 2
+      tools:
+        - type: web_search_preview
+
+  # Test with more searches
+  - id: openai:responses:o4-mini-deep-research
+    label: Detailed Research (5 searches)
+    config:
+      max_output_tokens: 10000
+      max_tool_calls: 5
+      tools:
+        - type: web_search_preview
+
+tests:
+  - vars:
+      query: What is the current stable version of React.js?
+    assert:
+      - type: contains-any
+        value: [React, version, '18', '19', stable]
+      - type: contains
+        value: Web Search
+
+  - vars:
+      query: What are the main features of TypeScript 5.0?
+    assert:
+      - type: contains-any
+        value: [TypeScript, '5.0', features, decorators]
+      - type: contains
+        value: Web Search
+
+  - vars:
+      query: What is the latest Python version and its release date?
+    assert:
+      - type: contains-any
+        value: [Python, version, '3.1', '3.2', release]
+      - type: contains
+        value: Web Search

examples/openai-deep-research/mcp-server-example.md

@@ -0,0 +1,367 @@
+# Building a Deep Research Compatible MCP Server
+
+Deep research models require MCP servers that implement a specific search and fetch interface. This guide shows you how to build a compatible server.
+
+## Required Interface
+
+Your MCP server must provide exactly two tools:
+
+1. **search** - Searches your data and returns results
+2. **fetch** - Retrieves full content for a specific document
+
+## Example Implementation
+
+Here's a minimal Express.js server that implements the required interface:
+
+```javascript
+const express = require('express');
+const app = express();
+app.use(express.json());
+
+// Sample data store
+const documents = {
+  doc1: {
+    id: 'doc1',
+    title: 'Q1 Sales Report',
+    content: 'Total sales for Q1 were $1.2M, up 15% from last year...',
+    metadata: { department: 'sales', date: '2025-03-31' },
+  },
+  doc2: {
+    id: 'doc2',
+    title: 'Product Roadmap 2025',
+    content: 'Key features planned: AI integration, mobile app redesign...',
+    metadata: { department: 'product', date: '2025-01-15' },
+  },
+};
+
+// MCP endpoint
+app.post('/mcp', async (req, res) => {
+  const { method, params } = req.body;
+
+  // List available tools
+  if (method === 'tools/list') {
+    return res.json({
+      tools: [
+        {
+          name: 'search',
+          description: 'Search internal documents',
+          inputSchema: {
+            type: 'object',
+            properties: {
+              query: { type: 'string', description: 'Search query' },
+              limit: { type: 'number', description: 'Max results', default: 10 },
+            },
+            required: ['query'],
+          },
+        },
+        {
+          name: 'fetch',
+          description: 'Fetch document by ID',
+          inputSchema: {
+            type: 'object',
+            properties: {
+              id: { type: 'string', description: 'Document ID' },
+            },
+            required: ['id'],
+          },
+        },
+      ],
+    });
+  }
+
+  // Handle tool calls
+  if (method === 'tools/call') {
+    const { name, arguments: args } = params;
+
+    if (name === 'search') {
+      // Simple search implementation
+      const query = args.query.toLowerCase();
+      const results = Object.values(documents)
+        .filter(
+          (doc) =>
+            doc.title.toLowerCase().includes(query) || doc.content.toLowerCase().includes(query),
+        )
+        .slice(0, args.limit || 10)
+        .map((doc) => ({
+          id: doc.id,
+          title: doc.title,
+          snippet: doc.content.substring(0, 100) + '...',
+          metadata: doc.metadata,
+        }));
+
+      return res.json({
+        content: [
+          {
+            type: 'text',
+            text: JSON.stringify({ results }, null, 2),
+          },
+        ],
+      });
+    }
+
+    if (name === 'fetch') {
+      const doc = documents[args.id];
+      if (!doc) {
+        return res.json({
+          content: [
+            {
+              type: 'text',
+              text: 'Document not found',
+            },
+          ],
+          isError: true,
+        });
+      }
+
+      return res.json({
+        content: [
+          {
+            type: 'text',
+            text: JSON.stringify(doc, null, 2),
+          },
+        ],
+      });
+    }
+
+    return res.status(400).json({ error: 'Unknown tool' });
+  }
+
+  return res.status(400).json({ error: 'Unknown method' });
+});
+
+app.listen(3000, () => {
+  console.log('MCP server running on http://localhost:3000');
+});
+```
+
+## Python Example with FastAPI
+
+```python
+from fastapi import FastAPI, HTTPException
+from pydantic import BaseModel
+from typing import Dict, List, Any, Optional
+import json
+
+app = FastAPI()
+
+# Sample data store
+documents = {
+    "doc1": {
+        "id": "doc1",
+        "title": "Q1 Sales Report",
+        "content": "Total sales for Q1 were $1.2M, up 15% from last year...",
+        "metadata": {"department": "sales", "date": "2025-03-31"}
+    },
+    "doc2": {
+        "id": "doc2",
+        "title": "Product Roadmap 2025",
+        "content": "Key features planned: AI integration, mobile app redesign...",
+        "metadata": {"department": "product", "date": "2025-01-15"}
+    }
+}
+
+class MCPRequest(BaseModel):
+    method: str
+    params: Optional[Dict[str, Any]] = None
+
+@app.post("/mcp")
+async def mcp_endpoint(request: MCPRequest):
+    if request.method == "tools/list":
+        return {
+            "tools": [
+                {
+                    "name": "search",
+                    "description": "Search internal documents",
+                    "inputSchema": {
+                        "type": "object",
+                        "properties": {
+                            "query": {"type": "string", "description": "Search query"},
+                            "limit": {"type": "number", "description": "Max results", "default": 10}
+                        },
+                        "required": ["query"]
+                    }
+                },
+                {
+                    "name": "fetch",
+                    "description": "Fetch document by ID",
+                    "inputSchema": {
+                        "type": "object",
+                        "properties": {
+                            "id": {"type": "string", "description": "Document ID"}
+                        },
+                        "required": ["id"]
+                    }
+                }
+            ]
+        }
+
+    elif request.method == "tools/call":
+        tool_name = request.params.get("name")
+        args = request.params.get("arguments", {})
+
+        if tool_name == "search":
+            query = args.get("query", "").lower()
+            limit = args.get("limit", 10)
+
+            results = []
+            for doc in documents.values():
+                if query in doc["title"].lower() or query in doc["content"].lower():
+                    results.append({
+                        "id": doc["id"],
+                        "title": doc["title"],
+                        "snippet": doc["content"][:100] + "...",
+                        "metadata": doc["metadata"]
+                    })
+
+            return {
+                "content": [
+                    {
+                        "type": "text",
+                        "text": json.dumps({"results": results[:limit]}, indent=2)
+                    }
+                ]
+            }
+
+        elif tool_name == "fetch":
+            doc_id = args.get("id")
+            doc = documents.get(doc_id)
+
+            if not doc:
+                return {
+                    "content": [
+                        {
+                            "type": "text",
+                            "text": "Document not found"
+                        }
+                    ],
+                    "isError": True
+                }
+
+            return {
+                "content": [
+                    {
+                        "type": "text",
+                        "text": json.dumps(doc, indent=2)
+                    }
+                ]
+            }
+
+        raise HTTPException(status_code=400, detail="Unknown tool")
+
+    raise HTTPException(status_code=400, detail="Unknown method")
+
+if __name__ == "__main__":
+    import uvicorn
+    uvicorn.run(app, host="0.0.0.0", port=3000)
+```
+
+## Integration with Deep Research
+
+Configure your deep research model to use your MCP server:
+
+```yaml
+providers:
+  - id: openai:responses:o3-deep-research
+    config:
+      max_output_tokens: 100000
+      tools:
+        - type: web_search_preview # Required
+        - type: mcp
+          server_label: internal_docs
+          server_url: http://localhost:3000/mcp
+          require_approval: never # Required for deep research
+          headers:
+            Authorization: Bearer your-api-key
+```
+
+## Best Practices
+
+1. **Efficient Search**: Implement proper indexing (e.g., Elasticsearch, PostgreSQL full-text search)
+2. **Result Ranking**: Return most relevant results first
+3. **Metadata**: Include useful metadata in search results
+4. **Error Handling**: Return clear error messages for debugging
+5. **Authentication**: Secure your MCP server with API keys or OAuth
+6. **Rate Limiting**: Implement rate limits to prevent abuse
+7. **Logging**: Log all requests for debugging and monitoring
+
+## Testing Your MCP Server
+
+Test your server using curl:
+
+```bash
+# List tools
+curl -X POST http://localhost:3000/mcp \
+  -H "Content-Type: application/json" \
+  -d '{"method": "tools/list"}'
+
+# Search
+curl -X POST http://localhost:3000/mcp \
+  -H "Content-Type: application/json" \
+  -d '{
+    "method": "tools/call",
+    "params": {
+      "name": "search",
+      "arguments": {"query": "sales"}
+    }
+  }'
+
+# Fetch
+curl -X POST http://localhost:3000/mcp \
+  -H "Content-Type: application/json" \
+  -d '{
+    "method": "tools/call",
+    "params": {
+      "name": "fetch",
+      "arguments": {"id": "doc1"}
+    }
+  }'
+```
+
+## Advanced Features
+
+### Connecting to Real Data Sources
+
+Replace the sample data with connections to your actual systems:
+
+```javascript
+// Example: Connect to PostgreSQL
+const { Pool } = require('pg');
+const pool = new Pool({
+  connectionString: process.env.DATABASE_URL,
+});
+
+// In your search handler
+if (name === 'search') {
+  const result = await pool.query(
+    'SELECT id, title, content FROM documents WHERE to_tsvector(content) @@ plainto_tsquery($1) LIMIT $2',
+    [args.query, args.limit || 10],
+  );
+  // Format and return results...
+}
+```
+
+### Adding Filters
+
+Enhance search with filters:
+
+```javascript
+{
+  name: 'search',
+  inputSchema: {
+    type: 'object',
+    properties: {
+      query: { type: 'string' },
+      filters: {
+        type: 'object',
+        properties: {
+          department: { type: 'string' },
+          dateFrom: { type: 'string', format: 'date' },
+          dateTo: { type: 'string', format: 'date' }
+        }
+      }
+    }
+  }
+}
+```
+
+Remember: Deep research models will automatically use your search and fetch tools to gather information needed to answer user queries. The better your search implementation, the better the research results will be.

examples/openai-deep-research/promptfooconfig.yaml

@@ -0,0 +1,21 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: OpenAI Deep Research API
+prompts:
+  - 'Research and provide a comprehensive answer about {{topic}}'
+providers:
+  - id: openai:responses:o4-mini-deep-research
+    config:
+      max_output_tokens: 50000 # Deep research models need high token limits
+      tools:
+        - type: web_search_preview
+tests:
+  - vars:
+      topic: recent advancements in machine learning
+    assert:
+      - type: contains-any
+        value: [machine learning, ML, neural networks, deep learning]
+  - vars:
+      topic: current space exploration missions
+    assert:
+      - type: contains-any
+        value: [space, NASA, SpaceX, Mars, moon]