chore: import upstream snapshot with attribution

2026-07-13 13:24:08 +08:00
commit 0d3cb498a3
5438 changed files with 1316560 additions and 0 deletions
@@ -0,0 +1,8 @@
+# integration-vercel (Vercel)
+
+Examples for using promptfoo with Vercel AI services.
+
+## Examples
+
+- [ai-sdk](./ai-sdk/) - Vercel AI SDK integration
+- [ai-gateway](./ai-gateway/) - Vercel AI Gateway integration
@@ -0,0 +1,43 @@
+# integration-vercel/ai-gateway (Vercel AI Gateway Example)
+
+This example demonstrates how to use [Vercel AI Gateway](https://vercel.com/docs/ai-sdk/ai-gateway) to access multiple AI providers through a unified API.
+
+## Prerequisites
+
+1. A Vercel account with AI Gateway enabled
+2. Your Vercel AI Gateway API key
+
+## Setup
+
+Set the required environment variable:
+
+```bash
+export VERCEL_AI_GATEWAY_API_KEY=your_api_key
+```
+
+## Running the Example
+
+```bash
+npx promptfoo@latest init --example integration-vercel/ai-gateway
+npx promptfoo eval
+```
+
+Or run directly:
+
+```bash
+npx promptfoo eval -c examples/integration-vercel/ai-gateway/promptfooconfig.yaml
+```
+
+## What This Example Does
+
+The configuration compares responses from three different providers, all accessed through Vercel AI Gateway:
+
+- **OpenAI** (gpt-4o-mini)
+- **Anthropic** (claude-haiku-4.5)
+- **Google** (gemini-2.5-flash)
+
+Each provider answers questions about technical topics, and the assertions verify that responses contain relevant keywords.
+
+## Documentation
+
+See the [Vercel AI Gateway provider documentation](https://www.promptfoo.dev/docs/providers/vercel) for more details.
@@ -0,0 +1,66 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+# Vercel AI Gateway Example
+#
+# This example demonstrates using Vercel AI Gateway to access multiple
+# AI providers through a unified API with 0% markup.
+#
+# Required environment variables:
+#   VERCEL_AI_GATEWAY_API_KEY - Your Vercel AI Gateway API key
+#
+# Run with:
+#   npx promptfoo eval -c examples/integration-vercel/ai-gateway/promptfooconfig.yaml
+
+description: Vercel AI Gateway multi-provider comparison
+
+prompts:
+  - 'Explain {{topic}} in simple terms, in about 2-3 sentences.'
+
+providers:
+  # OpenAI via Vercel AI Gateway
+  - id: vercel:openai/gpt-4o-mini
+    config:
+      temperature: 0.7
+      maxTokens: 200
+
+  # Anthropic via Vercel AI Gateway
+  - id: vercel:anthropic/claude-haiku-4.5
+    config:
+      temperature: 0.7
+      maxTokens: 200
+
+  # Google via Vercel AI Gateway
+  - id: vercel:google/gemini-2.5-flash
+    config:
+      temperature: 0.7
+      maxTokens: 200
+
+tests:
+  - vars:
+      topic: machine learning
+    assert:
+      - type: contains-any
+        value:
+          - algorithm
+          - data
+          - learn
+          - pattern
+
+  - vars:
+      topic: quantum computing
+    assert:
+      - type: contains-any
+        value:
+          - qubit
+          - quantum
+          - superposition
+          - computer
+
+  - vars:
+      topic: blockchain
+    assert:
+      - type: contains-any
+        value:
+          - block
+          - chain
+          - ledger
+          - decentralized
@@ -0,0 +1,124 @@
+# integration-vercel/ai-sdk (Vercel AI SDK Provider)
+
+Demonstrates dynamic prompt construction using the [Vercel AI SDK](https://ai-sdk.dev) with promptfoo's provider prompt reporting feature.
+
+## Why This Matters
+
+Modern LLM applications dynamically construct prompts with:
+
+- **System instructions** tailored to the task
+- **Few-shot examples** selected at runtime
+- **Retrieved context** from RAG pipelines
+- **User preferences** and safety guardrails
+
+Without prompt reporting, promptfoo shows `{{topic}}` as the prompt, making it impossible to debug what was actually sent or run assertions on the real prompt content.
+
+## How It Works
+
+The provider reports the actual prompt it sent using the `prompt` field:
+
+```javascript
+return {
+  output: result.text,
+  prompt: [
+    { role: 'system', content: dynamicSystemPrompt },
+    { role: 'user', content: dynamicUserPrompt },
+  ],
+};
+```
+
+## Features Demonstrated
+
+| Feature               | Description                                                    |
+| --------------------- | -------------------------------------------------------------- |
+| **Multiple personas** | `expert`, `coder`, `analyst` with different system prompts     |
+| **Task types**        | `explain`, `compare`, `troubleshoot` with different structures |
+| **Context injection** | RAG-style context added to prompts                             |
+| **Template filling**  | Variables like `{{domain}}`, `{{audience}}` filled dynamically |
+
+## Running the Example
+
+```bash
+npx promptfoo@latest init --example integration-vercel/ai-sdk
+cd integration-vercel/ai-sdk
+npm install
+export OPENAI_API_KEY=sk-...
+npx promptfoo@latest eval
+npx promptfoo@latest view
+```
+
+## What You'll See
+
+In the promptfoo UI, click any result to see **"Actual Prompt Sent"** showing the full dynamically-constructed prompt instead of just `{{topic}}`.
+
+**Input:**
+
+```yaml
+vars:
+  topic: quantum entanglement
+  persona: expert
+  domain: quantum physics
+  audience: college students
+```
+
+**Actual Prompt Sent:**
+
+```text
+System: You are a world-class expert in quantum physics.
+
+Your communication style:
+- Clear and precise explanations
+- Use analogies for complex concepts
+- Include concrete examples
+- Acknowledge limitations honestly
+
+Your audience: college students
+
+User: Explain quantum entanglement in a way that's accessible and engaging.
+
+Focus on:
+1. Core concepts and why they matter
+2. Real-world applications
+3. Common misconceptions to avoid
+```
+
+## Files
+
+| File                   | Description                                                   |
+| ---------------------- | ------------------------------------------------------------- |
+| `aiSdkProvider.mjs`    | Provider using Vercel AI SDK with dynamic prompt construction |
+| `promptfooconfig.yaml` | Test cases showcasing different personas and task types       |
+| `package.json`         | Dependencies (`ai`, `@ai-sdk/openai`)                         |
+
+## Adapting for Your Use Case
+
+The pattern works with any framework:
+
+```javascript
+// LangChain
+const chain = prompt.pipe(model);
+const result = await chain.invoke(input);
+return {
+  output: result,
+  prompt: prompt.format(input),
+};
+
+// Custom RAG
+const context = await retrieveContext(query);
+const fullPrompt = `Context: ${context}\n\nQuestion: ${query}`;
+const result = await llm.generate(fullPrompt);
+return {
+  output: result,
+  prompt: fullPrompt,
+};
+```
+
+## Learn More
+
+- [Vercel AI SDK Documentation](https://ai-sdk.dev)
+- [promptfoo Custom Providers](https://promptfoo.dev/docs/providers/custom-api)
+- [Promptfoo tracing and trajectory assertions](https://promptfoo.dev/docs/tracing/)
+
+## Tool Telemetry
+
+If you enable Vercel AI SDK `experimental_telemetry` for tool-calling workflows, Promptfoo trajectory assertions can normalize the SDK's tool-call spans from `ai.toolCall.name` plus the matching `ai.toolCall.args`, `ai.toolCall.arguments`, or `ai.toolCall.input` attributes.
@@ -0,0 +1,192 @@
+/**
+ * Vercel AI SDK Provider with Dynamic Prompt Reporting
+ *
+ * This provider demonstrates a real-world pattern: dynamically constructing
+ * prompts based on context, then reporting the actual prompt back to promptfoo.
+ *
+ * Why this matters:
+ * - Without prompt reporting, promptfoo shows "{{topic}}" as the prompt
+ * - With prompt reporting, you see the full system instructions and context
+ * - This enables prompt-based assertions and debugging
+ *
+ * The Vercel AI SDK (https://ai-sdk.dev) is the TypeScript toolkit for AI apps
+ * with 20M+ monthly downloads, supporting OpenAI, Anthropic, Google, and more.
+ *
+ * Usage:
+ *   npm install ai @ai-sdk/openai
+ *   OPENAI_API_KEY=sk-... npx promptfoo@latest eval
+ */
+import { openai } from '@ai-sdk/openai';
+import { generateText } from 'ai';
+
+// =============================================================================
+// PROMPT TEMPLATES - These simulate what frameworks like LangChain do
+// =============================================================================
+
+const SYSTEM_TEMPLATES = {
+  expert: `You are a world-class expert in {{domain}}.
+
+Your communication style:
+- Clear and precise explanations
+- Use analogies for complex concepts
+- Include concrete examples
+- Acknowledge limitations honestly
+
+Your audience: {{audience}}`,
+
+  coder: `You are an expert software engineer specializing in {{domain}}.
+
+Guidelines:
+- Write clean, idiomatic code
+- Explain your reasoning
+- Consider edge cases
+- Suggest best practices
+
+Target experience level: {{audience}}`,
+
+  analyst: `You are a data analyst specializing in {{domain}}.
+
+Your approach:
+- Ground claims in evidence
+- Quantify when possible
+- Consider multiple perspectives
+- Identify key insights
+
+Report format: {{format}}`,
+};
+
+const USER_TEMPLATES = {
+  explain: `Explain {{topic}} in a way that's accessible and engaging.
+
+Focus on:
+1. Core concepts and why they matter
+2. Real-world applications
+3. Common misconceptions to avoid`,
+
+  compare: `Compare and contrast: {{topic}}
+
+Structure your response as:
+1. Key similarities
+2. Key differences
+3. When to use each
+4. Recommendations`,
+
+  troubleshoot: `Help troubleshoot: {{topic}}
+
+Provide:
+1. Common causes
+2. Diagnostic steps
+3. Solutions ranked by likelihood
+4. Prevention strategies`,
+};
+
+// =============================================================================
+// DYNAMIC PROMPT BUILDER - The core of this example
+// =============================================================================
+
+function buildPrompt(rawPrompt, vars) {
+  // Determine the best template based on the task
+  const taskType = vars.task_type || 'explain';
+  const persona = vars.persona || 'expert';
+
+  // Get templates
+  const systemTemplate = SYSTEM_TEMPLATES[persona] || SYSTEM_TEMPLATES.expert;
+  const userTemplate = USER_TEMPLATES[taskType] || USER_TEMPLATES.explain;
+
+  // Fill in template variables
+  const fillTemplate = (template, variables) => {
+    return template.replace(/\{\{(\w+)\}\}/g, (match, key) => {
+      return variables[key] || match;
+    });
+  };
+
+  const templateVars = {
+    topic: rawPrompt,
+    domain: vars.domain || 'the requested topic',
+    audience: vars.audience || 'general audience',
+    format: vars.format || 'clear prose',
+    ...vars,
+  };
+
+  const systemPrompt = fillTemplate(systemTemplate, templateVars);
+  const userPrompt = fillTemplate(userTemplate, templateVars);
+
+  // Add any retrieved context (simulating RAG)
+  let contextAddition = '';
+  if (vars.context) {
+    contextAddition = `\n\nRelevant context:\n${vars.context}`;
+  }
+
+  // Add few-shot examples if provided
+  let examplesAddition = '';
+  if (vars.examples) {
+    examplesAddition = `\n\nExamples for reference:\n${vars.examples}`;
+  }
+
+  return {
+    system: systemPrompt,
+    user: userPrompt + contextAddition + examplesAddition,
+  };
+}
+
+// =============================================================================
+// PROVIDER CLASS
+// =============================================================================
+
+export default class AiSdkProvider {
+  constructor(options = {}) {
+    this.config = options.config || {};
+    this.modelId = this.config.model || 'gpt-4o-mini';
+    this.temperature = this.config.temperature ?? 0.7;
+  }
+
+  id() {
+    return `ai-sdk:${this.modelId}`;
+  }
+
+  async callApi(prompt, context) {
+    const vars = context.vars || {};
+
+    // Build the dynamic prompt
+    const { system, user } = buildPrompt(prompt, vars);
+
+    // Construct the messages array (what we'll actually send)
+    const messages = [
+      { role: 'system', content: system },
+      { role: 'user', content: user },
+    ];
+
+    try {
+      // Call the LLM using Vercel AI SDK
+      const result = await generateText({
+        model: openai(this.modelId),
+        messages,
+        temperature: this.temperature,
+        maxOutputTokens: this.config.maxOutputTokens,
+      });
+
+      return {
+        output: result.text,
+
+        // THE KEY FEATURE: Report what prompt was actually sent
+        // This enables:
+        // 1. UI shows "Actual Prompt Sent" instead of the template
+        // 2. Assertions can check the real prompt content
+        // 3. Moderation runs on the actual prompt
+        prompt: messages,
+
+        tokenUsage: {
+          prompt: result.usage?.inputTokens,
+          completion: result.usage?.outputTokens,
+          total: result.usage?.totalTokens,
+        },
+      };
+    } catch (error) {
+      return {
+        error: `AI SDK error: ${error.message}`,
+        // Still report the prompt even on error - useful for debugging
+        prompt: messages,
+      };
+    }
+  }
+}
@@ -0,0 +1,11 @@
+{
+  "name": "vercel-ai-sdk",
+  "version": "1.0.0",
+  "license": "MIT",
+  "private": true,
+  "type": "module",
+  "dependencies": {
+    "ai": "^6.0.190",
+    "@ai-sdk/openai": "^3.0.41"
+  }
+}
@@ -0,0 +1,100 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+#
+# Vercel AI SDK Example
+#
+# Demonstrates dynamic prompt construction with the Vercel AI SDK.
+# The provider builds prompts based on persona, task type, and context,
+# then reports the actual prompt sent to the LLM.
+#
+# Setup:
+#   cd examples/integration-vercel/ai-sdk
+#   npm install
+#   export OPENAI_API_KEY=sk-...
+#
+# Run:
+#   npx promptfoo@latest eval
+
+description: Vercel AI SDK with dynamic prompt construction
+
+providers:
+  - id: file://./aiSdkProvider.mjs
+    config:
+      model: gpt-4o-mini
+      temperature: 0.7
+
+prompts:
+  - '{{topic}}'
+
+tests:
+  # Expert persona with explain task
+  - description: Quantum computing for students
+    vars:
+      topic: quantum entanglement
+      persona: expert
+      task_type: explain
+      domain: quantum physics
+      audience: college students
+    assert:
+      - type: llm-rubric
+        value: explains quantum entanglement clearly with appropriate examples
+
+  # Coder persona with explain task
+  - description: Async/await for junior devs
+    vars:
+      topic: async/await patterns in JavaScript
+      persona: coder
+      task_type: explain
+      domain: JavaScript
+      audience: junior developers
+    assert:
+      - type: icontains
+        value: async
+      - type: icontains
+        value: await
+
+  # Analyst persona with comparison task
+  - description: Compare SQL vs NoSQL
+    vars:
+      topic: SQL databases vs NoSQL databases
+      persona: analyst
+      task_type: compare
+      domain: database systems
+      format: structured comparison with pros/cons
+    assert:
+      - type: icontains
+        value: SQL
+      - type: llm-rubric
+        value: provides balanced comparison of both database types
+
+  # Expert with troubleshooting task
+  - description: Debug memory leaks
+    vars:
+      topic: memory leaks in Node.js applications
+      persona: coder
+      task_type: troubleshoot
+      domain: Node.js performance
+      audience: senior engineers
+    assert:
+      - type: icontains
+        value: memory
+      - type: llm-rubric
+        value: provides actionable debugging steps
+
+  # With RAG-style context injection
+  - description: Explain with context (RAG simulation)
+    vars:
+      topic: transformer architecture
+      persona: expert
+      task_type: explain
+      domain: machine learning
+      audience: ML engineers
+      context: |
+        From "Attention Is All You Need" (2017):
+        The Transformer uses self-attention to compute representations
+        of its input and output without using sequence-aligned RNNs or
+        convolution.
+    assert:
+      - type: icontains
+        value: attention
+      - type: llm-rubric
+        value: references or builds upon the provided context