chore: import upstream snapshot with attribution

2026-07-13 13:24:08 +08:00
commit 0d3cb498a3
5438 changed files with 1316560 additions and 0 deletions
@@ -0,0 +1,111 @@
+# integration-langfuse (Langfuse Prompt Management)
+
+This example demonstrates how to use Langfuse prompt management with labels in promptfoo.
+
+## Why use labels?
+
+Labels allow you to:
+
+- Deploy new prompt versions without changing your code
+- Use different prompts for different environments (production, staging, development)
+- A/B test different prompt versions
+- Roll back to previous versions quickly
+
+## Setup
+
+1. **Install dependencies:**
+
+   ```bash
+   npm install langfuse
+   ```
+
+2. **Set environment variables:**
+
+   ```bash
+   export LANGFUSE_PUBLIC_KEY="your-public-key"
+   export LANGFUSE_SECRET_KEY="your-secret-key"
+   export LANGFUSE_HOST="https://cloud.langfuse.com"
+   ```
+
+3. **Create prompts in Langfuse:**
+   - Log into your Langfuse dashboard
+   - Create prompts named `customer-support` and `email-writer`
+   - Add prompt templates with variables like `{{customer_name}}`, `{{issue}}`, etc.
+   - Assign labels to your prompt versions (e.g., `production`, `staging`, `latest`)
+
+## Initialize this example
+
+To get started with this example:
+
+```bash
+npx promptfoo@latest init --example integration-langfuse
+cd integration-langfuse
+```
+
+## Running the example
+
+```bash
+npx promptfoo@latest eval
+```
+
+## Expected outputs
+
+After running the eval, you should see:
+
+- Comparison of responses from different labeled prompts (production vs staging)
+- Performance metrics for each prompt version
+- Test results showing how different providers handle the same prompts
+- A detailed evaluation report demonstrating label-based prompt management
+
+## Prompt format reference
+
+### Two syntax options
+
+Promptfoo supports two ways to reference Langfuse prompts with labels:
+
+1. **Explicit @ syntax** (recommended for clarity)
+
+   ```yaml
+   langfuse://prompt-name@label:type
+   ```
+
+2. **Auto-detection with : syntax**
+
+   ```yaml
+   langfuse://prompt-name:version-or-label:type
+   ```
+
+   - Numeric values (e.g., `1`, `2`, `3`) are treated as versions
+   - String values (e.g., `production`, `staging`) are treated as labels
+
+### Format components
+
+- `prompt-name`: The name of your prompt in Langfuse
+- `version`: Specific version number (1, 2, 3, etc.)
+- `label`: Label assigned to a prompt version
+- `type`: Either `text` or `chat` (defaults to `text`)
+
+## Example prompt references
+
+```yaml
+# Explicit @ syntax for labels (recommended)
+langfuse://customer-support@production:chat
+langfuse://customer-support@staging:chat
+langfuse://email-writer@latest
+
+# Auto-detection with : syntax
+langfuse://customer-support:production:chat  # String → label
+langfuse://customer-support:staging          # String → label
+langfuse://email-writer:latest:text          # "latest" → label
+
+# Version references (numeric values)
+langfuse://customer-support:1:chat           # Numeric → version
+langfuse://email-writer:3:text               # Numeric → version
+```
+
+## Best practices
+
+1. **Use labels for production** - Avoid hardcoding version numbers
+2. **Test in staging first** - Use a `staging` label to test changes
+3. **Use descriptive labels** - `production`, `staging`, `experiment-a`, `tenant-xyz`
+4. **Monitor performance** - Track metrics for different prompt versions
@@ -0,0 +1,61 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+
+description: 'Langfuse prompt management with labels - demonstrates version-controlled prompt deployment'
+
+# Example: Using Langfuse prompts with labels
+#
+# This example demonstrates how to reference Langfuse prompts using labels
+# instead of version numbers. This allows you to update prompts in Langfuse
+# without changing your promptfoo configuration.
+#
+# Prerequisites:
+# 1. Set up Langfuse environment variables:
+#    - LANGFUSE_PUBLIC_KEY
+#    - LANGFUSE_SECRET_KEY
+#    - LANGFUSE_HOST
+# 2. Create prompts in Langfuse with the names used below
+# 3. Assign labels to your prompt versions (e.g., "production", "staging")
+
+prompts:
+  # Reference prompts by label using @ syntax (explicit)
+  - langfuse://customer-support@production:chat
+  - langfuse://customer-support@staging:chat
+
+  # Reference prompts by label using : syntax (auto-detected)
+  - langfuse://customer-support:production:chat # String detected as label
+  - langfuse://customer-support:staging:chat # String detected as label
+
+  # You can still use version numbers (numeric values)
+  - langfuse://customer-support:1:chat # Numeric detected as version
+
+  # Text prompts with labels (both syntaxes work)
+  - langfuse://email-writer@production # @ syntax
+  - langfuse://email-writer:latest:text # : syntax (latest treated as label)
+  - langfuse://email-writer:production # : syntax (auto-detected as label)
+
+providers:
+  - openai:gpt-4o
+  - openai:gpt-4.1-mini
+
+tests:
+  - vars:
+      customer_name: 'Alice Johnson'
+      issue: "I can't log into my account"
+      company: 'Acme Corp'
+      tone: 'friendly and professional'
+
+  - vars:
+      customer_name: 'Bob Smith'
+      issue: "My order hasn't arrived yet"
+      company: 'Acme Corp'
+      tone: 'empathetic and helpful'
+
+# Default assertions that apply to all tests
+defaultTest:
+  assert:
+    - type: contains
+      value: '{{customer_name}}'
+    - type: not-contains
+      value: 'error'
+    - type: llm-rubric
+      value: "The response should be {{tone}} and address the customer's issue: {{issue}}"