chore: import upstream snapshot with attribution

examples/eval-conversation-relevance/README.md

@@ -0,0 +1,86 @@
+# eval-conversation-relevance (Conversation Relevance)
+
+You can run this example with:
+
+```bash
+npx promptfoo@latest init --example eval-conversation-relevance
+cd eval-conversation-relevance
+```
+
+This example demonstrates how to use the `conversation-relevance` assertion to evaluate whether chatbot responses remain relevant throughout a conversation.
+
+## What is Conversation Relevance?
+
+The conversation relevance metric evaluates whether each response in a conversation is relevant to the context and previous messages. It uses a sliding window approach to analyze conversation segments.
+
+## Running the Example
+
+1. Install dependencies:
+
+   ```bash
+   npm install -g promptfoo
+   ```
+
+2. Set your OpenAI API key:
+
+   ```bash
+   export OPENAI_API_KEY=your-api-key
+   ```
+
+3. Run the evaluation:
+   ```bash
+   promptfoo eval
+   ```
+
+## Example Test Cases
+
+### 1. Single-turn Evaluation
+
+Tests basic relevance for a single query-response pair about travel to Paris.
+
+### 2. Multi-turn Travel Conversation
+
+Evaluates a complete conversation about travel planning where all responses should be relevant.
+
+### 3. Conversation with Irrelevant Response
+
+Demonstrates detection of an off-topic response (stock market comment) in the middle of a conversation about wedding planning.
+
+### 4. Technical Support Conversation
+
+Shows a high-quality technical support conversation with a high relevance threshold (0.95).
+
+## Configuration Options
+
+- `threshold`: Minimum score required to pass (0-1)
+- `config.windowSize`: Number of messages in each sliding window (default: 5)
+- `provider`: Override the default grading model
+
+## Interpreting Results
+
+- **Score**: Proportion of conversation windows deemed relevant
+- **Pass/Fail**: Based on whether the score meets the threshold
+- **Reason**: Explanation when responses are found irrelevant
+
+## Tips
+
+1. Use lower thresholds (0.7-0.8) for general conversations
+2. Use higher thresholds (0.9-0.95) for specialized domains like technical support
+3. Adjust window size based on conversation complexity
+4. Consider using more capable models (GPT-4) for grading complex conversations
+
+## How Scoring Works
+
+The metric evaluates each message position using a sliding window approach. For example, with a 5-message conversation and window size of 3:
+
+- Window 1: Message 1 only (evaluates if Response 1 is relevant)
+- Window 2: Messages 1-2 (evaluates if Response 2 is relevant given context)
+- Window 3: Messages 1-3 (evaluates if Response 3 is relevant given context)
+- Window 4: Messages 2-4 (evaluates if Response 4 is relevant given context)
+- Window 5: Messages 3-5 (evaluates if Response 5 is relevant given context)
+
+Each window evaluates whether the LAST assistant response in that window is relevant. The final score is:
+
+```text
+Score = Number of Relevant Windows / Total Number of Windows
+```

examples/eval-conversation-relevance/promptfooconfig.yaml

@@ -0,0 +1,90 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+# Example configuration for testing conversation relevance
+# This metric evaluates whether chatbot responses remain relevant throughout a conversation
+
+description: 'Conversation Relevance Example'
+
+prompts:
+  - |
+    You are a helpful travel assistant specializing in European destinations.
+    Respond to the user's query: {{message}}
+
+providers:
+  - openai:gpt-4o
+  - openai:gpt-4.1-mini
+
+# Example with single turn evaluation
+tests:
+  - description: 'Single-turn travel question'
+    vars:
+      message: "What's the best time to visit Paris?"
+    assert:
+      - type: conversation-relevance
+        threshold: 0.9
+
+  - description: 'Multi-turn travel conversation'
+    vars:
+      _conversation:
+        - input: "I'm planning a trip to Europe"
+          output: 'That sounds exciting! Which countries are you thinking of visiting?'
+        - input: 'France and Italy'
+          output: 'Great choices! France and Italy offer amazing culture, food, and history. When are you planning to travel?'
+        - input: 'Next summer, probably July'
+          output: 'July is peak season with warm weather but also crowds. Consider booking accommodations early.'
+        - input: 'What about the weather?'
+          output: 'In July, expect warm to hot weather. France averages 20-25°C (68-77°F), Italy can be hotter at 25-30°C (77-86°F).'
+    assert:
+      - type: conversation-relevance
+        threshold: 0.9
+        config:
+          windowSize: 3
+
+  - description: 'Conversation with irrelevant response'
+    vars:
+      _conversation:
+        - input: 'I need help planning my wedding'
+          output: "Congratulations! I'd be happy to help plan your wedding. What's your vision?"
+        - input: 'We want an outdoor ceremony'
+          output: 'Outdoor ceremonies can be beautiful! Consider the season and have a backup plan for weather.'
+        - input: 'What about flowers?'
+          output: 'The stock market has been volatile lately with tech stocks leading the decline.' # Irrelevant
+        - input: 'I meant wedding flowers'
+          output: 'Oh, for wedding flowers, popular choices include roses, peonies, and hydrangeas. Consider seasonal blooms for better prices.'
+    assert:
+      - type: conversation-relevance
+        threshold: 0.7 # Lower threshold as we expect one irrelevant response
+
+  - description: 'Technical support conversation'
+    vars:
+      _conversation:
+        - input: "My computer won't start"
+          output: "I'm sorry to hear that. Let's troubleshoot. What happens when you press the power button?"
+        - input: 'Nothing, no lights or sounds'
+          output: 'This suggests a power issue. First, check if the power cable is properly connected to both the computer and wall outlet.'
+        - input: "I checked, it's plugged in"
+          output: 'Try a different power outlet to rule out outlet issues. Also, check if the power supply switch (usually on the back) is in the ON position.'
+        - input: 'Still nothing'
+          output: "The power supply might have failed. If it's a laptop, try removing the battery and running on AC power only. For desktops, you may need a technician to test the PSU."
+    assert:
+      - type: conversation-relevance
+        threshold: 0.95 # High threshold for technical support
+        config:
+          windowSize: 4
+
+# Example with custom provider for grading
+defaultTest:
+  options:
+    provider: openai:gpt-4o # Use GPT-4o for more accurate relevance assessment
+
+
+# You can also test with different configurations
+# tests:
+#   - description: 'Test with smaller window size'
+#     vars:
+#       _conversation: [...]
+#     assert:
+#       - type: conversation-relevance
+#         threshold: 0.8
+#         config:
+#           windowSize: 2  # Only look at 2 messages at a time
+