chore: import upstream snapshot with attribution

examples/eval-python-assert/README.md

@@ -0,0 +1,111 @@
+# eval-python-assert (Python Assertions)
+
+Example configurations for testing LLM outputs using Python assertions with promptfoo.
+
+You can run this example with:
+
+```bash
+npx promptfoo@latest init --example eval-python-assert
+cd eval-python-assert
+```
+
+## Purpose
+
+This example demonstrates how to use Python assertions for custom output validation with:
+
+- External Python files with assertion functions
+- Inline Python code directly in configuration files
+- Configuration-based assertions with custom parameters
+- Different assertion return formats (boolean, score, detailed results)
+
+## Prerequisites
+
+- Python 3.7+ installed and available in your PATH
+- OpenAI API key (or other LLM provider)
+
+## Environment Variables
+
+- `OPENAI_API_KEY` - Your OpenAI API key (required)
+
+## Configurations
+
+This example includes two different approaches:
+
+### External Python Files (`promptfooconfig-external.yaml`)
+
+Uses external Python files for complex assertion logic:
+
+- `promptfooconfig-external.yaml` - Configuration with external Python assertions
+- `assert.py` - Basic assertion function with detailed scoring
+- `assert_with_config.py` - Configuration-based assertion function
+
+### Inline Python Code (`promptfooconfig-inline.yaml`)
+
+Demonstrates inline Python assertions directly in the configuration:
+
+- `promptfooconfig-inline.yaml` - Configuration with inline Python code
+- Shows simple boolean checks and complex scoring logic
+
+## Running the Examples
+
+1. **External Python assertions example:**
+
+   ```sh
+   promptfoo eval -c promptfooconfig-external.yaml
+   ```
+
+2. **Inline Python assertions example:**
+
+   ```sh
+   promptfoo eval -c promptfooconfig-inline.yaml
+   ```
+
+3. **View results:**
+
+   ```sh
+   promptfoo view
+   ```
+
+## Python Assertion Patterns
+
+### Basic Boolean Return
+
+```python
+def get_assert(output, context):
+    return "expected_word" in output.lower()
+```
+
+### Score-Based Return
+
+```python
+def get_assert(output, context):
+    if "perfect" in output.lower():
+        return 1.0
+    elif "good" in output.lower():
+        return 0.5
+    else:
+        return 0.0
+```
+
+### Detailed Result Object
+
+```python
+def get_assert(output, context):
+    return {
+        "pass": True,
+        "score": 0.8,
+        "reason": "Contains expected content",
+        "namedScores": {"quality": 0.9, "relevance": 0.7}
+    }
+```
+
+## Expected Results
+
+- **External example**: Shows advanced assertion patterns with detailed scoring and configuration support
+- **Inline example**: Demonstrates quick assertions and simple validation logic
+
+## Learn More
+
+- [Python Assertions Documentation](https://www.promptfoo.dev/docs/configuration/expected-outputs/python/)
+- [promptfoo Configuration Guide](https://www.promptfoo.dev/docs/configuration/guide/)
+- [Assertion Types](https://www.promptfoo.dev/docs/configuration/expected-outputs/)

examples/eval-python-assert/assert.py

@@ -0,0 +1,33 @@
+def get_assert(output, context):
+    """
+    Custom function that grades an LLM output.
+    """
+
+    # You can return a bool, number, or dict
+    if "banana" not in output.lower():
+        return False
+
+    if "yellow" not in output.lower():
+        return 0.5
+
+    # Snake_case field names are automatically converted to camelCase
+    return {
+        "pass_": True,
+        "score": 0.75,
+        "reason": "Good banana content",
+        "named_scores": {"banana_quality": 0.8, "color_accuracy": 0.7},
+        "component_results": [
+            {
+                "pass_": "bananas" in output.lower(),
+                "score": 0.5,
+                "reason": "Contains banana",
+                "named_scores": {"banana_mentions": 1.0},
+            },
+            {
+                "pass_": "yellow" in output.lower(),
+                "score": 0.5,
+                "reason": "Contains yellow",
+                "named_scores": {"color_mentions": 0.66},
+            },
+        ],
+    }

examples/eval-python-assert/assert_with_config.py

@@ -0,0 +1,16 @@
+def get_assert(output, context):
+    print("Prompt:", context["prompt"])
+    print("Vars", context["vars"]["topic"])
+    print("Context", context)
+    print("Config", context.get("config", {}))
+
+    test_configuration = context.get("config", {})
+    canonical_fruit_list = test_configuration.get("fruitList", [])
+    assert_passed = False
+
+    for fruit in canonical_fruit_list:
+        if fruit in output.lower():
+            assert_passed = True
+            break
+
+    return assert_passed

examples/eval-python-assert/promptfooconfig-external.yaml

@@ -0,0 +1,48 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: External Python assertion files for output validation
+
+prompts:
+  - 'Tell me about {{topic}} in three words'
+providers:
+  - openai:gpt-4.1-mini
+
+tests:
+  - vars:
+      topic: yellow fruits
+    assert:
+      - type: python
+        # Optionally specify a function name. Defaults to `get_assert` if not provided.
+        value: file://assert.py:get_assert
+  - vars:
+      topic: fruits high in potassium
+    assert:
+      - type: python
+        value: file://assert.py
+  - vars:
+      topic: fruits that are long and skinny
+    assert:
+      - type: python
+        value: file://assert.py
+  - vars:
+      topic: fruits that smell bad
+    assert:
+      - type: python
+        value: file://assert.py
+  - vars:
+      topic: fruits that originated in south america such as passion fruit, banana, pineapple and guava
+    assert:
+      - type: python
+        value: file://assert_with_config.py
+        config:
+          fruitList:
+            - passion fruit
+            - pineapple
+            - guava
+            - papaya
+            - açaí
+            - strawberry
+      - type: python
+        value: file://assert_with_config.py
+        config:
+          fruitList:
+            - banana

examples/eval-python-assert/promptfooconfig-inline.yaml

@@ -0,0 +1,34 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: Inline Python assertions for output validation
+
+prompts:
+  - 'Write a tweet about {{topic}}'
+
+providers:
+  - openai:gpt-4.1-mini
+
+tests:
+  - vars:
+      topic: bananas
+    assert:
+      - type: python
+        value: "context['vars']['topic'] in output"
+
+  - vars:
+      topic: potatoes
+    assert:
+      - type: python
+        value: |
+          # Insert your scoring logic here...
+          if output == 'Expected output':
+              return {
+                'pass': True,
+                'score': 0.5,
+                'reason': 'Looks good to me',
+              }
+          else:
+              return {
+                'pass': False,
+                'score': 0,
+                'reason': 'Did not contain expected output',
+              }