chore: import upstream snapshot with attribution
CI / Shell Format Check (push) Has been cancelled
CI / Check Ruby (3.4) (push) Has been cancelled
CI / CI Config (push) Has been cancelled
CI / Test on Node ${{ matrix.node }} and ${{ matrix.os }}${{ matrix.shard && format(' (shard {0}/3)', matrix.shard) || '' }} (push) Has been cancelled
CI / Build on Node ${{ matrix.node }} (push) Has been cancelled
CI / Style Check (push) Has been cancelled
CI / Generate Assets (push) Has been cancelled
CI / Check Python (3.14) (push) Has been cancelled
CI / Check Python (3.9) (push) Has been cancelled
CI / Build Docs (push) Has been cancelled
CI / Code Scan Action (push) Has been cancelled
CI / Site tests (push) Has been cancelled
CI / webui tests (push) Has been cancelled
CI / Run Integration Tests (push) Has been cancelled
CI / Run Smoke Tests (push) Has been cancelled
CI / Go Tests (push) Has been cancelled
CI / Share Test (push) Has been cancelled
CI / Redteam (Production API) (push) Has been cancelled
CI / Redteam (Staging API) (push) Has been cancelled
CI / GitHub Actions Lint (push) Has been cancelled
CI / Check Ruby (3.0) (push) Has been cancelled
release-please / release-please (push) Has been cancelled
release-please / build (push) Has been cancelled
release-please / publish-npm (push) Has been cancelled
release-please / publish-npm-backfill (push) Has been cancelled
release-please / docker (push) Has been cancelled
release-please / publish-code-scan-action (push) Has been cancelled
release-please / attest-code-scan-action (push) Has been cancelled
Deploy local.promptfoo.app / Deploy to Cloudflare Pages (push) Has been cancelled
Test and Publish Multi-arch Docker Image / test (push) Has been cancelled
Test and Publish Multi-arch Docker Image / build-docker-and-push-digests (map[digest-suffix:linux-amd64 platform:linux/amd64 runner:ubuntu-latest]) (push) Has been cancelled
Test and Publish Multi-arch Docker Image / build-docker-and-push-digests (map[digest-suffix:linux-arm64 platform:linux/arm64 runner:ubuntu-24.04-arm]) (push) Has been cancelled
Test and Publish Multi-arch Docker Image / merge-docker-digests (push) Has been cancelled
Test and Publish Multi-arch Docker Image / Attest Multi-arch Image (push) Has been cancelled
Validate Renovate Config / Validate Renovate Configuration (push) Has been cancelled
CI / Shell Format Check (push) Has been cancelled
CI / Check Ruby (3.4) (push) Has been cancelled
CI / CI Config (push) Has been cancelled
CI / Test on Node ${{ matrix.node }} and ${{ matrix.os }}${{ matrix.shard && format(' (shard {0}/3)', matrix.shard) || '' }} (push) Has been cancelled
CI / Build on Node ${{ matrix.node }} (push) Has been cancelled
CI / Style Check (push) Has been cancelled
CI / Generate Assets (push) Has been cancelled
CI / Check Python (3.14) (push) Has been cancelled
CI / Check Python (3.9) (push) Has been cancelled
CI / Build Docs (push) Has been cancelled
CI / Code Scan Action (push) Has been cancelled
CI / Site tests (push) Has been cancelled
CI / webui tests (push) Has been cancelled
CI / Run Integration Tests (push) Has been cancelled
CI / Run Smoke Tests (push) Has been cancelled
CI / Go Tests (push) Has been cancelled
CI / Share Test (push) Has been cancelled
CI / Redteam (Production API) (push) Has been cancelled
CI / Redteam (Staging API) (push) Has been cancelled
CI / GitHub Actions Lint (push) Has been cancelled
CI / Check Ruby (3.0) (push) Has been cancelled
release-please / release-please (push) Has been cancelled
release-please / build (push) Has been cancelled
release-please / publish-npm (push) Has been cancelled
release-please / publish-npm-backfill (push) Has been cancelled
release-please / docker (push) Has been cancelled
release-please / publish-code-scan-action (push) Has been cancelled
release-please / attest-code-scan-action (push) Has been cancelled
Deploy local.promptfoo.app / Deploy to Cloudflare Pages (push) Has been cancelled
Test and Publish Multi-arch Docker Image / test (push) Has been cancelled
Test and Publish Multi-arch Docker Image / build-docker-and-push-digests (map[digest-suffix:linux-amd64 platform:linux/amd64 runner:ubuntu-latest]) (push) Has been cancelled
Test and Publish Multi-arch Docker Image / build-docker-and-push-digests (map[digest-suffix:linux-arm64 platform:linux/arm64 runner:ubuntu-24.04-arm]) (push) Has been cancelled
Test and Publish Multi-arch Docker Image / merge-docker-digests (push) Has been cancelled
Test and Publish Multi-arch Docker Image / Attest Multi-arch Image (push) Has been cancelled
Validate Renovate Config / Validate Renovate Configuration (push) Has been cancelled
This commit is contained in:
@@ -0,0 +1,111 @@
|
||||
# eval-python-assert (Python Assertions)
|
||||
|
||||
Example configurations for testing LLM outputs using Python assertions with promptfoo.
|
||||
|
||||
You can run this example with:
|
||||
|
||||
```bash
|
||||
npx promptfoo@latest init --example eval-python-assert
|
||||
cd eval-python-assert
|
||||
```
|
||||
|
||||
## Purpose
|
||||
|
||||
This example demonstrates how to use Python assertions for custom output validation with:
|
||||
|
||||
- External Python files with assertion functions
|
||||
- Inline Python code directly in configuration files
|
||||
- Configuration-based assertions with custom parameters
|
||||
- Different assertion return formats (boolean, score, detailed results)
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Python 3.7+ installed and available in your PATH
|
||||
- OpenAI API key (or other LLM provider)
|
||||
|
||||
## Environment Variables
|
||||
|
||||
- `OPENAI_API_KEY` - Your OpenAI API key (required)
|
||||
|
||||
## Configurations
|
||||
|
||||
This example includes two different approaches:
|
||||
|
||||
### External Python Files (`promptfooconfig-external.yaml`)
|
||||
|
||||
Uses external Python files for complex assertion logic:
|
||||
|
||||
- `promptfooconfig-external.yaml` - Configuration with external Python assertions
|
||||
- `assert.py` - Basic assertion function with detailed scoring
|
||||
- `assert_with_config.py` - Configuration-based assertion function
|
||||
|
||||
### Inline Python Code (`promptfooconfig-inline.yaml`)
|
||||
|
||||
Demonstrates inline Python assertions directly in the configuration:
|
||||
|
||||
- `promptfooconfig-inline.yaml` - Configuration with inline Python code
|
||||
- Shows simple boolean checks and complex scoring logic
|
||||
|
||||
## Running the Examples
|
||||
|
||||
1. **External Python assertions example:**
|
||||
|
||||
```sh
|
||||
promptfoo eval -c promptfooconfig-external.yaml
|
||||
```
|
||||
|
||||
2. **Inline Python assertions example:**
|
||||
|
||||
```sh
|
||||
promptfoo eval -c promptfooconfig-inline.yaml
|
||||
```
|
||||
|
||||
3. **View results:**
|
||||
|
||||
```sh
|
||||
promptfoo view
|
||||
```
|
||||
|
||||
## Python Assertion Patterns
|
||||
|
||||
### Basic Boolean Return
|
||||
|
||||
```python
|
||||
def get_assert(output, context):
|
||||
return "expected_word" in output.lower()
|
||||
```
|
||||
|
||||
### Score-Based Return
|
||||
|
||||
```python
|
||||
def get_assert(output, context):
|
||||
if "perfect" in output.lower():
|
||||
return 1.0
|
||||
elif "good" in output.lower():
|
||||
return 0.5
|
||||
else:
|
||||
return 0.0
|
||||
```
|
||||
|
||||
### Detailed Result Object
|
||||
|
||||
```python
|
||||
def get_assert(output, context):
|
||||
return {
|
||||
"pass": True,
|
||||
"score": 0.8,
|
||||
"reason": "Contains expected content",
|
||||
"namedScores": {"quality": 0.9, "relevance": 0.7}
|
||||
}
|
||||
```
|
||||
|
||||
## Expected Results
|
||||
|
||||
- **External example**: Shows advanced assertion patterns with detailed scoring and configuration support
|
||||
- **Inline example**: Demonstrates quick assertions and simple validation logic
|
||||
|
||||
## Learn More
|
||||
|
||||
- [Python Assertions Documentation](https://www.promptfoo.dev/docs/configuration/expected-outputs/python/)
|
||||
- [promptfoo Configuration Guide](https://www.promptfoo.dev/docs/configuration/guide/)
|
||||
- [Assertion Types](https://www.promptfoo.dev/docs/configuration/expected-outputs/)
|
||||
Reference in New Issue
Block a user