chore: import upstream snapshot with attribution
CI / Shell Format Check (push) Has been cancelled
CI / Check Ruby (3.4) (push) Has been cancelled
CI / CI Config (push) Has been cancelled
CI / Test on Node ${{ matrix.node }} and ${{ matrix.os }}${{ matrix.shard && format(' (shard {0}/3)', matrix.shard) || '' }} (push) Has been cancelled
CI / Build on Node ${{ matrix.node }} (push) Has been cancelled
CI / Style Check (push) Has been cancelled
CI / Generate Assets (push) Has been cancelled
CI / Check Python (3.14) (push) Has been cancelled
CI / Check Python (3.9) (push) Has been cancelled
CI / Build Docs (push) Has been cancelled
CI / Code Scan Action (push) Has been cancelled
CI / Site tests (push) Has been cancelled
CI / webui tests (push) Has been cancelled
CI / Run Integration Tests (push) Has been cancelled
CI / Run Smoke Tests (push) Has been cancelled
CI / Go Tests (push) Has been cancelled
CI / Share Test (push) Has been cancelled
CI / Redteam (Production API) (push) Has been cancelled
CI / Redteam (Staging API) (push) Has been cancelled
CI / GitHub Actions Lint (push) Has been cancelled
CI / Check Ruby (3.0) (push) Has been cancelled
release-please / release-please (push) Has been cancelled
release-please / build (push) Has been cancelled
release-please / publish-npm (push) Has been cancelled
release-please / publish-npm-backfill (push) Has been cancelled
release-please / docker (push) Has been cancelled
release-please / publish-code-scan-action (push) Has been cancelled
release-please / attest-code-scan-action (push) Has been cancelled
Deploy local.promptfoo.app / Deploy to Cloudflare Pages (push) Has been cancelled
Test and Publish Multi-arch Docker Image / test (push) Has been cancelled
Test and Publish Multi-arch Docker Image / build-docker-and-push-digests (map[digest-suffix:linux-amd64 platform:linux/amd64 runner:ubuntu-latest]) (push) Has been cancelled
Test and Publish Multi-arch Docker Image / build-docker-and-push-digests (map[digest-suffix:linux-arm64 platform:linux/arm64 runner:ubuntu-24.04-arm]) (push) Has been cancelled
Test and Publish Multi-arch Docker Image / merge-docker-digests (push) Has been cancelled
Test and Publish Multi-arch Docker Image / Attest Multi-arch Image (push) Has been cancelled
Validate Renovate Config / Validate Renovate Configuration (push) Has been cancelled
CI / Shell Format Check (push) Has been cancelled
CI / Check Ruby (3.4) (push) Has been cancelled
CI / CI Config (push) Has been cancelled
CI / Test on Node ${{ matrix.node }} and ${{ matrix.os }}${{ matrix.shard && format(' (shard {0}/3)', matrix.shard) || '' }} (push) Has been cancelled
CI / Build on Node ${{ matrix.node }} (push) Has been cancelled
CI / Style Check (push) Has been cancelled
CI / Generate Assets (push) Has been cancelled
CI / Check Python (3.14) (push) Has been cancelled
CI / Check Python (3.9) (push) Has been cancelled
CI / Build Docs (push) Has been cancelled
CI / Code Scan Action (push) Has been cancelled
CI / Site tests (push) Has been cancelled
CI / webui tests (push) Has been cancelled
CI / Run Integration Tests (push) Has been cancelled
CI / Run Smoke Tests (push) Has been cancelled
CI / Go Tests (push) Has been cancelled
CI / Share Test (push) Has been cancelled
CI / Redteam (Production API) (push) Has been cancelled
CI / Redteam (Staging API) (push) Has been cancelled
CI / GitHub Actions Lint (push) Has been cancelled
CI / Check Ruby (3.0) (push) Has been cancelled
release-please / release-please (push) Has been cancelled
release-please / build (push) Has been cancelled
release-please / publish-npm (push) Has been cancelled
release-please / publish-npm-backfill (push) Has been cancelled
release-please / docker (push) Has been cancelled
release-please / publish-code-scan-action (push) Has been cancelled
release-please / attest-code-scan-action (push) Has been cancelled
Deploy local.promptfoo.app / Deploy to Cloudflare Pages (push) Has been cancelled
Test and Publish Multi-arch Docker Image / test (push) Has been cancelled
Test and Publish Multi-arch Docker Image / build-docker-and-push-digests (map[digest-suffix:linux-amd64 platform:linux/amd64 runner:ubuntu-latest]) (push) Has been cancelled
Test and Publish Multi-arch Docker Image / build-docker-and-push-digests (map[digest-suffix:linux-arm64 platform:linux/arm64 runner:ubuntu-24.04-arm]) (push) Has been cancelled
Test and Publish Multi-arch Docker Image / merge-docker-digests (push) Has been cancelled
Test and Publish Multi-arch Docker Image / Attest Multi-arch Image (push) Has been cancelled
Validate Renovate Config / Validate Renovate Configuration (push) Has been cancelled
This commit is contained in:
@@ -0,0 +1,104 @@
|
||||
# provider-replicate/comprehensive (Comprehensive Replicate Testing)
|
||||
|
||||
You can run this example with:
|
||||
|
||||
```bash
|
||||
npx promptfoo@latest init --example provider-replicate/comprehensive
|
||||
cd provider-replicate/comprehensive
|
||||
```
|
||||
|
||||
This example demonstrates the full capabilities of the Replicate provider in promptfoo, including text generation and image generation.
|
||||
|
||||
## Environment Variables
|
||||
|
||||
This example requires:
|
||||
|
||||
- `REPLICATE_API_TOKEN` - Your Replicate API key (get one at https://replicate.com/account/api-tokens)
|
||||
|
||||
Set this in your environment:
|
||||
|
||||
```bash
|
||||
export REPLICATE_API_TOKEN=r8_your_api_token_here
|
||||
```
|
||||
|
||||
## Features Demonstrated
|
||||
|
||||
### 1. Text Generation with Llama 3
|
||||
|
||||
- Uses Meta's Llama 3 8B Instruct model
|
||||
- Configurable temperature, token limits, and sampling parameters
|
||||
- Demonstrates creative writing and explanatory tasks
|
||||
|
||||
### 2. Image Generation with SDXL
|
||||
|
||||
- Uses Stable Diffusion XL for high-quality image generation
|
||||
- Custom resolution settings (512x512 for faster generation)
|
||||
- Tests both photorealistic and artistic styles
|
||||
|
||||
## Running the Example
|
||||
|
||||
1. Set your Replicate API token:
|
||||
|
||||
```bash
|
||||
export REPLICATE_API_TOKEN=your_token_here
|
||||
```
|
||||
|
||||
2. Run the evaluation:
|
||||
|
||||
```bash
|
||||
promptfoo eval
|
||||
```
|
||||
|
||||
3. View the results:
|
||||
```bash
|
||||
promptfoo view
|
||||
```
|
||||
|
||||
## Understanding the Configuration
|
||||
|
||||
### Provider Configuration
|
||||
|
||||
```yaml
|
||||
providers:
|
||||
- id: replicate:meta/meta-llama-3-8b-instruct
|
||||
config:
|
||||
temperature: 0.5 # Balanced creativity
|
||||
max_new_tokens: 200 # Limit response length
|
||||
```
|
||||
|
||||
### Assertion Types
|
||||
|
||||
The example uses various assertion types:
|
||||
|
||||
- `contains-any`: Checks for specific keywords
|
||||
- `javascript`: Custom validation logic
|
||||
- `is-valid-openai-image-generation-response`: Validates image URLs
|
||||
|
||||
## Customization Ideas
|
||||
|
||||
1. **Try Different Models**:
|
||||
- Text: `replicate:meta/meta-llama-3-70b-instruct` (larger, more capable)
|
||||
- Images: `replicate:image:playgroundai/playground-v2.5-1024px-aesthetic`
|
||||
|
||||
2. **Adjust Parameters**:
|
||||
- Increase `temperature` for more creative outputs
|
||||
- Change image dimensions for different aspect ratios
|
||||
- Modify `num_inference_steps` for quality vs speed tradeoff
|
||||
|
||||
3. **Add More Tests**:
|
||||
- Code generation tasks
|
||||
- Language translation
|
||||
- Style transfer for images
|
||||
- Different artistic styles
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
- **Rate Limits**: Replicate has rate limits; add delays between tests if needed
|
||||
- **Timeouts**: Large models may take 30-60 seconds; the provider handles polling automatically
|
||||
- **Image URLs**: Generated images are temporary; save them if needed for long-term use
|
||||
|
||||
## Cost Considerations
|
||||
|
||||
- Text generation: ~$0.0005 per 1K tokens (Llama 3 8B)
|
||||
- Image generation: ~$0.012 per image (SDXL)
|
||||
- Check current pricing at https://replicate.com/pricing
|
||||
@@ -0,0 +1,64 @@
|
||||
# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
|
||||
description: Comprehensive Replicate provider testing
|
||||
|
||||
prompts:
|
||||
- '{{task}}'
|
||||
|
||||
providers:
|
||||
# Text generation with Llama 4
|
||||
- id: replicate:meta/llama-4-scout-instruct
|
||||
config:
|
||||
temperature: 0.5
|
||||
max_new_tokens: 200
|
||||
|
||||
# Image generation with SDXL
|
||||
- id: replicate:image:stability-ai/sdxl:7762fd07cf82c948538e41f63f77d685e02b063e37e496e96eefd46c929f9bdc
|
||||
config:
|
||||
width: 512
|
||||
height: 512
|
||||
num_inference_steps: 25
|
||||
|
||||
tests:
|
||||
# Text generation tests
|
||||
- vars:
|
||||
task: 'Write a haiku about mountains'
|
||||
assert:
|
||||
- type: contains-any
|
||||
value: ['mountain', 'peak', 'high', 'snow', 'summit']
|
||||
- type: javascript
|
||||
value: |
|
||||
// Basic check for haiku-like structure
|
||||
const lines = output.split('\n').filter(line => line.trim().length > 0);
|
||||
return {
|
||||
pass: lines.length >= 3,
|
||||
reason: `Found ${lines.length} lines, expected at least 3 for haiku`
|
||||
};
|
||||
provider: replicate:meta/llama-4-scout-instruct
|
||||
|
||||
- vars:
|
||||
task: 'Explain quantum computing in one paragraph'
|
||||
assert:
|
||||
- type: contains-any
|
||||
value: ['quantum', 'qubit', 'superposition', 'entanglement']
|
||||
- type: javascript
|
||||
value: 'output.length > 100'
|
||||
provider: replicate:meta/llama-4-scout-instruct
|
||||
|
||||
# Image generation test
|
||||
- vars:
|
||||
task: 'A serene mountain landscape at sunset with a lake reflection'
|
||||
assert:
|
||||
- type: javascript
|
||||
value: |
|
||||
// Check that the output contains markdown image format
|
||||
return output && typeof output === 'string' && output.includes(';
|
||||
provider: replicate:image:stability-ai/sdxl:7762fd07cf82c948538e41f63f77d685e02b063e37e496e96eefd46c929f9bdc
|
||||
|
||||
- vars:
|
||||
task: 'Abstract geometric art with vibrant colors and sharp angles'
|
||||
assert:
|
||||
- type: javascript
|
||||
value: |
|
||||
// Check that the output contains markdown image format
|
||||
return output && typeof output === 'string' && output.includes(';
|
||||
provider: replicate:image:stability-ai/sdxl:7762fd07cf82c948538e41f63f77d685e02b063e37e496e96eefd46c929f9bdc
|
||||
Reference in New Issue
Block a user