chore: import upstream snapshot with attribution
CI / Shell Format Check (push) Has been cancelled
CI / Check Ruby (3.4) (push) Has been cancelled
CI / CI Config (push) Has been cancelled
CI / Test on Node ${{ matrix.node }} and ${{ matrix.os }}${{ matrix.shard && format(' (shard {0}/3)', matrix.shard) || '' }} (push) Has been cancelled
CI / Build on Node ${{ matrix.node }} (push) Has been cancelled
CI / Style Check (push) Has been cancelled
CI / Generate Assets (push) Has been cancelled
CI / Check Python (3.14) (push) Has been cancelled
CI / Check Python (3.9) (push) Has been cancelled
CI / Build Docs (push) Has been cancelled
CI / Code Scan Action (push) Has been cancelled
CI / Site tests (push) Has been cancelled
CI / webui tests (push) Has been cancelled
CI / Run Integration Tests (push) Has been cancelled
CI / Run Smoke Tests (push) Has been cancelled
CI / Go Tests (push) Has been cancelled
CI / Share Test (push) Has been cancelled
CI / Redteam (Production API) (push) Has been cancelled
CI / Redteam (Staging API) (push) Has been cancelled
CI / GitHub Actions Lint (push) Has been cancelled
CI / Check Ruby (3.0) (push) Has been cancelled
release-please / release-please (push) Has been cancelled
release-please / build (push) Has been cancelled
release-please / publish-npm (push) Has been cancelled
release-please / publish-npm-backfill (push) Has been cancelled
release-please / docker (push) Has been cancelled
release-please / publish-code-scan-action (push) Has been cancelled
release-please / attest-code-scan-action (push) Has been cancelled
Deploy local.promptfoo.app / Deploy to Cloudflare Pages (push) Has been cancelled
Test and Publish Multi-arch Docker Image / test (push) Has been cancelled
Test and Publish Multi-arch Docker Image / build-docker-and-push-digests (map[digest-suffix:linux-amd64 platform:linux/amd64 runner:ubuntu-latest]) (push) Has been cancelled
Test and Publish Multi-arch Docker Image / build-docker-and-push-digests (map[digest-suffix:linux-arm64 platform:linux/arm64 runner:ubuntu-24.04-arm]) (push) Has been cancelled
Test and Publish Multi-arch Docker Image / merge-docker-digests (push) Has been cancelled
Test and Publish Multi-arch Docker Image / Attest Multi-arch Image (push) Has been cancelled
Validate Renovate Config / Validate Renovate Configuration (push) Has been cancelled

This commit is contained in:
wehub-resource-sync
2026-07-13 13:24:08 +08:00
commit 0d3cb498a3
5438 changed files with 1316560 additions and 0 deletions
@@ -0,0 +1,104 @@
# provider-replicate/comprehensive (Comprehensive Replicate Testing)
You can run this example with:
```bash
npx promptfoo@latest init --example provider-replicate/comprehensive
cd provider-replicate/comprehensive
```
This example demonstrates the full capabilities of the Replicate provider in promptfoo, including text generation and image generation.
## Environment Variables
This example requires:
- `REPLICATE_API_TOKEN` - Your Replicate API key (get one at https://replicate.com/account/api-tokens)
Set this in your environment:
```bash
export REPLICATE_API_TOKEN=r8_your_api_token_here
```
## Features Demonstrated
### 1. Text Generation with Llama 3
- Uses Meta's Llama 3 8B Instruct model
- Configurable temperature, token limits, and sampling parameters
- Demonstrates creative writing and explanatory tasks
### 2. Image Generation with SDXL
- Uses Stable Diffusion XL for high-quality image generation
- Custom resolution settings (512x512 for faster generation)
- Tests both photorealistic and artistic styles
## Running the Example
1. Set your Replicate API token:
```bash
export REPLICATE_API_TOKEN=your_token_here
```
2. Run the evaluation:
```bash
promptfoo eval
```
3. View the results:
```bash
promptfoo view
```
## Understanding the Configuration
### Provider Configuration
```yaml
providers:
- id: replicate:meta/meta-llama-3-8b-instruct
config:
temperature: 0.5 # Balanced creativity
max_new_tokens: 200 # Limit response length
```
### Assertion Types
The example uses various assertion types:
- `contains-any`: Checks for specific keywords
- `javascript`: Custom validation logic
- `is-valid-openai-image-generation-response`: Validates image URLs
## Customization Ideas
1. **Try Different Models**:
- Text: `replicate:meta/meta-llama-3-70b-instruct` (larger, more capable)
- Images: `replicate:image:playgroundai/playground-v2.5-1024px-aesthetic`
2. **Adjust Parameters**:
- Increase `temperature` for more creative outputs
- Change image dimensions for different aspect ratios
- Modify `num_inference_steps` for quality vs speed tradeoff
3. **Add More Tests**:
- Code generation tasks
- Language translation
- Style transfer for images
- Different artistic styles
## Troubleshooting
- **Rate Limits**: Replicate has rate limits; add delays between tests if needed
- **Timeouts**: Large models may take 30-60 seconds; the provider handles polling automatically
- **Image URLs**: Generated images are temporary; save them if needed for long-term use
## Cost Considerations
- Text generation: ~$0.0005 per 1K tokens (Llama 3 8B)
- Image generation: ~$0.012 per image (SDXL)
- Check current pricing at https://replicate.com/pricing
@@ -0,0 +1,64 @@
# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
description: Comprehensive Replicate provider testing
prompts:
- '{{task}}'
providers:
# Text generation with Llama 4
- id: replicate:meta/llama-4-scout-instruct
config:
temperature: 0.5
max_new_tokens: 200
# Image generation with SDXL
- id: replicate:image:stability-ai/sdxl:7762fd07cf82c948538e41f63f77d685e02b063e37e496e96eefd46c929f9bdc
config:
width: 512
height: 512
num_inference_steps: 25
tests:
# Text generation tests
- vars:
task: 'Write a haiku about mountains'
assert:
- type: contains-any
value: ['mountain', 'peak', 'high', 'snow', 'summit']
- type: javascript
value: |
// Basic check for haiku-like structure
const lines = output.split('\n').filter(line => line.trim().length > 0);
return {
pass: lines.length >= 3,
reason: `Found ${lines.length} lines, expected at least 3 for haiku`
};
provider: replicate:meta/llama-4-scout-instruct
- vars:
task: 'Explain quantum computing in one paragraph'
assert:
- type: contains-any
value: ['quantum', 'qubit', 'superposition', 'entanglement']
- type: javascript
value: 'output.length > 100'
provider: replicate:meta/llama-4-scout-instruct
# Image generation test
- vars:
task: 'A serene mountain landscape at sunset with a lake reflection'
assert:
- type: javascript
value: |
// Check that the output contains markdown image format
return output && typeof output === 'string' && output.includes('![') && output.includes('](http');
provider: replicate:image:stability-ai/sdxl:7762fd07cf82c948538e41f63f77d685e02b063e37e496e96eefd46c929f9bdc
- vars:
task: 'Abstract geometric art with vibrant colors and sharp angles'
assert:
- type: javascript
value: |
// Check that the output contains markdown image format
return output && typeof output === 'string' && output.includes('![') && output.includes('](http');
provider: replicate:image:stability-ai/sdxl:7762fd07cf82c948538e41f63f77d685e02b063e37e496e96eefd46c929f9bdc