chore: import upstream snapshot with attribution

examples/provider-replicate/image-generation/.gitignore

@@ -0,0 +1,6 @@
+# Downloaded images directory
+images/
+
+# OS files
+.DS_Store
+Thumbs.db 

examples/provider-replicate/image-generation/README.md

@@ -0,0 +1,149 @@
+# provider-replicate/image-generation (State-of-the-Art Image Generation)
+
+You can run this example with:
+
+```bash
+npx promptfoo@latest init --example provider-replicate/image-generation
+cd provider-replicate/image-generation
+```
+
+This example demonstrates state-of-the-art image generation using Replicate's latest models, particularly the FLUX family from Black Forest Labs.
+
+## Features
+
+This example tests:
+
+- **FLUX 1.1 Pro Ultra** - The highest quality model supporting up to 4MP images
+- **FLUX 1.1 Pro Ultra (Raw Mode)** - For authentic, photorealistic results
+- **FLUX Dev** - Open-source version for commercial use
+- **FLUX Dev Realism** - Specialized for photorealistic outputs
+- **Stable Diffusion XL** - For comparison with previous generation
+
+## Environment Setup
+
+1. Get a Replicate API token from https://replicate.com/account/api-tokens
+2. Set the environment variable:
+   ```bash
+   export REPLICATE_API_TOKEN=r8_your_token_here
+   ```
+
+## Running the Example
+
+### Full Test Suite (8 different image types across 5 models = 40 images)
+
+```bash
+promptfoo eval
+```
+
+### Quick Test (2 images with the best model)
+
+```bash
+promptfoo eval --filter-providers flux-1.1-pro-ultra --max-concurrency 1
+```
+
+### Test Specific Image Types
+
+```bash
+# Test only portraits
+promptfoo eval --filter-tests "Photorealistic portrait"
+
+# Test only landscapes and architecture
+promptfoo eval --filter-tests "landscape|architecture"
+```
+
+## Image Types Tested
+
+1. **Photorealistic Portrait** - Professional headshots
+2. **Artistic Landscape** - Traditional painting style
+3. **Architectural Visualization** - Modern building photography
+4. **Product Photography** - Commercial product shots
+5. **Abstract Art** - Expressionist paintings
+6. **Science Fiction** - Cyberpunk cityscapes
+7. **Food Photography** - Culinary presentation
+8. **Wildlife Photography** - Nature and animals
+
+## Model Comparison
+
+| Model                    | Best For        | Speed  | Cost          | Resolution |
+| ------------------------ | --------------- | ------ | ------------- | ---------- |
+| FLUX 1.1 Pro Ultra       | Highest quality | Fast   | $0.06/image   | Up to 4MP  |
+| FLUX 1.1 Pro Ultra (Raw) | Photorealism    | Fast   | $0.06/image   | Up to 4MP  |
+| FLUX Dev                 | General use     | Medium | ~$0.02/image  | 1024x1024  |
+| FLUX Dev Realism         | Photorealistic  | Medium | ~$0.028/image | 1024x1024  |
+| SDXL                     | Artistic styles | Fast   | ~$0.01/image  | 1024x1024  |
+
+## Configuration Options
+
+You can customize generation parameters:
+
+```yaml
+providers:
+  - id: replicate:image:black-forest-labs/flux-dev
+    config:
+      width: 1344 # Image width
+      height: 768 # Image height
+      num_outputs: 1 # Number of images to generate
+      guidance: 3.5 # How closely to follow the prompt (1-20)
+      num_inference_steps: 28 # Quality vs speed tradeoff
+      output_format: 'png' # png, webp, or jpg
+      seed: 42 # For reproducible results
+```
+
+## Important: Image URL Expiration
+
+:::warning
+**Replicate image URLs expire after approximately 24 hours.** To preserve generated images, use the included `save-images.js` hook that automatically downloads all images during evaluation.
+:::
+
+## Automatic Image Downloads
+
+This example includes a `save-images.js` hook that automatically downloads all generated images to an `images/` directory. To enable it:
+
+```yaml
+# Add to any config file
+extensions:
+  - file://save-images.js:hook
+```
+
+Or run with the included configuration:
+
+```bash
+promptfoo eval -c promptfooconfig-with-download.yaml
+```
+
+Downloaded images will be saved as:
+
+- `images/flux-dev-red-apple-on-a-white-table-2025-07-28T12-30-45.png`
+- `images/sdxl-portrait-of-elderly-man-2025-07-28T12-31-02.png`
+
+## Tips
+
+1. **For best quality**: Use FLUX 1.1 Pro Ultra
+2. **For photorealism**: Use FLUX 1.1 Pro Ultra with `raw: true`
+3. **For commercial use with downloaded weights**: Use FLUX Dev
+4. **For artistic styles**: SDXL often performs better than FLUX
+5. **Always download images** if you need them later - URLs expire!
+
+## Viewing Results
+
+After running the evaluation:
+
+1. Check the terminal for immediate results
+2. Open the generated web UI for visual comparison
+3. Images are displayed inline with markdown formatting
+
+## Cost Estimation
+
+Running the full test suite (40 images) costs approximately:
+
+- FLUX 1.1 Pro Ultra: 16 images × $0.06 = $0.96
+- FLUX Dev models: 16 images × ~$0.025 = $0.40
+- SDXL: 8 images × $0.01 = $0.08
+- **Total: ~$1.44**
+
+## Troubleshooting
+
+1. **Rate limits**: Replicate has rate limits. Use `--delay 1000` to add delays between requests or `--max-concurrency 1` to run sequentially
+2. **Timeouts**: Some models take 20-30 seconds on first run (cold start). The provider handles polling automatically
+3. **API errors**: Ensure your Replicate API token is valid and has credits
+4. **Model not found**: Check the model ID matches one from https://replicate.com/explore

examples/provider-replicate/image-generation/promptfooconfig-with-download.yaml

@@ -0,0 +1,55 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: Image generation with automatic downloads
+
+# Enable the download hook
+extensions:
+  - file://save-images.js:hook
+
+prompts:
+  - 'Generate an image: {{imagePrompt}}'
+
+providers:
+  # FLUX Dev - Best balance of quality and cost
+  - id: replicate:image:black-forest-labs/flux-dev
+    config:
+      width: 1024
+      height: 1024
+      num_outputs: 1
+      guidance: 3.5
+      num_inference_steps: 28
+
+  # SDXL for comparison
+  - id: replicate:image:stability-ai/sdxl:7762fd07cf82c948538e41f63f77d685e02b063e37e496e96eefd46c929f9bdc
+    label: sdxl
+    config:
+      width: 1024
+      height: 1024
+      num_inference_steps: 25
+
+tests:
+  # Simple object
+  - vars:
+      imagePrompt: 'A red apple on a white table, professional product photography, soft lighting'
+    assert:
+      - type: javascript
+        value: |
+          // Check markdown image format
+          return output && output.includes('![') && output.includes('](http');
+
+  # Portrait
+  - vars:
+      imagePrompt: 'Portrait of a smiling elderly man with grey beard, warm lighting, bokeh background'
+    assert:
+      - type: javascript
+        value: |
+          // Check markdown image format
+          return output && output.includes('![') && output.includes('](http');
+
+  # Landscape
+  - vars:
+      imagePrompt: 'Serene mountain lake at sunrise, mirror reflection, pine trees, photorealistic'
+    assert:
+      - type: javascript
+        value: |
+          // Check markdown image format
+          return output && output.includes('![') && output.includes('](http');

examples/provider-replicate/image-generation/promptfooconfig.yaml

@@ -0,0 +1,124 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: Test state-of-the-art image generation models on Replicate
+
+prompts:
+  - 'Generate an image: {{imagePrompt}}'
+
+providers:
+  # FLUX 1.1 Pro Ultra - Highest quality, up to 4MP images
+  - id: replicate:image:black-forest-labs/flux-1.1-pro-ultra
+    config:
+      width: 1024
+      height: 1024
+      aspect_ratio: '1:1'
+      output_format: 'webp'
+
+  # FLUX 1.1 Pro Ultra - Raw mode for photorealistic results
+  - id: replicate:image:black-forest-labs/flux-1.1-pro-ultra
+    label: flux-1.1-pro-ultra-raw
+    config:
+      width: 1024
+      height: 1024
+      aspect_ratio: '1:1'
+      output_format: 'webp'
+      raw: true # Enable raw mode for more authentic photography
+
+  # FLUX Dev - Open source version
+  - id: replicate:image:black-forest-labs/flux-dev
+    config:
+      width: 1024
+      height: 1024
+      num_outputs: 1
+      guidance: 3.5
+      num_inference_steps: 28
+
+  # FLUX Dev Realism - Specialized for photorealistic images
+  - id: replicate:image:xlabs-ai/flux-dev-realism
+    config:
+      width: 1024
+      height: 1024
+      num_outputs: 1
+      num_inference_steps: 28
+      guidance_scale: 3.5
+
+  # Stable Diffusion XL for comparison
+  - id: replicate:image:stability-ai/sdxl:7762fd07cf82c948538e41f63f77d685e02b063e37e496e96eefd46c929f9bdc
+    label: sdxl
+    config:
+      width: 1024
+      height: 1024
+      num_inference_steps: 25
+      guidance_scale: 7.5
+
+tests:
+  # Photorealistic portrait
+  - vars:
+      imagePrompt: 'Professional headshot of a confident business woman, natural lighting, shallow depth of field, shot on Canon R5, 85mm lens'
+    assert:
+      - type: javascript
+        value: |
+          // Check that the output contains markdown image format
+          return output && typeof output === 'string' && output.includes('![') && output.includes('](http');
+
+  # Artistic landscape
+  - vars:
+      imagePrompt: 'Majestic mountain landscape at golden hour, dramatic clouds, painted in the style of Albert Bierstadt, oil on canvas, highly detailed'
+    assert:
+      - type: javascript
+        value: |
+          // Check that the output contains markdown image format
+          return output && typeof output === 'string' && output.includes('![') && output.includes('](http');
+
+  # Architectural visualization
+  - vars:
+      imagePrompt: 'Modern minimalist house with large glass windows, concrete and wood exterior, surrounded by lush greenery, architectural photography, dusk lighting'
+    assert:
+      - type: javascript
+        value: |
+          // Check that the output contains markdown image format
+          return output && typeof output === 'string' && output.includes('![') && output.includes('](http');
+
+  # Product photography
+  - vars:
+      imagePrompt: 'Luxury watch floating in air with dramatic lighting, black background, professional product photography, ultra sharp focus, commercial style'
+    assert:
+      - type: javascript
+        value: |
+          // Check that the output contains markdown image format
+          return output && typeof output === 'string' && output.includes('![') && output.includes('](http');
+
+  # Abstract art
+  - vars:
+      imagePrompt: 'Abstract expressionist painting with vibrant blues and oranges, dynamic brushstrokes, inspired by Jackson Pollock and Mark Rothko'
+    assert:
+      - type: javascript
+        value: |
+          // Check that the output contains markdown image format
+          return output && typeof output === 'string' && output.includes('![') && output.includes('](http');
+
+  # Science fiction scene
+  - vars:
+      imagePrompt: 'Futuristic cyberpunk cityscape at night, neon lights reflecting on wet streets, flying cars, holographic advertisements, blade runner aesthetic'
+    assert:
+      - type: javascript
+        value: |
+          // Check that the output contains markdown image format
+          return output && typeof output === 'string' && output.includes('![') && output.includes('](http');
+
+  # Food photography
+  - vars:
+      imagePrompt: 'Gourmet sushi platter, minimalist presentation, soft natural lighting, shallow depth of field, overhead shot, restaurant quality'
+    assert:
+      - type: javascript
+        value: |
+          // Check that the output contains markdown image format
+          return output && typeof output === 'string' && output.includes('![') && output.includes('](http');
+
+  # Wildlife photography
+  - vars:
+      imagePrompt: 'Majestic lion in African savanna during sunset, sharp focus on eyes, bokeh background, National Geographic style wildlife photography'
+    assert:
+      - type: javascript
+        value: |
+          // Check that the output contains markdown image format
+          return output && typeof output === 'string' && output.includes('![') && output.includes('](http');

examples/provider-replicate/image-generation/quick-test.yaml

@@ -0,0 +1,54 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: Quick test of Replicate image generation (2 models, 3 prompts)
+
+prompts:
+  - 'Generate an image: {{imagePrompt}}'
+
+providers:
+  # FLUX Dev - Best balance of quality and cost
+  - id: replicate:image:black-forest-labs/flux-dev
+    config:
+      width: 1024
+      height: 1024
+      num_outputs: 1
+      guidance: 3.5
+      num_inference_steps: 28
+
+  # SDXL for comparison
+  - id: replicate:image:stability-ai/sdxl:7762fd07cf82c948538e41f63f77d685e02b063e37e496e96eefd46c929f9bdc
+    label: sdxl
+    config:
+      width: 1024
+      height: 1024
+      num_inference_steps: 25
+
+tests:
+  # Simple object
+  - vars:
+      imagePrompt: 'A red apple on a white table, professional product photography, soft lighting'
+    assert:
+      - type: javascript
+        value: |
+          // Check that the output contains markdown image format
+          if (typeof output !== 'string') {
+            return false;
+          }
+          return output.includes('![') && output.includes('](http');
+
+  # Portrait
+  - vars:
+      imagePrompt: 'Portrait of a smiling elderly man with grey beard, warm lighting, bokeh background'
+    assert:
+      - type: javascript
+        value: |
+          // Check markdown image format
+          return output && output.includes('![') && output.includes('](http');
+
+  # Landscape
+  - vars:
+      imagePrompt: 'Serene mountain lake at sunrise, mirror reflection, pine trees, photorealistic'
+    assert:
+      - type: javascript
+        value: |
+          // Check markdown image format
+          return output && output.includes('![') && output.includes('](http');

examples/provider-replicate/image-generation/save-images.js

@@ -0,0 +1,66 @@
+const fs = require('fs');
+const path = require('path');
+
+// For Node >= 20, fetch is available globally
+const { fetch } = globalThis;
+
+/**
+ * Downloads and saves Replicate generated images after each test
+ */
+module.exports = {
+  async hook(hookName, context) {
+    // Only run for afterEach hook and when we have an output
+    if (hookName !== 'afterEach') {
+      return;
+    }
+
+    // Extract URL from markdown image format
+    const output = context.result?.response?.output;
+    if (!output || typeof output !== 'string') {
+      return;
+    }
+
+    const match = output.match(/!\[.*?\]\((.*?)\)/);
+    const imageUrl = match?.[1];
+    if (!imageUrl || !imageUrl.includes('replicate.delivery')) {
+      return;
+    }
+
+    try {
+      // Create images directory if it doesn't exist
+      const imagesDir = path.join(__dirname, 'images');
+      await fs.promises.mkdir(imagesDir, { recursive: true });
+
+      // Extract provider and model info for filename
+      const providerId = context.provider?.id || 'unknown';
+      const modelParts = providerId.split(':');
+      const modelFullName = modelParts[modelParts.length - 1] || 'model';
+      const modelName = modelFullName.split('/').pop()?.split(':')[0] || 'model';
+
+      // Generate filename from prompt and timestamp
+      const imagePrompt = context.test.vars?.imagePrompt || 'unnamed';
+      const timestamp = new Date().toISOString().replace(/[:.]/g, '-');
+      const sanitizedPrompt = imagePrompt
+        .substring(0, 50) // Limit length
+        .replace(/[^a-z0-9\s-]/gi, '')
+        .trim()
+        .replace(/\s+/g, '-')
+        .toLowerCase();
+      const filename = `${modelName}-${sanitizedPrompt}-${timestamp}.png`;
+      const filepath = path.join(imagesDir, filename);
+
+      // Download and save the image
+      const response = await fetch(imageUrl);
+      if (!response.ok) {
+        throw new Error(`HTTP error: ${response.status}`);
+      }
+
+      const buffer = await response.arrayBuffer();
+      await fs.promises.writeFile(filepath, Buffer.from(buffer));
+
+      console.log(`✓ Saved image: ${filename}`);
+    } catch (error) {
+      console.error(`❌ Failed to save image: ${error.message}`);
+    }
+  },
+};