wehub-resource-sync
0d3cb498a3
Deploy local.promptfoo.app / Deploy to Cloudflare Pages (push) Waiting to run
Test and Publish Multi-arch Docker Image / test (push) Waiting to run
Test and Publish Multi-arch Docker Image / build-docker-and-push-digests (map[digest-suffix:linux-amd64 platform:linux/amd64 runner:ubuntu-latest]) (push) Blocked by required conditions
Test and Publish Multi-arch Docker Image / build-docker-and-push-digests (map[digest-suffix:linux-arm64 platform:linux/arm64 runner:ubuntu-24.04-arm]) (push) Blocked by required conditions
Test and Publish Multi-arch Docker Image / merge-docker-digests (push) Blocked by required conditions
Test and Publish Multi-arch Docker Image / Attest Multi-arch Image (push) Blocked by required conditions
Validate Renovate Config / Validate Renovate Configuration (push) Waiting to run
CI / Shell Format Check (push) Has been cancelled
CI / Check Ruby (3.4) (push) Has been cancelled
CI / CI Config (push) Has been cancelled
CI / Test on Node ${{ matrix.node }} and ${{ matrix.os }}${{ matrix.shard && format(' (shard {0}/3)', matrix.shard) || '' }} (push) Has been cancelled
CI / Build on Node ${{ matrix.node }} (push) Has been cancelled
CI / Style Check (push) Has been cancelled
CI / Generate Assets (push) Has been cancelled
CI / Check Python (3.14) (push) Has been cancelled
CI / Check Python (3.9) (push) Has been cancelled
CI / Build Docs (push) Has been cancelled
CI / Code Scan Action (push) Has been cancelled
CI / Site tests (push) Has been cancelled
CI / webui tests (push) Has been cancelled
CI / Run Integration Tests (push) Has been cancelled
CI / Run Smoke Tests (push) Has been cancelled
CI / Go Tests (push) Has been cancelled
CI / Share Test (push) Has been cancelled
CI / Redteam (Production API) (push) Has been cancelled
CI / Redteam (Staging API) (push) Has been cancelled
CI / GitHub Actions Lint (push) Has been cancelled
CI / Check Ruby (3.0) (push) Has been cancelled
release-please / release-please (push) Has been cancelled
release-please / build (push) Has been cancelled
release-please / publish-npm (push) Has been cancelled
release-please / publish-npm-backfill (push) Has been cancelled
release-please / docker (push) Has been cancelled
release-please / publish-code-scan-action (push) Has been cancelled
release-please / attest-code-scan-action (push) Has been cancelled
openai-images (OpenAI Image Generation Example)
You can run this example with:
npx promptfoo@latest init --example openai-images
cd openai-images
A simple example showing how to evaluate OpenAI's current image generation models (GPT Image 2, GPT Image 1.5, GPT Image 1, GPT Image 1 Mini) with promptfoo.
Quick Start
# Create this example
npx promptfoo@latest init --example openai-images
# Set your API key
export OPENAI_API_KEY=your-key-here
# Run the evaluation
promptfoo eval
# View the results
promptfoo view
What's in this Example
- Compares GPT Image 2, GPT Image 1.5, GPT Image 1, and GPT Image 1 Mini outputs
- Uses text-to-image generation through the Image API generations endpoint
- Tests artistic style prompts across different models
- Configures different image sizes and quality settings
- Tests with different subjects
Supported Models
GPT Image 2 (Recommended)
OpenAI's latest image generation model with flexible custom sizes and improved output controls.
providers:
- id: openai:image:gpt-image-2
config:
size: 1024x1024 # common sizes, custom WIDTHxHEIGHT, auto
quality: low # low, medium, high, auto
background: opaque # opaque, auto
output_format: webp # png, jpeg, webp
output_compression: 80 # 0-100, only set with jpeg/webp
moderation: auto # auto, low
n: 1 # 1-10 images
user: promptfoo-user # optional end-user identifier
GPT Image 1.5
High-quality GPT Image model with strong instruction following, prompt adherence, and photorealistic quality. Uses token-based pricing.
providers:
- id: openai:image:gpt-image-1.5
config:
size: 1024x1024 # 1024x1024, 1024x1536, 1536x1024, auto
quality: low # low, medium, high, auto
background: transparent # transparent, opaque, auto
output_format: webp # png, jpeg, webp
output_compression: 80 # 0-100, only set with jpeg/webp
moderation: auto # auto, low
GPT Image 1
High-quality image generation model with superior instruction following and text rendering.
providers:
- id: openai:image:gpt-image-1
config:
size: 1024x1024 # 1024x1024, 1024x1536, 1536x1024, auto
quality: low # low, medium, high, auto
background: transparent # transparent, opaque, auto
output_format: webp # png, jpeg, webp
output_compression: 80 # 0-100, only set with jpeg/webp
moderation: auto # auto, low
GPT Image 1 Mini
Cost-efficient version of GPT Image 1 with the same capabilities at lower cost.
providers:
- id: openai:image:gpt-image-1-mini
config:
size: 1024x1024 # 1024x1024, 1024x1536, 1536x1024, auto
quality: low # low, medium, high, auto
background: transparent # transparent, opaque, auto
output_format: webp # png, jpeg, webp
output_compression: 80 # 0-100, only set with jpeg/webp
moderation: auto # auto, low
DALL-E 3 and DALL-E 2
dall-e-3 and dall-e-2 remain supported by the provider for backward compatibility, but they are deprecated by OpenAI. Use gpt-image-2, gpt-image-1.5, gpt-image-1, or gpt-image-1-mini for new evals.
Promptfoo's openai:image provider currently supports text-to-image generation. Image edits/reference inputs, variations, and streaming partial images are not part of this example and are rejected if configured.
Pricing
| Model | Quality | Size | Price per image |
|---|---|---|---|
| GPT Image 2 | Low | 1024x1024 | $0.006 |
| GPT Image 2 | Medium | 1024x1024 | $0.053 |
| GPT Image 2 | High | 1024x1024 | $0.211 |
| GPT Image 1.5 | Low | 1024x1024 | ~$0.064 |
| GPT Image 1.5 | Medium | 1024x1024 | ~$0.128 |
| GPT Image 1.5 | High | 1024x1024 | ~$0.192 |
| GPT Image 1 | Low | 1024x1024 | $0.011 |
| GPT Image 1 | Medium | 1024x1024 | $0.042 |
| GPT Image 1 | High | 1024x1024 | $0.167 |
| GPT Image 1 Mini | Low | 1024x1024 | $0.005 |
| GPT Image 1 Mini | Medium | 1024x1024 | $0.011 |
| GPT Image 1 Mini | High | 1024x1024 | $0.036 |
Note: Prices shown are 1024x1024 output image estimates and do not include input text tokens. For GPT Image 2 auto quality or custom sizes, promptfoo preserves OpenAI usage metadata and leaves cost unset instead of guessing.