wehub-resource-sync
0d3cb498a3
Deploy local.promptfoo.app / Deploy to Cloudflare Pages (push) Waiting to run
Test and Publish Multi-arch Docker Image / test (push) Waiting to run
Test and Publish Multi-arch Docker Image / build-docker-and-push-digests (map[digest-suffix:linux-amd64 platform:linux/amd64 runner:ubuntu-latest]) (push) Blocked by required conditions
Test and Publish Multi-arch Docker Image / build-docker-and-push-digests (map[digest-suffix:linux-arm64 platform:linux/arm64 runner:ubuntu-24.04-arm]) (push) Blocked by required conditions
Test and Publish Multi-arch Docker Image / merge-docker-digests (push) Blocked by required conditions
Test and Publish Multi-arch Docker Image / Attest Multi-arch Image (push) Blocked by required conditions
Validate Renovate Config / Validate Renovate Configuration (push) Waiting to run
CI / Shell Format Check (push) Has been cancelled
CI / Check Ruby (3.4) (push) Has been cancelled
CI / CI Config (push) Has been cancelled
CI / Test on Node ${{ matrix.node }} and ${{ matrix.os }}${{ matrix.shard && format(' (shard {0}/3)', matrix.shard) || '' }} (push) Has been cancelled
CI / Build on Node ${{ matrix.node }} (push) Has been cancelled
CI / Style Check (push) Has been cancelled
CI / Generate Assets (push) Has been cancelled
CI / Check Python (3.14) (push) Has been cancelled
CI / Check Python (3.9) (push) Has been cancelled
CI / Build Docs (push) Has been cancelled
CI / Code Scan Action (push) Has been cancelled
CI / Site tests (push) Has been cancelled
CI / webui tests (push) Has been cancelled
CI / Run Integration Tests (push) Has been cancelled
CI / Run Smoke Tests (push) Has been cancelled
CI / Go Tests (push) Has been cancelled
CI / Share Test (push) Has been cancelled
CI / Redteam (Production API) (push) Has been cancelled
CI / Redteam (Staging API) (push) Has been cancelled
CI / GitHub Actions Lint (push) Has been cancelled
CI / Check Ruby (3.0) (push) Has been cancelled
release-please / release-please (push) Has been cancelled
release-please / build (push) Has been cancelled
release-please / publish-npm (push) Has been cancelled
release-please / publish-npm-backfill (push) Has been cancelled
release-please / docker (push) Has been cancelled
release-please / publish-code-scan-action (push) Has been cancelled
release-please / attest-code-scan-action (push) Has been cancelled
config-tests-per-provider (Isolated Test Suites)
This example demonstrates how to run different tests against different providers in a single evaluation using the providers filter on test cases.
What You'll Learn
- How to filter which providers run specific tests
- How to run separate test suites without cross-pollination
- Using wildcards and labels for provider matching
- Using
defaultTestto set provider filters globally
Setup
npx promptfoo@latest init --example config-tests-per-provider
npx promptfoo@latest eval
The Problem
Without filtering, promptfoo creates a cross-product of all tests × providers. If you have 2 tests and 2 providers, you get 4 test cases. This example shows how to run specific tests only against specific providers.
Features Demonstrated
Provider Filtering
Each test case can specify which providers it should run against:
tests:
- vars:
question: 'Quick math question'
providers:
- fast-model # Only runs on fast-model
assert:
- type: latency
threshold: 2000
- vars:
question: 'Complex reasoning'
providers:
- smart-model # Only runs on smart-model
Matching Patterns
The providers filter supports multiple matching patterns:
| Pattern | Example | Matches |
|---|---|---|
| Label | fast-model |
Provider with label: fast-model |
| Exact ID | openai:gpt-4 |
Provider with that exact ID |
| Wildcard | openai:* |
All providers starting with openai: |
| Prefix | openai |
All providers starting with openai: |
Default Filters
Use defaultTest.providers to apply filters to all tests:
defaultTest:
providers:
- openai:* # All tests default to OpenAI only
tests:
- vars:
question: 'Question 1'
- vars:
question: 'Question 2'
providers:
- anthropic:* # Override for this test only
Result
Instead of 6 test cases (3 tests × 2 providers), this example runs exactly 3 test cases:
- Quick facts test → fast-model only
- Complex reasoning test → smart-model only
- Basic test → both providers