wehub-resource-sync
0d3cb498a3
CI / Shell Format Check (push) Has been cancelled
CI / Check Ruby (3.4) (push) Has been cancelled
CI / CI Config (push) Has been cancelled
CI / Test on Node ${{ matrix.node }} and ${{ matrix.os }}${{ matrix.shard && format(' (shard {0}/3)', matrix.shard) || '' }} (push) Has been cancelled
CI / Build on Node ${{ matrix.node }} (push) Has been cancelled
CI / Style Check (push) Has been cancelled
CI / Generate Assets (push) Has been cancelled
CI / Check Python (3.14) (push) Has been cancelled
CI / Check Python (3.9) (push) Has been cancelled
CI / Build Docs (push) Has been cancelled
CI / Code Scan Action (push) Has been cancelled
CI / Site tests (push) Has been cancelled
CI / webui tests (push) Has been cancelled
CI / Run Integration Tests (push) Has been cancelled
CI / Run Smoke Tests (push) Has been cancelled
CI / Go Tests (push) Has been cancelled
CI / Share Test (push) Has been cancelled
CI / Redteam (Production API) (push) Has been cancelled
CI / Redteam (Staging API) (push) Has been cancelled
CI / GitHub Actions Lint (push) Has been cancelled
CI / Check Ruby (3.0) (push) Has been cancelled
release-please / release-please (push) Has been cancelled
release-please / build (push) Has been cancelled
release-please / publish-npm (push) Has been cancelled
release-please / publish-npm-backfill (push) Has been cancelled
release-please / docker (push) Has been cancelled
release-please / publish-code-scan-action (push) Has been cancelled
release-please / attest-code-scan-action (push) Has been cancelled
Deploy local.promptfoo.app / Deploy to Cloudflare Pages (push) Has been cancelled
Test and Publish Multi-arch Docker Image / test (push) Has been cancelled
Test and Publish Multi-arch Docker Image / build-docker-and-push-digests (map[digest-suffix:linux-amd64 platform:linux/amd64 runner:ubuntu-latest]) (push) Has been cancelled
Test and Publish Multi-arch Docker Image / build-docker-and-push-digests (map[digest-suffix:linux-arm64 platform:linux/arm64 runner:ubuntu-24.04-arm]) (push) Has been cancelled
Test and Publish Multi-arch Docker Image / merge-docker-digests (push) Has been cancelled
Test and Publish Multi-arch Docker Image / Attest Multi-arch Image (push) Has been cancelled
Validate Renovate Config / Validate Renovate Configuration (push) Has been cancelled
config-tests-per-prompt (Filter Tests by Prompt)
This example demonstrates how to run specific tests only with specific prompts using the test-level prompts field.
Overview
By default, promptfoo runs each test against all prompts (a cartesian product). The prompts field on test cases lets you break this pattern and associate specific tests with specific prompts.
Use Cases
- Different assertion criteria for different prompt styles
- Testing prompt-specific behaviors
- Organizing large test suites with mixed prompt types
Quick Start
npx promptfoo@latest init --example config-tests-per-prompt
npx promptfoo@latest eval
Key Features
- Exact matching: Reference prompts by label or ID
- Wildcard patterns:
Math:*matchesMath:Basic,Math:Advanced - Strict validation: Invalid references error at config load time
- defaultTest inheritance: Set a default prompt filter for all tests
Configuration
The example uses the echo provider for demonstration. In production, replace with your preferred LLM provider.
prompts:
- id: factual
label: Factual Assistant
raw: 'You are factual. Answer: {{question}}'
- id: creative
label: Creative Writer
raw: 'Be creative. Answer: {{question}}'
tests:
- vars:
question: 'What is 2+2?'
prompts:
- Factual Assistant # Only runs with this prompt
assert:
- type: contains
value: '4'
Learn More
See the Filtering Tests by Prompt documentation for more details.