chore: import upstream snapshot with attribution

2026-07-13 13:24:08 +08:00
commit 0d3cb498a3
5438 changed files with 1316560 additions and 0 deletions
@@ -0,0 +1,85 @@
+# config-tests-per-provider (Isolated Test Suites)
+
+This example demonstrates how to run different tests against different providers in a single evaluation using the `providers` filter on test cases.
+
+## What You'll Learn
+
+- How to filter which providers run specific tests
+- How to run separate test suites without cross-pollination
+- Using wildcards and labels for provider matching
+- Using `defaultTest` to set provider filters globally
+
+## Setup
+
+```bash
+npx promptfoo@latest init --example config-tests-per-provider
+npx promptfoo@latest eval
+```
+
+## The Problem
+
+Without filtering, promptfoo creates a cross-product of all tests × providers. If you have 2 tests and 2 providers, you get 4 test cases. This example shows how to run specific tests only against specific providers.
+
+## Features Demonstrated
+
+### Provider Filtering
+
+Each test case can specify which providers it should run against:
+
+```yaml
+tests:
+  - vars:
+      question: 'Quick math question'
+    providers:
+      - fast-model # Only runs on fast-model
+    assert:
+      - type: latency
+        threshold: 2000
+
+  - vars:
+      question: 'Complex reasoning'
+    providers:
+      - smart-model # Only runs on smart-model
+```
+
+### Matching Patterns
+
+The `providers` filter supports multiple matching patterns:
+
+| Pattern  | Example        | Matches                               |
+| -------- | -------------- | ------------------------------------- |
+| Label    | `fast-model`   | Provider with `label: fast-model`     |
+| Exact ID | `openai:gpt-4` | Provider with that exact ID           |
+| Wildcard | `openai:*`     | All providers starting with `openai:` |
+| Prefix   | `openai`       | All providers starting with `openai:` |
+
+### Default Filters
+
+Use `defaultTest.providers` to apply filters to all tests:
+
+```yaml
+defaultTest:
+  providers:
+    - openai:* # All tests default to OpenAI only
+
+tests:
+  - vars:
+      question: 'Question 1'
+  - vars:
+      question: 'Question 2'
+    providers:
+      - anthropic:* # Override for this test only
+```
+
+## Result
+
+Instead of 6 test cases (3 tests × 2 providers), this example runs exactly 3 test cases:
+
+- Quick facts test → fast-model only
+- Complex reasoning test → smart-model only
+- Basic test → both providers
+
+## Learn More
+
+- [Test Case Configuration](/docs/configuration/test-cases#filtering-tests-by-provider)
+- [Provider Configuration](/docs/configuration/providers)
@@ -0,0 +1,42 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: Isolated test suites per provider
+
+prompts:
+  - 'Answer concisely: {{question}}'
+
+providers:
+  - id: openai:gpt-4.1-mini
+    label: fast-model
+  - id: openai:gpt-4.1
+    label: smart-model
+
+tests:
+  # Quick tests - only run on fast-model (cheaper, faster)
+  - description: Quick facts test
+    vars:
+      question: 'What is 2 + 2?'
+    providers:
+      - fast-model
+    assert:
+      - type: contains
+        value: '4'
+      - type: latency
+        threshold: 3000
+
+  # Complex tests - only run on smart-model (more capable)
+  - description: Complex reasoning test
+    vars:
+      question: 'Explain why the sky appears blue using physics concepts.'
+    providers:
+      - smart-model
+    assert:
+      - type: llm-rubric
+        value: 'Accurately explains Rayleigh scattering and wavelength concepts'
+
+  # Basic test - runs on both providers (no filter)
+  - description: Basic capability test
+    vars:
+      question: 'What is the capital of France?'
+    assert:
+      - type: contains
+        value: 'Paris'