promptfoo--promptfoo/site/docs/configuration/test-cases.md

---
sidebar_position: 12
sidebar_label: Test Cases
title: Test Case Configuration - Variables, Assertions, and Data
description: Configure test cases for LLM evaluation with variables, assertions, CSV data, and dynamic generation. Learn inline tests, external files, and media support.
keywords:
  [
    test cases,
    LLM testing,
    evaluation data,
    assertions,
    CSV tests,
    variables,
    dynamic testing,
    test automation,
  ]
pagination_prev: configuration/prompts
pagination_next: configuration/scenarios
---

# Test Case Configuration

Define evaluation scenarios with variables, assertions, and test data.

## Inline Tests

The simplest way to define tests is directly in your config:

```yaml title="promptfooconfig.yaml"
tests:
  - vars:
      question: 'What is the capital of France?'
    assert:
      - type: contains
        value: 'Paris'

  - vars:
      question: 'What is 2 + 2?'
    assert:
      - type: equals
        value: '4'
```

### Test Structure

Each test case can include:

```yaml
tests:
  - description: 'Optional test description'
    vars:
      # Variables to substitute in prompts
      var1: value1
      var2: value2
    assert:
      # Expected outputs and validations
      - type: contains
        value: 'expected text'
    metadata:
      # Filterable metadata
      category: math
      difficulty: easy
```

### Repeating an Individual Test

Set `options.repeat` to a positive integer to run one test case multiple times:

```yaml title="promptfooconfig.yaml"
tests:
  - description: 'Sample a nondeterministic response'
    vars:
      question: 'Write a short greeting'
    options:
      repeat: 3
```

The per-test value overrides `--repeat`, `commandLineOptions.repeat`, or
`evaluateOptions.repeat` for that test. Other tests continue to use the global repeat count.
Repeat indexes use separate cache entries; add `--no-cache` when every run must call the provider.

### Filtering Tests by Provider

Control which providers run specific tests using the `providers` field. This allows you to run different test suites against different models in a single evaluation:

```yaml
providers:
  - id: openai:gpt-3.5-turbo
    label: fast-model
  - id: openai:gpt-4
    label: smart-model

tests:
  # Only run on fast-model
  - vars:
      question: 'What is 2 + 2?'
    providers:
      - fast-model
    assert:
      - type: equals
        value: '4'

  # Only run on smart-model
  - vars:
      question: 'Explain quantum entanglement'
    providers:
      - smart-model
    assert:
      - type: llm-rubric
        value: 'Provides accurate physics explanation'
```

**Matching syntax:**

| Pattern        | Matches                                                              |
| -------------- | -------------------------------------------------------------------- |
| `fast-model`   | Exact label match                                                    |
| `openai:gpt-4` | Exact provider ID match                                              |
| `openai:*`     | Wildcard - any provider starting with `openai:`                      |
| `openai`       | Legacy prefix - matches `openai:gpt-4`, `openai:gpt-3.5-turbo`, etc. |

**Apply to all tests using `defaultTest`:**

```yaml
defaultTest:
  providers:
    - openai:* # All tests default to OpenAI providers only

tests:
  - vars:
      question: 'Simple question'
  - vars:
      question: 'Complex question'
    providers:
      - smart-model # Override default for this test
```

**Edge cases:**

- **No filter**: Without the `providers` field, the test runs against all providers (cross-product behavior)
- **Empty array**: `providers: []` means the test runs on no providers and is effectively skipped
- **Stacking with providerPromptMap**: When both `providers` and `providerPromptMap` are set, they filter together—a provider must match both to run
- **CLI `--filter-providers`**: If you use `--filter-providers` to filter providers at the CLI level, validation only sees the filtered providers. Tests referencing providers excluded by `--filter-providers` will fail validation

### Filtering Tests by Prompt

By default, each test runs against all prompts (a cartesian product). You can use the `prompts` field to restrict a test to specific prompts:

```yaml
prompts:
  - id: prompt-factual
    label: Factual Assistant
    raw: 'You are a factual assistant. Answer: {{question}}'
  - id: prompt-creative
    label: Creative Writer
    raw: 'You are a creative writer. Answer: {{question}}'

providers:
  - openai:gpt-4o-mini

tests:
  # This test only runs with the Factual Assistant prompt
  - vars:
      question: 'What is the capital of France?'
    prompts:
      - Factual Assistant
    assert:
      - type: contains
        value: 'Paris'

  # This test only runs with the Creative Writer prompt
  - vars:
      question: 'Write a poem about Paris'
    prompts:
      - prompt-creative # You can reference by ID or label
    assert:
      - type: llm-rubric
        value: 'Contains poetic language'

  # This test runs with all prompts (default behavior)
  - vars:
      question: 'Hello'
```

The `prompts` field accepts:

- **Exact labels**: `prompts: ['Factual Assistant']`
- **Exact IDs**: `prompts: ['prompt-factual']`
- **Wildcard patterns**: `prompts: ['Math:*']` matches `Math:Basic`, `Math:Advanced`, etc.
- **Prefix patterns**: `prompts: ['Math']` matches `Math:Basic`, `Math:Advanced` (legacy syntax)

:::note

Invalid prompt references will cause an error at config load time. This strict validation catches typos early.

:::

You can also set a default prompt filter in `defaultTest`:

```yaml
defaultTest:
  prompts:
    - Factual Assistant

tests:
  # Inherits prompts: ['Factual Assistant'] from defaultTest
  - vars:
      question: 'What is 2+2?'

  # Override to use a different prompt
  - vars:
      question: 'Write a story'
    prompts:
      - Creative Writer
```

## External Test Files

For larger test suites, store tests in separate files:

```yaml title="promptfooconfig.yaml"
tests: file://tests.yaml
```

Or load multiple files:

```yaml
tests:
  - file://basic_tests.yaml
  - file://advanced_tests.yaml
  - file://edge_cases/*.yaml
```

## CSV Format

CSV or Excel (XLSX) files are ideal for bulk test data:

```yaml title="promptfooconfig.yaml"
tests: file://test_cases.csv
```

```yaml title="promptfooconfig.yaml"
tests: file://test_cases.xlsx
```

### Basic CSV

```csv title="test_cases.csv"
question,expectedAnswer
"What is 2+2?","4"
"What is the capital of France?","Paris"
"Who wrote Romeo and Juliet?","Shakespeare"
```

Variables are automatically mapped from column headers.

### Excel (XLSX/XLS) Support

Excel files (.xlsx and .xls) are supported as an optional feature. To use Excel files:

1. Install the `read-excel-file` package as a peer dependency:

   ```bash
   npm install read-excel-file
   ```

2. Use Excel files just like CSV files:
   ```yaml title="promptfooconfig.yaml"
   tests: file://test_cases.xlsx
   ```

**Multi-sheet support:** By default, only the first sheet is used. To specify a different sheet, use the `#` syntax:

- `file://test_cases.xlsx#Sheet2` - Select sheet by name
- `file://test_cases.xlsx#2` - Select sheet by 1-based index (2 = second sheet)

```yaml title="promptfooconfig.yaml"
# Use a specific sheet by name
tests: file://test_cases.xlsx#DataSheet

# Or by index (1-based)
tests: file://test_cases.xlsx#2
```

### XLSX Example

Your Excel file should have column headers in the first row, with each subsequent row representing a test case:

| question                       | expectedAnswer |
| ------------------------------ | -------------- |
| What is 2+2?                   | 4              |
| What is the capital of France? | Paris          |
| Name a primary color           | blue           |

**Tips for Excel files:**

- First row must contain column headers
- Column names become variable names in your prompts
- Empty cells are treated as empty strings
- Use `__expected` columns for assertions (same as CSV)

### CSV with Assertions

Use special `__expected` columns for assertions:

```csv title="test_cases.csv"
input,__expected
"Hello world","contains: Hello"
"Calculate 5 * 6","equals: 30"
"What's the weather?","llm-rubric: Provides weather information"
```

Values without a type prefix default to `equals`:

| `__expected` value                | Assertion type               |
| --------------------------------- | ---------------------------- |
| `Paris`                           | `equals`                     |
| `contains:Paris`                  | `contains`                   |
| `factuality:The capital is Paris` | `factuality`                 |
| `similar(0.8):Hello there`        | `similar` with 0.8 threshold |

Multiple assertions:

```csv title="test_cases.csv"
question,__expected1,__expected2,__expected3
"What is 2+2?","equals: 4","contains: four","javascript: output.length < 10"
```

:::note
**contains-any**, **icontains-any**, **contains-all**, and **icontains-all** expect comma-delimited values inside the `__expected` column. Surrounding whitespace around each value is trimmed.

```csv title="test_cases.csv"
translated_text,__expected
"<span>Hola</span> <b>mundo</b>","contains-any: <b>,</span>"
```

If you write `"contains-any: <b> </span>"`, promptfoo treats `<b> </span>` as a single search term rather than two separate tags.

To match a value that itself contains a comma, wrap that value in double quotes. Because the assertion is inside a quoted CSV cell, write each of those wrapping quotes twice:

```csv title="test_cases.csv"
text,__expected
"1,000 items in stock","contains-all: ""1,000"",in stock"
```

Here `"1,000"` is a single search term (the comma is preserved), while `in stock` is a second term. These quoting rules apply to `__expected` columns in CSV, XLSX, and Google Sheets. JSON and JSONL test files should use the structured `assert` object form instead.

At the assertion-string level, escape a literal double quote inside a quoted value as `\"` or `""`. In a CSV file, remember to apply CSV escaping as well by doubling every double quote in the cell.
:::

### Special CSV Columns

| Column                                                      | Purpose                                                  | Example                                                           |
| ----------------------------------------------------------- | -------------------------------------------------------- | ----------------------------------------------------------------- |
| `__expected`                                                | Single assertion                                         | `contains: Paris`                                                 |
| `__expected1`, `__expected2`, ...                           | Multiple assertions                                      | `equals: 42`                                                      |
| `__description`                                             | Test description                                         | `Basic math test`                                                 |
| `__prefix`                                                  | Prepend to prompt                                        | `You must answer: `                                               |
| `__suffix`                                                  | Append to prompt                                         | ` (be concise)`                                                   |
| `__metric`                                                  | Display name in reports (does not change assertion type) | `accuracy`                                                        |
| `__threshold`                                               | Pass threshold (applies to all asserts)                  | `0.8`                                                             |
| `__metadata:*`                                              | Filterable metadata                                      | See below                                                         |
| `__config:__expected:<key>` or `__config:__expectedN:<key>` | Set configuration for all or specific assertions         | `__config:__expected:threshold`, `__config:__expected2:threshold` |

Using `__metadata` without a key is not supported. Specify the metadata field like `__metadata:category`.
If a CSV file includes a `__metadata` column without a key, Promptfoo logs a warning and ignores the column.

### Metadata in CSV

Add filterable metadata:

```csv title="test_cases.csv"
question,__expected,__metadata:category,__metadata:difficulty
"What is 2+2?","equals: 4","math","easy"
"Explain quantum physics","llm-rubric: Accurate explanation","science","hard"
```

Array metadata with `[]`:

```csv
topic,__metadata:tags[]
"Machine learning","ai,technology,data science"
"Climate change","environment,science,global\,warming"
```

Filter tests:

```bash
promptfoo eval --filter-metadata category=math
promptfoo eval --filter-metadata difficulty=easy
promptfoo eval --filter-metadata tags=ai

# Multiple filters use AND logic (tests must match ALL conditions)
promptfoo eval --filter-metadata category=math --filter-metadata difficulty=easy
```

### JSON in CSV

Include structured data:

```csv title="test_cases.csv"
query,context,__expected
"What's the temperature?","{""location"":""NYC"",""units"":""celsius""}","contains: celsius"
```

Access in prompts:

```yaml
prompts:
  - 'Query: {{query}}, Location: {{(context | load).location}}'
```

### CSV with defaultTest

Apply the same assertions to all tests loaded from a CSV file using [`defaultTest`](/docs/configuration/guide#default-test-cases):

```yaml title="promptfooconfig.yaml"
defaultTest:
  assert:
    - type: factuality
      value: '{{reference_answer}}'
  options:
    provider: openai:gpt-5.2

tests: file://tests.csv
```

```csv title="tests.csv"
question,reference_answer
"What does GPT stand for?","Generative Pre-trained Transformer"
"What is the capital of France?","Paris is the capital of France"
```

Use regular column names (like `reference_answer`) instead of `__expected` when referencing values in `defaultTest` assertions. The `__expected` column automatically creates assertions per row.

## Dynamic Test Generation

Generate tests programmatically:

### JavaScript/TypeScript

```yaml title="promptfooconfig.yaml"
tests: file://generate_tests.js
```

```javascript title="generate_tests.js"
module.exports = async function () {
  // Fetch data, compute test cases, etc.
  const testCases = [];

  for (let i = 1; i <= 10; i++) {
    testCases.push({
      description: `Test case ${i}`,
      vars: {
        number: i,
        squared: i * i,
      },
      assert: [
        {
          type: 'contains',
          value: String(i * i),
        },
      ],
    });
  }

  return testCases;
};
```

### Python

```yaml title="promptfooconfig.yaml"
tests: file://generate_tests.py:create_tests
```

```python title="generate_tests.py"
import json

def create_tests():
    test_cases = []

    # Load test data from database, API, etc.
    test_data = load_test_data()

    for item in test_data:
        test_cases.append({
            "vars": {
                "input": item["input"],
                "context": item["context"]
            },
            "assert": [{
                "type": "contains",
                "value": item["expected"]
            }]
        })

    return test_cases
```

### With Configuration

Pass configuration to generators:

```yaml title="promptfooconfig.yaml"
tests:
  - path: file://generate_tests.py:create_tests
    config:
      dataset: 'validation'
      category: 'math'
      sample_size: 100
```

```python title="generate_tests.py"
def create_tests(config):
    dataset = config.get('dataset', 'train')
    category = config.get('category', 'all')
    size = config.get('sample_size', 50)

    # Use configuration to generate tests
    return generate_test_cases(dataset, category, size)
```

## JSON/JSONL Format

### JSON Array

```json title="tests.json"
[
  {
    "vars": {
      "topic": "artificial intelligence"
    },
    "assert": [
      {
        "type": "contains",
        "value": "AI"
      }
    ]
  },
  {
    "vars": {
      "topic": "climate change"
    },
    "assert": [
      {
        "type": "llm-rubric",
        "value": "Discusses environmental impact"
      }
    ]
  }
]
```

### JSONL (One test per line)

```jsonl title="tests.jsonl"
{"vars": {"x": 5, "y": 3}, "assert": [{"type": "equals", "value": "8"}]}
{"vars": {"x": 10, "y": 7}, "assert": [{"type": "equals", "value": "17"}]}
```

## Loading Media Files

Include images, PDFs, and other files as variables:

```yaml title="promptfooconfig.yaml"
tests:
  - vars:
      image: file://images/chart.png
      document: file://docs/report.pdf
      data: file://data/config.yaml
```

### Path Resolution

`file://` paths are resolved relative to your **config file's directory**, not the current working directory. This ensures consistent behavior regardless of where you run `promptfoo` from:

```yaml title="src/tests/promptfooconfig.yaml"
tests:
  - vars:
      # Resolved as src/tests/data/input.json
      data: file://./data/input.json

      # Also works - resolved as src/tests/data/input.json
      data2: file://data/input.json

      # Parent directory - resolved as src/shared/context.json
      shared: file://../shared/context.json
```

Without the `file://` prefix, values are passed as plain strings to your provider.

### Supported File Types

| Type                    | Handling            | Usage             |
| ----------------------- | ------------------- | ----------------- |
| Images (png, jpg, etc.) | Converted to base64 | Vision models     |
| Videos (mp4, etc.)      | Converted to base64 | Multimodal models |
| PDFs                    | Text extraction     | Document analysis |
| Text files              | Loaded as string    | Any use case      |
| YAML/JSON               | Parsed to object    | Structured data   |

### Example: Vision Model Test

```yaml
tests:
  - vars:
      image: file://test_image.jpg
      question: 'What objects are in this image?'
    assert:
      - type: contains
        value: 'dog'
```

In your prompt:

```json
[
  {
    "role": "user",
    "content": [
      { "type": "text", "text": "{{question}}" },
      {
        "type": "image_url",
        "image_url": {
          "url": "data:image/jpeg;base64,{{image}}"
        }
      }
    ]
  }
]
```

## Best Practices

### 1. Organize Test Data

```
project/
├── promptfooconfig.yaml
├── prompts/
│   └── main_prompt.txt
└── tests/
    ├── basic_functionality.csv
    ├── edge_cases.yaml
    └── regression_tests.json
```

### 2. Use Descriptive Names

```yaml
tests:
  - description: 'Test French translation with formal tone'
    vars:
      text: 'Hello'
      language: 'French'
      tone: 'formal'
```

### 3. Group Related Tests

```yaml
# Use metadata for organization
tests:
  - vars:
      query: 'Reset password'
    metadata:
      feature: authentication
      priority: high
```

### 4. Combine Approaches

```yaml
tests:
  # Quick smoke tests inline
  - vars:
      test: 'quick check'

  # Comprehensive test suite from file
  - file://tests/full_suite.csv

  # Dynamic edge case generation
  - file://tests/generate_edge_cases.js
```

## Common Patterns

### A/B Testing Variables

```csv title="ab_tests.csv"
message_style,greeting,__expected
"formal","Good morning","contains: Good morning"
"casual","Hey there","contains: Hey"
"friendly","Hello!","contains: Hello"
```

### Error Handling Tests

```yaml
tests:
  - description: 'Handle empty input'
    vars:
      input: ''
    assert:
      - type: contains
        value: 'provide more information'
```

### Performance Tests

```yaml
tests:
  - vars:
      prompt: 'Simple question'
    assert:
      - type: latency
        threshold: 1000 # milliseconds
```

### Passing Arrays to Assertions

By default, array variables expand into multiple test cases. To pass an array directly to assertions like `contains-any`, disable variable expansion:

```yaml
defaultTest:
  options:
    disableVarExpansion: true
  assert:
    - type: contains-any
      value: '{{expected_values}}'

tests:
  - description: 'Check for any valid response'
    vars:
      expected_values: ['option1', 'option2', 'option3']
```

## External Data Sources

### Google Sheets

See [Google Sheets integration](/docs/integrations/google-sheets) for details on loading test data directly from spreadsheets.

### SharePoint

See [SharePoint integration](/docs/integrations/sharepoint) for details on loading test data from Microsoft SharePoint document libraries.

### Azure Blob Storage

Promptfoo can read test sets directly from Azure Blob Storage:

```yaml
tests: az://myaccount/evals/tests.json
```

Use `az://<account>/<container>/<blob>`. Promptfoo supports CSV, JSON, JSONL, YAML, and YML test-set blobs. Blob names may keep the original extension and append a suffix, such as `tests.json.<sha256>`.

Authentication uses the first available option:

1. A SAS query string on the URI, such as `az://myaccount/evals/tests.json?<sas-token>`
2. `AZURE_STORAGE_CONNECTION_STRING`
3. Azure identity credentials through `DefaultAzureCredential`, such as Azure CLI login, managed identity, or service principal environment variables

When using `AZURE_STORAGE_CONNECTION_STRING`, the storage account comes from the connection string. Keep the `az://` account segment aligned with that account so the URI remains self-describing; Promptfoo rejects clearly mismatched `AccountName` values. Query strings are interpreted as SAS tokens and must include `sig`.

SAS query strings and `DefaultAzureCredential` use the standard public Azure Blob endpoint for the named account. For Azure Government, Azure operated by 21Vianet, or custom blob endpoints, use `AZURE_STORAGE_CONNECTION_STRING` with the appropriate `EndpointSuffix` or explicit `BlobEndpoint`.

Blob-hosted YAML, JSON, and JSONL files are treated as remote test-case data. Promptfoo does not expand local file references or provider references found inside those blob contents.

### HuggingFace Datasets

See [HuggingFace Datasets](/docs/configuration/huggingface-datasets) for instructions on importing test cases from existing datasets.