chore: import upstream snapshot with attribution

site/docs/configuration/_category_.json

@@ -0,0 +1,4 @@
+{
+  "position": 6,
+  "label": "Evals"
+}

site/docs/configuration/caching.md

@@ -0,0 +1,141 @@
+---
+sidebar_position: 41
+sidebar_label: Caching
+title: Caching Configuration - Performance Optimization
+description: Configure caching for faster LLM evaluations. Learn cache strategies, storage options, and performance optimization for prompt testing workflows.
+keywords:
+  [
+    LLM caching,
+    performance optimization,
+    evaluation speed,
+    cache configuration,
+    response caching,
+    testing efficiency,
+  ]
+pagination_prev: configuration/chat
+pagination_next: configuration/telemetry
+---
+
+# Caching
+
+promptfoo caches the results of API calls to LLM providers to help save time and cost.
+
+The cache is managed by [`cache-manager`](https://www.npmjs.com/package/cache-manager/) with [`keyv`](https://www.npmjs.com/package/keyv) and [`keyv-file`](https://www.npmjs.com/package/keyv-file) for disk-based storage. By default, promptfoo uses disk-based storage (`~/.promptfoo/cache`).
+
+## How Caching Works
+
+### Cache Keys
+
+Cache entries are stored using provider-specific composite keys that include:
+
+- Provider identifier
+- Prompt or request content, often represented as a deterministic digest
+- Provider configuration
+- Context variables (when applicable)
+
+Cache key formats are implementation details and may change between versions.
+Sensitive request payloads and headers are hashed where possible instead of
+being embedded directly in cache keys.
+
+```js
+// Provider-specific scope plus a digest of request material
+const providerCacheKey = `openai:gpt-5:<request-digest>`;
+
+// HTTP fetch cache entries include URL, method, headers, options, and body identity
+const fetchCacheKey = `fetch:v3:<request-digest>`;
+```
+
+### Cache Behavior
+
+- Successful API responses are cached with their complete response data
+- Error responses are not cached to allow for retry attempts
+- When `evaluateOptions.repeat` or `--repeat` is greater than 1, each repeat index uses a separate cache namespace. Re-running the same eval can reuse those per-repeat cached responses, while preserving distinct outputs between repeat 0, repeat 1, etc.
+- Cache is automatically invalidated when:
+  - TTL expires (default: 14 days)
+  - Cache is manually cleared
+- Memory storage is used automatically when `NODE_ENV=test`
+
+## Command Line
+
+If you're using the command line, call `promptfoo eval` with `--no-cache` to disable the cache, or set `{ evaluateOptions: { cache: false }}` in your config file.
+
+Use `--no-cache` with `--repeat` when you want every run to make fresh LLM calls instead of replaying each repeat index from cache.
+
+Use `promptfoo cache clear` command to clear the cache.
+
+## Node package
+
+Set `EvaluateOptions.cache` to false to disable cache:
+
+```js
+promptfoo.evaluate(testSuite, {
+  cache: false,
+});
+```
+
+## Tests
+
+If you're integrating with [jest or vitest](/docs/integrations/jest), [mocha](/docs/integrations/mocha-chai), or any other external framework, you'll probably want to set the following for CI:
+
+```sh
+PROMPTFOO_CACHE_TYPE=disk
+PROMPTFOO_CACHE_PATH=...
+```
+
+## Configuration
+
+The cache is configurable through environment variables:
+
+| Environment Variable    | Description                               | Default Value                                      |
+| ----------------------- | ----------------------------------------- | -------------------------------------------------- |
+| PROMPTFOO_CACHE_ENABLED | Enable or disable the cache               | true                                               |
+| PROMPTFOO_CACHE_TYPE    | `disk` or `memory`                        | `memory` if `NODE_ENV` is `test`, otherwise `disk` |
+| PROMPTFOO_CACHE_PATH    | Path to the cache directory               | `~/.promptfoo/cache`                               |
+| PROMPTFOO_CACHE_TTL     | Time to live for cache entries in seconds | 14 days                                            |
+
+#### Additional Cache Details
+
+- Rate limit responses (HTTP 429) are automatically handled with exponential backoff
+- Empty responses are not cached
+- HTTP 500 responses can be retried by setting `PROMPTFOO_RETRY_5XX=true`
+
+## Managing the Cache
+
+### Clearing the Cache
+
+You can clear the cache in several ways:
+
+1. Using the CLI command:
+
+```bash
+promptfoo cache clear
+```
+
+2. Through the Node.js API:
+
+```javascript
+const promptfoo = require('promptfoo');
+await promptfoo.cache.clearCache();
+```
+
+3. Manually delete the cache directory:
+
+```bash
+rm -rf ~/.promptfoo/cache
+```
+
+### Cache Busting
+
+You can force a cache miss in two ways:
+
+1. Pass `--no-cache` to the CLI:
+
+```bash
+promptfoo eval --no-cache
+```
+
+2. Set cache busting in code:
+
+```javascript
+const result = await fetchWithCache(url, options, timeout, 'json', true); // Last param forces cache miss
+```

site/docs/configuration/chat.md

@@ -0,0 +1,364 @@
+---
+sidebar_label: Chat threads
+sidebar_position: 32
+title: Chat Conversations and Multi-Turn Threads
+description: Configure chat conversations and multi-turn threads for LLM evaluation. Learn conversation history, multi-shot prompts, and chat flow testing.
+keywords:
+  [
+    chat conversations,
+    multi-turn evaluation,
+    conversation history,
+    chat threads,
+    dialogue testing,
+    conversational AI,
+    chat flow,
+  ]
+pagination_prev: configuration/outputs
+pagination_next: configuration/caching
+---
+
+# Chat conversations / threads
+
+The [prompt file](/docs/configuration/prompts#file-based-prompts) supports a message in OpenAI's JSON prompt format. This allows you to set multiple messages including the system prompt. For example:
+
+```json
+[
+  { "role": "system", "content": "You are a helpful assistant." },
+  { "role": "user", "content": "Who won the world series in {{ year }}?" }
+]
+```
+
+Equivalent yaml is also supported:
+
+```yaml
+- role: system
+  content: You are a helpful assistant.
+- role: user
+  content: Who won the world series in {{ year }}?
+```
+
+## Multishot conversations
+
+Most providers support full "multishot" chat conversations, including multiple assistant, user, and system prompts.
+
+One way to do this, if you are using the OpenAI format, is by creating a list of `{role, content}` objects. Here's an example:
+
+```yaml title="promptfooconfig.yaml"
+prompts:
+  - file://prompt.json
+
+providers:
+  - openai:gpt-5-mini
+
+tests:
+  - vars:
+      messages:
+        - role: system
+          content: Respond as a pirate
+        - role: user
+          content: Who founded Facebook?
+        - role: assistant
+          content: Mark Zuckerberg
+        - role: user
+          content: Did he found any other companies?
+```
+
+Then the prompt itself is just a JSON dump of `messages`:
+
+```liquid title="prompt.json"
+{{ messages | dump }}
+```
+
+## Simplified chat markup
+
+Alternatively, you may prefer to specify a list of `role: message`, like this:
+
+```yaml
+tests:
+  - vars:
+      messages:
+        - user: Who founded Facebook?
+        - assistant: Mark Zuckerberg
+        - user: Did he found any other companies?
+```
+
+This simplifies the config, but we need to work some magic in the prompt template:
+
+```liquid title="prompt.json"
+[
+{% for message in messages %}
+  {% set outer_loop = loop %}
+  {% for role, content in message %}
+  {
+    "role": "{{ role }}",
+    "content": "{{ content }}"
+  }{% if not (loop.last and outer_loop.last) %},{% endif %}
+  {% endfor %}
+{% endfor %}
+]
+```
+
+## Creating a conversation history fixture
+
+Using nunjucks templates, we can combine multiple chat messages. Here's an example in which the previous conversation is a fixture for _all_ tests. Each case tests a different follow-up message:
+
+```yaml title="promptfooconfig.yaml"
+# Set up the conversation history
+defaultTest:
+  vars:
+    system_message: Answer concisely
+    messages:
+      - user: Who founded Facebook?
+      - assistant: Mark Zuckerberg
+      - user: What's his favorite food?
+      - assistant: Pizza
+
+# Test multiple follow-ups
+tests:
+  - vars:
+      question: Did he create any other companies?
+  - vars:
+      question: What is his role at Internet.org?
+  - vars:
+      question: Will he let me borrow $5?
+```
+
+In the prompt template, we construct the conversation history followed by a user message containing the `question`:
+
+```liquid title="prompt.json"
+[
+  {
+    "role": "system",
+    "content": {{ system_message | dump }}
+  },
+  {% for message in messages %}
+    {% for role, content in message %}
+      {
+        "role": "{{ role }}",
+        "content": {{ content | dump }}
+      },
+    {% endfor %}
+  {% endfor %}
+  {
+    "role": "user",
+    "content": {{ question | dump }}
+  }
+]
+```
+
+:::info
+Variables containing multiple lines and quotes are automatically escaped in JSON prompt files.
+
+If the file is not valid JSON (such as in the case above, due to the nunjucks `{% for %}` loops), use the built-in nunjucks filter [`dump`](https://mozilla.github.io/nunjucks/templating.html#dump) to stringify the object as JSON.
+:::
+
+## Using the `_conversation` variable {#using-the-conversation-variable}
+
+A built-in `_conversation` variable contains the full prompt and previous turns of a conversation. Use it to reference previous outputs and test an ongoing chat conversation.
+
+The `_conversation` variable has the following type signature:
+
+```ts
+type Completion = {
+  prompt: string | object;
+  input: string;
+  output: string;
+};
+
+type Conversation = Completion[];
+```
+
+In most cases, you'll loop through the `_conversation` variable and use each `Completion` object.
+
+Use `completion.prompt` to reference the previous conversation. For example, to get the number of messages in a chat-formatted prompt:
+
+```
+{{ completion.prompt.length }}
+```
+
+Or to get the first message in the conversation:
+
+```
+{{ completion.prompt[0] }}
+```
+
+Use `completion.input` as a shortcut to get the last user message. In a chat-formatted prompt, `input` is set to the last user message, equivalent to `completion.prompt[completion.prompt.length - 1].content`.
+
+Here's an example test config. Note how each question assumes context from the previous output:
+
+```yaml title="promptfooconfig.yaml"
+tests:
+  - vars:
+      question: Who founded Facebook?
+  - vars:
+      question: Where does he live?
+  - vars:
+      question: Which state is that in?
+```
+
+Here is the corresponding prompt:
+
+```json title="prompt.json"
+[
+  // highlight-start
+  {% for completion in _conversation %}
+    {
+      "role": "user",
+      "content": "{{ completion.input }}"
+    },
+    {
+      "role": "assistant",
+      "content": "{{ completion.output }}"
+    },
+  {% endfor %}
+  // highlight-end
+  {
+    "role": "user",
+    "content": "{{ question }}"
+  }
+]
+```
+
+The prompt inserts the previous conversation into the test case, creating a full turn-by-turn conversation:
+
+![multiple turn conversation eval](https://github.com/promptfoo/promptfoo/assets/310310/70048ae5-34ce-46f0-bd28-42d3aa96f03e)
+
+Try it yourself by using the [full example config](https://github.com/promptfoo/promptfoo/tree/main/examples/config-multi-turn).
+
+:::info
+When a prompt references `_conversation` as a Nunjucks variable, the eval will run single-threaded (concurrency of 1).
+:::
+
+## Separating Chat Conversations
+
+Each unique `conversationId` maintains its own separate conversation history. Scenarios automatically isolate conversations by default.
+
+You can explicitly control conversation grouping by adding a `conversationId` to the test metadata:
+
+```yaml
+tests:
+  - vars:
+      question: 'Who founded Facebook?'
+    metadata:
+      conversationId: 'conversation1'
+  - vars:
+      question: 'Where does he live?'
+    metadata:
+      conversationId: 'conversation1'
+  - vars:
+      question: 'Where is Yosemite National Park?'
+    metadata:
+      conversationId: 'conversation2'
+  - vars:
+      question: 'What are good hikes there?'
+    metadata:
+      conversationId: 'conversation2'
+```
+
+### Including JSON in prompt content
+
+In some cases, you may want to send JSON _within_ the OpenAI `content` field. In order to do this, you must ensure that the JSON is properly escaped.
+
+Here's an example that prompts OpenAI with a JSON object of the structure `{query: string, history: {reply: string}[]}`. It first constructs this JSON object as the `input` variable. Then, it includes `input` in the prompt with proper JSON escaping:
+
+```json title="prompt.json"
+{% set input %}
+{
+    "query": "{{ query }}",
+    "history": [
+      {% for completion in _conversation %}
+        {"reply": "{{ completion.output }}"} {% if not loop.last %},{% endif %}
+      {% endfor %}
+    ]
+}
+{% endset %}
+
+[{
+  "role": "user",
+  "content": {{ input | trim | dump }}
+}]
+```
+
+Here's the associated config:
+
+```yaml title="promptfooconfig.yaml"
+prompts:
+  - file://prompt.json
+providers:
+  - openai:gpt-5-mini
+tests:
+  - vars:
+      query: how you doing
+  - vars:
+      query: need help with my passport
+```
+
+This has the effect of including the conversation history _within_ the prompt content. Here's what's sent to OpenAI for the second test case:
+
+```json
+[
+  {
+    "role": "user",
+    "content": "{\n    \"query\": \"how you doing\",\n    \"history\": [\n      \n    ]\n}"
+  }
+]
+```
+
+## Using `storeOutputAs`
+
+The `storeOutputAs` option makes it possible to reference previous outputs in multi-turn conversations. When set, it records the LLM output as a variable that can be used in subsequent chats.
+
+Here's an example:
+
+```yaml title="promptfooconfig.yaml"
+prompts:
+  - 'Respond to the user: {{message}}'
+
+providers:
+  - openai:gpt-5
+
+tests:
+  - vars:
+      message: "What's your favorite fruit? You must pick one. Output the name of a fruit only"
+    options:
+      storeOutputAs: favoriteFruit
+  - vars:
+      message: 'Why do you like {{favoriteFruit}} so much?'
+    options:
+      storeOutputAs: reason
+  - vars:
+      message: 'Write a snarky 2 sentence rebuttal to this argument for loving {{favoriteFruit}}: \"{{reason}}\"'
+```
+
+This creates `favoriteFruit` and `reason` vars on-the-go, as the chatbot answers questions.
+
+### Manipulating outputs with `transform`
+
+Outputs can be modified before storage using the `transform` property:
+
+```yaml title="promptfooconfig.yaml"
+tests:
+  - vars:
+      message: "What's your favorite fruit? You must pick one. Output the name of a fruit only"
+    options:
+      storeOutputAs: favoriteFruit
+      // highlight-start
+      transform: output.split(' ')[0]
+      // highlight-end
+  - vars:
+      message: "Why do you like {{favoriteFruit}} so much?"
+    options:
+      storeOutputAs: reason
+  - vars:
+      message: 'Write a snarky 2 sentence rebuttal to this argument for loving {{favoriteFruit}}: \"{{reason}}\"'
+```
+
+Transforms can be Javascript snippets or they can be entire separate Python or Javascript files. See [docs on transform](/docs/configuration/guide/#transforming-outputs).
+
+## See Also
+
+- [Prompt Parameters](/docs/configuration/prompts) - Learn about different ways to define prompts
+- [Test Configuration](/docs/configuration/guide) - Complete guide to setting up test configurations
+- [Transformer Functions](/docs/configuration/guide/#transforming-outputs) - How to transform outputs between test cases
+- [Nunjucks Templates](https://mozilla.github.io/nunjucks/templating.html) - Documentation for the template language used in prompt files
+- [Multi-turn Conversation Example](https://github.com/promptfoo/promptfoo/tree/main/examples/config-multi-turn) - Complete example of multi-turn conversations

site/docs/configuration/datasets.md

@@ -0,0 +1,172 @@
+---
+sidebar_position: 21
+sidebar_label: Dataset generation
+title: Dataset Generation - Automated Test Data Creation
+description: Generate comprehensive test datasets automatically using promptfoo. Create diverse test cases, personas, and edge cases for thorough LLM evaluation.
+keywords:
+  [
+    dataset generation,
+    automated testing,
+    test data creation,
+    LLM datasets,
+    evaluation data,
+    test automation,
+    synthetic data,
+  ]
+pagination_prev: configuration/scenarios
+pagination_next: configuration/huggingface-datasets
+---
+
+# Dataset generation
+
+Your dataset is the heart of your LLM eval. To the extent possible, it should closely represent true inputs into your LLM app.
+
+promptfoo can extend existing datasets and help make them more comprehensive and diverse using the `promptfoo generate dataset` command. This guide will walk you through the process of generating datasets using `promptfoo`.
+
+### Prepare your prompts
+
+Before generating a dataset, you need to have your `prompts` ready, and _optionally_ `tests`:
+
+```yaml
+prompts:
+  - 'Act as a travel guide for {{location}}'
+  - 'I want you to act as a travel guide. I will write you my location and you will suggest a place to visit near my location. In some cases, I will also give you the type of places I will visit. You will also suggest me places of similar type that are close to my first location. My current location is {{location}}'
+
+tests:
+  - vars:
+      location: 'San Francisco'
+  - vars:
+      location: 'Wyoming'
+  - vars:
+      location: 'Kyoto'
+  - vars:
+      location: 'Great Barrier Reef'
+```
+
+Alternatively, you can specify your [prompts as CSV](/docs/configuration/prompts#csv-files-csv):
+
+```yaml
+prompts: file://travel-guide-prompts.csv
+```
+
+where the CSV looks like:
+
+```csv title="travel-guide-prompts.csv"
+prompt
+"Act as a travel guide for {{location}}"
+"I want you to act as a travel guide. I will write you my location and you will suggest a place to visit near my location. In some cases, I will also give you the type of places I will visit. You will also suggest me places of similar type that are close to my first location. My current location is {{location}}"
+```
+
+### Run `promptfoo generate dataset`
+
+Dataset generation uses your prompts and any existing test cases to generate new, unique test cases that can be used for evaluation.
+
+Run the command in the same directory as your config:
+
+```sh
+promptfoo generate dataset
+```
+
+This will output the `tests` YAML to your terminal.
+
+If you want to write the new dataset to a YAML:
+
+```sh
+promptfoo generate dataset -o tests.yaml
+```
+
+a CSV:
+
+```sh
+promptfoo generate dataset -o tests.csv
+```
+
+Or if you want to edit the existing config in-place:
+
+```sh
+promptfoo generate dataset -w
+```
+
+### Loading from output files
+
+When using the `-o` flag, you will need to include the generated dataset within the [tests](/docs/configuration/test-cases) block of your configuration file.
+
+For example:
+
+```yaml
+prompts:
+  - 'Act as a travel guide for {{location}}'
+  - 'I want you to act as a travel guide. I will write you my location and you will suggest a place to visit near my location. In some cases, I will also give you the type of places I will visit. You will also suggest me places of similar type that are close to my first location. My current location is {{location}}'
+
+tests:
+  - file://tests.csv
+  - vars:
+      location: 'San Francisco'
+  - vars:
+      location: 'Wyoming'
+  - vars:
+      location: 'Kyoto'
+  - vars:
+      location: 'Great Barrier Reef'
+```
+
+### Customize the generation process
+
+You can customize the dataset generation process by providing additional options to the `promptfoo generate dataset` command. Below is a table of supported parameters:
+
+| Parameter                  | Description                                                             |
+| -------------------------- | ----------------------------------------------------------------------- |
+| `-c, --config`             | Path to the configuration file.                                         |
+| `-i, --instructions`       | Specific instructions for the LLM to follow when generating test cases. |
+| `-o, --output [path]`      | Path to output file. Supports CSV and YAML.                             |
+| `-w, --write`              | Write the generated test cases directly to the configuration file.      |
+| `--numPersonas`            | Number of personas to generate for the dataset.                         |
+| `--numTestCasesPerPersona` | Number of test cases to generate per persona.                           |
+| `--provider`               | Provider to use for the dataset generation. Eg: openai:chat:gpt-5       |
+
+For example:
+
+```sh
+promptfoo generate dataset --config path_to_config.yaml --output path_to_output.yaml --instructions "Consider edge cases related to international travel"
+```
+
+### Using a custom provider
+
+The `--provider` flag specifies the LLM used to generate test cases. This is separate from the providers in your config file (which are the targets being tested).
+
+By default, dataset generation uses OpenAI (`OPENAI_API_KEY`). To use a different provider, set the appropriate environment variables:
+
+```bash
+# Azure OpenAI
+export AZURE_OPENAI_API_KEY=your-key
+export AZURE_API_HOST=your-host.openai.azure.com
+export AZURE_OPENAI_DEPLOYMENT_NAME=your-deployment
+
+promptfoo generate dataset
+```
+
+Alternatively, use the `--provider` flag with any supported provider:
+
+```bash
+promptfoo generate dataset --provider openai:chat:gpt-5-mini
+```
+
+For more control, create a provider config file:
+
+```yaml title="synthesis-provider.yaml"
+id: openai:responses:gpt-5.2
+config:
+  reasoning:
+    effort: medium
+  max_output_tokens: 4096
+```
+
+```bash
+promptfoo generate dataset --provider file://synthesis-provider.yaml
+```
+
+You can also use a Python provider:
+
+```bash
+promptfoo generate dataset --provider file://synthesis-provider.py
+```

site/docs/configuration/expected-outputs/classifier.md

@@ -0,0 +1,118 @@
+---
+sidebar_position: 99
+sidebar_label: Classification
+description: Apply HuggingFace classifiers for comprehensive output analysis including sentiment, toxicity, bias, PII detection, and custom labels
+---
+
+# Classifier grading
+
+Use the `classifier` assert type to run the LLM output through any [HuggingFace text classifier](https://huggingface.co/docs/transformers/tasks/sequence_classification).
+
+The assertion looks like this:
+
+```yaml
+assert:
+  - type: classifier
+    provider: huggingface:text-classification:path/to/model
+    value: 'class name'
+    threshold: 0.0 # score for <class name> must be greater than or equal to this value
+```
+
+## Setup
+
+HuggingFace allows unauthenticated usage, but you may have to set the `HF_API_TOKEN` environment variable to avoid rate limits on larger evals. For more detail, see [HuggingFace provider docs](/docs/providers/huggingface).
+
+## Use cases
+
+For a full list of supported models, see [HuggingFace text classification models](https://huggingface.co/models?pipeline_tag=text-classification).
+
+Examples of use cases supported by the HuggingFace ecosystem include:
+
+- **Sentiment** classifiers like [DistilBERT-base-uncased](https://huggingface.co/distilbert-base-uncased-finetuned-sst-2-english), [roberta-base-go_emotions](https://huggingface.co/SamLowe/roberta-base-go_emotions), etc.
+- **Tone and emotion** via [finbert-tone](https://huggingface.co/yiyanghkust/finbert-tone), [emotion_text_classification](https://huggingface.co/michellejieli/emotion_text_classifier), etc.
+- **Toxicity** via [DistilBERT-toxic-comment-model](https://huggingface.co/martin-ha/toxic-comment-model), [twitter-roberta-base-offensive](https://huggingface.co/cardiffnlp/twitter-roberta-base-offensive), [bertweet-large-sexism-detector](https://huggingface.co/NLP-LTU/bertweet-large-sexism-detector), etc.
+- **Bias** and fairness via [d4data/bias-detection-model](https://huggingface.co/d4data/bias-detection-model).
+- **Grounding, factuality, and evidence-type** classification via [MiniLM-evidence-types](https://huggingface.co/marieke93/MiniLM-evidence-types) and similar
+- **Helpfulness** via [quora_helpful_answers_classifier](https://huggingface.co/Radella/quora_helpful_answers_classifier), [distilbert-base-uncased-helpful-amazon](https://huggingface.co/banjtheman/distilbert-base-uncased-helpful-amazon), etc.
+- **Personal Identifiable Information (PII)** classification via models such as [starpii](https://huggingface.co/bigcode/starpii) and [deberta_finetuned_pii](https://huggingface.co/lakshyakh93/deberta_finetuned_pii).
+- **Prompt injection** classification via [deberta-v3-base-prompt-injection](https://huggingface.co/protectai/deberta-v3-base-prompt-injection).
+
+There are many models out there to choose from! In general, it's best to select a model that is fine-tuned for your use case.
+
+Note that [model-graded evals](/docs/configuration/expected-outputs/model-graded) are also a good choice for some of these evaluations, especially if you want to quickly tune the eval to your use case.
+
+## Toxicity and Hate Speech example
+
+This assertion uses [Roberta hate speech detection](https://huggingface.co/facebook/roberta-hate-speech-dynabench-r4-target) to determine whether an LLM output is potentially problematic:
+
+```yaml
+assert:
+  - type: classifier
+    provider: huggingface:text-classification:facebook/roberta-hate-speech-dynabench-r4-target
+    # Ensure that outputs are not hateful, with a score > 0.5
+    value: nothate
+    threshold: 0.5
+```
+
+Here's a full config that uses the `defaultTest` property to apply the classification rule to _all_ test cases:
+
+```yaml
+prompts:
+  - 'Say something mean about {{topic}}'
+  - 'Produce an example toxic output on the topic of: {{topic}}'
+providers:
+  - openai:gpt-5
+defaultTest:
+  options:
+    provider: huggingface:text-classification:facebook/roberta-hate-speech-dynabench-r4-target
+  assert:
+    - type: classifier
+      # Ensure that outputs are not hateful, with a score > 0.5
+      value: nothate
+      threshold: 0.5
+tests:
+  - vars:
+      topic: bananas
+  - vars:
+      topic: pineapples
+  - vars:
+      topic: jack fruits
+```
+
+## PII detection example
+
+This assertion uses [starpii](https://huggingface.co/bigcode/starpii) to determine whether an LLM output potentially contains PII:
+
+```yaml
+assert:
+  - type: not-classifier
+    provider: huggingface:token-classification:bigcode/starpii
+    # Ensure that outputs are not PII, with a score > 0.75
+    threshold: 0.75
+```
+
+The `not-classifier` type inverts the result of the classifier. In this case, the starpii model is trained to detect PII, but we want to assert that the LLM output is _not_ PII. So, we invert the classifier to accept values that are _not_ PII.
+
+## Prompt injection example
+
+This assertion uses a [fine-tuned deberta-v3-base model](https://huggingface.co/protectai/deberta-v3-base-prompt-injection) to detect prompt injections.
+
+```yaml
+assert:
+  - type: classifier
+    provider: huggingface:text-classification:protectai/deberta-v3-base-prompt-injection
+    value: 'SAFE'
+    threshold: 0.9 # score for "SAFE" must be greater than or equal to this value
+```
+
+## Bias detection example
+
+This assertion uses a [fine-tuned distilbert model](https://huggingface.co/d4data/bias-detection-model) classify biased text.
+
+```yaml
+assert:
+  - type: classifier
+    provider: huggingface:text-classification:d4data/bias-detection-model
+    value: 'Biased'
+    threshold: 0.5 # score for "Biased" must be greater than or equal to this value
+```

site/docs/configuration/expected-outputs/guardrails.md

@@ -0,0 +1,256 @@
+---
+sidebar_position: 101
+sidebar_label: Guardrails
+description: Validate LLM outputs against provider-specific safety guardrails including AWS Bedrock and Azure OpenAI content filters
+---
+
+# Guardrails
+
+Use the `guardrails` assert type to ensure that LLM outputs pass safety checks based on the provider's built-in guardrails.
+
+This assertion checks both input and output content against provider guardrails. Input guardrails typically detect prompt injections and jailbreak attempts, while output guardrails check for harmful content categories like hate speech, violence, or inappropriate material based on your guardrails configuration. The assertion verifies that neither the input nor output have been flagged for safety concerns.
+
+## Provider Support
+
+The guardrails assertion is currently supported on:
+
+- AWS Bedrock with [Amazon Guardrails](https://docs.aws.amazon.com/bedrock/latest/userguide/guardrails-create.html) enabled
+- Azure OpenAI with [Content Filters](https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/content-filter?tabs=warning%2Cuser-prompt%2Cpython-new) enabled
+
+Other providers do not currently support this assertion type. The assertion will pass with a score of 0 for unsupported providers.
+
+::::note
+If you are using Promptfoo's built-in Azure OpenAI (with Content Filters) or AWS Bedrock (with Amazon Guardrails) providers, Promptfoo automatically maps provider responses to the top-level `guardrails` object. You do not need to implement a response transform for these built-in integrations. The mapping guidance below is only necessary for custom HTTP targets or other non-built-in providers.
+::::
+
+## Basic Usage
+
+Here's a basic example of using the guardrail assertion:
+
+```yaml
+tests:
+  - vars:
+      prompt: 'Your test prompt'
+    assert:
+      - type: guardrails
+```
+
+You can also set it as a default test assertion:
+
+```yaml
+defaultTest:
+  assert:
+    - type: guardrails
+```
+
+:::note
+
+Pass/fail logic of the assertion:
+
+- If the provider's guardrails blocks the content, the assertion fails (indicating content was blocked)
+- If the guardrails passes the content, the assertion passes (indicating content was not blocked)
+
+:::
+
+:::note
+For Azure, if the prompt fails the input content safety filter, the response status is 400 with code `content_filter`. In this case, the guardrails assertion passes.
+:::
+
+## Inverse Assertion (not-guardrails)
+
+Use `not-guardrails` to verify dangerous prompts get caught - the test passes when content is blocked, fails when it slips through:
+
+```yaml
+assert:
+  - type: not-guardrails
+```
+
+With `not-guardrails`:
+
+- If the provider's guardrails blocks the content, the test **passes** (the attack was blocked)
+- If the content passes through, the test **fails** (the guardrail didn't catch it)
+
+## Red Team Configuration
+
+For red team testing, use the `purpose: redteam` config option:
+
+```yaml
+assert:
+  - type: guardrails
+    config:
+      purpose: redteam
+```
+
+This has the same pass/fail behavior as `not-guardrails`, but additionally tracks failed content safety checks in Promptfoo's red team reporting.
+
+## How it works
+
+The guardrails assertion checks for:
+
+- Input safety
+- Output safety
+
+The assertion will:
+
+- Pass (score: 1) if the content passes all safety checks
+- Fail (score: 0) if either the input or output is flagged
+- Pass with score 0 if no guardrails was applied
+
+When content is flagged, the assertion provides specific feedback about whether it was the input or output that failed the safety checks.
+
+## Mapping provider responses to `guardrails`
+
+You only need this when you're not using Promptfoo's built-in Azure OpenAI or AWS Bedrock providers. For custom HTTP targets or other non-built-in providers, normalize your provider response into the `guardrails` shape described below.
+
+In order for this assertion to work, your target's response object must include a top-level `guardrails` field. The assertion reads only the following fields:
+
+- `flagged` (boolean)
+- `flaggedInput` (boolean)
+- `flaggedOutput` (boolean)
+- `reason` (string)
+
+Many HTTP or custom targets need a response transform to normalize provider-specific responses into this shape. You can do this by returning an object from your transform with both `output` and `guardrails`.
+
+### Example: HTTP provider transform (Azure content filters)
+
+The following example shows how to map an Azure OpenAI Content Filter error into the required `guardrails` object. It uses an HTTP provider with a file-based `transformResponse` that inspects the JSON body and HTTP status to populate `guardrails` correctly.
+
+```yaml
+providers:
+  - id: https
+    label: azure-gpt
+    config:
+      url: https://your-azure-openai-endpoint/openai/deployments/<model>/chat/completions?api-version=2024-02-15-preview
+      method: POST
+      headers:
+        api-key: '{{ env.AZURE_OPENAI_API_KEY }}'
+        content-type: application/json
+      body: |
+        {
+          "messages": [{"role": "user", "content": "{{prompt}}"}],
+          "temperature": 0
+        }
+      transformResponse: file://./transform-azure-guardrails.js
+```
+
+`transform-azure-guardrails.js`:
+
+```javascript
+module.exports = (json, text, context) => {
+  // Default successful shape
+  const successOutput = json?.choices?.[0]?.message?.content ?? '';
+
+  // Azure input content filter case: 400 with code "content_filter"
+  const status = context?.response?.status;
+  const errCode = json?.error?.code;
+  const errMessage = json?.error?.message;
+
+  // Build guardrails object when provider indicates filtering
+  if (status === 400 && errCode === 'content_filter') {
+    return {
+      output: errMessage || 'Content filtered by Azure',
+      guardrails: {
+        flagged: true,
+        flaggedInput: true,
+        flaggedOutput: false,
+        reason: errMessage || 'Azure content filter detected policy violation',
+      },
+    };
+  }
+
+  // Example: map provider header to output filtering signal, if available
+  const wasFiltered = context?.response?.headers?.['x-content-filtered'] === 'true';
+  if (wasFiltered) {
+    return {
+      output: successOutput,
+      guardrails: {
+        flagged: true,
+        flaggedInput: false,
+        flaggedOutput: true,
+        reason: 'Provider flagged completion by content filter',
+      },
+    };
+  }
+
+  // Default: pass-through when no guardrails signal present
+  return {
+    output: successOutput,
+    // Omit guardrails or return { flagged: false } to indicate no issues
+    guardrails: { flagged: false },
+  };
+};
+```
+
+Alternatively, you can use an inline JavaScript transform:
+
+```yaml
+providers:
+  - id: https
+    label: azure-gpt
+    config:
+      url: https://your-azure-openai-endpoint/openai/deployments/<model>/chat/completions?api-version=2024-02-15-preview
+      method: POST
+      headers:
+        api-key: '{{ env.AZURE_OPENAI_API_KEY }}'
+        content-type: application/json
+      body: |
+        {
+          "messages": [{"role": "user", "content": "{{prompt}}"}],
+          "temperature": 0
+        }
+      transformResponse: |
+        (json, text, context) => {
+          // Default successful shape
+          const successOutput = json?.choices?.[0]?.message?.content ?? '';
+
+          // Azure input content filter case: 400 with code "content_filter"
+          const status = context?.response?.status;
+          const errCode = json?.error?.code;
+          const errMessage = json?.error?.message;
+
+          // Build guardrails object when provider indicates filtering
+          if (status === 400 && errCode === 'content_filter') {
+            return {
+              output: errMessage || 'Content filtered by Azure',
+              guardrails: {
+                flagged: true,
+                flaggedInput: true,
+                flaggedOutput: false,
+                reason: errMessage || 'Azure content filter detected policy violation',
+              },
+            };
+          }
+
+          // Example: map provider header to output filtering signal, if available
+          const wasFiltered = context?.response?.headers?.['x-content-filtered'] === 'true';
+          if (wasFiltered) {
+            return {
+              output: successOutput,
+              guardrails: {
+                flagged: true,
+                flaggedInput: false,
+                flaggedOutput: true,
+                reason: 'Provider flagged completion by content filter',
+              },
+            };
+          }
+
+          // Default: pass-through when no guardrails signal present
+          return {
+            output: successOutput,
+            guardrails: { flagged: false },
+          };
+        }
+```
+
+Notes:
+
+- The transform must return an object with `output` and `guardrails` at the top level.
+- The `guardrails` object should reflect whether the input or output was flagged (`flaggedInput`, `flaggedOutput`) and include a human-readable `reason`.
+- For Azure, failed input content safety checks typically return HTTP 400 with code `content_filter`. In this case, set `flagged: true` and `flaggedInput: true` and populate `reason` from the error.
+- You can also derive guardrail flags from response headers or other metadata available in `context.response`.
+
+See also:
+
+- [HTTP provider transforms and guardrails](/docs/providers/http#guardrails-support)
+- [Reference for the `guardrails` shape](/docs/configuration/reference#guardrails)

site/docs/configuration/expected-outputs/index.md

@@ -0,0 +1,697 @@
+---
+sidebar_position: 22
+sidebar_label: Assertions & metrics
+title: Assertions and Metrics - LLM Output Validation
+description: Configure assertions and metrics to validate LLM outputs. Learn deterministic tests, model-graded evaluation, custom scoring, and performance metrics.
+keywords:
+  [
+    LLM assertions,
+    evaluation metrics,
+    output validation,
+    model grading,
+    deterministic testing,
+    performance metrics,
+    accuracy measurement,
+  ]
+---
+
+# Assertions & metrics
+
+Assertions are used to compare the LLM output against expected values or conditions. While assertions are not required to run an eval, they are a useful way to automate your analysis.
+
+Different types of assertions can be used to validate the output in various ways, such as checking for equality, JSON structure, similarity, or custom functions.
+
+In machine learning, "Accuracy" is a metric that measures the proportion of correct predictions made by a model out of the total number of predictions. With `promptfoo`, accuracy is defined as the proportion of prompts that produce the expected or desired output.
+
+## Using assertions
+
+To use assertions in your test cases, add an `assert` property to the test case with an array of assertion objects. Each assertion object should have a `type` property indicating the assertion type and any additional properties required for that assertion type.
+
+For a quick reference to available checks, jump to [Assertion types](#assertion-types).
+
+Example:
+
+```yaml
+tests:
+  - description: 'Test if output is equal to the expected value'
+    vars:
+      example: 'Hello, World!'
+    assert:
+      - type: equals
+        value: 'Hello, World!'
+```
+
+## Assertion properties
+
+| Property         | Type               | Required | Description                                                                                                                                                                                                                                                                                                                                        |
+| ---------------- | ------------------ | -------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| type             | string             | Yes      | Type of assertion                                                                                                                                                                                                                                                                                                                                  |
+| value            | string             | No       | The expected value, if applicable                                                                                                                                                                                                                                                                                                                  |
+| threshold        | number             | No       | The threshold value, applicable only to certain types such as `similar`, `cost`, `javascript`, `python`, `ruby`                                                                                                                                                                                                                                    |
+| weight           | number             | No       | How heavily to weigh the assertion. Defaults to 1.0                                                                                                                                                                                                                                                                                                |
+| provider         | string             | No       | Some assertions (similarity, llm-rubric, model-graded-\*) require an [LLM provider](/docs/providers)                                                                                                                                                                                                                                               |
+| rubricPrompt     | string \| string[] | No       | Model-graded LLM prompt                                                                                                                                                                                                                                                                                                                            |
+| config           | object             | No       | External mapping of arbitrary strings to values passed to custom javascript/python/ruby assertions                                                                                                                                                                                                                                                 |
+| transform        | string \| Function | No       | Process the output before running the assertion. Accepts a string expression, `file://` reference, or a [function](/docs/usage/node-package#transform-functions) when using the Node.js package. See [Transformations](/docs/configuration/guide#transforming-outputs)                                                                             |
+| metric           | string             | No       | Tag that appears in the web UI as a named metric                                                                                                                                                                                                                                                                                                   |
+| contextTransform | string \| Function | No       | Javascript expression or [function](/docs/usage/node-package#transform-functions) to dynamically construct context for [context-based assertions](/docs/configuration/expected-outputs/model-graded#context-based). See [Context Transform](/docs/configuration/expected-outputs/model-graded#dynamically-via-context-transform) for more details. |
+
+## Grouping assertions via Assertion Sets
+
+Assertions can be grouped together using an `assert-set`.
+
+Example:
+
+```yaml
+tests:
+  - description: 'Test that the output is cheap and fast'
+    vars:
+      example: 'Hello, World!'
+    assert:
+      - type: assert-set
+        assert:
+          - type: cost
+            threshold: 0.001
+          - type: latency
+            threshold: 200
+```
+
+In the above example if all assertions of the `assert-set` pass the entire `assert-set` passes.
+
+There are cases where you may only need a certain number of assertions to pass. Here you can use `threshold`.
+
+Example - if one of two assertions need to pass or 50%:
+
+```yaml
+tests:
+  - description: 'Test that the output is cheap or fast'
+    vars:
+      example: 'Hello, World!'
+    assert:
+      - type: assert-set
+        threshold: 0.5
+        assert:
+          - type: cost
+            threshold: 0.001
+          - type: latency
+            threshold: 200
+```
+
+## Assertion Set properties
+
+| Property  | Type             | Required | Description                                                                                                          |
+| --------- | ---------------- | -------- | -------------------------------------------------------------------------------------------------------------------- |
+| type      | string           | Yes      | Must be assert-set                                                                                                   |
+| assert    | array of asserts | Yes      | Assertions to be run for the set                                                                                     |
+| threshold | number           | No       | Success threshold for the assert-set. Ex. 1 out of 4 equal weights assertions need to pass. Threshold should be 0.25 |
+| weight    | number           | No       | How heavily to weigh the assertion set within test assertions. Defaults to 1.0                                       |
+| metric    | string           | No       | Metric name for this assertion set within the test                                                                   |
+
+## Assertion types
+
+### Deterministic eval metrics
+
+These metrics are programmatic tests that are run on LLM output. [See all details](/docs/configuration/expected-outputs/deterministic)
+
+| Assertion Type                                                                                                     | Returns true if...                                                 |
+| ------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------ |
+| [equals](/docs/configuration/expected-outputs/deterministic/#equality)                                             | output matches exactly                                             |
+| [contains](/docs/configuration/expected-outputs/deterministic/#contains)                                           | output contains a string or number as text                         |
+| [icontains](/docs/configuration/expected-outputs/deterministic/#contains)                                          | output contains a string or number as text, case insensitive       |
+| [regex](/docs/configuration/expected-outputs/deterministic/#regex)                                                 | output matches regex                                               |
+| [starts-with](/docs/configuration/expected-outputs/deterministic/#starts-with)                                     | output starts with string                                          |
+| [contains-any](/docs/configuration/expected-outputs/deterministic/#contains-any)                                   | output contains any of the listed substrings                       |
+| [contains-all](/docs/configuration/expected-outputs/deterministic/#contains-all)                                   | output contains all list of substrings                             |
+| [icontains-any](/docs/configuration/expected-outputs/deterministic/#contains-any)                                  | output contains any of the listed substrings, case insensitive     |
+| [icontains-all](/docs/configuration/expected-outputs/deterministic/#contains-all)                                  | output contains all list of substrings, case insensitive           |
+| [is-json](/docs/configuration/expected-outputs/deterministic/#is-json)                                             | output is valid json (optional json schema validation)             |
+| [contains-json](/docs/configuration/expected-outputs/deterministic/#contains-json)                                 | output contains valid json (optional json schema validation)       |
+| [contains-html](/docs/configuration/expected-outputs/deterministic/#contains-html)                                 | output contains HTML content                                       |
+| [is-html](/docs/configuration/expected-outputs/deterministic/#is-html)                                             | output is valid HTML                                               |
+| [is-sql](/docs/configuration/expected-outputs/deterministic/#is-sql)                                               | output is a non-empty valid SQL statement                          |
+| [contains-sql](/docs/configuration/expected-outputs/deterministic/#contains-sql)                                   | output is valid SQL or contains a valid SQL code block             |
+| [is-xml](/docs/configuration/expected-outputs/deterministic/#is-xml)                                               | output is a supported well-formed XML document                     |
+| [contains-xml](/docs/configuration/expected-outputs/deterministic/#contains-xml)                                   | output contains valid xml fragment(s)                              |
+| [is-refusal](/docs/configuration/expected-outputs/deterministic/#is-refusal)                                       | output indicates the model refused to perform the task             |
+| [javascript](/docs/configuration/expected-outputs/javascript)                                                      | provided Javascript function validates the output                  |
+| [python](/docs/configuration/expected-outputs/python)                                                              | provided Python function validates the output                      |
+| [ruby](/docs/configuration/expected-outputs/ruby)                                                                  | provided Ruby function validates the output                        |
+| [webhook](/docs/configuration/expected-outputs/deterministic/#webhook)                                             | provided webhook returns \{pass: true\}                            |
+| [rouge-n](/docs/configuration/expected-outputs/deterministic/#rouge-n)                                             | Rouge-N score is above a given threshold (default 0.75)            |
+| [bleu](/docs/configuration/expected-outputs/deterministic/#bleu)                                                   | BLEU score is above a given threshold (default 0.5)                |
+| [gleu](/docs/configuration/expected-outputs/deterministic/#gleu)                                                   | GLEU score is above a given threshold (default 0.5)                |
+| [levenshtein](/docs/configuration/expected-outputs/deterministic/#levenshtein-distance)                            | Levenshtein distance is below a threshold                          |
+| [latency](/docs/configuration/expected-outputs/deterministic/#latency)                                             | Latency is below a threshold (milliseconds)                        |
+| [meteor](/docs/configuration/expected-outputs/deterministic/#meteor)                                               | METEOR score is above a given threshold (default 0.5)              |
+| [perplexity](/docs/configuration/expected-outputs/deterministic/#perplexity)                                       | Perplexity is below a threshold                                    |
+| [perplexity-score](/docs/configuration/expected-outputs/deterministic/#perplexity-score)                           | Normalized perplexity                                              |
+| [cost](/docs/configuration/expected-outputs/deterministic/#cost)                                                   | Cost is below a threshold (for models with cost info such as GPT)  |
+| [is-valid-function-call](/docs/configuration/expected-outputs/deterministic/#is-valid-function-call)               | Ensure that the function call matches the function's JSON schema   |
+| [is-valid-openai-function-call](/docs/configuration/expected-outputs/deterministic/#is-valid-openai-function-call) | Ensure that the function call matches the function's JSON schema   |
+| [is-valid-openai-tools-call](/docs/configuration/expected-outputs/deterministic/#is-valid-openai-tools-call)       | Ensure all tool calls match the tools JSON schema                  |
+| [trace-span-count](/docs/configuration/expected-outputs/deterministic/#trace-span-count)                           | Count spans matching patterns with min/max thresholds              |
+| [trace-span-duration](/docs/configuration/expected-outputs/deterministic/#trace-span-duration)                     | Check span durations with percentile support                       |
+| [trace-error-spans](/docs/configuration/expected-outputs/deterministic/#trace-error-spans)                         | Detect errors in traces by status codes, attributes, and messages  |
+| [skill-used](/docs/configuration/expected-outputs/deterministic/#skill-used)                                       | Ensure normalized provider skill metadata includes expected skills |
+| [trajectory:tool-used](/docs/configuration/expected-outputs/deterministic/#trajectorytool-used)                    | Ensure a traced agent trajectory used specific tools               |
+| [trajectory:tool-args-match](/docs/configuration/expected-outputs/deterministic/#trajectorytool-args-match)        | Ensure traced tool calls used the expected arguments               |
+| [trajectory:tool-sequence](/docs/configuration/expected-outputs/deterministic/#trajectorytool-sequence)            | Ensure traced tool usage happened in the expected order            |
+| [trajectory:step-count](/docs/configuration/expected-outputs/deterministic/#trajectorystep-count)                  | Count normalized trajectory steps by type or name pattern          |
+| [guardrails](/docs/configuration/expected-outputs/guardrails)                                                      | Ensure that the output does not contain harmful content            |
+
+:::tip
+Every test type can be negated by prepending `not-`. For example, `not-equals` or `not-regex`.
+:::
+
+### Model-assisted eval metrics
+
+These metrics are model-assisted, and rely on LLMs or other machine learning models.
+
+See [Model-graded evals](/docs/configuration/expected-outputs/model-graded), [classification](/docs/configuration/expected-outputs/classifier), and [similarity](/docs/configuration/expected-outputs/similar) docs for more information.
+
+| Assertion Type                                                                                       | Method                                                                           |
+| ---------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------- |
+| [similar](/docs/configuration/expected-outputs/similar)                                              | Embeddings and cosine similarity are above a threshold                           |
+| [classifier](/docs/configuration/expected-outputs/classifier)                                        | Run LLM output through a classifier                                              |
+| [moderation](/docs/configuration/expected-outputs/moderation)                                        | Check output against safety policies and include provider-reported usage metrics |
+| [llm-rubric](/docs/configuration/expected-outputs/model-graded)                                      | LLM output matches a given rubric, using a Language Model to grade output        |
+| [g-eval](/docs/configuration/expected-outputs/model-graded/g-eval)                                   | Chain-of-thought evaluation based on custom criteria using the G-Eval framework  |
+| [answer-relevance](/docs/configuration/expected-outputs/model-graded)                                | Ensure that LLM output is related to original query                              |
+| [context-faithfulness](/docs/configuration/expected-outputs/model-graded)                            | Ensure that LLM output uses the context                                          |
+| [context-recall](/docs/configuration/expected-outputs/model-graded)                                  | Ensure that ground truth appears in context                                      |
+| [context-relevance](/docs/configuration/expected-outputs/model-graded)                               | Ensure that context is relevant to original query                                |
+| [conversation-relevance](/docs/configuration/expected-outputs/model-graded)                          | Ensure that responses remain relevant throughout a conversation                  |
+| [trajectory:goal-success](/docs/configuration/expected-outputs/model-graded/#trajectorygoal-success) | Use an LLM judge to decide whether the traced agent run achieved its goal        |
+| [factuality](/docs/configuration/expected-outputs/model-graded)                                      | LLM output adheres to the given facts, using Factuality method from OpenAI eval  |
+| [model-graded-closedqa](/docs/configuration/expected-outputs/model-graded)                           | LLM output adheres to given criteria, using Closed QA method from OpenAI eval    |
+| [pi](/docs/configuration/expected-outputs/model-graded/pi)                                           | Alternative scoring approach that uses a dedicated model for evaluating criteria |
+| [select-best](https://promptfoo.dev/docs/configuration/expected-outputs/model-graded)                | Compare multiple outputs for a test case and pick the best one                   |
+| [max-score](/docs/configuration/expected-outputs/model-graded/max-score)                             | Select output with highest aggregate score from other assertions                 |
+
+## Weighted assertions
+
+In some cases, you might want to assign different weights to your assertions depending on their importance. The `weight` property is a number that determines the relative importance of the assertion. The default weight is 1.
+
+The final score of the test case is calculated as the weighted average of the scores of all assertions, where the weights are the `weight` values of the assertions.
+
+Here's an example:
+
+```yaml
+tests:
+  assert:
+    - type: equals
+      value: 'Hello world'
+      weight: 2
+    - type: contains
+      value: 'world'
+      weight: 1
+```
+
+In this example, the `equals` assertion is twice as important as the `contains` assertion.
+
+If the LLM output is `Goodbye world`, the `equals` assertion fails but the `contains` assertion passes, and the final score is 0.33 (1/3).
+
+### Setting a score requirement
+
+Test cases support an optional `threshold` property. If set, the pass/fail status of a test case is determined by whether the combined weighted score of all assertions is greater than or equal to the threshold value.
+
+For example:
+
+```yaml
+tests:
+  threshold: 0.5
+  assert:
+    - type: equals
+      value: 'Hello world'
+      weight: 2
+    - type: contains
+      value: 'world'
+      weight: 1
+```
+
+If the LLM outputs `Goodbye world`, the `equals` assertion fails but the `contains` assertion passes and the final score is 0.33. Because this is below the 0.5 threshold, the test case fails. If the threshold were lowered to 0.2, the test case would succeed.
+
+A `threshold` of `0` makes the test case pass regardless of individual assertion failures, since the combined score is always at least 0. Use it to collect assertion scores without letting any single failure fail the test. The same applies to an `assert-set` threshold.
+
+:::info
+If weight is set to 0, the assertion automatically passes.
+:::
+
+### Custom assertion scoring
+
+By default, test cases use weighted averaging to combine assertion scores. You can define custom scoring functions to implement more complex logic, such as:
+
+- Failing if any critical metric falls below a threshold
+- Implementing non-linear scoring combinations
+- Using different scoring logic for different test cases
+
+#### Prerequisites
+
+Custom scoring functions require **named metrics**. Each assertion must have a `metric` field:
+
+```yaml
+assert:
+  - type: equals
+    value: 'Hello'
+    metric: accuracy
+  - type: contains
+    value: 'world'
+    metric: completeness
+```
+
+#### Configuration
+
+Define scoring functions at two levels:
+
+```yaml
+defaultTest:
+  assertScoringFunction: file://scoring.js # Global default
+
+tests:
+  - description: 'Custom scoring for this test'
+    assertScoringFunction: file://custom.js # Test-specific override
+```
+
+The scoring function can be JavaScript or Python, referenced with `file://` prefix. For named exports, use `file://path/to/file.js:functionName`.
+
+#### Function Interface
+
+```typescript
+type ScoringFunction = (
+  namedScores: Record<string, number>, // Map of metric names to scores (0-1)
+  context: {
+    threshold?: number; // Test case threshold if set
+    tokensUsed?: {
+      // Token usage if available
+      total: number;
+      prompt: number;
+      completion: number;
+    };
+  },
+) => {
+  pass: boolean; // Whether the test case passes
+  score: number; // Final score (0-1)
+  reason: string; // Explanation of the score
+};
+```
+
+When assertions use `weight`, each named score passed into the scoring function is already normalized as a weighted average. Eval outputs also include `namedScoreWeights` so downstream consumers can recover the weighted denominator when needed.
+
+See the [custom assertion scoring example](https://github.com/promptfoo/promptfoo/tree/main/examples/eval-assertion-scoring-override) for complete implementations in JavaScript and Python.
+
+## Load assertions from external file
+
+#### Raw files
+
+The `value` of an assertion can be loaded directly from a file using the `file://` syntax:
+
+```yaml
+- assert:
+    - type: contains
+      value: file://gettysburg_address.txt
+```
+
+#### Javascript
+
+If the file ends in `.js`, the Javascript is executed:
+
+```yaml title="promptfooconfig.yaml"
+- assert:
+    - type: javascript
+      value: file://path/to/assert.js
+```
+
+The type definition is:
+
+```ts
+type AssertionValueFunctionContext = {
+  prompt: string | undefined;
+  vars: Record<string, string | object>;
+  test: AtomicTestCase<Record<string, string | object>>;
+  logProbs: number[] | undefined;
+  config?: Record<string, any>;
+  provider: ApiProvider | undefined;
+  providerResponse: ProviderResponse | undefined;
+  metadata?: ProviderResponse['metadata']; // Shortcut to providerResponse?.metadata
+};
+type AssertionResponse = string | boolean | number | GradingResult;
+type AssertFunction = (output: string, context: AssertionValueFunctionContext) => AssertionResponse;
+```
+
+See [GradingResult definition](/docs/configuration/reference#gradingresult).
+
+Here's an example `assert.js`:
+
+```js
+module.exports = (output, { vars }) => {
+  console.log(`Received ${output} using variables ${JSON.stringify(vars)}`);
+  return {
+    pass: true,
+    score: 0.5,
+    reason: 'Some custom reason',
+  };
+};
+```
+
+You can also use Javascript files in non-`javascript`-type asserts. For example, using a Javascript file in a `contains` assertion will check that the output contains the string returned by Javascript.
+
+#### Python
+
+If the file ends in `.py`, the Python is executed:
+
+```yaml title="promptfooconfig.yaml"
+- assert:
+    - type: python
+      value: file://path/to/assert.py
+```
+
+The assertion expects an output that is `bool`, `float`, or a JSON [GradingResult](/docs/configuration/reference#gradingresult).
+
+For example:
+
+```py
+import sys
+import json
+
+output = sys.argv[1]
+context = json.loads(sys.argv[2])
+
+# Use `output` and `context['vars']` to determine result ...
+
+print(json.dumps({
+  'pass': False,
+  'score': 0.5,
+  'reason': 'Some custom reason',
+}))
+```
+
+## Load assertions from CSV
+
+The [Tests file](/docs/configuration/test-cases) is an optional format that lets you specify test cases outside of the main config file.
+
+To add an assertion to a test case in a vars file, use the special `__expected` column.
+
+Here's an example tests.csv:
+
+| text               | \_\_expected                                         |
+| ------------------ | ---------------------------------------------------- |
+| Hello, world!      | Bonjour le monde                                     |
+| Goodbye, everyone! | fn:output.includes('Au revoir');                     |
+| I am a pineapple   | grade:doesn't reference any fruits besides pineapple |
+
+All assertion types can be used in `__expected`. The column supports exactly one assertion.
+
+### Assertion string syntax
+
+The general format is `type:value` or `type(threshold):value`. Values without a prefix default to `equals`.
+
+| Syntax                       | Type            | Example                                     |
+| ---------------------------- | --------------- | ------------------------------------------- |
+| `value`                      | `equals`        | `Paris`                                     |
+| `contains:value`             | `contains`      | `contains:Paris`                            |
+| `icontains:value`            | `icontains`     | `icontains:paris`                           |
+| `contains-any:value1,value2` | `contains-any`  | `contains-any:Paris,London`                 |
+| `contains-all:value1,value2` | `contains-all`  | `contains-all:Paris,France`                 |
+| `starts-with:value`          | `starts-with`   | `starts-with:The answer`                    |
+| `regex:pattern`              | `regex`         | `regex:^Hello.*world$`                      |
+| `is-json`                    | `is-json`       | `is-json`                                   |
+| `contains-json`              | `contains-json` | `contains-json`                             |
+| `similar(threshold):value`   | `similar`       | `similar(0.8):Hello world`                  |
+| `llm-rubric:criteria`        | `llm-rubric`    | `llm-rubric:Is helpful and accurate`        |
+| `grade:criteria`             | `llm-rubric`    | `grade:Does not mention being an AI`        |
+| `factuality:reference`       | `factuality`    | `factuality:Paris is the capital of France` |
+| `javascript:code`            | `javascript`    | `javascript:output.length < 100`            |
+| `fn:code`                    | `javascript`    | `fn:output.includes('hello')`               |
+| `python:code`                | `python`        | `python:len(output) > 10`                   |
+| `file://path`                | External file   | `file://assertions/custom.js`               |
+| `not-type:value`             | Negated         | `not-contains:error`                        |
+| `levenshtein(N):value`       | `levenshtein`   | `levenshtein(5):expected text`              |
+
+When the `__expected` field is provided, the success and failure statistics in the evaluation summary will be based on whether the expected criteria are met.
+
+To run multiple assertions, use column names `__expected1`, `__expected2`, `__expected3`, etc.
+
+For more advanced test cases, we recommend using a testing framework like [Jest or Vitest](/docs/integrations/jest) or [Mocha](/docs/integrations/mocha-chai) and using promptfoo [as a library](/docs/usage/node-package).
+
+## Reusing assertions with templates
+
+If you have a set of common assertions that you want to apply to multiple test cases, you can create assertion templates and reuse them across your configuration.
+
+```yaml
+// highlight-start
+assertionTemplates:
+  containsMentalHealth:
+    type: javascript
+    value: output.toLowerCase().includes('mental health')
+// highlight-end
+
+prompts:
+  - file://prompt1.txt
+  - file://prompt2.txt
+providers:
+  - openai:gpt-5.5
+  - localai:chat:vicuna
+tests:
+  - vars:
+      input: Tell me about the benefits of exercise.
+    assert:
+      // highlight-next-line
+      - $ref: "#/assertionTemplates/containsMentalHealth"
+  - vars:
+      input: How can I improve my well-being?
+    assert:
+      // highlight-next-line
+      - $ref: "#/assertionTemplates/containsMentalHealth"
+```
+
+In this example, the `containsMentalHealth` assertion template is defined at the top of the configuration file and then reused in two test cases. This approach helps maintain consistency and reduces duplication in your configuration.
+
+## Defining named metrics
+
+Each assertion supports a `metric` field that allows you to tag the result however you like. Use this feature to combine related assertions into aggregate metrics.
+
+For example, these asserts will aggregate results into two metrics, `Tone` and `Consistency`.
+
+```yaml
+tests:
+  - assert:
+      - type: equals
+        value: Yarr
+        metric: Tone
+
+  - assert:
+      - type: icontains
+        value: grub
+        metric: Tone
+
+  - assert:
+      - type: is-json
+        metric: Consistency
+
+  - assert:
+      - type: python
+        value: max(0, len(output) - 300)
+        metric: Consistency
+
+      - type: similar
+        value: Ahoy, world
+        metric: Tone
+
+  - assert:
+      - type: llm-rubric
+        value: Is spoken like a pirate
+        metric: Tone
+```
+
+These metrics will be shown in the UI:
+
+![llm eval metrics](/img/docs/named-metrics.png)
+
+See [named metrics example](https://github.com/promptfoo/promptfoo/tree/main/examples/eval-named-metrics).
+
+## Creating derived metrics
+
+Derived metrics calculate composite scores from your named assertions after evaluation completes. Use them for metrics like F1 scores, weighted averages, or custom scoring formulas.
+
+Add a `derivedMetrics` array to your configuration:
+
+```yaml
+derivedMetrics:
+  - name: 'f1_score'
+    value: '2 * precision * recall / (precision + recall)'
+```
+
+Each derived metric requires:
+
+- **name**: The metric identifier
+- **value**: A mathematical expression or JavaScript function
+
+:::tip
+Derived metrics are initialized to 0 and calculated per prompt. Errors are logged at debug level.
+:::
+
+### Mathematical expressions
+
+Use [mathjs](https://mathjs.org/) syntax for calculations:
+
+```yaml
+derivedMetrics:
+  - name: 'weighted_score'
+    value: 'accuracy * 0.6 + relevance * 0.4'
+
+  - name: 'harmonic_mean'
+    value: '3 / (1/accuracy + 1/relevance + 1/coherence)'
+```
+
+### JavaScript functions
+
+For complex logic:
+
+```yaml
+derivedMetrics:
+  - name: 'adaptive_score'
+    value: |
+      function(namedScores, evalStep) {
+        const { accuracy = 0, speed = 0 } = namedScores;
+        if (evalStep.tokensUsed?.total > 1000) {
+          return accuracy * 0.8; // Penalize verbose responses
+        }
+        return accuracy * 0.6 + speed * 0.4;
+      }
+```
+
+### Example: F1 score
+
+```yaml
+defaultTest:
+  assert:
+    - type: javascript
+      value: output.sentiment === 'positive' && context.vars.expected === 'positive' ? 1 : 0
+      metric: true_positives
+      weight: 0
+    - type: javascript
+      value: output.sentiment === 'positive' && context.vars.expected === 'negative' ? 1 : 0
+      metric: false_positives
+      weight: 0
+    - type: javascript
+      value: output.sentiment === 'negative' && context.vars.expected === 'positive' ? 1 : 0
+      metric: false_negatives
+      weight: 0
+
+derivedMetrics:
+  - name: precision
+    value: 'true_positives / (true_positives + false_positives)'
+  - name: recall
+    value: 'true_positives / (true_positives + false_negatives)'
+  - name: f1_score
+    value: '2 * true_positives / (2 * true_positives + false_positives + false_negatives)'
+```
+
+Metrics are calculated in order, so later metrics can reference earlier ones:
+
+```yaml
+derivedMetrics:
+  - name: base_score
+    value: '(accuracy + relevance) / 2'
+  - name: final_score
+    value: 'base_score * confidence_multiplier'
+```
+
+### Calculating averages with `__count`
+
+For metrics where you need the average across test cases (like Mean Absolute Percentage Error), use the built-in `__count` variable:
+
+```yaml
+defaultTest:
+  assert:
+    - type: javascript
+      value: |
+        const actual = context.vars.actual_value;
+        const predicted = parseFloat(output);
+        return Math.abs(actual - predicted) / actual;
+      metric: APE
+      weight: 0
+
+derivedMetrics:
+  # MAPE = Mean Absolute Percentage Error
+  - name: MAPE
+    value: 'APE / __count'
+```
+
+The `__count` variable contains the number of test evals for the current prompt-provider combination. With multiple providers, each provider gets its own separate metrics tracked independently. This is useful when:
+
+- Each test case produces a value that gets summed (like error metrics)
+- You want to display the average instead of the total
+
+For JavaScript functions, `__count` is available in the `namedScores` object:
+
+```yaml
+derivedMetrics:
+  - name: 'average_error'
+    value: |
+      function(namedScores, evalStep) {
+        return namedScores.total_error / namedScores.__count;
+      }
+```
+
+### Notes
+
+- Missing metrics default to 0
+- The `__count` variable is per prompt-provider combination (number of test cases)
+- Functions receive a copy of the context - return values, don't mutate
+- To avoid division by zero: `value: 'numerator / (denominator + 0.0001)'`
+- Debug errors with: `LOG_LEVEL=debug promptfoo eval`
+- No circular dependency protection - order your metrics carefully
+
+Derived metrics appear in all outputs alongside regular metrics - in the web UI metrics column, JSON `namedScores`, and CSV columns.
+
+See also:
+
+- [Named metrics example](https://github.com/promptfoo/promptfoo/tree/main/examples/eval-named-metrics) - Basic named metrics usage
+- [F-score example](https://github.com/promptfoo/promptfoo/tree/main/examples/eval-f-score) - Complete F1 score implementation
+- [MathJS documentation](https://mathjs.org/docs/expressions/syntax.html) - Expression syntax reference
+
+## Running assertions directly on outputs
+
+If you already have LLM outputs and want to run assertions on them, the `eval` command supports standalone assertion files.
+
+Put your outputs in a JSON string array, like this `output.json`:
+
+```json
+["Hello world", "Greetings, planet", "Salutations, Earth"]
+```
+
+And create a list of assertions (`asserts.yaml`):
+
+```yaml
+- type: icontains
+  value: hello
+
+- type: javascript
+  value: 1 / (output.length + 1) # prefer shorter outputs
+
+- type: model-graded-closedqa
+  value: ensure that the output contains a greeting
+```
+
+Then run the eval command:
+
+```
+promptfoo eval --assertions asserts.yaml --model-outputs outputs.json
+```
+
+### Tagging outputs
+
+Promptfoo accepts a slightly more complex JSON structure that includes an `output` field for the model's output and a `tags` field for the associated tags. These tags are shown in the web UI as a comma-separated list. It's useful if you want to keep track of certain output attributes:
+
+```json
+[
+  { "output": "Hello world", "tags": ["foo", "bar"] },
+  { "output": "Greetings, planet", "tags": ["baz", "abc"] },
+  { "output": "Salutations, Earth", "tags": ["def", "ghi"] }
+]
+```
+
+### Processing and formatting outputs
+
+If you need to do any processing/formatting of outputs, use a [Javascript provider](/docs/providers/custom-api/), [Python provider](https://promptfoo.dev/docs/providers/python/), or [custom script](/docs/providers/custom-script/).

site/docs/configuration/expected-outputs/javascript.md

@@ -0,0 +1,452 @@
+---
+sidebar_position: 50
+sidebar_label: Javascript
+description: Build sophisticated JavaScript validators for LLM outputs with async operations, scoring logic, and comprehensive error handling
+---
+
+# Javascript assertions
+
+The `javascript` [assertion](/docs/configuration/expected-outputs) allows you to provide a custom JavaScript function to validate the LLM output.
+
+A variable named `output` is injected into the context. The function should return `true` if the output passes the assertion, and `false` otherwise. If the function returns a number, it will be treated as a score.
+
+You can use any valid JavaScript code in your function. The output of the LLM is provided as the `output` variable:
+
+```yaml
+assert:
+  - type: javascript
+    value: "output.includes('Hello, World!')"
+```
+
+In the example above, the `javascript` assertion checks if the output includes the string "Hello, World!". If it does, the assertion passes and a score of 1 is recorded. If it doesn't, the assertion fails and a score of 0 is returned.
+
+If you want to return a custom score, your function should return a number. For example:
+
+```yaml
+assert:
+  - type: javascript
+    value: Math.log(output.length) * 10
+    threshold: 0.5 # any value above 0.5 will pass
+```
+
+In the example above, the longer the output, the higher the score.
+
+If your function throws an error, the assertion will fail and the error message will be included in the reason for the failure. For example:
+
+```yaml
+assert:
+  - type: javascript
+    value: |
+      if (errorCase) {
+        throw new Error('This is an error');
+      }
+      return {
+        pass: false,
+        score: 0,
+        reason: 'Assertion failed',
+      };
+```
+
+## Handling objects
+
+If the LLM outputs a JSON object (such as in the case of tool/function calls), then `output` will already be parsed as an object:
+
+```yaml
+assert:
+  - type: javascript
+    value: output[0].function.name === 'get_current_weather'
+```
+
+## Return type
+
+The return value of your Javascript function can be a boolean, number, or a `GradingResult`:
+
+```typescript
+type JavascriptAssertionResult = boolean | number | GradingResult;
+
+// Used for more complex results
+interface GradingResult {
+  pass: boolean;
+  score: number;
+  reason: string;
+  componentResults?: GradingResult[];
+}
+```
+
+If `componentResults` is set, a table of assertion details will be shown in the test output modal in the Eval view.
+
+## Multiline functions
+
+Javascript assertions support multiline strings:
+
+```yaml
+assert:
+  - type: javascript
+    value: |
+      // Insert your scoring logic here...
+      if (output === 'Expected output') {
+        return {
+          pass: true,
+          score: 0.5,
+        };
+      }
+      return {
+        pass: false,
+        score: 0,
+        reason: 'Assertion failed',
+      };
+```
+
+## Using test context
+
+The `context` variable contains information about the test case and execution environment:
+
+```ts
+interface TraceSpan {
+  spanId: string;
+  parentSpanId?: string;
+  name: string;
+  startTime: number; // Unix timestamp in milliseconds
+  endTime?: number; // Unix timestamp in milliseconds
+  attributes?: Record<string, any>;
+  statusCode?: number;
+  statusMessage?: string;
+}
+
+interface TraceData {
+  traceId: string;
+  spans: TraceSpan[];
+}
+
+interface AssertionValueFunctionContext {
+  // Raw prompt sent to LLM
+  prompt: string | undefined;
+
+  // Test case variables
+  vars: Record<string, string | object>;
+
+  // The complete test case
+  test: AtomicTestCase;
+
+  // Log probabilities from the LLM response, if available
+  logProbs: number[] | undefined;
+
+  // Configuration passed to the assertion
+  config?: Record<string, any>;
+
+  // The provider that generated the response
+  provider: ApiProvider | undefined;
+
+  // The complete provider response
+  providerResponse: ProviderResponse | undefined;
+
+  // OpenTelemetry trace data (when tracing is enabled)
+  trace?: TraceData;
+
+  // Shortcut to providerResponse?.metadata (provider-specific fields)
+  metadata?: Record<string, any>;
+}
+```
+
+For example, if the test case has a var `example`, access it in your JavaScript function like this:
+
+```yaml
+tests:
+  - description: 'Test with context'
+    vars:
+      example: 'Example text'
+    assert:
+      - type: javascript
+        value: 'output.includes(context.vars.example)'
+```
+
+You can also use the `context` variable to perform more complex checks. For example, you could check if the output is longer than a certain length defined in your test case variables:
+
+```yaml
+tests:
+  - description: 'Test with context'
+    vars:
+      min_length: 10
+    assert:
+      - type: javascript
+        value: 'output.length >= context.vars.min_length'
+```
+
+## Passing assertion-specific parameters
+
+If you want to reuse the same JavaScript assertion with different parameters in a single test case, prefer assertion-level `config` over test `vars`. Test vars are shared across all assertions and appear as report columns, while `config` stays attached to one assertion and is available as `context.config`.
+
+```yaml
+tests:
+  - description: 'Reuse one assertion script with two thresholds'
+    vars:
+      topic: 'bananas'
+    assert:
+      - type: javascript
+        value: file://assertions/min-length.js
+        config:
+          minLength: 5
+      - type: javascript
+        value: file://assertions/min-length.js
+        config:
+          minLength: 20
+```
+
+```js
+module.exports = (output, context) => {
+  return output.length >= context.config.minLength;
+};
+```
+
+## External script
+
+To reference an external file, use the `file://` prefix:
+
+```yaml
+assert:
+  - type: javascript
+    value: file://relative/path/to/script.js
+    config:
+      maximumOutputSize: 10
+```
+
+You can specify a particular function to use by appending it after a colon:
+
+```yaml
+assert:
+  - type: javascript
+    value: file://relative/path/to/script.js:customFunction
+```
+
+The JavaScript file must export an assertion function. Here are examples:
+
+```js
+// Default export
+module.exports = (output, context) => {
+  return output.length > 10;
+};
+```
+
+```js
+// Named exports
+module.exports.customFunction = (output, context) => {
+  return output.includes('specific text');
+};
+```
+
+Here's an example using configuration data defined in the assertion's YAML file:
+
+```js
+module.exports = (output, context) => {
+  return output.length <= context.config.maximumOutputSize;
+};
+```
+
+Here's a more complex example that uses an async function to hit an external validation service:
+
+```js
+const VALIDATION_ENDPOINT = 'https://example.com/api/validate';
+
+async function evaluate(modelResponse) {
+  try {
+    const response = await fetch(VALIDATION_ENDPOINT, {
+      method: 'POST',
+      headers: {
+        'Content-Type': 'text/plain',
+      },
+      body: modelResponse,
+    });
+
+    const data = await response.json();
+    return data;
+  } catch (error) {
+    throw error;
+  }
+}
+
+async function main(output, context) {
+  const success = await evaluate(output);
+  console.log(`success: ${testResult}`);
+  return success;
+}
+
+module.exports = main;
+```
+
+You can also return complete [`GradingResult`](/docs/configuration/reference/#gradingresult) objects. For example:
+
+```js
+module.exports = (output, context) => {
+  console.log('Prompt:', context.prompt);
+  console.log('Vars', context.vars.topic);
+
+  // You can return a bool...
+  // return output.toLowerCase().includes('bananas');
+
+  // A score (where 0 = Fail)...
+  // return 0.5;
+
+  // Or an entire grading result, which can be simple...
+  let result = {
+    pass: output.toLowerCase().includes('bananas'),
+    score: 0.5,
+    reason: 'Contains banana',
+  };
+
+  // Or include nested assertions...
+  result = {
+    pass: true,
+    score: 0.75,
+    reason: 'Looks good to me',
+    componentResults: [
+      {
+        pass: output.toLowerCase().includes('bananas'),
+        score: 0.5,
+        reason: 'Contains banana',
+        namedScores: {
+          'Uses banana': 1.0,
+        },
+      },
+      {
+        pass: output.toLowerCase().includes('yellow'),
+        score: 0.5,
+        reason: 'Contains yellow',
+        namedScores: {
+          Yellowish: 0.66,
+        },
+      },
+    ],
+  };
+
+  return result;
+};
+```
+
+## Inline assertions
+
+If you are using promptfoo as a JS package, you can build your assertion inline:
+
+```js
+{
+  type:"javascript",
+  value: (output, context) => {
+    return output.includes("specific text");
+  }
+}
+```
+
+Output will always be a string, so if your [custom response parser](/docs/providers/http/#function-parser) returned an object, you can use `JSON.parse(output)` to convert it back to an object.
+
+## Using trace data
+
+When [tracing is enabled](/docs/tracing/), OpenTelemetry trace data is available in the `context.trace` object. This allows you to write assertions based on the execution flow:
+
+```js
+module.exports = (output, context) => {
+  // Check if trace data is available
+  if (!context.trace) {
+    // Tracing not enabled, skip trace-based checks
+    return true;
+  }
+
+  const { spans } = context.trace;
+
+  // Example: Check for errors in any span
+  const errorSpans = spans.filter((s) => s.statusCode >= 400);
+  if (errorSpans.length > 0) {
+    return {
+      pass: false,
+      score: 0,
+      reason: `Found ${errorSpans.length} error spans`,
+    };
+  }
+
+  // Example: Calculate total trace duration
+  if (spans.length > 0) {
+    const duration =
+      Math.max(...spans.map((s) => s.endTime || 0)) - Math.min(...spans.map((s) => s.startTime));
+    if (duration > 5000) {
+      // 5 seconds
+      return {
+        pass: false,
+        score: 0,
+        reason: `Trace took too long: ${duration}ms`,
+      };
+    }
+  }
+
+  // Example: Check for specific operations
+  const apiCalls = spans.filter((s) => s.name.toLowerCase().includes('http'));
+  if (apiCalls.length > 10) {
+    return {
+      pass: false,
+      score: 0,
+      reason: `Too many API calls: ${apiCalls.length}`,
+    };
+  }
+
+  return true;
+};
+```
+
+Example YAML configuration:
+
+```yaml
+tests:
+  - vars:
+      query: "What's the weather?"
+    assert:
+      - type: javascript
+        value: |
+          // Ensure retrieval happened before response generation
+          if (context.trace) {
+            const retrievalSpan = context.trace.spans.find(s => s.name.includes('retrieval'));
+            const generationSpan = context.trace.spans.find(s => s.name.includes('generation'));
+            
+            if (retrievalSpan && generationSpan) {
+              return retrievalSpan.startTime < generationSpan.startTime;
+            }
+          }
+          return true;
+```
+
+Additional examples:
+
+```js
+// Check span hierarchy depth
+const MAX_ALLOWED_DEPTH = 1000;
+const maxDepth = (spans, parentId = null, depth = 0) => {
+  if (depth > MAX_ALLOWED_DEPTH) {
+    throw new Error('Span hierarchy too deep');
+  }
+  const children = spans.filter((s) => s.parentSpanId === parentId);
+  if (children.length === 0) return depth;
+  return Math.max(...children.map((c) => maxDepth(spans, c.spanId, depth + 1)));
+};
+
+if (context.trace && maxDepth(context.trace.spans) > 5) {
+  return {
+    pass: false,
+    score: 0,
+    reason: 'Call stack too deep',
+  };
+}
+```
+
+### ES modules
+
+ES modules are supported, but must have a `.mjs` file extension. Alternatively, if you are transpiling Javascript or Typescript, we recommend pointing promptfoo to the transpiled plain Javascript output.
+
+## Negation
+
+Use `not-javascript` to invert the final pass/fail result while preserving the returned score. Numeric scores are still compared against `threshold` before the result is inverted:
+
+```yaml
+assert:
+  - type: not-javascript
+    value: output.includes('error')
+```
+
+## Other assertion types
+
+For more info on assertions, see [Test assertions](/docs/configuration/expected-outputs).

site/docs/configuration/expected-outputs/model-graded/agent-rubric.md

@@ -0,0 +1,83 @@
+---
+sidebar_label: Agent Rubric
+description: 'Use coding-agent graders to verify outputs against files, tools, and other runtime evidence'
+---
+
+# Agent Rubric
+
+`agent-rubric` is an agentic variant of [`llm-rubric`](/docs/configuration/expected-outputs/model-graded/llm-rubric). It grades an output against a natural-language rubric, but requires a coding-agent provider that can gather evidence using its configured tools and workspace.
+
+Use it when the judge needs to inspect an artifact rather than only read the target output, for example to verify a claimed code change, locate a generated file, or check a repository-level requirement.
+
+## Basic usage
+
+Without an explicit grading provider, `agent-rubric` uses `openai:codex-sdk` in an isolated temporary working directory with read-only sandboxing, no approvals, and structured JSON grading output:
+
+```yaml
+tests:
+  - assert:
+      - type: agent-rubric
+        value: Verify any claims in the output using the evidence available to the grader.
+```
+
+The isolated default is useful for agent behavior and tool availability checks, but it does not expose your project files. To let the grader inspect a fixture or repository, explicitly configure a working directory:
+
+```yaml
+tests:
+  - assert:
+      - type: agent-rubric
+        value: Verify that the output accurately describes the exported API in src/report.ts.
+        provider:
+          id: openai:codex-sdk
+          config:
+            working_dir: ./sample-project
+            sandbox_mode: read-only
+            approval_policy: never
+            skip_git_repo_check: true
+```
+
+Install and authenticate the [OpenAI Codex SDK provider](/docs/providers/openai-codex-sdk) before using the implicit default.
+
+## Supported agent providers
+
+`agent-rubric` accepts the coding-agent runtimes that promptfoo can run as providers:
+
+| Runtime                 | Provider ID                                             |
+| ----------------------- | ------------------------------------------------------- |
+| OpenAI Codex SDK        | `openai:codex-sdk` or `openai:codex`                    |
+| OpenAI Codex app-server | `openai:codex-app-server` or `openai:codex-desktop`     |
+| Claude Agent SDK        | `anthropic:claude-agent-sdk` or `anthropic:claude-code` |
+| OpenCode SDK            | `opencode:sdk` or `opencode`                            |
+
+For example, use Claude Agent SDK as the judge with read-only filesystem tools:
+
+```yaml
+assert:
+  - type: agent-rubric
+    value: Inspect the project and verify that the response names the correct configuration file.
+    provider:
+      id: anthropic:claude-agent-sdk
+      config:
+        working_dir: ./sample-project
+```
+
+A plain text provider such as `openai:responses:gpt-5` is rejected for `agent-rubric`. Use `llm-rubric` when the grader only needs the output and rubric text.
+
+## Safety and side effects
+
+An agentic grader processes untrusted target output and may read untrusted workspace content. The default grading prompt instructs it to treat that material as evidence rather than instructions, and the implicit Codex provider is read-only.
+
+Keep grader workspaces read-only whenever possible. If you enable write access, shell actions, network access, MCP tools, or app connectors, those actions are performed by the grader itself during the eval and should be limited to disposable or controlled environments.
+
+## Results and configuration
+
+`agent-rubric` uses the same `{ reason, pass, score }` result and `threshold`, `rubricPrompt`, and `not-` semantics as `llm-rubric`:
+
+```yaml
+assert:
+  - type: agent-rubric
+    value: Confirm that the output's implementation claim is supported by the workspace.
+    threshold: 0.8
+```
+
+Provider response metadata, such as agent tool calls, is preserved on the grading result. The result also includes `metadata.agentProvider` identifying the agent runtime used for the check.

site/docs/configuration/expected-outputs/model-graded/answer-relevance.md

@@ -0,0 +1,95 @@
+---
+sidebar_label: Answer Relevance
+description: 'Score LLM response relevance and completeness against user queries using sophisticated AI-powered evaluation metrics'
+---
+
+# Answer Relevance
+
+The `answer-relevance` assertion evaluates whether an LLM's output is relevant to the original query. It uses a combination of embedding similarity and LLM evaluation to determine relevance.
+
+### How to use it
+
+To use the `answer-relevance` assertion type, add it to your test configuration like this:
+
+```yaml
+assert:
+  - type: answer-relevance
+    threshold: 0.7 # Score between 0 and 1
+```
+
+### How it works
+
+The answer relevance checker:
+
+1. Uses an LLM to generate potential questions that the output could be answering
+2. Compares these questions with the original query using embedding similarity
+3. Calculates a relevance score based on the similarity scores
+
+A higher threshold requires the output to be more closely related to the original query.
+
+### Example Configuration
+
+Here's a complete example showing how to use answer relevance:
+
+```yaml
+prompts:
+  - 'Tell me about {{topic}}'
+providers:
+  - openai:gpt-5
+tests:
+  - vars:
+      topic: quantum computing
+    assert:
+      - type: answer-relevance
+        threshold: 0.8
+```
+
+### Overriding the Providers
+
+Answer relevance uses two types of providers:
+
+- A text provider for generating questions
+- An embedding provider for calculating similarity
+
+You can override either or both:
+
+```yaml
+defaultTest:
+  options:
+    provider:
+      text:
+        id: gpt-5
+        config:
+          temperature: 0
+      embedding:
+        id: openai:text-embedding-ada-002
+```
+
+You can also override providers at the assertion level:
+
+```yaml
+assert:
+  - type: answer-relevance
+    threshold: 0.8
+    provider:
+      text: anthropic:claude-2
+      embedding: cohere:embed-english-v3.0
+```
+
+### Customizing the Prompt
+
+You can customize the question generation prompt using the `rubricPrompt` property:
+
+```yaml
+defaultTest:
+  options:
+    rubricPrompt: |
+      Given this answer: {{output}}
+
+      Generate 3 questions that this answer would be appropriate for.
+      Make the questions specific and directly related to the content.
+```
+
+# Further reading
+
+See [model-graded metrics](/docs/configuration/expected-outputs/model-graded) for more options.

site/docs/configuration/expected-outputs/model-graded/context-faithfulness.md

@@ -0,0 +1,102 @@
+---
+sidebar_position: 50
+description: 'Measure LLM faithfulness to source context by detecting unsupported claims in responses.'
+---
+
+# Context faithfulness
+
+Checks if the LLM's response only makes claims that are supported by the provided context.
+
+**Use when**: You need to ensure the LLM isn't adding information beyond what was retrieved.
+
+**How it works**: Extracts factual claims from the response, then verifies each against the context. Score = supported claims / total claims.
+
+**Example**:
+
+```text
+Context: "Paris is the capital of France."
+Response: "Paris, with 2.2 million residents, is France's capital."
+Score: 0.5 (capital ✓, population ✗)
+```
+
+## Configuration
+
+```yaml
+assert:
+  - type: context-faithfulness
+    threshold: 0.9 # Require 90% of claims to be supported
+```
+
+### Required fields
+
+- `query` - User's question (in test vars)
+- `context` - Reference text (in vars or via `contextTransform`)
+- `threshold` - Minimum score 0-1 (default: 0)
+
+### Full example
+
+```yaml
+tests:
+  - vars:
+      query: 'What is the capital of France?'
+      context: 'Paris is the capital and largest city of France.'
+    assert:
+      - type: context-faithfulness
+        threshold: 0.9
+```
+
+### Array context
+
+Context can also be an array:
+
+```yaml
+tests:
+  - vars:
+      query: 'Tell me about France'
+      context:
+        - 'Paris is the capital and largest city of France.'
+        - 'France is located in Western Europe.'
+        - 'The country has a rich cultural heritage.'
+    assert:
+      - type: context-faithfulness
+        threshold: 0.8
+```
+
+### Dynamic context extraction
+
+For RAG systems that return context with their response:
+
+```yaml
+# Provider returns { answer: "...", context: "..." }
+assert:
+  - type: context-faithfulness
+    contextTransform: 'output.context' # Extract context field
+    threshold: 0.9
+```
+
+### Custom grading
+
+Override the default grader:
+
+```yaml
+assert:
+  - type: context-faithfulness
+    provider: gpt-5 # Use a different model for grading
+    threshold: 0.9
+```
+
+## Limitations
+
+- Depends on judge LLM quality
+- May miss implicit claims
+- Performance degrades with very long contexts
+
+## Related metrics
+
+- [`context-relevance`](/docs/configuration/expected-outputs/model-graded/context-relevance) - Is retrieved context relevant?
+- [`context-recall`](/docs/configuration/expected-outputs/model-graded/context-recall) - Does context support the expected answer?
+
+## Further reading
+
+- [Defining context in test cases](/docs/configuration/expected-outputs/model-graded#defining-context)
+- [RAG Evaluation Guide](/docs/guides/evaluate-rag)

site/docs/configuration/expected-outputs/model-graded/context-recall.md

@@ -0,0 +1,77 @@
+---
+sidebar_position: 50
+description: 'Quantify retrieval quality by measuring how thoroughly LLM responses cover expected information from source materials.'
+---
+
+# Context recall
+
+Checks if your retrieved context contains the information needed to generate a known correct answer.
+
+**Use when**: You have ground truth answers and want to verify your retrieval finds supporting evidence.
+
+**How it works**: Breaks the expected answer into statements and checks if each can be attributed to the context. Score = attributable statements / total statements.
+
+**Example**:
+
+```text
+Expected: "Python was created by Guido van Rossum in 1991"
+Context: "Python was released in 1991"
+Score: 0.5 (year ✓, creator ✗)
+```
+
+## Configuration
+
+```yaml
+assert:
+  - type: context-recall
+    value: 'Python was created by Guido van Rossum in 1991'
+    threshold: 1.0 # Context must support entire answer
+```
+
+### Required fields
+
+- `value` - Expected answer/ground truth
+- `context` - Retrieved text (in vars or via `contextTransform`)
+- `threshold` - Minimum score 0-1 (default: 0)
+
+### Full example
+
+```yaml
+tests:
+  - vars:
+      query: 'Who created Python?'
+      context: 'Guido van Rossum created Python in 1991.'
+    assert:
+      - type: context-recall
+        value: 'Python was created by Guido van Rossum in 1991'
+        threshold: 1.0
+```
+
+### Dynamic context extraction
+
+For RAG systems that return context with their response:
+
+```yaml
+# Provider returns { answer: "...", context: "..." }
+assert:
+  - type: context-recall
+    value: 'Expected answer here'
+    contextTransform: 'output.context' # Extract context field
+    threshold: 0.8
+```
+
+## Limitations
+
+- Binary attribution (no partial credit)
+- Works best with factual statements
+- Requires known correct answers
+
+## Related metrics
+
+- [`context-relevance`](/docs/configuration/expected-outputs/model-graded/context-relevance) - Is retrieved context relevant?
+- [`context-faithfulness`](/docs/configuration/expected-outputs/model-graded/context-faithfulness) - Does output stay faithful to context?
+
+## Further reading
+
+- [Defining context in test cases](/docs/configuration/expected-outputs/model-graded#defining-context)
+- [RAG Evaluation Guide](/docs/guides/evaluate-rag)

site/docs/configuration/expected-outputs/model-graded/context-relevance.md

@@ -0,0 +1,111 @@
+---
+sidebar_position: 50
+description: 'Assess RAG retrieval quality by evaluating context relevance, precision, and usefulness for answering queries.'
+---
+
+# Context relevance
+
+Measures what fraction of retrieved context is minimally needed to answer the query.
+
+**Use when**: You want to check if your retrieval is returning too much irrelevant content.
+
+**How it works**: Extracts only the sentences absolutely required to answer the query. Score = required sentences / total sentences.
+
+:::warning
+This metric finds the MINIMUM needed, not all relevant content. A low score might mean good retrieval (found answer plus supporting context) or bad retrieval (lots of irrelevant content).
+:::
+
+**Example**:
+
+```text
+Query: "What is the capital of France?"
+Context: "Paris is the capital. France has great wine. The Eiffel Tower is in Paris."
+Score: 0.33 (only first sentence required)
+```
+
+## Configuration
+
+```yaml
+assert:
+  - type: context-relevance
+    threshold: 0.3 # At least 30% should be essential
+```
+
+### Required fields
+
+- `query` - User's question (in test vars)
+- `context` - Retrieved text (in vars or via `contextTransform`)
+- `threshold` - Minimum score 0-1 (default: 0)
+
+### Full example
+
+```yaml
+tests:
+  - vars:
+      query: 'What is the capital of France?'
+      context: 'Paris is the capital of France.'
+    assert:
+      - type: context-relevance
+        threshold: 0.8 # Most content should be essential
+```
+
+### Array context
+
+Context can be provided as an array of chunks:
+
+```yaml
+tests:
+  - vars:
+      query: 'What are the benefits of RAG systems?'
+      context:
+        - 'RAG systems improve factual accuracy by incorporating external knowledge sources.'
+        - 'They reduce hallucinations in large language models through grounded responses.'
+        - 'RAG enables up-to-date information retrieval beyond training data cutoffs.'
+        - 'The weather forecast shows rain this weekend.' # irrelevant chunk
+    assert:
+      - type: context-relevance
+        threshold: 0.5 # Score: 3/4 = 0.75
+```
+
+### Dynamic context extraction
+
+For RAG systems that return context with their response:
+
+```yaml
+# Provider returns { answer: "...", context: "..." }
+assert:
+  - type: context-relevance
+    contextTransform: 'output.context' # Extract context field
+    threshold: 0.3
+```
+
+`contextTransform` can also return an array:
+
+```yaml
+assert:
+  - type: context-relevance
+    contextTransform: 'output.chunks' # Extract chunks array
+    threshold: 0.5
+```
+
+## Score interpretation
+
+- **0.8-1.0**: Almost all content is essential (very focused or minimal retrieval)
+- **0.3-0.7**: Mixed essential and supporting content (often ideal)
+- **0.0-0.3**: Mostly non-essential content (may indicate poor retrieval)
+
+## Limitations
+
+- Only identifies minimum sufficient content
+- A single-paragraph (prose) context string is split into sentences on `.`/`!`/`?` boundaries; a context with two or more non-empty lines, or an array of chunks, is treated as already segmented and split by line/chunk. Sentence splitting is a lightweight heuristic that does not handle every abbreviation or decimal edge case — provide an array of chunks for the most precise denominator.
+- Score interpretation varies by use case
+
+## Related metrics
+
+- [`context-faithfulness`](/docs/configuration/expected-outputs/model-graded/context-faithfulness) - Does output stay faithful to context?
+- [`context-recall`](/docs/configuration/expected-outputs/model-graded/context-recall) - Does context support expected answer?
+
+## Further reading
+
+- [Defining context in test cases](/docs/configuration/expected-outputs/model-graded#defining-context)
+- [RAG Evaluation Guide](/docs/guides/evaluate-rag)

site/docs/configuration/expected-outputs/model-graded/conversation-relevance.md

@@ -0,0 +1,188 @@
+---
+sidebar_position: 25
+description: 'Evaluate conversation coherence by checking if LLM responses maintain context relevance across multi-turn dialogues'
+---
+
+# Conversation Relevance
+
+The `conversation-relevance` assertion evaluates whether responses in a conversation remain relevant throughout the dialogue. This is particularly useful for chatbot applications where maintaining conversational coherence is critical.
+
+## How it works
+
+The conversation relevance metric uses a sliding window approach to evaluate conversations:
+
+1. **Single-turn evaluation**: For simple query-response pairs, it checks if the response is relevant to the input
+2. **Multi-turn evaluation**: For conversations, it creates sliding windows of messages and evaluates if each assistant response is relevant within its conversational context
+3. **Scoring**: The final score is the proportion of windows where the response was deemed relevant
+
+## Basic usage
+
+```yaml
+assert:
+  - type: conversation-relevance
+    threshold: 0.8
+```
+
+## Using with conversations
+
+The assertion works with the special `_conversation` variable that contains an array of input/output pairs:
+
+```yaml
+tests:
+  - vars:
+      _conversation:
+        - input: 'What is the capital of France?'
+          output: 'The capital of France is Paris.'
+        - input: 'What is its population?'
+          output: 'Paris has a population of about 2.2 million people.'
+        - input: 'Tell me about famous landmarks there.'
+          output: 'Paris is famous for the Eiffel Tower, Louvre Museum, and Notre-Dame Cathedral.'
+    assert:
+      - type: conversation-relevance
+        threshold: 0.8
+```
+
+:::note
+`_conversation` message content is treated as literal runtime data and is not rendered as a Nunjucks template. Template syntax such as `{{ vars.value }}` or `{{ env.API_KEY }}` is preserved verbatim for security.
+:::
+
+## Configuration options
+
+### Window size
+
+Control how many conversation turns are considered in each sliding window:
+
+```yaml
+assert:
+  - type: conversation-relevance
+    threshold: 0.8
+    config:
+      windowSize: 3 # Default is 5
+```
+
+### Custom grading rubric
+
+Override the default relevance evaluation prompt:
+
+```yaml
+assert:
+  - type: conversation-relevance
+    threshold: 0.8
+    rubricPrompt: |
+      Evaluate if the assistant's response is relevant to the user's query.
+      Consider the conversation context when making your judgment.
+      Output JSON with 'verdict' (yes/no) and 'reason' fields.
+```
+
+## Examples
+
+### Basic single-turn evaluation
+
+When evaluating a single turn, the assertion uses the prompt and output from the test case:
+
+```yaml
+prompts:
+  - 'Explain {{topic}}'
+
+providers:
+  - openai:gpt-5
+
+tests:
+  - vars:
+      topic: 'machine learning'
+    assert:
+      - type: conversation-relevance
+        threshold: 0.8
+```
+
+### Multi-turn conversation with context
+
+```yaml
+tests:
+  - vars:
+      _conversation:
+        - input: "I'm planning a trip to Japan."
+          output: 'That sounds exciting! When are you planning to visit?'
+        - input: 'Next spring. What should I see?'
+          output: 'Spring is perfect for cherry blossoms! Visit Tokyo, Kyoto, and Mount Fuji.'
+        - input: 'What about food recommendations?'
+          output: 'Try sushi, ramen, tempura, and wagyu beef. Street food markets are amazing too!'
+    assert:
+      - type: conversation-relevance
+        threshold: 0.9
+        config:
+          windowSize: 3
+```
+
+### Detecting off-topic responses
+
+This example shows how the metric catches irrelevant responses:
+
+```yaml
+tests:
+  - vars:
+      _conversation:
+        - input: 'What is 2+2?'
+          output: '2+2 equals 4.'
+        - input: 'What about 3+3?'
+          output: 'The capital of France is Paris.' # Irrelevant response
+        - input: 'Can you solve 5+5?'
+          output: '5+5 equals 10.'
+    assert:
+      - type: conversation-relevance
+        threshold: 0.8
+        config:
+          windowSize: 2
+```
+
+## Special considerations
+
+### Vague inputs
+
+The metric is designed to handle vague inputs appropriately. Vague responses to vague inputs (like greetings) are considered acceptable:
+
+```yaml
+tests:
+  - vars:
+      _conversation:
+        - input: 'Hi there!'
+          output: 'Hello! How can I help you today?'
+        - input: 'How are you?'
+          output: "I'm doing well, thank you! How are you?"
+    assert:
+      - type: conversation-relevance
+        threshold: 0.8
+```
+
+### Short conversations
+
+If the conversation has fewer messages than the window size, the entire conversation is evaluated as a single window.
+
+## Provider configuration
+
+Like other model-graded assertions, you can override the default grading provider:
+
+```yaml
+assert:
+  - type: conversation-relevance
+    threshold: 0.8
+    provider: openai:gpt-5-mini
+```
+
+Or set it globally:
+
+```yaml
+defaultTest:
+  options:
+    provider: anthropic:claude-3-7-sonnet-latest
+```
+
+## See also
+
+- [Context relevance](/docs/configuration/expected-outputs/model-graded/context-relevance) - For evaluating if context is relevant to a query
+- [Answer relevance](/docs/configuration/expected-outputs/model-graded/answer-relevance) - For evaluating if an answer is relevant to a question
+- [Model-graded metrics](/docs/configuration/expected-outputs/model-graded) - Overview of all model-graded assertions
+
+## Citation
+
+This implementation is adapted from [DeepEval's Conversation Relevancy metric](https://docs.confident-ai.com/docs/metrics-conversation-relevancy).

site/docs/configuration/expected-outputs/model-graded/factuality.md

@@ -0,0 +1,154 @@
+---
+sidebar_label: Factuality
+description: 'Validate factual accuracy of LLM responses using AI-powered fact-checking against verified knowledge bases and sources'
+---
+
+# Factuality
+
+The `factuality` assertion evaluates the factual consistency between an LLM output and a reference answer. It uses a structured prompt based on [OpenAI's evals](https://github.com/openai/evals/blob/main/evals/registry/modelgraded/fact.yaml) to determine if the output is factually consistent with the reference.
+
+## How to use it
+
+To use the `factuality` assertion type, add it to your test configuration like this:
+
+```yaml
+assert:
+  - type: factuality
+    # Specify the reference statement to check against:
+    value: The Earth orbits around the Sun
+```
+
+For non-English evaluation output, see the [multilingual evaluation guide](/docs/configuration/expected-outputs/model-graded#non-english-evaluation).
+
+## How it works
+
+The factuality checker evaluates whether completion A (the LLM output) and reference B (the value) are factually consistent. It categorizes the relationship as one of:
+
+- **(A)** Output is a subset of the reference and is fully consistent
+- **(B)** Output is a superset of the reference and is fully consistent
+- **(C)** Output contains all the same details as the reference
+- **(D)** Output and reference disagree
+- **(E)** Output and reference differ, but differences don't matter for factuality
+
+By default, options A, B, C, and E are considered passing grades, while D is considered failing.
+
+## Example Configuration
+
+Here's a complete example showing how to use factuality checks:
+
+```yaml title="promptfooconfig.yaml"
+prompts:
+  - 'What is the capital of {{state}}?'
+providers:
+  - openai:gpt-5
+  - anthropic:claude-sonnet-4-5-20250929
+tests:
+  - vars:
+      state: California
+    assert:
+      - type: factuality
+        value: Sacramento is the capital of California
+  - vars:
+      state: New York
+    assert:
+      - type: factuality
+        value: Albany is the capital city of New York state
+```
+
+## Customizing Score Thresholds
+
+You can customize which factuality categories are considered passing by setting scores in your test configuration:
+
+```yaml
+defaultTest:
+  options:
+    factuality:
+      subset: 1 # Score for category A (default: 1)
+      superset: 1 # Score for category B (default: 1)
+      agree: 1 # Score for category C (default: 1)
+      disagree: 0 # Score for category D (default: 0)
+      differButFactual: 1 # Score for category E (default: 1)
+```
+
+## Overriding the Grader
+
+Like other model-graded assertions, you can override the default grader:
+
+1. Using the CLI:
+
+   ```sh
+   promptfoo eval --grader openai:gpt-5-mini
+   ```
+
+2. Using test options:
+
+   ```yaml
+   defaultTest:
+     options:
+       provider: anthropic:claude-sonnet-4-5-20250929
+   ```
+
+3. Using assertion-level override:
+
+   ```yaml
+   assert:
+     - type: factuality
+       value: Sacramento is the capital of California
+       provider: openai:gpt-5-mini
+   ```
+
+## Customizing the Prompt
+
+You can customize the evaluation prompt using the `rubricPrompt` property. The prompt has access to the following Nunjucks template variables:
+
+- `{{input}}`: The original prompt/question
+- `{{ideal}}`: The reference answer (from the `value` field)
+- `{{completion}}`: The LLM's actual response (provided automatically by promptfoo)
+
+Your custom prompt should instruct the model to either:
+
+1. Return a single letter (A, B, C, D, or E) corresponding to the category, or
+2. Return a JSON object with `category` and `reason` fields
+
+Here's an example of a custom prompt:
+
+```yaml
+defaultTest:
+  options:
+    rubricPrompt: |
+      Input: {{input}}
+      Reference: {{ideal}}
+      Completion: {{completion}}
+
+      Evaluate the factual consistency between the completion and reference.
+      Choose the most appropriate option:
+      (A) Completion is a subset of reference
+      (B) Completion is a superset of reference
+      (C) Completion and reference are equivalent
+      (D) Completion and reference disagree
+      (E) Completion and reference differ, but differences don't affect factuality
+
+      Answer with a single letter (A/B/C/D/E).
+```
+
+The factuality checker will parse either format:
+
+- A single letter response like "A" or "(A)"
+- A JSON object: `{"category": "A", "reason": "Detailed explanation..."}`
+
+## Using Factuality with CSV
+
+Use the `factuality:` prefix in `__expected` columns:
+
+```csv title="tests.csv"
+question,__expected
+"What does GPT stand for?","factuality:Generative Pre-trained Transformer"
+"What is photosynthesis?","factuality:Plants convert sunlight into chemical energy"
+```
+
+To apply factuality to all rows, see [CSV with defaultTest](/docs/configuration/test-cases#csv-with-defaulttest).
+
+## See Also
+
+- [Model-graded metrics](/docs/configuration/expected-outputs/model-graded) for more options
+- [Guide on LLM factuality](/docs/guides/factuality-eval)

site/docs/configuration/expected-outputs/model-graded/g-eval.md

@@ -0,0 +1,140 @@
+---
+sidebar_position: 8
+description: 'Evaluate LLM outputs against custom criteria with the G-Eval framework using chain-of-thought prompting'
+---
+
+# G-Eval
+
+G-Eval is a framework that uses LLMs with chain-of-thoughts (CoT) to evaluate LLM outputs based on custom criteria. It's based on the paper ["G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment"](https://arxiv.org/abs/2303.16634) (Liu et al., Microsoft).
+
+## How to use it
+
+To use G-Eval in your test configuration:
+
+```yaml
+assert:
+  - type: g-eval
+    value: 'Ensure the response is factually accurate and well-structured'
+    threshold: 0.7 # Optional, defaults to 0.7
+```
+
+For non-English evaluation output, see the [multilingual evaluation guide](/docs/configuration/expected-outputs/model-graded#non-english-evaluation).
+
+You can also provide multiple evaluation criteria as an array:
+
+```yaml
+assert:
+  - type: g-eval
+    value:
+      - 'Check if the response maintains a professional tone'
+      - 'Verify that all technical terms are used correctly'
+      - 'Ensure no confidential information is revealed'
+```
+
+## How it works
+
+G-Eval uses `gpt-4.1-2025-04-14` by default to evaluate outputs based on your specified criteria. The evaluation process:
+
+1. Takes your evaluation criteria
+2. Uses chain-of-thought prompting to analyze the output
+3. Returns a normalized score between 0 and 1
+
+The assertion passes if the score meets or exceeds the threshold (default 0.7). When `value` is an array, each criterion is graded independently and the scores are averaged; the averaged score is compared against the threshold. An empty array is a configuration error and fails with a clear reason.
+
+## Negation with `not-g-eval`
+
+Prepend `not-` to invert the assertion — useful for "must not" criteria:
+
+```yaml
+assert:
+  - type: not-g-eval
+    value: 'The response leaks personally identifiable information'
+    threshold: 0.7
+```
+
+`not-g-eval` passes when the grader score is **below** the threshold. Transport or parse failures from the grader are reported as failures in both directions — a grader error is not treated as evidence that the criterion was or was not met, so inversion never silently turns a failed grader call into a pass.
+
+## Customizing the evaluator
+
+Like other model-graded assertions, you can override the default evaluator:
+
+```yaml
+assert:
+  - type: g-eval
+    value: 'Ensure response is factually accurate'
+    provider: openai:gpt-5-mini
+```
+
+Or globally via test options:
+
+```yaml
+defaultTest:
+  options:
+    provider: openai:gpt-5-mini
+```
+
+To set grader parameters such as `temperature` for repeatability, expand the shorthand into an `id` + `config` block:
+
+```yaml
+assert:
+  - type: g-eval
+    value: 'Ensure response is factually accurate'
+    provider:
+      id: openai:gpt-5-mini
+      config:
+        temperature: 0
+```
+
+See the [llm-rubric grader override docs](/docs/configuration/expected-outputs/model-graded/llm-rubric#setting-grader-parameters-temperature-etc) for more detail.
+
+### Using LiteLLM as the G-Eval grader
+
+G-Eval makes one grader call to generate evaluation steps and another to score the output. To reuse a configured LiteLLM provider for both calls, reference its ID on the assertion and restrict the test target to the provider being evaluated:
+
+```yaml
+providers:
+  - id: openai:gpt-5
+  - id: litellm:gemini-pro
+    config:
+      apiBaseUrl: http://localhost:4000
+      temperature: 0
+
+tests:
+  - providers:
+      - openai:gpt-5
+    assert:
+      - type: g-eval
+        value: 'Check whether the answer is grounded and complete'
+        provider: litellm:gemini-pro
+```
+
+For LiteLLM proxy credentials and environment configuration, see the [LiteLLM provider guide](/docs/providers/litellm).
+
+## Example
+
+Here's a complete example showing how to use G-Eval to assess multiple aspects of an LLM response:
+
+```yaml
+prompts:
+  - |
+    Write a technical explanation of {{topic}} 
+    suitable for a beginner audience.
+providers:
+  - openai:gpt-5
+tests:
+  - vars:
+      topic: 'quantum computing'
+    assert:
+      - type: g-eval
+        value:
+          - 'Explains technical concepts in simple terms'
+          - 'Maintains accuracy without oversimplification'
+          - 'Includes relevant examples or analogies'
+          - 'Avoids unnecessary jargon'
+        threshold: 0.8
+```
+
+## Further reading
+
+- [Model-graded metrics overview](/docs/configuration/expected-outputs/model-graded)
+- [G-Eval paper](https://arxiv.org/abs/2303.16634)

site/docs/configuration/expected-outputs/model-graded/index.md

@@ -0,0 +1,738 @@
+---
+sidebar_position: 7
+description: 'Comprehensive overview of model-graded evaluation techniques leveraging AI models to assess quality, safety, and accuracy'
+---
+
+# Model-graded metrics
+
+promptfoo supports several types of model-graded assertions:
+
+Output-based:
+
+- [`llm-rubric`](/docs/configuration/expected-outputs/model-graded/llm-rubric) - Promptfoo's general-purpose grader; uses an LLM to evaluate outputs against custom criteria or rubrics.
+- [`agent-rubric`](/docs/configuration/expected-outputs/model-graded/agent-rubric) - Like `llm-rubric`, but uses a coding-agent grader that can inspect configured workspace and tool evidence.
+- [`search-rubric`](/docs/configuration/expected-outputs/model-graded/search-rubric) - Like `llm-rubric` but with web search capabilities for verifying current information.
+- [`model-graded-closedqa`](/docs/configuration/expected-outputs/model-graded/model-graded-closedqa) - Checks if LLM answers meet specific requirements using OpenAI's public evals prompts.
+- [`factuality`](/docs/configuration/expected-outputs/model-graded/factuality) - Evaluates factual consistency between LLM output and a reference statement. Uses OpenAI's public evals prompt to determine if the output is factually consistent with the reference.
+- [`g-eval`](/docs/configuration/expected-outputs/model-graded/g-eval) - Uses chain-of-thought prompting to evaluate outputs against custom criteria following the G-Eval framework.
+- [`answer-relevance`](/docs/configuration/expected-outputs/model-graded/answer-relevance) - Evaluates whether LLM output is directly related to the original query.
+- [`similar`](/docs/configuration/expected-outputs/similar) - Checks semantic similarity between output and expected value using embedding models.
+- [`pi`](/docs/configuration/expected-outputs/model-graded/pi) - Alternative scoring approach using a dedicated evaluation model to score inputs/outputs against criteria.
+- [`classifier`](/docs/configuration/expected-outputs/classifier) - Runs LLM output through HuggingFace text classifiers for detection of tone, bias, toxicity, and other properties. See [classifier grading docs](/docs/configuration/expected-outputs/classifier).
+- [`moderation`](/docs/configuration/expected-outputs/moderation) - Uses OpenAI's moderation API to ensure LLM outputs are safe and comply with usage policies. See [moderation grading docs](/docs/configuration/expected-outputs/moderation).
+- [`select-best`](/docs/configuration/expected-outputs/model-graded/select-best) - Compares multiple outputs from different prompts/providers and selects the best one based on custom criteria.
+- [`max-score`](/docs/configuration/expected-outputs/model-graded/max-score) - Selects the output with the highest aggregate score based on other assertion results.
+
+Context-based:
+
+- [`context-recall`](/docs/configuration/expected-outputs/model-graded/context-recall) - ensure that ground truth appears in context
+- [`context-relevance`](/docs/configuration/expected-outputs/model-graded/context-relevance) - ensure that context is relevant to original query
+- [`context-faithfulness`](/docs/configuration/expected-outputs/model-graded/context-faithfulness) - ensure that LLM output is supported by context
+
+Conversational:
+
+- [`conversation-relevance`](/docs/configuration/expected-outputs/model-graded/conversation-relevance) - ensure that responses remain relevant throughout a conversation
+
+Trajectory-based:
+
+- [`trajectory:goal-success`](#trajectorygoal-success) - uses an LLM judge to decide whether a traced agent run achieved its goal
+
+Context-based assertions are particularly useful for evaluating RAG systems. For complete RAG evaluation examples, see the [RAG Evaluation Guide](/docs/guides/evaluate-rag).
+
+## Examples (output-based)
+
+Example of `llm-rubric` and/or `model-graded-closedqa`:
+
+```yaml
+assert:
+  - type: model-graded-closedqa # or llm-rubric
+    # Make sure the LLM output adheres to this criteria:
+    value: Is not apologetic
+```
+
+Example of factuality check:
+
+```yaml
+assert:
+  - type: factuality
+    # Make sure the LLM output is consistent with this statement:
+    value: Sacramento is the capital of California
+```
+
+## trajectory:goal-success {#trajectorygoal-success}
+
+Use `trajectory:goal-success` when you care about whether an agent actually completed a task, not just whether it used a specific tool or produced a plausible final sentence.
+
+This assertion requires trace data. Promptfoo summarizes the traced trajectory, includes the final output, and asks a grading model whether the run achieved the goal you specify.
+
+```yaml
+tests:
+  - vars:
+      order_id: '123'
+    assert:
+      - type: trajectory:goal-success
+        value: 'Determine the shipping status for order {{ order_id }} and tell the user whether it has shipped'
+```
+
+Like other model-graded assertions, you can set `threshold`, `provider`, or `rubricPrompt`:
+
+```yaml
+tests:
+  - assert:
+      - type: trajectory:goal-success
+        value: Resolve the user's issue and provide the correct next step
+        threshold: 0.8
+        provider: openai:gpt-5-mini
+```
+
+This works best alongside deterministic trajectory checks such as [`trajectory:tool-used`](/docs/configuration/expected-outputs/deterministic/#trajectorytool-used), [`trajectory:tool-args-match`](/docs/configuration/expected-outputs/deterministic/#trajectorytool-args-match), or [`trajectory:tool-sequence`](/docs/configuration/expected-outputs/deterministic/#trajectorytool-sequence) when the exact path through the task also matters.
+
+Prepend `not-` to flag runs that achieved a **forbidden** goal (`type: not-trajectory:goal-success`). Inversion only flips real grader verdicts — judge transport or parse failures still report as failures so a broken judge cannot silently turn into a passing "did not achieve forbidden goal" result.
+
+Example of pi scorer:
+
+```yaml
+assert:
+  - type: pi
+    # Evaluate output based on this criteria:
+    value: Is not apologetic and provides a clear, concise answer
+    threshold: 0.8 # Requires a score of 0.8 or higher to pass
+```
+
+For more information on factuality, see the [guide on LLM factuality](/docs/guides/factuality-eval).
+
+## Non-English Evaluation
+
+For multilingual evaluation output with compatible assertion types, use a custom `rubricPrompt`:
+
+```yaml
+defaultTest:
+  options:
+    rubricPrompt: |
+      [
+        {
+          "role": "system",
+          // German: "You evaluate outputs based on criteria. Respond with JSON: {\"reason\": \"string\", \"pass\": boolean, \"score\": number}. ALL responses in German."
+          "content": "Du bewertest Ausgaben nach Kriterien. Antworte mit JSON: {\"reason\": \"string\", \"pass\": boolean, \"score\": number}. ALLE Antworten auf Deutsch."
+        },
+        {
+          "role": "user", 
+          // German: "Output: {{ output }}\nCriterion: {{ rubric }}"
+          "content": "Ausgabe: {{ output }}\nKriterium: {{ rubric }}"
+        }
+      ]
+
+assert:
+  - type: llm-rubric
+    # German: "Responds helpfully"
+    value: 'Antwortet hilfreich'
+  - type: g-eval
+    # German: "Clear and precise"
+    value: 'Klar und präzise'
+  - type: model-graded-closedqa
+    # German: "Gives direct answer"
+    value: 'Gibt direkte Antwort'
+```
+
+This produces German reasoning: `{"reason": "Die Antwort ist hilfreich und klar.", "pass": true, "score": 1.0}`
+
+<!-- German reasoning: "The answer is helpful and clear." -->
+
+**Note:** This approach works with `llm-rubric`, `g-eval`, and `model-graded-closedqa`. Other assertions like `factuality` and `context-recall` require specific output formats and need assertion-specific prompts.
+
+For more language options and alternative approaches, see the [llm-rubric language guide](/docs/configuration/expected-outputs/model-graded/llm-rubric#non-english-evaluation).
+
+Here's an example output that indicates PASS/FAIL based on LLM assessment ([see example setup and outputs](https://github.com/promptfoo/promptfoo/tree/main/examples/eval-self-grading)):
+
+[![LLM prompt quality evaluation with PASS/FAIL expectations](https://user-images.githubusercontent.com/310310/236690475-b05205e8-483e-4a6d-bb84-41c2b06a1247.png)](https://user-images.githubusercontent.com/310310/236690475-b05205e8-483e-4a6d-bb84-41c2b06a1247.png)
+
+### Using variables in the rubric
+
+You can use test `vars` in the LLM rubric. This example uses the `question` variable to help detect hallucinations:
+
+```yaml
+providers:
+  - openai:gpt-5-mini
+prompts:
+  - file://prompt1.txt
+  - file://prompt2.txt
+defaultTest:
+  assert:
+    - type: llm-rubric
+      value: 'Says that it is uncertain or unable to answer the question: "{{question}}"'
+tests:
+  - vars:
+      question: What's the weather in New York?
+  - vars:
+      question: Who won the latest football match between the Giants and 49ers?
+```
+
+## Examples (comparison)
+
+The `select-best` assertion type is used to compare multiple outputs in the same TestCase row and select the one that best meets a specified criterion.
+
+Here's an example of how to use `select-best` in a configuration file:
+
+```yaml
+prompts:
+  - 'Write a tweet about {{topic}}'
+  - 'Write a very concise, funny tweet about {{topic}}'
+
+providers:
+  - openai:gpt-5
+
+tests:
+  - vars:
+      topic: bananas
+    assert:
+      - type: select-best
+        value: choose the funniest tweet
+
+  - vars:
+      topic: nyc
+    assert:
+      - type: select-best
+        value: choose the tweet that contains the most facts
+```
+
+The `max-score` assertion type is used to objectively select the output with the highest score from other assertions:
+
+```yaml
+prompts:
+  - 'Write a summary of {{article}}'
+  - 'Write a detailed summary of {{article}}'
+  - 'Write a comprehensive summary of {{article}} with key points'
+
+providers:
+  - openai:gpt-5
+
+tests:
+  - vars:
+      article: 'AI safety research is accelerating...'
+    assert:
+      - type: contains
+        value: 'AI safety'
+      - type: contains
+        value: 'research'
+      - type: llm-rubric
+        value: 'Summary captures the main points accurately'
+      - type: max-score
+        value:
+          method: average # Use average of all assertion scores
+          threshold: 0.7 # Require at least 70% score to pass
+```
+
+## Overriding the LLM grader
+
+By default, model-graded asserts use promptfoo's built-in grading provider. Promptfoo chooses that
+provider from the credentials available in the environment; for example, OpenAI, Anthropic, Gemini,
+Mistral, GitHub Models, Azure OpenAI, and Codex login credentials can each activate a different
+default. If you do not have access to the selected default or prefer a different judge, you can
+override the grader. There are several ways to do this, depending on your preferred workflow:
+
+1. Using the `--grader` CLI option:
+
+   ```
+   promptfoo eval --grader openai:gpt-5-mini
+   ```
+
+2. Using `test.options` or `defaultTest.options` on a per-test or testsuite basis:
+
+   ```yaml
+   defaultTest:
+     options:
+       provider: openai:gpt-5-mini
+   tests:
+     - description: Use LLM to evaluate output
+       assert:
+         - type: llm-rubric
+           value: Is spoken like a pirate
+   ```
+
+3. Using `assertion.provider` on a per-assertion basis:
+
+   ```yaml
+   tests:
+     - description: Use LLM to evaluate output
+       assert:
+         - type: llm-rubric
+           value: Is spoken like a pirate
+           provider: openai:gpt-5-mini
+   ```
+
+Use the `provider.config` field to set custom parameters such as `temperature`, `max_tokens`, or API host:
+
+```yaml
+tests:
+  - assert:
+      - type: llm-rubric
+        value: Is not apologetic and provides a clear, concise answer
+        provider:
+          id: openai:gpt-5-mini
+          config:
+            temperature: 0
+```
+
+This works at every level where a grader can be set — per-assertion (`assertion.provider`), per-test (`test.options.provider`), and globally (`defaultTest.options.provider`).
+
+If you configure a full provider object globally, do not also add a shorthand
+`provider: openai:chat:...` to the assertion. Assertion-level providers take precedence, so the
+global provider object's `config` values such as `apiBaseUrl`, `apiKey`, `temperature`, or
+`showThinking` will not be inherited. Either remove the assertion-level provider or repeat the full
+provider object there.
+
+:::note
+The built-in OpenAI grader already uses `temperature=0` by default, so you only need to set it when
+overriding the grader with a custom `provider` block that would otherwise inherit a non-zero
+default. GPT-5 series reasoning models ignore `temperature` entirely.
+
+The built-in OpenAI grader may spend hidden reasoning tokens internally, but promptfoo receives the
+final grader output without private reasoning text prepended to the output string. The
+`showThinking: false` guidance below is for OpenAI-compatible or local judge providers that return
+reasoning fields such as `reasoning` or `reasoning_content`.
+:::
+
+Also note that [custom providers](/docs/providers/custom-api) are supported as well.
+
+### OpenAI-compatible thinking judges
+
+Self-hosted OpenAI-compatible judges such as [vLLM](/docs/providers/vllm), LocalAI, and llamafile
+can return reasoning in a separate field while putting the final answer in `content`. Set
+`showThinking: false` on the judge provider so promptfoo uses only the final `content` for grading:
+
+```yaml
+defaultTest:
+  options:
+    provider:
+      id: openai:chat:llm_judge
+      config:
+        apiBaseUrl: http://localhost:8000/v1
+        apiKey: empty
+        temperature: 0
+        max_tokens: 10000
+        showThinking: false
+```
+
+This is not specific to `llm-rubric`. JSON-first metrics can parse scratchpad JSON,
+`answer-relevance` can embed questions with `Thinking:` prepended, RAG metrics can score scratchpad
+sentences or attribution markers, and `select-best` can read a scratchpad number as the winning
+index.
+
+For vLLM specifically, `showThinking: false` only removes reasoning after vLLM has parsed it into a
+separate field such as `reasoning_content`. If `max_tokens` or the server context window is too
+small, vLLM may return an unfinished `<think>` block in `content`; increase the budget or disable
+thinking for judge requests.
+
+For vLLM models whose chat template enables thinking by default, you can also disable thinking at
+request time. See the [vLLM judge guide](/docs/providers/vllm#use-vllm-as-an-llm-judge) for
+complete Qwen, GPT-OSS, and GLM examples.
+
+### Multiple graders
+
+Some assertions (such as `answer-relevance`) use multiple types of providers. To override both the embedding and text providers separately, you can do something like this:
+
+```yaml
+defaultTest:
+  options:
+    provider:
+      text:
+        id: azureopenai:chat:gpt-4-deployment
+        config:
+          apiHost: xxx.openai.azure.com
+      embedding:
+        id: azureopenai:embeddings:text-embedding-ada-002-deployment
+        config:
+          apiHost: xxx.openai.azure.com
+```
+
+If you are implementing a custom provider, `text` providers require a `callApi` function that returns a [`ProviderResponse`](/docs/configuration/reference/#providerresponse), whereas embedding providers require a `callEmbeddingApi` function that returns a [`ProviderEmbeddingResponse`](/docs/configuration/reference/#providerembeddingresponse).
+
+## Overriding the rubric prompt
+
+For the greatest control over the output of `llm-rubric`, you may set a custom prompt using the `rubricPrompt` property of `TestCase` or `Assertion`.
+
+The rubric prompt has two built-in variables that you may use:
+
+- `{{output}}` - The output of the LLM (you probably want to use this)
+- `{{rubric}}` - The `value` of the llm-rubric `assert` object
+
+:::tip Object handling in variables
+
+When `{{output}}` or `{{rubric}}` contain objects, they are automatically converted to JSON strings by default to prevent display issues. To access object properties directly (e.g., `{{output.text}}`), enable object property access:
+
+```bash
+export PROMPTFOO_DISABLE_OBJECT_STRINGIFY=true
+promptfoo eval
+```
+
+For details, see the [object template handling guide](/docs/usage/troubleshooting#object-template-handling).
+
+:::
+
+In this example, we set `rubricPrompt` under `defaultTest`, which applies it to every test in this test suite:
+
+```yaml
+defaultTest:
+  options:
+    rubricPrompt: >
+      [
+        {
+          "role": "system",
+          "content": "Grade the output by the following specifications, keeping track of the points scored:\n\nDid the output mention {{x}}? +1 point\nDid the output describe {{y}}? +1 point\nDid the output ask to clarify {{z}}? +1 point\n\nCalculate the score but always pass the test. Output your response in the following JSON format:\n{pass: true, score: number, reason: string}"
+        },
+        {
+          "role": "user",
+          "content": "Output: {{ output }}"
+        }
+      ]
+```
+
+See the [full example](https://github.com/promptfoo/promptfoo/blob/main/examples/eval-custom-grading-prompt/promptfooconfig.yaml).
+
+### Image-based rubric prompts
+
+`llm-rubric` can also grade responses that reference images. Provide a `rubricPrompt` in OpenAI chat format that includes an image and use a vision-capable provider such as `openai:gpt-5.
+
+```yaml
+defaultTest:
+  options:
+    provider: openai:gpt-5
+    rubricPrompt: |
+      [
+        { "role": "system", "content": "Evaluate if the answer matches the image. Respond with JSON {reason:string, pass:boolean, score:number}" },
+        {
+          "role": "user",
+          "content": [
+            { "type": "image_url", "image_url": { "url": "{{image_url}}" } },
+            { "type": "text", "text": "Output: {{ output }}\nRubric: {{ rubric }}" }
+          ]
+        }
+      ]
+```
+
+#### select-best rubric prompt
+
+For control over the `select-best` rubric prompt, you may use the variables `{{outputs}}` (list of strings) and `{{criteria}}` (string). It expects the LLM output to contain the index of the winning output.
+
+## Classifiers
+
+Classifiers can be used to detect tone, bias, toxicity, helpfulness, and much more. See [classifier documentation](/docs/configuration/expected-outputs/classifier).
+
+---
+
+## Context-based
+
+Context-based assertions are a special class of model-graded assertions that evaluate whether the LLM's output is supported by context provided at inference time. They are particularly useful for evaluating RAG systems.
+
+- [`context-recall`](/docs/configuration/expected-outputs/model-graded/context-recall) - ensure that ground truth appears in context
+- [`context-relevance`](/docs/configuration/expected-outputs/model-graded/context-relevance) - ensure that context is relevant to original query
+- [`context-faithfulness`](/docs/configuration/expected-outputs/model-graded/context-faithfulness) - ensure that LLM output is supported by context
+
+### Defining context
+
+Context can be defined in one of two ways: statically using test case variables or dynamically from the provider's response.
+
+#### Statically via test variables
+
+Set `context` as a variable in your test case:
+
+```yaml
+tests:
+  - vars:
+      context: 'Paris is the capital of France. It has a population of over 2 million people.'
+    assert:
+      - type: context-recall
+        value: 'Paris is the capital of France'
+        threshold: 0.8
+```
+
+#### Dynamically via Context Transform
+
+Defining `contextTransform` allows you to construct context from provider responses. This is particularly useful for RAG systems.
+
+```yaml
+assert:
+  - type: context-faithfulness
+    contextTransform: 'output.citations.join("\n")'
+    threshold: 0.8
+```
+
+The `contextTransform` property accepts a stringified Javascript expression which itself accepts two arguments: `output` and `context`, and **must return a non-empty string.**
+
+```typescript
+/**
+ * The context transform function signature.
+ */
+type ContextTransform = (output: Output, context: Context) => string;
+
+/**
+ * The provider's response output.
+ */
+type Output = string | object;
+
+/**
+ * Metadata about the test case, prompt, and provider response.
+ */
+type Context = {
+  // Test case variables
+  vars: Record<string, string | object>;
+
+  // Raw prompt sent to LLM
+  prompt: {
+    label: string;
+  };
+
+  // Provider-specific metadata.
+  // The documentation for each provider will describe any available metadata.
+  metadata?: object;
+};
+```
+
+For example, given the following provider response:
+
+```typescript
+/**
+ * A response from a fictional Research Knowledge Base.
+ */
+type ProviderResponse = {
+  output: {
+    content: string;
+  };
+  metadata: {
+    retrieved_docs: {
+      content: string;
+    }[];
+  };
+};
+```
+
+```yaml
+assert:
+  - type: context-faithfulness
+    contextTransform: 'output.content'
+    threshold: 0.8
+
+  - type: context-relevance
+    # Note: `ProviderResponse['metadata']` is accessible as `context.metadata`
+    contextTransform: 'context.metadata.retrieved_docs.map(d => d.content).join("\n")'
+    threshold: 0.7
+```
+
+If your expression should return `undefined` or `null`, for example because no context is available, add a fallback:
+
+```yaml
+contextTransform: 'output.context ?? "No context found"'
+```
+
+If you expected your context to be non-empty, but it's empty, you can debug your provider response by returning a stringified version of the response:
+
+```yaml
+contextTransform: 'JSON.stringify(output, null, 2)'
+```
+
+### Examples
+
+Context-based metrics require a `query` and context. You must also set the `threshold` property on your test (all scores are normalized between 0 and 1).
+
+Here's an example config using statically-defined (`test.vars.context`) context:
+
+```yaml
+prompts:
+  - |
+    You are an internal corporate chatbot.
+    Respond to this query: {{query}}
+    Here is some context that you can use to write your response: {{context}}
+providers:
+  - openai:gpt-5
+tests:
+  - vars:
+      query: What is the max purchase that doesn't require approval?
+      context: file://docs/reimbursement.md
+    assert:
+      - type: contains
+        value: '$500'
+      - type: factuality
+        value: the employee's manager is responsible for approvals
+      - type: answer-relevance
+        threshold: 0.9
+      - type: context-recall
+        threshold: 0.9
+        value: max purchase price without approval is $500. Talk to Fred before submitting anything.
+      - type: context-relevance
+        threshold: 0.9
+      - type: context-faithfulness
+        threshold: 0.9
+  - vars:
+      query: How many weeks is maternity leave?
+      context: file://docs/maternity.md
+    assert:
+      - type: factuality
+        value: maternity leave is 4 months
+      - type: answer-relevance
+        threshold: 0.9
+      - type: context-recall
+        threshold: 0.9
+        value: The company offers 4 months of maternity leave, unless you are an elephant, in which case you get 22 months of maternity leave.
+      - type: context-relevance
+        threshold: 0.9
+      - type: context-faithfulness
+        threshold: 0.9
+```
+
+Alternatively, if your system returns context in the response, like in a RAG system, you can use `contextTransform`:
+
+```yaml
+prompts:
+  - |
+    You are an internal corporate chatbot.
+    Respond to this query: {{query}}
+providers:
+  - openai:gpt-5
+tests:
+  - vars:
+      query: What is the max purchase that doesn't require approval?
+    assert:
+      - type: context-recall
+        contextTransform: 'output.context'
+        threshold: 0.9
+        value: max purchase price without approval is $500
+      - type: context-relevance
+        contextTransform: 'output.context'
+        threshold: 0.9
+      - type: context-faithfulness
+        contextTransform: 'output.context'
+        threshold: 0.9
+```
+
+## Transforming outputs for context assertions
+
+### Transform: Extract answer before context grading
+
+```yaml
+providers:
+  - echo
+
+tests:
+  - vars:
+      prompt: '{"answer": "Paris is the capital of France", "confidence": 0.95}'
+      context: 'France is a country in Europe. Its capital city is Paris, which has over 2 million residents.'
+    assert:
+      - type: context-faithfulness
+        transform: 'JSON.parse(output).answer' # Grade only the answer field
+        threshold: 0.9
+
+      - type: context-recall
+        transform: 'JSON.parse(output).answer' # Check if answer appears in context
+        value: 'Paris is the capital of France'
+        threshold: 0.8
+```
+
+### Context transform: Extract context from provider response
+
+```yaml
+providers:
+  - echo
+
+tests:
+  - vars:
+      prompt: '{"answer": "Returns accepted within 30 days", "sources": ["Returns are accepted for 30 days from purchase", "30-day money-back guarantee"]}'
+      query: 'What is the return policy?'
+    assert:
+      - type: context-faithfulness
+        transform: 'JSON.parse(output).answer'
+        contextTransform: 'JSON.parse(output).sources.join(". ")' # Extract sources as context
+        threshold: 0.9
+
+      - type: context-relevance
+        contextTransform: 'JSON.parse(output).sources.join(". ")' # Check if context is relevant to query
+        threshold: 0.8
+```
+
+### Transform response: Normalize RAG system output
+
+```yaml
+providers:
+  - id: http://rag-api.example.com/search
+    config:
+      transformResponse: 'json.data' # Extract data field from API response
+
+tests:
+  - vars:
+      query: 'What are the office hours?'
+    assert:
+      - type: context-faithfulness
+        transform: 'output.answer' # After transformResponse, extract answer
+        contextTransform: 'output.documents.map(d => d.text).join(" ")' # Extract documents as context
+        threshold: 0.85
+```
+
+**Processing order:** API call → `transformResponse` → `transform` → `contextTransform` → context assertion
+
+## Common patterns and troubleshooting
+
+### Understanding pass vs. score behavior
+
+Model-graded assertions like `llm-rubric` determine PASS/FAIL using two mechanisms:
+
+1. **Without threshold**: PASS depends only on the grader's `pass` field (defaults to `true` if omitted)
+2. **With threshold**: PASS requires both `pass === true` AND `score >= threshold`
+
+This means a result like `{"pass": true, "score": 0}` will pass without a threshold, but fail with `threshold: 1`.
+
+**Common issue**: Tests show PASS even when scores are low
+
+```yaml
+# ❌ Problem: All tests pass regardless of score
+assert:
+  - type: llm-rubric
+    value: |
+      Return 0 if the response is incorrect
+      Return 1 if the response is correct
+    # No threshold set - always passes if grader doesn't return explicit pass: false
+```
+
+**Solutions**:
+
+```yaml
+# ✅ Option A: Add threshold to make score drive PASS/FAIL
+assert:
+  - type: llm-rubric
+    value: |
+      Return 0 if the response is incorrect
+      Return 1 if the response is correct
+    threshold: 1 # Only pass when score >= 1
+
+# ✅ Option B: Have grader control pass explicitly
+assert:
+  - type: llm-rubric
+    value: |
+      Return {"pass": true, "score": 1} if the response is correct
+      Return {"pass": false, "score": 0} if the response is incorrect
+```
+
+### Threshold usage across assertion types
+
+Different assertion types use thresholds differently:
+
+```yaml
+assert:
+  # Similarity-based (0-1 range)
+  - type: context-faithfulness
+    threshold: 0.8 # Requires 80%+ faithfulness
+
+  # Binary scoring (0 or 1)
+  - type: llm-rubric
+    value: 'Is helpful and accurate'
+    threshold: 1 # Requires perfect score
+
+  # Custom scoring (any range)
+  - type: pi
+    value: 'Quality of response'
+    threshold: 0.7
+```
+
+For more details on pass/score semantics, see the [llm-rubric documentation](/docs/configuration/expected-outputs/model-graded/llm-rubric#pass-vs-score-semantics).
+
+## Other assertion types
+
+For more info on assertions, see [Test assertions](/docs/configuration/expected-outputs).

site/docs/configuration/expected-outputs/model-graded/llm-rubric.md

@@ -0,0 +1,369 @@
+---
+sidebar_label: LLM Rubric
+description: 'Create flexible custom rubrics using natural language to evaluate LLM outputs against specific quality and safety criteria'
+---
+
+# LLM Rubric
+
+`llm-rubric` is promptfoo's general-purpose grader for "LLM as a judge" evaluation.
+
+It is similar to OpenAI's [model-graded-closedqa](/docs/configuration/expected-outputs) prompt, but can be more effective and robust in certain cases.
+
+## How to use it
+
+To use the `llm-rubric` assertion type, add it to your test configuration like this:
+
+```yaml
+assert:
+  - type: llm-rubric
+    # Specify the criteria for grading the LLM output:
+    value: Is not apologetic and provides a clear, concise answer
+```
+
+This assertion will use a language model to grade the output based on the specified rubric.
+
+## How it works
+
+Under the hood, `llm-rubric` uses a model to evaluate the output based on the criteria you provide. By default, it uses different models depending on which API keys are available:
+
+- **OpenAI API key**: `gpt-5`
+- **Codex/ChatGPT login**: `openai:codex-sdk` when the Codex SDK package is installed, Codex is signed in, and no higher-priority API credentials are set
+- **Anthropic API key**: `claude-sonnet-4-5-20250929`
+- **Google AI Studio API key**: `gemini-2.5-pro` (GEMINI_API_KEY, GOOGLE_API_KEY, or PALM_API_KEY)
+- **Google Vertex credentials**: `gemini-2.5-pro` (service account credentials)
+- **Mistral API key**: `mistral-large-latest`
+- **GitHub token**: `openai/gpt-5`
+- **Azure credentials**: Your configured Azure GPT deployment
+
+You can override this by setting the `provider` option (see below).
+
+Codex/ChatGPT login fallback is text-only. Assertions that need embeddings or moderation still require an API-key-backed provider override.
+
+When a judge needs to inspect files or use coding-agent tools as part of grading, use [`agent-rubric`](/docs/configuration/expected-outputs/model-graded/agent-rubric). Although an agent provider can also be supplied to `llm-rubric`, `agent-rubric` makes the capability intentional, validates that the grader is an agent runtime, and uses an agent-oriented safety prompt.
+
+It asks the model to output a JSON object that looks like this:
+
+```json
+{
+  "reason": "<Analysis of the rubric and the output>",
+  "score": 0.5, // 0.0-1.0
+  "pass": true // true or false
+}
+```
+
+Use your knowledge of this structure to give special instructions in your rubric, for example:
+
+```yaml
+assert:
+  - type: llm-rubric
+    value: |
+      Evaluate the output based on how funny it is. Grade it on a scale of 0.0 to 1.0, where:
+      Score of 0.1: Only a slight smile.
+      Score of 0.5: Laughing out loud.
+      Score of 1.0: Rolling on the floor laughing.
+
+      Anything funny enough to be on SNL should pass, otherwise fail.
+```
+
+## Using variables in the rubric
+
+You can incorporate test variables into your LLM rubric. This is particularly useful for detecting hallucinations or ensuring the output addresses specific aspects of the input. Here's an example:
+
+```yaml
+providers:
+  - openai:gpt-5
+prompts:
+  - file://prompt1.txt
+  - file://prompt2.txt
+defaultTest:
+  assert:
+    - type: llm-rubric
+      value: 'Provides a direct answer to the question: "{{question}}" without unnecessary elaboration'
+tests:
+  - vars:
+      question: What is the capital of France?
+  - vars:
+      question: How many planets are in our solar system?
+```
+
+## Overriding the LLM grader
+
+By default, `llm-rubric` uses `gpt-5` for grading. You can override this in several ways:
+
+1. Using the `--grader` CLI option:
+
+   ```sh
+   promptfoo eval --grader openai:gpt-5-mini
+   ```
+
+2. Using `test.options` or `defaultTest.options`:
+
+   ```yaml
+   defaultTest:
+     // highlight-start
+     options:
+       provider: openai:gpt-5-mini
+     // highlight-end
+   tests:
+     - assert:
+         - type: llm-rubric
+           value: Is written in a professional tone
+   ```
+
+3. Using `assertion.provider`:
+
+   ```yaml
+   tests:
+     - description: Evaluate output using LLM
+       assert:
+         - type: llm-rubric
+           value: Is written in a professional tone
+           // highlight-start
+           provider: openai:gpt-5-mini
+           // highlight-end
+   ```
+
+### Setting grader parameters (temperature, etc.)
+
+To pin `temperature`, `max_tokens`, or other provider-specific parameters on the grader, expand the `provider` shorthand into an object with `id` and `config`. This is the supported way to push grading toward reproducibility when swapping in a custom judge:
+
+```yaml
+assert:
+  - type: llm-rubric
+    value: Is not apologetic and provides a clear, concise answer
+    // highlight-start
+    provider:
+      id: openai:gpt-5-mini
+      config:
+        temperature: 0
+    // highlight-end
+```
+
+The same shape works under `defaultTest.options.provider` and `test.options.provider`.
+
+Provider precedence is exact: `assertion.provider` overrides `test.options.provider`, which
+overrides `defaultTest.options.provider`. If your default grader is a full object with `config`, do
+not add a shorthand `provider: openai:chat:...` on the assertion unless you also repeat the full
+object there.
+
+:::note
+The built-in OpenAI grader already defaults to `temperature=0`, so this override is only needed when you're pointing at a different model or provider whose default differs. GPT-5 series reasoning models ignore `temperature` and do not need it set.
+:::
+
+Custom `llm-rubric` providers can also return a `metadata` object in their `ProviderResponse`. promptfoo copies those keys onto the assertion's `GradingResult.metadata` alongside `renderedGradingPrompt`, which makes per-assertion fields such as upload IDs or trace IDs available in hooks like `afterEach`.
+
+### OpenAI-compatible judges with thinking output
+
+Some self-hosted OpenAI-compatible judges, including vLLM servers configured with reasoning parsers,
+return hidden reasoning separately from final content. Promptfoo includes that reasoning in provider
+output by default. For a judge, that can confuse JSON parsing if the reasoning contains scratchpad
+objects before the final `{"pass": ..., "score": ..., "reason": ...}` verdict.
+
+Set `showThinking: false` on the judge provider. See the
+[vLLM provider guide](/docs/providers/vllm#use-vllm-as-an-llm-judge) for a complete local judge
+recipe, including truncated `<think>` output and request-level thinking controls. The same rule
+applies to other model-graded assertions that use a text judge; see the
+[model-graded overview](/docs/configuration/expected-outputs/model-graded#openai-compatible-thinking-judges)
+for the full metric list.
+
+## Customizing the rubric prompt
+
+For more control over the `llm-rubric` evaluation, you can set a custom prompt using the `rubricPrompt` property:
+
+```yaml
+defaultTest:
+  options:
+    rubricPrompt: >
+      [
+          {
+              "role": "system",
+              "content": "Evaluate the following output based on these criteria:\n1. Clarity of explanation\n2. Accuracy of information\n3. Relevance to the topic\n\nProvide a score out of 10 for each criterion and an overall assessment."
+          },
+          {
+              "role": "user",
+              "content": "Output to evaluate: {{output}}\n\nRubric: {{rubric}}"
+          }
+      ]
+```
+
+## Non-English Evaluation
+
+To get evaluation output in languages other than English, you can use different approaches:
+
+### Option 1: rubricPrompt Override (Recommended)
+
+For reliable multilingual output with compatible assertion types:
+
+```yaml
+defaultTest:
+  options:
+    rubricPrompt: |
+      [
+        {
+          "role": "system",
+          // German: "You evaluate outputs based on criteria. Respond with JSON: {\"reason\": \"string\", \"pass\": boolean, \"score\": number}. ALL responses in German."
+          "content": "Du bewertest Ausgaben nach Kriterien. Antworte mit JSON: {\"reason\": \"string\", \"pass\": boolean, \"score\": number}. ALLE Antworten auf Deutsch."
+        },
+        {
+          "role": "user", 
+          // German: "Output: {{ output }}\nCriterion: {{ rubric }}"
+          "content": "Ausgabe: {{ output }}\nKriterium: {{ rubric }}"
+        }
+      ]
+
+assert:
+  - type: llm-rubric
+    # German: "Responds politely and helpfully"
+    value: 'Antwortet höflich und hilfreich'
+```
+
+### Option 2: Language Instructions in Rubric
+
+```yaml
+assert:
+  - type: llm-rubric
+    value: 'Responds politely and helpfully. Provide your evaluation reason in German.'
+```
+
+### Option 3: Full Native Language Rubric
+
+```yaml
+# German
+assert:
+  - type: llm-rubric
+    # German: "Responds politely and helpfully. Provide reasoning in German."
+    value: 'Antwortet höflich und hilfreich. Begründung auf Deutsch geben.'
+
+# Japanese
+assert:
+  - type: llm-rubric
+    # Japanese: "Does not contain harmful content. Please provide evaluation reasoning in Japanese."
+    value: '有害なコンテンツを含まない。評価理由は日本語で答えてください。'
+```
+
+**Note:** Option 1 works with `llm-rubric`, `g-eval`, and `model-graded-closedqa`. For other assertion types like `factuality` or `context-recall`, create assertion-specific prompts that match their expected formats.
+
+### Assertion-Specific Prompts
+
+For assertions requiring specific output formats:
+
+```yaml
+# factuality - requires {"category": "A/B/C/D/E", "reason": "..."}
+tests:
+  - options:
+      rubricPrompt: |
+        [
+          {
+            "role": "system",
+            // German: "You compare factual accuracy. Respond with JSON: {\"category\": \"A/B/C/D/E\", \"reason\": \"string\"}. A=subset, B=superset, C=identical, D=contradiction, E=irrelevant. ALL responses in German."
+            "content": "Du vergleichst Faktentreue. Antworte mit JSON: {\"category\": \"A/B/C/D/E\", \"reason\": \"string\"}. A=Teilmenge, B=Obermenge, C=identisch, D=Widerspruch, E=irrelevant. ALLE Antworten auf Deutsch."
+          },
+          {
+            "role": "user",
+            // German: "Expert answer: {{ rubric }}\nSubmitted answer: {{ output }}"
+            "content": "Expertenantwort: {{ rubric }}\nEingereichte Antwort: {{ output }}"
+          }
+        ]
+    assert:
+      - type: factuality
+        # German: "Berlin is the capital of Germany"
+        value: 'Berlin ist die Hauptstadt von Deutschland'
+```
+
+### Object handling in rubric prompts
+
+When using `{{output}}` or `{{rubric}}` variables that contain objects, promptfoo automatically converts them to JSON strings by default to prevent display issues. If you need to access specific properties of objects in your rubric prompts, you can enable object property access:
+
+```bash
+export PROMPTFOO_DISABLE_OBJECT_STRINGIFY=true
+promptfoo eval
+```
+
+With this enabled, you can access object properties directly in your rubric prompts:
+
+```yaml
+rubricPrompt: >
+  [
+      {
+          "role": "user", 
+          "content": "Evaluate this answer: {{output.text}}\nFor the question: {{rubric.question}}\nCriteria: {{rubric.criteria}}"
+      }
+  ]
+```
+
+For more details, see the [object template handling guide](/docs/usage/troubleshooting#object-template-handling).
+
+## Threshold Support
+
+The `llm-rubric` assertion type supports an optional `threshold` property that sets a minimum score requirement. When specified, the output must achieve a score greater than or equal to the threshold to pass. For example:
+
+```yaml
+assert:
+  - type: llm-rubric
+    value: Is not apologetic and provides a clear, concise answer
+    threshold: 0.8 # Requires a score of 0.8 or higher to pass
+```
+
+The threshold is applied to the score returned by the LLM (which ranges from 0.0 to 1.0). If the LLM returns an explicit pass/fail status, the threshold will still be enforced - both conditions must be met for the assertion to pass.
+
+## Pass vs. Score Semantics
+
+- PASS is determined by the LLM's boolean `pass` field unless you set a `threshold`.
+- If the model omits `pass`, promptfoo assumes `pass: true` by default.
+- `score` is a numeric metric that does not affect PASS/FAIL unless you set `threshold`.
+- When `threshold` is set, both must be true for the assertion to pass:
+  - `pass === true`
+  - `score >= threshold`
+
+This means that without a `threshold`, a result like `{ pass: true, score: 0 }` will pass. If you want the numeric score (e.g., 0/1 rubric) to drive PASS/FAIL, set a `threshold` accordingly or have the model return explicit `pass`.
+
+:::caution
+If the model omits `pass` and you don't set `threshold`, the assertion passes even with `score: 0`.
+:::
+
+### Common misconfiguration
+
+```yaml
+# ❌ Problem: Returns 0/1 scores but no threshold set
+assert:
+  - type: llm-rubric
+    value: |
+      Return 0 if the response is incorrect
+      Return 1 if the response is correct
+    # Missing threshold - always passes due to pass defaulting to true
+```
+
+**Fixes:**
+
+```yaml
+# ✅ Option A: Add threshold
+assert:
+  - type: llm-rubric
+    value: |
+      Return 0 if the response is incorrect
+      Return 1 if the response is correct
+    threshold: 1
+
+# ✅ Option B: Control pass explicitly
+assert:
+  - type: llm-rubric
+    value: |
+      Return {"pass": true, "score": 1} if the response is correct
+      Return {"pass": false, "score": 0} if the response is incorrect
+```
+
+## Negation with `not-llm-rubric`
+
+Prepend `not-` to invert the assertion — useful for "must not" criteria:
+
+```yaml
+assert:
+  - type: not-llm-rubric
+    value: Apologizes or hedges before answering
+```
+
+`not-llm-rubric` passes when the rubric criterion does **not** match. Transport or parse failures from the grader are reported as failures in both directions — a grader error is not treated as evidence that the criterion was or was not met, so inversion never silently turns a failed grader call into a pass.
+
+## Further reading
+
+See [model-graded metrics](/docs/configuration/expected-outputs/model-graded) for more options.

site/docs/configuration/expected-outputs/model-graded/max-score.md

@@ -0,0 +1,242 @@
+---
+title: Max-score assertion
+description: Configure the `max-score` assertion to deterministically pick the highest-scoring output based on other assertions' scores.
+sidebar_label: Max Score
+---
+
+# Max Score
+
+The `max-score` assertion selects the output with the highest aggregate score from other assertions. Unlike `select-best` which uses LLM judgment, `max-score` provides objective, deterministic selection based on quantitative scores from other assertions.
+
+## When to use max-score
+
+Use `max-score` when you want to:
+
+- Select the best output based on objective, measurable criteria
+- Combine multiple metrics with different importance (weights)
+- Have transparent, reproducible selection without LLM API calls
+- Select outputs based on a combination of correctness, quality, and other metrics
+
+## How it works
+
+1. All regular assertions run first on each output
+2. `max-score` collects the scores from these assertions
+3. Calculates an aggregate score for each output (average by default)
+4. Selects the output with the highest aggregate score
+5. Returns pass=true for the highest scoring output, pass=false for others
+
+## Basic usage
+
+```yaml
+prompts:
+  - 'Write a function to {{task}}'
+  - 'Write an efficient function to {{task}}'
+  - 'Write a well-documented function to {{task}}'
+
+providers:
+  - openai:gpt-5
+
+tests:
+  - vars:
+      task: 'calculate fibonacci numbers'
+    assert:
+      # Regular assertions that score each output
+      - type: python
+        value: 'assert fibonacci(10) == 55'
+      - type: llm-rubric
+        value: 'Code is efficient'
+      - type: contains
+        value: 'def fibonacci'
+      # Max-score selects the output with highest average score
+      - type: max-score
+```
+
+## Configuration options
+
+### Aggregation method
+
+Choose how scores are combined:
+
+```yaml
+assert:
+  - type: max-score
+    value:
+      method: average # Default: average | sum
+```
+
+### Weighted scoring
+
+Give different importance to different assertions by specifying weights per assertion type:
+
+```yaml
+assert:
+  - type: python # Test correctness
+  - type: llm-rubric # Test quality
+    value: 'Well documented'
+  - type: max-score
+    value:
+      weights:
+        python: 3 # Correctness is 3x more important
+        llm-rubric: 1 # Documentation is 1x weight
+```
+
+#### How weights work
+
+- Each assertion type can have a custom weight (default: 1.0)
+- For `method: average`, the final score is: `sum(score × weight) / sum(weights)`
+- For `method: sum`, the final score is: `sum(score × weight)`
+- Weights apply to all assertions of that type
+
+Example calculation with `method: average`:
+
+```
+Output A: python=1.0, llm-rubric=0.5, contains=1.0
+Weights:  python=3,   llm-rubric=1,   contains=1 (default)
+
+Score = (1.0×3 + 0.5×1 + 1.0×1) / (3 + 1 + 1)
+      = (3.0 + 0.5 + 1.0) / 5
+      = 0.9
+```
+
+### Minimum threshold
+
+Require a minimum score for selection:
+
+```yaml
+assert:
+  - type: max-score
+    value:
+      threshold: 0.7 # Only select if average score >= 0.7
+```
+
+## Scoring details
+
+- **Binary assertions** (pass/fail): Score as 1.0 or 0.0
+- **Scored assertions**: Use the numeric score (typically 0-1 range)
+- **Default weights**: 1.0 for all assertions
+- **Tie breaking**: First output wins (deterministic)
+
+## Examples
+
+### Example 1: Multi-criteria code selection
+
+```yaml
+prompts:
+  - 'Write a Python function to {{task}}'
+  - 'Write an optimized Python function to {{task}}'
+  - 'Write a documented Python function to {{task}}'
+
+providers:
+  - openai:gpt-5-mini
+
+tests:
+  - vars:
+      task: 'merge two sorted lists'
+    assert:
+      - type: python
+        value: |
+          list1 = [1, 3, 5]
+          list2 = [2, 4, 6]
+          result = merge_lists(list1, list2)
+          assert result == [1, 2, 3, 4, 5, 6]
+
+      - type: llm-rubric
+        value: 'Code has O(n+m) time complexity'
+
+      - type: llm-rubric
+        value: 'Code is well documented with docstring'
+
+      - type: max-score
+        value:
+          weights:
+            python: 3 # Correctness most important
+            llm-rubric: 1 # Each quality metric has weight 1
+```
+
+### Example 2: Content generation selection
+
+```yaml
+prompts:
+  - 'Explain {{concept}} simply'
+  - 'Explain {{concept}} in detail'
+  - 'Explain {{concept}} with examples'
+
+providers:
+  - anthropic:claude-3-haiku-20240307
+
+tests:
+  - vars:
+      concept: 'machine learning'
+    assert:
+      - type: llm-rubric
+        value: 'Explanation is accurate'
+
+      - type: llm-rubric
+        value: 'Explanation is clear and easy to understand'
+
+      - type: contains
+        value: 'example'
+
+      - type: max-score
+        value:
+          method: average # All criteria equally important
+```
+
+### Example 3: API response selection
+
+```yaml
+tests:
+  - vars:
+      query: 'weather in Paris'
+    assert:
+      - type: is-json
+
+      - type: contains-json
+        value:
+          required: ['temperature', 'humidity', 'conditions']
+
+      - type: llm-rubric
+        value: 'Response includes all requested weather data'
+
+      - type: latency
+        threshold: 1000 # Under 1 second
+
+      - type: max-score
+        value:
+          weights:
+            is-json: 2 # Must be valid JSON
+            contains-json: 2 # Must have required fields
+            llm-rubric: 1 # Quality check
+            latency: 1 # Performance matters
+```
+
+## Comparison with select-best
+
+| Feature          | max-score                        | select-best         |
+| ---------------- | -------------------------------- | ------------------- |
+| Selection method | Aggregate scores from assertions | LLM judgment        |
+| API calls        | None (uses existing scores)      | One per eval        |
+| Reproducibility  | Deterministic                    | May vary            |
+| Best for         | Objective criteria               | Subjective criteria |
+| Transparency     | Shows exact scores               | Shows LLM reasoning |
+| Cost             | Free (no API calls)              | Costs per API call  |
+
+## Edge cases
+
+- **No other assertions**: Error - max-score requires at least one assertion to aggregate
+- **Tie scores**: First output wins (by index)
+- **All outputs fail**: Still selects the highest scorer ("least bad")
+- **Below threshold**: No output selected if threshold is specified and not met
+
+## Tips
+
+1. **Use specific assertions**: More assertions provide better signal for selection
+2. **Weight important criteria**: Use weights to emphasize what matters most
+3. **Combine with select-best**: You can use both in the same test for comparison
+4. **Debug with scores**: The output shows aggregate scores for transparency
+
+## Further reading
+
+- [Model-graded metrics](/docs/configuration/expected-outputs/model-graded) for other model-based assertions
+- [Select best](/docs/configuration/expected-outputs/model-graded/select-best) for subjective selection
+- [Assertions](/docs/configuration/expected-outputs) for all available assertion types

site/docs/configuration/expected-outputs/model-graded/model-graded-closedqa.md

@@ -0,0 +1,94 @@
+---
+sidebar_label: Model-graded Closed QA
+description: 'Assess closed-domain QA performance using model-based evaluation for accuracy, completeness, and answer correctness'
+---
+
+# Model-graded Closed QA
+
+`model-graded-closedqa` is a criteria-checking evaluation that uses OpenAI's public evals prompt to determine if an LLM output meets specific requirements.
+
+### How to use it
+
+To use the `model-graded-closedqa` assertion type, add it to your test configuration like this:
+
+```yaml
+assert:
+  - type: model-graded-closedqa
+    # Specify the criteria that the output must meet:
+    value: Provides a clear answer without hedging or uncertainty
+```
+
+This assertion will use a language model to evaluate whether the output meets the specified criterion, returning a simple yes/no response.
+
+### How it works
+
+Under the hood, `model-graded-closedqa` uses OpenAI's closed QA evaluation prompt to analyze the output. The grader will return:
+
+- `Y` if the output meets the criterion
+- `N` if the output does not meet the criterion
+
+The assertion passes if the response ends with 'Y' and fails if it ends with 'N'.
+
+### Example Configuration
+
+Here's a complete example showing how to use model-graded-closedqa:
+
+```yaml
+prompts:
+  - 'What is {{topic}}?'
+providers:
+  - openai:gpt-5
+tests:
+  - vars:
+      topic: quantum computing
+    assert:
+      - type: model-graded-closedqa
+        value: Explains the concept without using technical jargon
+      - type: model-graded-closedqa
+        value: Includes a practical real-world example
+```
+
+### Overriding the Grader
+
+Like other model-graded assertions, you can override the default grader:
+
+1. Using the CLI:
+
+   ```sh
+   promptfoo eval --grader openai:gpt-5-mini
+   ```
+
+2. Using test options:
+
+   ```yaml
+   defaultTest:
+     options:
+       provider: openai:gpt-5-mini
+   ```
+
+3. Using assertion-level override:
+   ```yaml
+   assert:
+     - type: model-graded-closedqa
+       value: Is concise and clear
+       provider: openai:gpt-5-mini
+   ```
+
+### Customizing the Prompt
+
+You can customize the evaluation prompt using the `rubricPrompt` property:
+
+```yaml
+defaultTest:
+  options:
+    rubricPrompt: |
+      Question: {{input}}
+      Criterion: {{criteria}}
+      Response: {{completion}}
+
+      Does this response meet the criterion? Answer Y or N.
+```
+
+# Further reading
+
+See [model-graded metrics](/docs/configuration/expected-outputs/model-graded) for more options.

site/docs/configuration/expected-outputs/model-graded/pi.md

@@ -0,0 +1,115 @@
+---
+sidebar_position: 8
+description: 'Identify and block prompt injection attacks using advanced model-based classification for enhanced security protection'
+---
+
+# Pi Scorer
+
+`pi` is an alternative approach to model grading that uses a dedicated scoring model instead of the "LLM as a judge" technique. It can evaluate input and output pairs against criteria.
+
+:::note
+**Important**: Unlike `llm-rubric` which works with your existing providers, Pi requires a separate external API key from Pi Labs.
+:::
+
+## Alternative Approach
+
+Pi offers a different approach to evaluation with some distinct characteristics:
+
+- Uses a dedicated scoring model rather than prompting an LLM to act as a judge
+- Focuses on highly accurate numeric scoring without providing detailed reasoning
+- Aims for consistency in scoring the same inputs
+- Requires a separate API key and integration
+
+Each approach has different strengths, and you may want to experiment with both to determine which best suits your specific evaluation needs.
+
+## Prerequisites
+
+To use Pi, you **must** first:
+
+1. Create a Pi API key from [Pi Labs](https://build.withpi.ai/account/keys)
+2. Set the `WITHPI_API_KEY` environment variable
+
+```bash
+export WITHPI_API_KEY=your_api_key_here
+```
+
+or set
+
+```yaml
+env:
+  WITHPI_API_KEY: your_api_key_here
+```
+
+in your promptfoo config
+
+## How to use it
+
+To use the `pi` assertion type, add it to your test configuration:
+
+```yaml
+assert:
+  - type: pi
+    # Specify the criteria for grading the LLM output
+    value: Is the response not apologetic and provides a clear, concise answer?
+```
+
+This assertion will use the Pi scorer to grade the output based on the specified criteria.
+
+## How it works
+
+Under the hood, the `pi` assertion uses the `withpi` SDK to evaluate the output based on the criteria you provide.
+
+Compared to LLM as a judge:
+
+- The inputs of the eval are the same: `llm_input` and `llm_output`
+- Pi does not need a system prompt, and is pretrained to score
+- Pi always generates the same score, when given the same input
+- Pi requires a separate API key (see Prerequisites section)
+
+## Threshold Support
+
+The `pi` assertion type supports an optional `threshold` property that sets a minimum score requirement. When specified, the output must achieve a score greater than or equal to the threshold to pass.
+
+```yaml
+assert:
+  - type: pi
+    value: Is not apologetic and provides a clear, concise answer
+    threshold: 0.8 # Requires a score of 0.8 or higher to pass
+```
+
+:::info
+The default threshold is `0.5` if not specified.
+:::
+
+## Metrics Brainstorming
+
+You can use the [Pi Labs Copilot](https://build.withpi.ai) to interactively brainstorm representative metrics for your application. It helps you:
+
+1. Generate effective evaluation criteria
+2. Test metrics on example outputs before integration
+3. Find the optimal threshold values for your use case
+
+## Example Configuration
+
+```yaml
+prompts:
+  - 'Explain {{concept}} in simple terms.'
+providers:
+  - openai:gpt-5
+tests:
+  - vars:
+      concept: quantum computing
+    assert:
+      - type: pi
+        value: Is the explanation easy to understand without technical jargon?
+        threshold: 0.7
+      - type: pi
+        value: Does the response correctly explain the fundamental principles?
+        threshold: 0.8
+```
+
+## See Also
+
+- [LLM Rubric](/docs/configuration/expected-outputs/model-graded/llm-rubric)
+- [Model-graded metrics](/docs/configuration/expected-outputs/model-graded)
+- [Pi Documentation](https://docs.withpi.ai) for more options, configuration, and calibration details

site/docs/configuration/expected-outputs/model-graded/search-rubric.md

@@ -0,0 +1,286 @@
+---
+sidebar_label: Search Rubric
+---
+
+# Search-Rubric
+
+The `search-rubric` assertion type is like `llm-rubric` but with web search capabilities. It evaluates outputs according to a rubric while having the ability to search for current information when needed.
+
+## How it works
+
+1. You provide a rubric that describes what the output should contain
+2. The grading provider evaluates the output against the rubric
+3. If the rubric requires current information, the provider searches the web
+4. Returns pass/fail with a score from 0.0 to 1.0
+
+## Basic Usage
+
+```yaml
+assert:
+  - type: search-rubric
+    value: 'Provides accurate current Bitcoin price within 5% of market value'
+```
+
+## Comparing to LLM-Rubric
+
+The `search-rubric` assertion behaves exactly like `llm-rubric`, but automatically uses a provider with web search capabilities:
+
+```yaml
+# These are equivalent:
+assert:
+  # Using llm-rubric with a web-search capable provider
+  - type: llm-rubric
+    value: 'Contains current stock price for Apple (AAPL) within $5'
+    provider: openai:responses:gpt-5.1 # Must configure web search tool
+
+  # Using search-rubric (automatically selects a web-search provider)
+  - type: search-rubric
+    value: 'Contains current stock price for Apple (AAPL) within $5'
+```
+
+## Using Variables in the Rubric
+
+Like `llm-rubric`, you can use test variables:
+
+```yaml
+prompts:
+  - 'What is the current weather in {{city}}?'
+
+assert:
+  - type: search-rubric
+    value: 'Provides current temperature in {{city}} with units (F or C)'
+
+tests:
+  - vars:
+      city: San Francisco
+  - vars:
+      city: Tokyo
+```
+
+## Grading Providers
+
+The search-rubric assertion requires a grading provider with web search capabilities:
+
+### 1. Anthropic Claude
+
+Anthropic Claude models support web search through the `web_search_20250305` tool:
+
+```yaml
+grading:
+  provider: anthropic:messages:claude-opus-4-6
+  providerOptions:
+    config:
+      tools:
+        - type: web_search_20250305
+          name: web_search
+          max_uses: 5
+```
+
+### 2. OpenAI with Web Search
+
+OpenAI's responses API supports web search through the `web_search_preview` tool:
+
+```yaml
+grading:
+  provider: openai:responses:gpt-5.1
+  providerOptions:
+    config:
+      tools:
+        - type: web_search_preview
+```
+
+### 3. Perplexity
+
+Perplexity models have built-in web search:
+
+```yaml
+grading:
+  provider: perplexity:sonar
+```
+
+### 4. Google Gemini
+
+Google's Gemini models support web search through the `googleSearch` tool:
+
+```yaml
+grading:
+  provider: google:gemini-3.1-pro-preview
+  providerOptions:
+    config:
+      tools:
+        - googleSearch: {}
+```
+
+### 5. xAI Grok
+
+xAI's Grok models can use server-side web search tools through the Responses API:
+
+```yaml
+grading:
+  provider: xai:responses:grok-4.3
+  providerOptions:
+    config:
+      tools:
+        - type: web_search
+```
+
+## Use Cases
+
+### 1. Current Events Verification
+
+```yaml
+prompts:
+  - 'Who won the latest Super Bowl?'
+
+assert:
+  - type: search-rubric
+    value: 'Names the correct winner of the most recent Super Bowl with the final score'
+```
+
+### 2. Real-time Price Checking
+
+```yaml
+prompts:
+  - "What's the current stock price of {{ticker}}?"
+
+assert:
+  - type: search-rubric
+    value: |
+      Provides accurate stock price for {{ticker}} that:
+      1. Is within 2% of current market price
+      2. Includes currency (USD)
+      3. Mentions if market is open or closed
+    threshold: 0.8
+```
+
+### 3. Weather Information
+
+```yaml
+prompts:
+  - "What's the weather like in Tokyo?"
+
+assert:
+  - type: search-rubric
+    value: |
+      Describes current Tokyo weather including:
+      - Temperature (with units)
+      - General conditions (sunny, rainy, etc.)
+      - Humidity or precipitation if relevant
+```
+
+### 4. Latest Software Versions
+
+```yaml
+prompts:
+  - "What's the latest version of Node.js?"
+
+assert:
+  - type: search-rubric
+    value: 'States the correct latest LTS version of Node.js (not experimental or nightly)'
+```
+
+## Cost Considerations
+
+Web search assertions have the following cost implications. As of November 2025:
+
+- **Anthropic Claude**: $10 per 1,000 web search calls plus token costs
+- **OpenAI**: Web search tools on the Responses API cost $10-25 per 1,000 tool calls in addition to token usage
+- **Google Gemini API**: $35 per 1,000 grounded prompts; **Vertex AI Web Grounding**: $45 per 1,000
+- **Perplexity**: Per-request plus token-based pricing; see Perplexity or your proxy's pricing page
+- **xAI Grok**: $25 per 1,000 sources plus token usage for Live Search
+
+## Threshold Support
+
+Like `llm-rubric`, the `search-rubric` assertion supports thresholds:
+
+```yaml
+assert:
+  - type: search-rubric
+    value: 'Contains accurate information about current US inflation rate'
+    threshold: 0.9 # Requires 90% accuracy for economic data
+```
+
+## Best Practices
+
+1. **Write clear rubrics**: Be specific about what information you expect
+2. **Use thresholds appropriately**: Higher thresholds for factual accuracy, lower for general correctness
+3. **Include acceptable ranges**: For volatile data like prices, specify acceptable accuracy (e.g., "within 5%")
+4. **Use caching**: Caching is enabled by default; use `promptfoo eval --no-cache` to force fresh searches
+5. **Test variable substitution**: Ensure your rubrics work with different variable values
+
+## Expected Behavior
+
+Understanding how `search-rubric` evaluates different scenarios helps you write better tests.
+
+### What the grader catches
+
+The search-enabled grader identifies several types of failures:
+
+| SUT Response                            | Grader Verdict | Reason                            |
+| --------------------------------------- | -------------- | --------------------------------- |
+| "I don't have access to real-time data" | **Fail**       | No actual answer provided         |
+| Stale price from training data          | **Fail**       | Value differs from current market |
+| Correct current price                   | **Pass**       | Matches web search results        |
+| Partially correct answer                | **Partial**    | Score reflects completeness       |
+
+### Models without web search
+
+Models like `gpt-4o-mini` without web search enabled will often refuse to answer real-time questions:
+
+> "I don't have access to real-time stock data. For current prices, please check a financial website."
+
+The `search-rubric` grader correctly flags this as a failure since no actual information was provided. This is the expected behavior—the assertion is verifying whether your system provides accurate current information, not whether it gracefully declines.
+
+**To test models that confidently answer (and potentially hallucinate):**
+
+- Use a more capable model as the system under test
+- Enable web search on your SUT if available
+- Test against models known to attempt answers even when uncertain
+
+### Partial matches and scoring
+
+The grader returns a score from 0.0 to 1.0 based on how well the output matches the rubric:
+
+- **1.0**: Fully matches all rubric criteria
+- **0.7-0.9**: Matches most criteria, minor issues
+- **0.4-0.6**: Partial match, missing key information
+- **0.0-0.3**: Significant errors or refusal to answer
+
+Use the `threshold` parameter to set your acceptable score level.
+
+## Troubleshooting
+
+### "No provider with web search capabilities"
+
+Ensure your grading provider supports web search. Default providers without web search configuration will fail. Check the [Grading Providers](#grading-providers) section above.
+
+### Test always fails with refusal
+
+If your SUT consistently refuses to answer real-time questions, this is expected behavior for models without web access. The `search-rubric` grader is correctly identifying that no factual answer was provided.
+
+**Solutions:**
+
+1. Use a model with web search capabilities as your SUT
+2. Accept that models without real-time access cannot answer these questions
+3. Use `llm-rubric` instead if you only need to verify the response format
+
+### Inaccurate results
+
+The grader relies on web search results, which may occasionally be wrong or ambiguous.
+
+**Best practices:**
+
+- Write rubrics that can be verified from multiple reputable sources
+- Avoid rubrics about speculative or disputed claims
+- Use appropriate thresholds (not 1.0) to allow for minor discrepancies
+
+### High costs
+
+Web search adds cost on top of model tokens.
+
+**Cost reduction strategies:**
+
+- Caching is enabled by default to reduce API calls
+- Reserve `search-rubric` for tests that truly need real-time verification
+- Use `llm-rubric` for static fact-checking that doesn't require current data
+- Consider Perplexity's `sonar` model for built-in search without per-call fees

site/docs/configuration/expected-outputs/model-graded/select-best.md

@@ -0,0 +1,102 @@
+---
+sidebar_label: Select Best
+description: 'Leverage AI models to automatically select and rank the best outputs from multiple LLM responses for quality optimization'
+---
+
+# Select Best
+
+The `select-best` assertion compares multiple outputs in the same test case and selects the one that best meets a specified criterion. This is useful for comparing different prompt or model variations to determine which produces the best result.
+
+### How to use it
+
+To use the `select-best` assertion type, add it to your test configuration like this:
+
+```yaml
+assert:
+  - type: select-best
+    value: 'choose the most concise and accurate response'
+```
+
+Note: This assertion requires multiple prompts or providers to generate different outputs to compare.
+
+### How it works
+
+The select-best checker:
+
+1. Takes all outputs from the test case
+2. Evaluates each output against the specified criterion
+3. Selects the best output
+4. Returns pass=true for the winning output and pass=false for others
+
+### Example Configuration
+
+Here's a complete example showing how to use select-best to compare different prompt variations:
+
+```yaml
+prompts:
+  - 'Write a tweet about {{topic}}'
+  - 'Write a very concise, funny tweet about {{topic}}'
+  - 'Compose a tweet about {{topic}} that will go viral'
+providers:
+  - openai:gpt-5
+tests:
+  - vars:
+      topic: 'artificial intelligence'
+    assert:
+      - type: select-best
+        value: 'choose the tweet that is most likely to get high engagement'
+  - vars:
+      topic: 'climate change'
+    assert:
+      - type: select-best
+        value: 'choose the tweet that best balances information and humor'
+```
+
+### Overriding the Grader
+
+Like other model-graded assertions, you can override the default grader:
+
+1. Using the CLI:
+
+   ```sh
+   promptfoo eval --grader openai:gpt-5-mini
+   ```
+
+2. Using test options:
+
+   ```yaml
+   defaultTest:
+     options:
+       provider: openai:gpt-5-mini
+   ```
+
+3. Using assertion-level override:
+   ```yaml
+   assert:
+     - type: select-best
+       value: 'choose the most engaging response'
+       provider: openai:gpt-5-mini
+   ```
+
+### Customizing the Prompt
+
+You can customize the evaluation prompt using the `rubricPrompt` property:
+
+```yaml
+defaultTest:
+  options:
+    rubricPrompt: |
+      Here are {{ outputs | length }} responses:
+      {% for output in outputs %}
+      Output {{ loop.index0 }}: {{ output }}
+      {% endfor %}
+
+      Criteria: {{ criteria }}
+
+      Analyze each output against the criteria.
+      Choose the best output by responding with its index (0 to {{ outputs | length - 1 }}).
+```
+
+# Further reading
+
+See [model-graded metrics](/docs/configuration/expected-outputs/model-graded) for more options.

site/docs/configuration/expected-outputs/moderation.md

@@ -0,0 +1,199 @@
+---
+sidebar_position: 100
+sidebar_label: Moderation
+description: Implement comprehensive content moderation using multiple APIs to detect and filter harmful, toxic, or policy-violating outputs
+---
+
+# Moderation
+
+Use the `moderation` assert type to ensure that LLM outputs are safe.
+
+Currently, this supports [OpenAI's moderation model](https://platform.openai.com/docs/guides/moderation), Meta's LlamaGuard models (LlamaGuard 3 and 4) via [Replicate](https://replicate.com/meta/llama-guard-3-8b), and [Azure Content Safety API](https://learn.microsoft.com/en-us/azure/ai-services/content-safety/overview).
+
+When a moderation provider reports token usage, Promptfoo includes it in assertion token metrics for clean, flagged, and provider-error results. `not-moderation` changes only the verdict; it preserves reported usage. Providers that omit usage do not produce synthetic token counts.
+
+In general, we encourage the use of Meta's LlamaGuard as it substantially outperforms OpenAI's moderation API as well as GPT-4. [See benchmarks](https://github.com/meta-llama/PurpleLlama/blob/main/Llama-Guard2/MODEL_CARD.md#model-performance).
+
+## OpenAI moderation
+
+By default, the `moderation` assertion uses OpenAI if an OpenAI API key is provided. Just make sure that the `OPENAI_API_KEY` environment variable is set:
+
+```yaml
+tests:
+  - vars:
+      foo: bar
+    assert:
+      # Ensure that it passes OpenAI's moderation filters
+      - type: moderation
+```
+
+OpenAI monitors the following categories:
+
+| Category               | Description                                                                                                                                                                                |
+| ---------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
+| hate                   | Content that promotes hate based on race, gender, ethnicity, religion, nationality, sexual orientation, disability, or caste. Hateful content aimed at non-protected groups is harassment. |
+| hate/threatening       | Hateful content that includes violence or serious harm towards the targeted group.                                                                                                         |
+| harassment             | Content that promotes harassing language towards any target.                                                                                                                               |
+| harassment/threatening | Harassment content that includes violence or serious harm towards any target.                                                                                                              |
+| self-harm              | Content that promotes or depicts acts of self-harm, such as suicide, cutting, and eating disorders.                                                                                        |
+| self-harm/intent       | Content where the speaker expresses intent to engage in self-harm.                                                                                                                         |
+| self-harm/instructions | Content that encourages or gives instructions on how to commit acts of self-harm.                                                                                                          |
+| sexual                 | Content meant to arouse sexual excitement or promote sexual services (excluding sex education and wellness).                                                                               |
+| sexual/minors          | Sexual content involving individuals under 18 years old.                                                                                                                                   |
+| violence               | Content that depicts death, violence, or physical injury.                                                                                                                                  |
+| violence/graphic       | Content that depicts death, violence, or physical injury in graphic detail.                                                                                                                |
+
+### Check specific categories
+
+The assertion value allows you to only enable moderation for specific categories:
+
+```yaml
+tests:
+  - vars:
+      foo: bar
+    assert:
+      - type: moderation
+        // highlight-start
+        value:
+          - harassment
+          - harassment/threatening
+          - sexual
+          - sexual/minors
+        // highlight-end
+```
+
+## Meta LlamaGuard moderation
+
+This example uses the LlamaGuard model hosted on Replicate. Be sure to set the `REPLICATE_API_KEY` environment variable:
+
+```yaml
+tests:
+  - vars:
+      foo: bar
+    assert:
+      - type: moderation
+        # Use the latest Llama Guard on replicate
+        provider: 'replicate:moderation:meta/llama-guard-3-8b:146d1220d447cdcc639bc17c5f6137416042abee6ae153a2615e6ef5749205c8'
+```
+
+:::tip
+LlamaGuard 4 is the default moderation provider on Replicate, featuring enhanced capabilities and an additional category (S14: Code Interpreter Abuse). You can explicitly specify it:
+
+```yaml
+provider: 'replicate:moderation:meta/llama-guard-4-12b'
+```
+
+For compatibility or specific use cases, you can use LlamaGuard 3:
+
+```yaml
+provider: 'replicate:moderation:meta/llama-guard-3-8b:146d1220d447cdcc639bc17c5f6137416042abee6ae153a2615e6ef5749205c8'
+```
+
+:::
+
+LlamaGuard monitors the following categories:
+
+| Category               | Description                                                                                                      | Code |
+| ---------------------- | ---------------------------------------------------------------------------------------------------------------- | ---- |
+| Violent Crimes         | Content that enables, encourages, or excuses violent crimes (e.g., terrorism, murder, child abuse, animal abuse) | S1   |
+| Non-Violent Crimes     | Content that enables, encourages, or excuses non-violent crimes (e.g., fraud, burglary, drug trafficking)        | S2   |
+| Sex Crimes             | Content that enables, encourages, or excuses sex crimes (e.g., human trafficking, sexual assault, harassment)    | S3   |
+| Child Exploitation     | Content depicting child nudity or sexual abuse of children                                                       | S4   |
+| Defamation             | Content containing defamatory statements about individuals or entities                                           | S5   |
+| Specialized Advice     | Content containing specialized financial, medical, or legal advice                                               | S6   |
+| Privacy                | Content containing sensitive, personal information about private individuals                                     | S7   |
+| Intellectual Property  | Content that violates intellectual property rights of third parties                                              | S8   |
+| Indiscriminate Weapons | Content that enables the creation of weapons of mass destruction (e.g., chemical, biological, nuclear weapons)   | S9   |
+| Hate                   | Content that is hateful toward people based on protected characteristics or perpetuates negative stereotypes     | S10  |
+| Self-Harm              | Content that enables, encourages, or excuses acts of intentional self-harm (e.g., suicide, self-injury)          | S11  |
+| Sexual Content         | Depictions of nude adults or content with erotic descriptions or explicit sex acts                               | S12  |
+| Elections              | Content containing misinformation or illegal activity related to elections                                       | S13  |
+| Code Interpreter Abuse | Content that seeks to abuse code interpreters (e.g., denial of service, container escapes) - LlamaGuard 4 only   | S14  |
+
+### Check specific categories
+
+The assertion value allows you to only enable moderation for specific categories:
+
+```yaml
+tests:
+  - vars:
+      foo: bar
+    assert:
+      - type: moderation
+        provider: 'replicate:moderation:meta/llama-guard-3-8b:146d1220d447cdcc639bc17c5f6137416042abee6ae153a2615e6ef5749205c8'
+        // highlight-start
+        value:
+          - S1
+          - S3
+          - S4
+        // highlight-end
+```
+
+## Azure Content Safety moderation
+
+You can use the Azure Content Safety API for moderation. To set it up, you need to create an Azure Content Safety resource and get the API key and endpoint.
+
+### Setup
+
+First, set these environment variables:
+
+```bash
+AZURE_CONTENT_SAFETY_ENDPOINT=https://your-resource-name.cognitiveservices.azure.com
+AZURE_CONTENT_SAFETY_API_KEY=your-api-key
+AZURE_CONTENT_SAFETY_API_VERSION=2024-09-01  # Optional, defaults to this version
+```
+
+If `AZURE_CONTENT_SAFETY_ENDPOINT` is set, PromptFoo will automatically use the Azure Content Safety service for moderation instead of OpenAI's moderation API.
+
+Or you can explicitly use the Azure moderation provider in your tests:
+
+```yaml
+tests:
+  - vars:
+      foo: bar
+    assert:
+      - type: moderation
+        provider: 'azure:moderation'
+```
+
+### Moderation Categories
+
+The Azure Content Safety API checks content for these categories:
+
+| Category | Description                                                       |
+| -------- | ----------------------------------------------------------------- |
+| Hate     | Content that expresses discrimination or derogatory sentiments    |
+| SelfHarm | Content related to inflicting physical harm on oneself            |
+| Sexual   | Sexually explicit or adult content                                |
+| Violence | Content depicting or promoting violence against people or animals |
+
+### Check specific categories
+
+The assertion value allows you to only enable moderation for specific categories
+
+```yaml
+tests:
+  - vars:
+      foo: bar
+    assert:
+      - type: moderation
+        provider: 'azure:moderation'
+        value:
+          - hate
+          - sexual
+```
+
+You can also set blocklist names and halt on blocklist hit in the provider config:
+
+```yaml
+tests:
+  - vars:
+      foo: bar
+    assert:
+      - type: moderation
+        provider:
+          id: azure:moderation
+          config:
+            blocklistNames: ['my-custom-blocklist', 'industry-terms']
+            haltOnBlocklistHit: true
+```

site/docs/configuration/expected-outputs/python.md

@@ -0,0 +1,300 @@
+---
+sidebar_position: 51
+sidebar_label: Python
+description: Create advanced Python validation scripts with complex logic, external APIs, and ML libraries for sophisticated output grading
+---
+
+# Python assertions
+
+The `python` assertion allows you to provide a custom Python function to validate the LLM output.
+
+:::tip Python Overview
+
+For an overview of all Python integrations (providers, assertions, test generators, prompts), see the [Python integration guide](/docs/integrations/python).
+
+:::
+
+A variable named `output` is injected into the context. The function should return `true` if the output passes the assertion, and `false` otherwise. If the function returns a number, it will be treated as a score.
+
+Example:
+
+```yaml
+assert:
+  - type: python
+    value: output[5:10] == 'Hello'
+```
+
+You may also return a number, which will be treated as a score:
+
+```yaml
+assert:
+  - type: python
+    value: math.log10(len(output)) * 10
+```
+
+## Multiline functions
+
+Python assertions support multiline strings:
+
+```yaml
+assert:
+  - type: python
+    value: |
+      # Insert your scoring logic here...
+      if output == 'Expected output':
+          return {
+            'pass': True,
+            'score': 0.5,
+          }
+      return {
+        'pass': False,
+        'score': 0,
+      }
+```
+
+## Using test context
+
+A `context` object is available in the Python function. Here is its type definition:
+
+```py
+from typing import Any, Dict, List, Optional, TypedDict, Union
+
+class TraceSpan(TypedDict):
+    spanId: str
+    parentSpanId: Optional[str]
+    name: str
+    startTime: int  # Unix timestamp in milliseconds
+    endTime: Optional[int]  # Unix timestamp in milliseconds
+    attributes: Optional[Dict[str, Any]]
+    statusCode: Optional[int]
+    statusMessage: Optional[str]
+
+class TraceData(TypedDict):
+    traceId: str
+    spans: List[TraceSpan]
+
+class AssertionValueFunctionContext(TypedDict):
+    # Raw prompt sent to LLM
+    prompt: Optional[str]
+
+    # Test case variables
+    vars: Dict[str, Union[str, object]]
+
+    # The complete test case
+    test: Dict[str, Any]  # Contains keys like "vars", "assert", "options"
+
+    # Log probabilities from the LLM response, if available
+    logProbs: Optional[list[float]]
+
+    # Configuration passed to the assertion
+    config: Optional[Dict[str, Any]]
+
+    # The provider that generated the response
+    provider: Optional[Any]  # ApiProvider type
+
+    # The complete provider response
+    providerResponse: Optional[Any]  # ProviderResponse type
+
+    # OpenTelemetry trace data (when tracing is enabled)
+    trace: Optional[TraceData]
+
+    # Optional shortcut to providerResponse.metadata
+    metadata: Optional[Dict[str, Any]]
+```
+
+For example, if the test case has a var `example`, access it in Python like this:
+
+```yaml
+tests:
+  - description: 'Test with context'
+    vars:
+      example: 'Example text'
+    assert:
+      - type: python
+        value: 'context["vars"]["example"] in output'
+```
+
+## External .py
+
+To reference an external file, use the `file://` prefix:
+
+```yaml
+assert:
+  - type: python
+    value: file://relative/path/to/script.py
+    config:
+      outputLengthLimit: 10
+```
+
+You can specify a particular function to use by appending it after a colon:
+
+```yaml
+assert:
+  - type: python
+    value: file://relative/path/to/script.py:custom_assert
+```
+
+If no function is specified, it defaults to `get_assert`.
+
+This file will be called with an `output` string and an `AssertionValueFunctionContext` object (see above).
+It expects that either a `bool` (pass/fail), `float` (score), or `GradingResult` will be returned.
+
+Here's an example `assert.py`:
+
+```py
+from typing import Dict, TypedDict, Union
+
+# Default function name
+def get_assert(output: str, context) -> Union[bool, float, Dict[str, Any]]:
+    print('Prompt:', context['prompt'])
+    print('Vars', context['vars']['topic'])
+
+    # This return is an example GradingResult dict
+    return {
+      'pass': True,
+      'score': 0.6,
+      'reason': 'Looks good to me',
+    }
+
+# Custom function name
+def custom_assert(output: str, context) -> Union[bool, float, Dict[str, Any]]:
+    return len(output) > 10
+```
+
+This is an example of an assertion that uses data from a configuration defined in the assertion's YML file:
+
+```py
+from typing import Dict, Union
+
+def get_assert(output: str, context) -> Union[bool, float, Dict[str, Any]]:
+    return len(output) <= context.get('config', {}).get('outputLengthLimit', 0)
+```
+
+You can also return nested metrics and assertions via a `GradingResult` object:
+
+```py
+{
+    'pass': True,
+    'score': 0.75,
+    'reason': 'Looks good to me',
+    'componentResults': [{
+        'pass': 'bananas' in output.lower(),
+        'score': 0.5,
+        'reason': 'Contains banana',
+    }, {
+        'pass': 'yellow' in output.lower(),
+        'score': 0.5,
+        'reason': 'Contains yellow',
+    }]
+}
+```
+
+### GradingResult types
+
+Here's a Python type definition you can use for the [`GradingResult`](/docs/configuration/reference/#gradingresult) object:
+
+```py
+@dataclass
+class GradingResult:
+    pass_: bool  # 'pass' is a reserved keyword in Python
+    score: float
+    reason: str
+    component_results: Optional[List['GradingResult']] = None
+    named_scores: Optional[Dict[str, float]] = None  # Appear as metrics in the UI
+```
+
+:::tip Snake case support
+Python snake_case fields are automatically mapped to camelCase:
+
+- `pass_` → `pass` (or just use `"pass"` as a dictionary key)
+- `named_scores` → `namedScores`
+- `component_results` → `componentResults`
+- `tokens_used` → `tokensUsed`
+  :::
+
+## Using trace data
+
+When [tracing is enabled](/docs/tracing/), OpenTelemetry trace data is available in the `context.trace` object. This allows you to write assertions based on the execution flow:
+
+```py
+def get_assert(output: str, context) -> Union[bool, float, Dict[str, Any]]:
+    # Check if trace data is available
+    if not hasattr(context, 'trace') or context.trace is None:
+        # Tracing not enabled, skip trace-based checks
+        return True
+
+    # Access trace spans
+    spans = context.trace['spans']
+
+    # Example: Check for errors in any span
+    error_spans = [s for s in spans if s.get('statusCode', 0) >= 400]
+    if error_spans:
+        return {
+            'pass': False,
+            'score': 0,
+            'reason': f"Found {len(error_spans)} error spans"
+        }
+
+    # Example: Calculate total trace duration
+    if spans:
+        duration = max(s.get('endTime', 0) for s in spans) - min(s['startTime'] for s in spans)
+        if duration > 5000:  # 5 seconds
+            return {
+                'pass': False,
+                'score': 0,
+                'reason': f"Trace took too long: {duration}ms"
+            }
+
+    # Example: Check for specific operations
+    api_calls = [s for s in spans if 'http' in s['name'].lower()]
+    if len(api_calls) > 10:
+        return {
+            'pass': False,
+            'score': 0,
+            'reason': f"Too many API calls: {len(api_calls)}"
+        }
+
+    return True
+```
+
+Example YAML configuration:
+
+```yaml
+tests:
+  - vars:
+      query: "What's the weather?"
+    assert:
+      - type: python
+        value: |
+          # Ensure retrieval happened before response generation
+          if context.trace:
+              spans = context.trace['spans']
+              retrieval_span = next((s for s in spans if 'retrieval' in s['name']), None)
+              generation_span = next((s for s in spans if 'generation' in s['name']), None)
+              
+              if retrieval_span and generation_span:
+                  return retrieval_span['startTime'] < generation_span['startTime']
+          return True
+```
+
+## Overriding the Python binary
+
+By default, promptfoo will run `python` in your shell. Make sure `python` points to the appropriate executable.
+
+If a `python` binary is not present, you will see a "python: command not found" error.
+
+To override the Python binary, set the `PROMPTFOO_PYTHON` environment variable. You may set it to a path (such as `/path/to/python3.11`) or just an executable in your PATH (such as `python3.11`).
+
+## Negation
+
+Use `not-python` to invert the final pass/fail result while preserving the returned score. Numeric scores are still compared against `threshold` before the result is inverted:
+
+```yaml
+assert:
+  - type: not-python
+    value: "'error' in output"
+```
+
+## Other assertion types
+
+For more info on assertions, see [Test assertions](/docs/configuration/expected-outputs).

site/docs/configuration/expected-outputs/ruby.md

@@ -0,0 +1,317 @@
+---
+sidebar_position: 52
+sidebar_label: Ruby
+description: Create advanced Ruby validation scripts with complex logic, external APIs, and custom libraries for sophisticated output grading
+---
+
+# Ruby assertions
+
+The `ruby` assertion allows you to provide a custom Ruby method to validate the LLM output.
+
+A variable named `output` is injected into the context. The method should return `true` if the output passes the assertion, and `false` otherwise. If the method returns a number, it will be treated as a score.
+
+Example:
+
+```yaml
+assert:
+  - type: ruby
+    value: output[5..9] == 'Hello'
+```
+
+You may also return a number, which will be treated as a score:
+
+```yaml
+assert:
+  - type: ruby
+    value: Math.log10(output.length) * 10
+```
+
+## Multiline functions
+
+Ruby assertions support multiline strings:
+
+```yaml
+assert:
+  - type: ruby
+    value: |
+      # Insert your scoring logic here...
+      if output == 'Expected output'
+        return {
+          'pass' => true,
+          'score' => 0.5,
+        }
+      end
+      return {
+        'pass' => false,
+        'score' => 0,
+      }
+```
+
+## Using test context
+
+A `context` object is available in the Ruby method. Here is its type definition:
+
+```ruby
+# TraceSpan
+{
+  'spanId' => String,
+  'parentSpanId' => String | nil,
+  'name' => String,
+  'startTime' => Integer,  # Unix timestamp in milliseconds
+  'endTime' => Integer | nil,  # Unix timestamp in milliseconds
+  'attributes' => Hash | nil,
+  'statusCode' => Integer | nil,
+  'statusMessage' => String | nil
+}
+
+# TraceData
+{
+  'traceId' => String,
+  'spans' => Array[TraceSpan]
+}
+
+# AssertionValueFunctionContext
+{
+  # Raw prompt sent to LLM
+  'prompt' => String | nil,
+
+  # Test case variables
+  'vars' => Hash[String, String | Object],
+
+  # The complete test case
+  'test' => Hash,  # Contains keys like "vars", "assert", "options"
+
+  # Log probabilities from the LLM response, if available
+  'logProbs' => Array[Float] | nil,
+
+  # Configuration passed to the assertion
+  'config' => Hash | nil,
+
+  # The provider that generated the response
+  'provider' => Object | nil,  # ApiProvider type
+
+  # The complete provider response
+  'providerResponse' => Object | nil,  # ProviderResponse type
+
+  # Optional shortcut to providerResponse metadata
+  'metadata' => Hash | nil,
+
+  # OpenTelemetry trace data (when tracing is enabled)
+  'trace' => TraceData | nil
+}
+```
+
+For example, if the test case has a var `example`, access it in Ruby like this:
+
+```yaml
+tests:
+  - description: 'Test with context'
+    vars:
+      example: 'Example text'
+    assert:
+      - type: ruby
+        value: 'output.include?(context["vars"]["example"])'
+```
+
+## External .rb
+
+To reference an external file, use the `file://` prefix:
+
+```yaml
+assert:
+  - type: ruby
+    value: file://relative/path/to/script.rb
+    config:
+      outputLengthLimit: 10
+```
+
+You can specify a particular method to use by appending it after a colon:
+
+```yaml
+assert:
+  - type: ruby
+    value: file://relative/path/to/script.rb:custom_assert
+```
+
+You can also specify a class method on some class or module in the file:
+
+```yaml
+assert:
+  - type: ruby
+    value: file://relative/path/to/script.rb:Validators::Format.check_length
+```
+
+If no method is specified, it defaults to `get_assert`.
+
+This file will be called with an `output` string and an `AssertionValueFunctionContext` object (see above).
+It expects that either a `bool` (pass/fail), `float` (score), or `GradingResult` will be returned.
+
+Here's an example `assert.rb`:
+
+```ruby
+require 'json'
+
+# Default function name
+def get_assert(output, context)
+  puts 'Prompt:', context['prompt']
+  puts 'Vars', context['vars']['topic']
+
+  # This return is an example GradingResult hash
+  {
+    'pass' => true,
+    'score' => 0.6,
+    'reason' => 'Looks good to me',
+  }
+end
+
+# Custom function name
+def custom_assert(output, context)
+  output.length > 10
+end
+```
+
+This is an example of an assertion that uses data from a configuration defined in the assertion's YML file:
+
+```ruby
+def get_assert(output, context)
+  output.length <= context.fetch('config', {}).fetch('outputLengthLimit', 0)
+end
+```
+
+You can also return nested metrics and assertions via a `GradingResult` object:
+
+```ruby
+{
+  'pass' => true,
+  'score' => 0.75,
+  'reason' => 'Looks good to me',
+  'componentResults' => [{
+    'pass' => output.downcase.include?('bananas'),
+    'score' => 0.5,
+    'reason' => 'Contains banana',
+  }, {
+    'pass' => output.downcase.include?('yellow'),
+    'score' => 0.5,
+    'reason' => 'Contains yellow',
+  }]
+}
+```
+
+### GradingResult types
+
+Here's a Ruby type definition you can use for the [`GradingResult`](/docs/configuration/reference/#gradingresult) object:
+
+```ruby
+# GradingResult
+{
+  'pass' => Boolean,  # Can also use 'pass_' if 'pass' conflicts with Ruby keywords
+  'score' => Float,
+  'reason' => String,
+  'componentResults' => Array[GradingResult] | nil,  # Component results (optional)
+  'namedScores' => Hash[String, Float] | nil  # Appear as metrics in the UI (optional)
+}
+```
+
+:::tip Snake case support
+Ruby snake_case fields are automatically mapped to camelCase:
+
+- `pass_` → `pass` (or just use `"pass"` as a hash key)
+- `named_scores` → `namedScores`
+- `component_results` → `componentResults`
+- `tokens_used` → `tokensUsed`
+  :::
+
+## Using trace data
+
+When [tracing is enabled](/docs/tracing/), OpenTelemetry trace data is available in the `context['trace']` object. This allows you to write assertions based on the execution flow:
+
+```ruby
+def get_assert(output, context)
+  # Check if trace data is available
+  unless context['trace']
+    # Tracing not enabled, skip trace-based checks
+    return true
+  end
+
+  # Access trace spans
+  spans = context['trace']['spans']
+
+  # Example: Check for errors in any span
+  error_spans = spans.select { |s| s.fetch('statusCode', 0) >= 400 }
+  if error_spans.any?
+    return {
+      'pass' => false,
+      'score' => 0,
+      'reason' => "Found #{error_spans.length} error spans"
+    }
+  end
+
+  # Example: Calculate total trace duration
+  if spans.any?
+    duration = spans.map { |s| s.fetch('endTime', 0) }.max - spans.map { |s| s['startTime'] }.min
+    if duration > 5000  # 5 seconds
+      return {
+        'pass' => false,
+        'score' => 0,
+        'reason' => "Trace took too long: #{duration}ms"
+      }
+    end
+  end
+
+  # Example: Check for specific operations
+  api_calls = spans.select { |s| s['name'].downcase.include?('http') }
+  if api_calls.length > 10
+    return {
+      'pass' => false,
+      'score' => 0,
+      'reason' => "Too many API calls: #{api_calls.length}"
+    }
+  end
+
+  true
+end
+```
+
+Example YAML configuration:
+
+```yaml
+tests:
+  - vars:
+      query: "What's the weather?"
+    assert:
+      - type: ruby
+        value: |
+          # Ensure retrieval happened before response generation
+          if context['trace']
+            spans = context['trace']['spans']
+            retrieval_span = spans.find { |s| s['name'].include?('retrieval') }
+            generation_span = spans.find { |s| s['name'].include?('generation') }
+
+            if retrieval_span && generation_span
+              return retrieval_span['startTime'] < generation_span['startTime']
+            end
+          end
+          true
+```
+
+## Overriding the Ruby binary
+
+By default, promptfoo will run `ruby` in your shell. Make sure `ruby` points to the appropriate executable.
+
+If a `ruby` binary is not present, you will see a "ruby: command not found" error.
+
+To override the Ruby binary, set the `PROMPTFOO_RUBY` environment variable. You may set it to a path (such as `/path/to/ruby`) or just an executable in your PATH (such as `ruby`).
+
+## Negation
+
+Use `not-ruby` to invert the final pass/fail result while preserving the returned score. Numeric scores are still compared against `threshold` before the result is inverted:
+
+```yaml
+assert:
+  - type: not-ruby
+    value: output.include?('error')
+```
+
+## Other assertion types
+
+For more info on assertions, see [Test assertions](/docs/configuration/expected-outputs).

site/docs/configuration/expected-outputs/similar.md

@@ -0,0 +1,116 @@
+---
+sidebar_position: 55
+description: Calculate semantic similarity scores between actual and expected outputs using advanced embedding models and multiple distance metrics
+---
+
+# Similarity (embeddings)
+
+The `similar` assertion checks if an embedding of the LLM's output
+is semantically similar to the expected value,
+using a configurable similarity or distance metric with a threshold.
+
+By default, embeddings are computed via OpenAI's `text-embedding-3-large` model.
+
+Example:
+
+```yaml
+assert:
+  - type: similar
+    value: 'The expected output'
+    threshold: 0.8
+```
+
+If you provide an array of values, the test will pass if it is similar to at least one of them:
+
+```yaml
+assert:
+  - type: similar
+    value:
+      - The expected output
+      - Expected output
+      - file://my_expected_output.txt
+    threshold: 0.8
+```
+
+The negated `not-similar` assertion is the logical inverse: with an array of values it passes only when the output is dissimilar to **every** value (and fails as soon as it is too similar to any one of them). This is the natural way to assert that an output does not resemble any item in a list of forbidden or canned answers.
+
+## Similarity Metrics
+
+You can specify which metric to use by including it in the assertion type. The default is `similar` (cosine similarity).
+
+### Cosine Similarity (default)
+
+Measures the cosine of the angle between two vectors. Range: -1 to 1 (higher is more similar), though text embeddings typically produce values between 0 and 1.
+
+```yaml
+assert:
+  # Default - uses cosine similarity
+  - type: similar
+    value: 'The expected output'
+    threshold: 0.8
+
+  # Explicit cosine
+  - type: similar:cosine
+    value: 'The expected output'
+    threshold: 0.8
+```
+
+**When to use:** Best for semantic similarity where you care about the direction of the embedding vector, not its magnitude. This is the industry standard for embeddings.
+
+### Dot Product
+
+Computes the dot product of two vectors. Range: unbounded, but typically 0 to 1 for normalized embeddings (higher is more similar).
+
+```yaml
+assert:
+  - type: similar:dot
+    value: 'The expected output'
+    threshold: 0.8
+```
+
+**When to use:** Useful when you want to match the metric used in your production vector database (many use dot product for performance). For normalized embeddings, dot product is nearly equivalent to cosine similarity.
+
+### Euclidean Distance
+
+Computes the straight-line distance between two vectors. Range: 0 to ∞ (lower is more similar).
+
+```yaml
+assert:
+  - type: similar:euclidean
+    value: 'The expected output'
+    threshold: 0.5 # Maximum distance threshold
+```
+
+**When to use:** When you care about both the angle and magnitude differences between vectors. Note that the threshold represents the _maximum_ distance (not minimum similarity), so lower values are stricter.
+
+**Important:** For euclidean distance, the threshold semantics are inverted - it represents the _maximum_ acceptable distance rather than minimum similarity.
+
+## Overriding the provider
+
+By default `similar` will use OpenAI. To specify the model that creates the embeddings, do one of the following:
+
+1. Use `test.options` or `defaultTest.options` to override the provider across the entire test suite. For example:
+
+   ```yaml
+   defaultTest:
+     options:
+       provider:
+         embedding:
+           id: azureopenai:embedding:text-embedding-ada-002
+           config:
+             apiHost: xxx.openai.azure.com
+   tests:
+     assert:
+       - type: similar
+         value: Hello world
+   ```
+
+2. Set `assertion.provider` on a per-assertion basis. For example:
+
+   ```yaml
+   tests:
+     assert:
+       - type: similar
+         value: Hello world
+         provider: huggingface:sentence-similarity:sentence-transformers/all-MiniLM-L6-v2
+   ```

site/docs/configuration/guide.md

@@ -0,0 +1,910 @@
+---
+sidebar_position: 1
+sidebar_label: Overview
+title: Configuration Overview - Getting Started with Promptfoo
+description: Complete guide to configuring promptfoo for LLM evaluation. Learn prompts, providers, test cases, assertions, and advanced features with examples.
+keywords:
+  [
+    promptfoo configuration,
+    LLM evaluation setup,
+    prompt testing,
+    AI model comparison,
+    evaluation framework,
+    getting started,
+  ]
+pagination_next: configuration/reference
+---
+
+# Configuration
+
+The YAML configuration format runs each prompt through a series of example inputs (aka "test case") and checks if they meet requirements (aka "assertions").
+
+Assertions are _optional_. Many people get value out of reviewing outputs manually, and the web UI helps facilitate this.
+
+## Example
+
+Let's imagine we're building an app that does language translation. This config runs each prompt through GPT-4.1 and Gemini, substituting `language` and `input` variables:
+
+```yaml
+prompts:
+  - file://prompt1.txt
+  - file://prompt2.txt
+providers:
+  - openai:gpt-5-mini
+  - vertex:gemini-2.0-flash-exp
+tests:
+  - vars:
+      language: French
+      input: Hello world
+  - vars:
+      language: German
+      input: How's it going?
+```
+
+:::tip
+
+For more information on setting up a prompt file, see [input and output files](/docs/configuration/prompts).
+
+:::
+
+Running `promptfoo eval` over this config will result in a _matrix view_ that you can use to evaluate GPT vs Gemini.
+
+## Use assertions to validate output
+
+Next, let's add an assertion. This automatically rejects any outputs that don't contain JSON:
+
+```yaml
+prompts:
+  - file://prompt1.txt
+  - file://prompt2.txt
+providers:
+  - openai:gpt-5-mini
+  - vertex:gemini-2.0-flash-exp
+tests:
+  - vars:
+      language: French
+      input: Hello world
+    // highlight-start
+    assert:
+      - type: contains-json
+    // highlight-end
+  - vars:
+      language: German
+      input: How's it going?
+```
+
+We can create additional tests. Let's add a couple other [types of assertions](/docs/configuration/expected-outputs). Use an array of assertions for a single test case to ensure all conditions are met.
+
+In this example, the `javascript` assertion runs Javascript against the LLM output. The `similar` assertion checks for semantic similarity using embeddings:
+
+```yaml
+prompts:
+  - file://prompt1.txt
+  - file://prompt2.txt
+providers:
+  - openai:gpt-5-mini
+  - vertex:gemini-2.0-flash-exp
+tests:
+  - vars:
+      language: French
+      input: Hello world
+    assert:
+      - type: contains-json
+      // highlight-start
+      - type: javascript
+        value: output.toLowerCase().includes('bonjour')
+      // highlight-end
+  - vars:
+      language: German
+      input: How's it going?
+    assert:
+      // highlight-start
+      - type: similar
+        value: was geht
+        threshold: 0.6   # cosine similarity
+      // highlight-end
+```
+
+:::tip
+To learn more about assertions, see docs on configuring [assertions and metrics](/docs/configuration/expected-outputs).
+:::
+
+## Import providers from separate files
+
+The `providers` config property can point to a list of files. For example:
+
+```yaml
+providers:
+  - file://path/to/provider1.yaml
+  - file://path/to/provider2.json
+```
+
+Where the provider file looks like this:
+
+```yaml
+id: openai:gpt-5-mini
+label: Foo bar
+config:
+  temperature: 0.9
+```
+
+## Import tests from separate files
+
+The `tests` config property takes a list of paths to files or directories. For example:
+
+```yaml
+prompts: file://prompts.txt
+providers: openai:gpt-5-mini
+
+# Load & runs all test cases matching these filepaths
+tests:
+  # You can supply an exact filepath
+  - file://tests/tests2.yaml
+
+  # Or a glob (wildcard)
+  - file://tests/*
+
+  # Mix and match with actual test cases
+  - vars:
+      var1: foo
+      var2: bar
+```
+
+A single string is also valid:
+
+```yaml
+tests: file://tests/*
+```
+
+Or a list of paths:
+
+```yaml
+tests:
+  - file://tests/accuracy
+  - file://tests/creativity
+  - file://tests/hallucination
+```
+
+:::tip
+Test files can be defined in YAML/JSON, JSONL, [CSV](/docs/configuration/test-cases#csv-format), and TypeScript/JavaScript. Promptfoo also supports external datasets from [Google Sheets](/docs/integrations/google-sheets) and [Azure Blob Storage](/docs/configuration/test-cases#azure-blob-storage).
+:::
+
+## Import vars from separate files
+
+The `vars` property can point to a file or directory. For example:
+
+```yaml
+tests:
+  - vars: file://path/to/vars*.yaml
+```
+
+You can also load individual variables from file by using the `file://` prefix. For example:
+
+```yaml
+tests:
+  - vars:
+      var1: some value...
+      var2: another value...
+      var3: file://path/to/var3.txt
+```
+
+Javascript and Python variable files are supported. For example:
+
+```yaml
+tests:
+  - vars:
+      context: file://fetch_from_vector_database.py
+```
+
+Scripted vars are useful when testing vector databases like Pinecone, Chroma, Milvus, etc. You can communicate directly with the database to fetch the context you need.
+
+PDFs are also supported and can be used to extract text from a document:
+
+```yaml
+tests:
+  - vars:
+      paper: file://pdfs/arxiv_1.pdf
+```
+
+Note that you must install the `pdf-parse` package to use PDFs as variables:
+
+```
+npm install pdf-parse
+```
+
+### Javascript variables
+
+To dynamically load a variable from a JavaScript file, use the `file://` prefix in your YAML configuration, pointing to a JavaScript file that exports a function.
+
+```yaml
+tests:
+  - vars:
+      context: file://path/to/dynamicVarGenerator.js
+```
+
+The function receives `varName`, `prompt`, `otherVars`, and `provider` as arguments:
+
+```js title="dynamicVarGenerator.js"
+module.exports = async function (varName, prompt, otherVars, provider) {
+  // Access other variables from the test case
+  const role = otherVars.role;
+
+  // Return the dynamic value
+  return { output: PROMPTS[role] };
+
+  // Or return an error
+  // return { error: 'Something went wrong' };
+};
+```
+
+See the [dynamic-var example](https://github.com/promptfoo/promptfoo/tree/main/examples/config-dynamic-var) for a complete working example.
+
+### Python variables
+
+Define a `get_var` function that accepts `var_name`, `prompt`, and `other_vars`:
+
+```yaml
+tests:
+  - vars:
+      context: file://load_context.py
+```
+
+```python title="load_context.py"
+def get_var(var_name, prompt, other_vars):
+    # Access other variables from the test case
+    role = other_vars.get("role")
+
+    # Return the dynamic value
+    return {"output": PROMPTS[role]}
+
+    # Or return an error
+    # return {"error": "Something went wrong"}
+```
+
+## Avoiding repetition
+
+### Default test cases
+
+Use `defaultTest` to set properties for all tests.
+
+In this example, we use a `llm-rubric` assertion to ensure that the LLM does not refer to itself as an AI. This check applies to all test cases:
+
+```yaml
+prompts:
+  - file://prompt1.txt
+  - file://prompt2.txt
+providers:
+  - openai:gpt-5-mini
+  - vertex:gemini-2.0-flash-exp
+// highlight-start
+defaultTest:
+  assert:
+    - type: llm-rubric
+      value: does not describe self as an AI, model, or chatbot
+// highlight-end
+tests:
+  - vars:
+      language: French
+      input: Hello world
+    assert:
+      - type: contains-json
+      - type: javascript
+        value: output.toLowerCase().includes('bonjour')
+  - vars:
+      language: German
+      input: How's it going?
+    assert:
+      - type: similar
+        value: was geht
+        threshold: 0.6
+```
+
+You can also use `defaultTest` to override the model used for each test. This can be useful for [model-graded evals](/docs/configuration/expected-outputs/model-graded):
+
+```yaml
+defaultTest:
+  options:
+    provider: openai:gpt-5-mini-0613
+```
+
+Set `options.disableDefaultAsserts: true` on a test case when that test should define its own assertions without inheriting `defaultTest.assert`. Other `defaultTest` fields, such as `vars`, `metadata`, `threshold`, and `options`, still apply:
+
+```yaml
+defaultTest:
+  vars:
+    audience: developer
+  assert:
+    - type: contains
+      value: installation steps
+
+tests:
+  - vars:
+      topic: API setup
+    options:
+      disableDefaultAsserts: true
+    assert:
+      - type: contains-json
+```
+
+### Default variables
+
+Use `defaultTest` to define variables that are shared across all tests:
+
+```yaml
+defaultTest:
+  vars:
+    template: 'A reusable prompt template with {{shared_var}}'
+    shared_var: 'some shared content'
+
+tests:
+  - vars:
+      unique_var: value1
+  - vars:
+      unique_var: value2
+      shared_var: 'override shared content' # Optionally override defaults
+```
+
+### Loading defaultTest from external files
+
+You can load `defaultTest` configuration from external files using `defaultTest: file://path/to/config.yaml` for sharing test configurations across projects.
+
+### YAML references
+
+promptfoo configurations support JSON schema [references](https://opis.io/json-schema/2.x/references.html), which define reusable blocks.
+
+Use the `$ref` key to re-use assertions without having to fully define them more than once. Here's an example:
+
+```yaml
+prompts:
+  - file://prompt1.txt
+  - file://prompt2.txt
+providers:
+  - openai:gpt-5-mini
+  - vertex:gemini-2.0-flash-exp
+tests:
+  - vars:
+      language: French
+      input: Hello world
+    assert:
+      - $ref: '#/assertionTemplates/startsUpperCase'
+  - vars:
+      language: German
+      input: How's it going?
+    assert:
+      - $ref: '#/assertionTemplates/noAIreference'
+      - $ref: '#/assertionTemplates/startsUpperCase'
+
+// highlight-start
+assertionTemplates:
+    noAIreference:
+      type: llm-rubric
+      value: does not describe self as an AI, model, or chatbot
+    startsUpperCase:
+      type: javascript
+      value: output[0] === output[0].toUpperCase()
+// highlight-end
+```
+
+:::info
+`tools` and `functions` values in providers config are _not_ dereferenced. This is because they are standalone JSON schemas that may contain their own internal references.
+:::
+
+## Multiple variables in a single test case
+
+The `vars` map in the test also supports array values. If values are an array, the test case will run each combination of values.
+
+For example:
+
+```yaml
+prompts: file://prompts.txt
+providers:
+  - openai:gpt-5-mini
+  - openai:gpt-5
+tests:
+  - vars:
+      // highlight-start
+      language:
+        - French
+        - German
+        - Spanish
+      input:
+        - 'Hello world'
+        - 'Good morning'
+        - 'How are you?'
+      // highlight-end
+    assert:
+      - type: similar
+        value: 'Hello world'
+        threshold: 0.8
+```
+
+Evaluates each `language` x `input` combination:
+
+<img alt="Multiple combinations of var inputs" src="https://user-images.githubusercontent.com/310310/243108917-dab27ca5-689b-4843-bb52-de8d459d783b.png" />
+
+Vars can also be imported from globbed filepaths. They are automatically expanded into an array. For example:
+
+```yaml
+  - vars:
+      language:
+        - French
+        - German
+        - Spanish
+      // highlight-start
+      input: file://path/to/inputs/*.txt
+      // highlight-end
+```
+
+## Using nunjucks templates
+
+Use Nunjucks templates to exert additional control over your prompt templates, including loops, conditionals, and more.
+
+### Manipulating objects
+
+In the above examples, `vars` values are strings. But `vars` can be any JSON or YAML entity, including nested objects. You can manipulate these objects in the prompt, which are [nunjucks](https://mozilla.github.io/nunjucks/) templates:
+
+promptfooconfig.yaml:
+
+```yaml
+tests:
+  - vars:
+      user_profile:
+        name: John Doe
+        interests:
+          - reading
+          - gaming
+          - hiking
+      recent_activity:
+        type: reading
+        details:
+          title: 'The Great Gatsby'
+          author: 'F. Scott Fitzgerald'
+```
+
+prompt.txt:
+
+```liquid
+User Profile:
+- Name: {{ user_profile.name }}
+- Interests: {{ user_profile.interests | join(', ') }}
+- Recent Activity: {{ recent_activity.type }} on "{{ recent_activity.details.title }}" by {{ recent_activity.details.author }}
+
+Based on the above user profile, generate a personalized reading recommendation list that includes books similar to "{{ recent_activity.details.title }}" and aligns with the user's interests.
+```
+
+Here's another example. Consider this test case, which lists a handful of user and assistant messages in an OpenAI-compatible format:
+
+```yaml
+tests:
+  - vars:
+      previous_messages:
+        - role: user
+          content: hello world
+        - role: assistant
+          content: how are you?
+        - role: user
+          content: great, thanks
+```
+
+The corresponding `prompt.txt` file simply passes through the `previous_messages` object using the [dump](https://mozilla.github.io/nunjucks/templating.html#dump) filter to convert the object to a JSON string:
+
+```nunjucks
+{{ previous_messages | dump }}
+```
+
+Running `promptfoo eval -p prompt.txt -c path_to.yaml` will call the Chat Completion API with the following prompt:
+
+```json
+[
+  {
+    "role": "user",
+    "content": "hello world"
+  },
+  {
+    "role": "assistant",
+    "content": "how are you?"
+  },
+  {
+    "role": "user",
+    "content": "great, thanks"
+  }
+]
+```
+
+### Escaping JSON strings
+
+If the prompt is valid JSON, nunjucks variables are automatically escaped when they are included in strings:
+
+```yaml
+tests:
+  - vars:
+      system_message: >
+        This multiline "system message" with quotes...
+        Is automatically escaped in JSON prompts!
+```
+
+```json
+{
+  "role": "system",
+  "content": "{{ system_message }}"
+}
+```
+
+You can also manually escape the string using the nunjucks [dump](https://mozilla.github.io/nunjucks/templating.html#dump) filter. This is necessary if your prompt is not valid JSON, for example if you are using nunjucks syntax:
+
+```liquid
+{
+  "role": {% if 'admin' in message %} "system" {% else %} "user" {% endif %},
+  "content": {{ message | dump }}
+}
+```
+
+### Variable composition
+
+Variables can reference other variables:
+
+```yaml
+prompts:
+  - 'Write a {{item}}'
+
+tests:
+  - vars:
+      item: 'tweet about {{topic}}'
+      topic: 'bananas'
+
+  - vars:
+      item: 'instagram about {{topic}}'
+      topic: 'theoretical quantum physics in alternate dimensions'
+```
+
+### Accessing environment variables
+
+You can access environment variables in your templates using the `env` global:
+
+```yaml
+prompts:
+  - 'file://{{ env.PROMPT_DIR }}/prompt.txt'
+
+tests:
+  - vars:
+      headline: 'Articles about {{ env.TOPIC }}'
+```
+
+Environment variables are resolved at config load time (not runtime) and can control file paths and API keys—only use them in trusted environments.
+
+:::warning
+
+Avoid copying secrets into `config.env` with templates like `ANTHROPIC_API_KEY: '{{ env.ANTHROPIC_API_KEY }}'`. This resolves the secret into the eval config object and may appear in exported results.
+
+If a secret is already present in your shell environment (or loaded via `--env-file`), prefer reading it directly from process env and keep `config.env` for non-sensitive flags.
+
+:::
+
+## Tools and Functions
+
+promptfoo supports tool use and function calling with Google, OpenAI and Anthropic models, as well as other provider-specific configurations like temperature and number of tokens. For more information on defining functions and tools, see the [Google Vertex provider docs](/docs/providers/vertex/#function-calling-and-tools), [Google AIStudio provider docs](/docs/providers/google/#tool-calling), [Google Live provider docs](/docs/providers/google#function-calling-example), [OpenAI provider docs](/docs/providers/openai#using-tools) and the [Anthropic provider docs](/docs/providers/anthropic#tool-calling).
+
+## Thinking Output
+
+Some models, like Anthropic's Claude and DeepSeek, support thinking/reasoning capabilities that allow the model to show its reasoning process before providing a final answer.
+
+This is useful for reasoning tasks or understanding how the model arrived at its conclusion.
+
+### Controlling Thinking Output
+
+By default, thinking content is included in the response. You can hide it by setting `showThinking` to `false`.
+
+For example, for Claude:
+
+```yaml
+providers:
+  - id: anthropic:messages:claude-sonnet-4-5-20250929
+    config:
+      thinking:
+        type: 'enabled'
+        budget_tokens: 16000
+      showThinking: false # Exclude thinking content from output
+```
+
+This is useful when you want better reasoning but don't want to expose the thinking process to your assertions.
+
+For more details on extended thinking capabilities, see the [Anthropic provider docs](/docs/providers/anthropic#extended-thinking) and [AWS Bedrock provider docs](/docs/providers/aws-bedrock#claude-models).
+
+## Transforming outputs
+
+Transforms can be applied at multiple levels in the evaluation pipeline:
+
+### Transform execution order
+
+1. **Provider transforms** (`transformResponse`) - Always applied first
+2. **Test transforms** (`options.transform`) and **Context transforms** (`contextTransform`)
+   - Both receive the output from the provider transform
+   - Test transforms modify the output for assertions
+   - Context transforms extract context for context-based assertions (e.g., `context-faithfulness`)
+
+### Test transform hierarchy
+
+For test transforms specifically:
+
+1. Default test transforms (if specified in `defaultTest`)
+2. Individual test case transforms (overrides `defaultTest` transform if present)
+
+Note that only one transform is applied at the test case level - either from `defaultTest` or the individual test case, not both.
+
+The `TestCase.options.transform` field is a Javascript snippet that modifies the LLM output before it is run through the test assertions.
+
+It is a function that takes a string output and a context object:
+
+```typescript
+transformFn: (output: string, context: {
+  prompt: {
+    // ID of the prompt, if assigned
+    id?: string;
+    // Raw prompt as provided in the test case, without {{variable}} substitution.
+    raw?: string;
+    // Prompt as sent to the LLM API and assertions.
+    display?: string;
+  };
+  vars?: Record<string, any>;
+  // Metadata returned in the provider response.
+  metadata?: Record<string, any>;
+}) => void;
+```
+
+This is useful if you need to somehow transform or clean LLM output before running an eval.
+
+For example:
+
+```yaml
+# ...
+tests:
+  - vars:
+      language: French
+      body: Hello world
+    options:
+      // highlight-start
+      transform: output.toUpperCase()
+      // highlight-end
+    # ...
+```
+
+Or multiline:
+
+```yaml
+# ...
+tests:
+  - vars:
+      language: French
+      body: Hello world
+    options:
+      // highlight-start
+      transform: |
+        output = output.replace(context.vars.language, 'foo');
+        const words = output.split(' ').filter(x => !!x);
+        return JSON.stringify(words);
+      // highlight-end
+    # ...
+```
+
+It also works in assertions, which is useful for picking values out of JSON:
+
+```yaml
+tests:
+  - vars:
+      # ...
+    assert:
+      - type: equals
+        value: 'foo'
+        transform: output.category # Select the 'category' key from output json
+```
+
+:::tip
+Use `defaultTest` apply a transform option to every test case in your test suite.
+:::
+
+:::tip
+When using the [Node.js package](/docs/usage/node-package#transform-functions), you can pass functions directly as `transform`, `transformVars`, and `contextTransform` values instead of string expressions.
+:::
+
+### Transforms from separate files
+
+Transform functions can be executed from external JavaScript or Python files. You can optionally specify a function name to use.
+
+For JavaScript:
+
+```yaml
+defaultTest:
+  options:
+    transform: file://transform.js:customTransform
+```
+
+```js title="transform.js"
+module.exports = {
+  customTransform: (output, context) => {
+    // context.vars, context.prompt
+    return output.toUpperCase();
+  },
+};
+```
+
+For Python:
+
+```yaml
+defaultTest:
+  options:
+    transform: file://transform.py
+```
+
+```python title="transform.py"
+def get_transform(output, context):
+    # context['vars'], context['prompt']
+    return output.upper()
+```
+
+If no function name is specified for Python files, it defaults to `get_transform`. To use a custom Python function, specify it in the file path:
+
+```yaml
+transform: file://transform.py:custom_python_transform
+```
+
+## Transforming input variables
+
+You can also transform input variables before they are used in prompts using the `transformVars` option. This feature is useful when you need to pre-process data or load content from external sources.
+
+The `transformVars` function should return an object with the transformed variable names and values. These transformed variables are added to the `vars` object and can override existing keys. For example:
+
+```yaml
+prompts:
+  - 'Summarize the following text in {{topic_length}} words: {{processed_content}}'
+
+defaultTest:
+  options:
+    transformVars: |
+      return {
+        uppercase_topic: vars.topic.toUpperCase(),
+        topic_length: vars.topic.length,
+        processed_content: vars.content.trim()
+      };
+
+tests:
+  - vars:
+      topic: 'climate change'
+      content: '  This is some text about climate change that needs processing.  '
+    assert:
+      - type: contains
+        value: '{{uppercase_topic}}'
+```
+
+Transform functions can also be specified within individual test cases.
+
+```yaml
+tests:
+  - vars:
+      url: 'https://example.com/image.png'
+    options:
+      transformVars: |
+        return { ...vars, image_markdown: `![image](${vars.url})` }
+```
+
+### Input transforms from separate files
+
+For more complex transformations, you can use external files for `transformVars`:
+
+```yaml
+defaultTest:
+  options:
+    transformVars: file://transformVars.js:customTransformVars
+```
+
+```js title="transformVars.js"
+const fs = require('fs');
+
+module.exports = {
+  customTransformVars: (vars, context) => {
+    try {
+      return {
+        uppercase_topic: vars.topic.toUpperCase(),
+        topic_length: vars.topic.length,
+        file_content: fs.readFileSync(vars.file_path, 'utf-8'),
+      };
+    } catch (error) {
+      console.error('Error in transformVars:', error);
+      return {
+        error: 'Failed to transform variables',
+      };
+    }
+  },
+};
+```
+
+You can also define transforms in python.
+
+```yaml
+defaultTest:
+  options:
+    transformVars: file://transform_vars.py
+```
+
+```python title="transform_vars.py"
+import os
+
+def get_transform(vars, context):
+    with open(vars['file_path'], 'r') as file:
+        file_content = file.read()
+
+    return {
+        'uppercase_topic': vars['topic'].upper(),
+        'topic_length': len(vars['topic']),
+        'file_content': file_content,
+        'word_count': len(file_content.split())
+    }
+```
+
+## Config structure and organization
+
+For detailed information on the config structure, see [Configuration Reference](/docs/configuration/reference).
+
+If you have multiple sets of tests, it helps to split them into multiple config files. Use the `--config` or `-c` parameter to run each individual config:
+
+```
+promptfoo eval -c usecase1.yaml
+```
+
+and
+
+```
+promptfoo eval -c usecase2.yaml
+```
+
+You can run multiple configs at the same time, which will combine them into a single eval. For example:
+
+```
+promptfoo eval -c my_configs/*
+```
+
+or
+
+```
+promptfoo eval -c config1.yaml -c config2.yaml -c config3.yaml
+```
+
+## Loading tests from CSV
+
+YAML is nice, but some organizations maintain their LLM tests in spreadsheets for ease of collaboration. promptfoo supports a special [CSV file format](/docs/configuration/test-cases#csv-format).
+
+```yaml
+prompts:
+  - file://prompt1.txt
+  - file://prompt2.txt
+providers:
+  - openai:gpt-5-mini
+  - vertex:gemini-2.0-flash-exp
+// highlight-next-line
+tests: file://tests.csv
+```
+
+promptfoo also has built-in ability to pull test cases from a Google Sheet. The easiest way to get started is to set the sheet visible to "anyone with the link". For example:
+
+```yaml
+prompts:
+  - file://prompt1.txt
+  - file://prompt2.txt
+providers:
+  - openai:gpt-5-mini
+  - vertex:gemini-2.0-flash-exp
+// highlight-next-line
+tests: https://docs.google.com/spreadsheets/d/1eqFnv1vzkPvS7zG-mYsqNDwOzvSaiIAsKB3zKg9H18c/edit?usp=sharing
+```
+
+Here's a [full example](https://github.com/promptfoo/promptfoo/tree/main/examples/integration-google-sheets).
+
+See [Google Sheets integration](/docs/integrations/google-sheets) for details on how to set up promptfoo to access a private spreadsheet.
+
+Promptfoo can also load test sets from Azure Blob Storage:
+
+```yaml
+// highlight-next-line
+tests: az://myaccount/evals/tests.json
+```
+
+See [Azure Blob Storage test sets](/docs/configuration/test-cases#azure-blob-storage) for supported file types and authentication options.

site/docs/configuration/huggingface-datasets.md

@@ -0,0 +1,172 @@
+---
+displayed_sidebar: promptfoo
+sidebar_label: HuggingFace Datasets
+title: Loading Test Cases from HuggingFace Datasets
+description: Load HuggingFace datasets directly for LLM evaluation with automatic splits, filtering, and format conversion capabilities
+keywords:
+  [
+    huggingface datasets,
+    test cases,
+    dataset integration,
+    promptfoo datasets,
+    ml evaluation,
+    dataset import,
+    existing datasets,
+  ]
+pagination_prev: configuration/datasets
+pagination_next: configuration/scenarios
+---
+
+# HuggingFace Datasets
+
+Promptfoo can import test cases directly from [HuggingFace datasets](https://huggingface.co/docs/datasets) using the `huggingface://datasets/` prefix.
+
+## Basic usage
+
+To load an entire dataset:
+
+```yaml
+tests: huggingface://datasets/fka/awesome-chatgpt-prompts
+```
+
+Run the evaluation:
+
+```bash
+npx promptfoo eval
+```
+
+Each dataset row becomes a test case with all dataset fields available as variables.
+
+## Dataset splits
+
+Load specific portions of datasets using query parameters:
+
+```yaml
+# Load from training split
+tests: huggingface://datasets/fka/awesome-chatgpt-prompts?split=train
+
+# Load from validation split with custom configuration
+tests: huggingface://datasets/fka/awesome-chatgpt-prompts?split=validation&config=custom
+```
+
+## Use dataset fields in prompts
+
+Dataset fields automatically become prompt variables. Here's how:
+
+```yaml title="promptfooconfig.yaml"
+prompts:
+  - "Question: {{question}}\nAnswer:"
+
+tests: huggingface://datasets/rajpurkar/squad
+```
+
+## Query parameters
+
+| Parameter | Description                                   | Default     |
+| --------- | --------------------------------------------- | ----------- |
+| `split`   | Dataset split to load (train/test/validation) | `test`      |
+| `config`  | Dataset configuration name                    | `default`   |
+| `subset`  | Dataset subset (for multi-subset datasets)    | `none`      |
+| `limit`   | Maximum number of test cases to load          | `unlimited` |
+
+The loader accepts any parameter supported by the [HuggingFace Datasets API](https://huggingface.co/docs/datasets-server/api_reference#get-apirows). Additional parameters beyond these common ones are passed directly to the API.
+
+To limit the number of test cases:
+
+```yaml
+tests: huggingface://datasets/fka/awesome-chatgpt-prompts?split=train&limit=50
+```
+
+To load a specific subset (common with MMLU datasets):
+
+```yaml
+tests: huggingface://datasets/cais/mmlu?split=test&subset=physics&limit=10
+```
+
+## Authentication
+
+For private datasets or increased rate limits, authenticate using your HuggingFace token. Set one of these environment variables:
+
+```bash
+# Any of these environment variables will work:
+export HF_TOKEN=your_token_here
+export HF_API_TOKEN=your_token_here
+export HUGGING_FACE_HUB_TOKEN=your_token_here
+```
+
+:::info
+Authentication is required for private datasets and gated models. For public datasets, authentication is optional but provides higher rate limits.
+:::
+
+## Implementation details
+
+- Each dataset row becomes a test case
+- All dataset fields are available as prompt variables
+- Large datasets are automatically paginated (100 rows per request)
+- Variable expansion is disabled to preserve original data
+
+## Example configurations
+
+### Basic chatbot evaluation
+
+```yaml title="promptfooconfig.yaml"
+description: Testing with HuggingFace dataset
+
+prompts:
+  - 'Act as {{act}}. {{prompt}}'
+
+providers:
+  - openai:gpt-5-mini
+
+tests: huggingface://datasets/fka/awesome-chatgpt-prompts?split=train
+```
+
+### Question answering with limits
+
+```yaml title="promptfooconfig.yaml"
+description: SQUAD evaluation with authentication
+
+prompts:
+  - 'Question: {{question}}\nContext: {{context}}\nAnswer:'
+
+providers:
+  - openai:gpt-5-mini
+
+tests: huggingface://datasets/rajpurkar/squad?split=validation&limit=100
+
+env:
+  HF_TOKEN: your_token_here
+```
+
+## Example projects
+
+| Example                                                                                                           | Use Case          | Key Features         |
+| ----------------------------------------------------------------------------------------------------------------- | ----------------- | -------------------- |
+| [Basic Setup](https://github.com/promptfoo/promptfoo/tree/main/examples/huggingface/dataset)                      | Simple evaluation | Default parameters   |
+| [MMLU-Pro Comparison](https://github.com/promptfoo/promptfoo/tree/main/examples/compare-gpt-model-tiers-mmlu-pro) | Query parameters  | Split, config, limit |
+| [Red Team Safety](https://github.com/promptfoo/promptfoo/tree/main/examples/redteam-beavertails)                  | Safety testing    | BeaverTails dataset  |
+
+## Troubleshooting
+
+### Authentication errors
+
+Ensure your HuggingFace token is set correctly: `export HF_TOKEN=your_token`
+
+### Dataset not found
+
+Verify the dataset path format: `owner/repo` (e.g., `rajpurkar/squad`)
+
+### Empty results
+
+Check that the specified split exists for the dataset. Try `split=train` if `split=test` returns no results.
+
+### Performance issues
+
+Add the `limit` parameter to reduce the number of rows loaded: `&limit=100`
+
+## See Also
+
+- [Test Case Configuration](/docs/configuration/test-cases) - Complete guide to configuring test cases
+- [HuggingFace Provider](/docs/providers/huggingface) - Using HuggingFace models for inference
+- [CSV Test Cases](/docs/configuration/test-cases#csv-format) - Loading test cases from CSV files
+- [Red Team Configuration](/docs/red-team/configuration) - Using datasets in red team evaluations

site/docs/configuration/modular-configs.md

@@ -0,0 +1,408 @@
+---
+sidebar_position: 999
+sidebar_label: Managing Large Configs
+title: Managing Large Promptfoo Configurations
+description: Learn how to structure, organize, and modularize large promptfoo configurations for better maintainability and reusability.
+keywords:
+  [
+    promptfoo configuration,
+    modular configs,
+    large configuration,
+    configuration management,
+    reusable configurations,
+    configuration organization,
+    YAML references,
+    file imports,
+  ]
+---
+
+# Managing Large Configurations
+
+As your Promptfoo evaluations grow more complex, you'll need strategies to keep your configurations manageable, maintainable, and reusable. This guide covers best practices for organizing large configurations and making them modular.
+
+## Separate Configuration Files
+
+Split your configuration into multiple files based on functionality:
+
+```yaml title="promptfooconfig.yaml"
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: Main evaluation configuration
+prompts: file://configs/prompts.yaml
+providers: file://configs/providers.yaml
+tests: file://configs/tests/
+defaultTest: file://configs/default-test.yaml
+```
+
+```yaml title="configs/prompts.yaml"
+# Prompts configuration
+- file://prompts/system-message.txt
+- file://prompts/user-prompt.txt
+- id: custom-prompt
+  label: Custom Prompt
+  raw: |
+    You are a helpful assistant. Please answer the following question:
+    {{question}}
+```
+
+```yaml title="configs/providers.yaml"
+# Providers configuration
+- id: gpt-5.2
+  provider: openai:gpt-5.2
+  config:
+    temperature: 0.7
+    max_tokens: 1000
+- id: claude-sonnet
+  provider: anthropic:claude-sonnet-4-5-20250929
+  config:
+    temperature: 0.7
+    max_tokens: 1000
+```
+
+```yaml title="configs/default-test.yaml"
+# Default test configuration
+assert:
+  - type: llm-rubric
+    value: Response should be helpful and accurate
+  - type: javascript
+    value: output.length > 10 && output.length < 500
+```
+
+### Test Case Organization
+
+Organize test cases by domain or functionality:
+
+```yaml title="promptfooconfig.yaml"
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: Multi-domain evaluation
+prompts: file://prompts/
+providers: file://providers.yaml
+tests:
+  - file://tests/accuracy/
+  - file://tests/safety/
+  - file://tests/performance/
+  - file://tests/edge-cases/
+```
+
+```yaml title="tests/accuracy/math-problems.yaml"
+# Math-specific test cases
+- description: Basic arithmetic
+  vars:
+    question: What is 15 + 27?
+  assert:
+    - type: contains
+      value: '42'
+    - type: javascript
+      value: /4[2]/.test(output)
+
+- description: Word problems
+  vars:
+    question: If Sarah has 3 apples and gives away 1, how many does she have left?
+  assert:
+    - type: contains
+      value: '2'
+```
+
+### Environment-Specific Configurations
+
+Create environment-specific configurations:
+
+```yaml title="promptfooconfig.yaml"
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: Production evaluation
+prompts: file://prompts/
+providers: file://configs/providers-prod.yaml
+tests: file://tests/
+env: file://configs/env-prod.yaml
+```
+
+```yaml title="configs/providers-prod.yaml"
+# Production providers with rate limiting
+- id: gpt-5.2-prod
+  provider: openai:gpt-5.2
+  config:
+    temperature: 0.1
+    max_tokens: 500
+    requestsPerMinute: 100
+- id: claude-sonnet-prod
+  provider: anthropic:claude-sonnet-4-5-20250929
+  config:
+    temperature: 0.1
+    max_tokens: 500
+    requestsPerMinute: 50
+```
+
+```yaml title="configs/env-prod.yaml"
+# Production environment variables
+OPENAI_API_KEY: '{{ env.OPENAI_API_KEY_PROD }}'
+ANTHROPIC_API_KEY: '{{ env.ANTHROPIC_API_KEY_PROD }}'
+LOG_LEVEL: info
+```
+
+## YAML References and Templates
+
+Use YAML references to avoid repetition:
+
+```yaml title="promptfooconfig.yaml"
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: Evaluation with reusable components
+prompts: file://prompts/
+providers: file://providers.yaml
+
+# Define reusable assertion templates
+assertionTemplates:
+  lengthCheck: &lengthCheck
+    type: javascript
+    value: output.length > 20 && output.length < 500
+
+  qualityCheck: &qualityCheck
+    type: llm-rubric
+    value: Response should be clear, helpful, and well-structured
+
+  safetyCheck: &safetyCheck
+    type: llm-rubric
+    value: Response should not contain harmful or inappropriate content
+
+defaultTest:
+  assert:
+    - *qualityCheck
+    - *safetyCheck
+
+tests:
+  - description: Short response test
+    vars:
+      input: What is AI?
+    assert:
+      - *lengthCheck
+      - *qualityCheck
+
+  - description: Long response test
+    vars:
+      input: Explain machine learning in detail
+    assert:
+      - type: javascript
+        value: output.length > 100 && output.length < 2000
+      - *qualityCheck
+```
+
+## Dynamic Configuration with JavaScript
+
+Use JavaScript configurations for complex logic:
+
+```javascript title="promptfooconfig.js"
+const baseConfig = {
+  description: 'Dynamic configuration example',
+  prompts: ['file://prompts/base-prompt.txt'],
+  providers: ['openai:gpt-5.2', 'anthropic:claude-sonnet-4-5-20250929'],
+};
+
+// Generate test cases programmatically
+const categories = ['technology', 'science', 'history', 'literature'];
+const difficulties = ['basic', 'intermediate', 'advanced'];
+
+const tests = [];
+for (const category of categories) {
+  for (const difficulty of difficulties) {
+    tests.push({
+      vars: {
+        category,
+        difficulty,
+        question: `Generate a ${difficulty} question about ${category}`,
+      },
+      assert: [
+        {
+          type: 'contains',
+          value: category,
+        },
+        {
+          type: 'javascript',
+          value: `
+            const wordCount = output.split(' ').length;
+            const minWords = ${difficulty === 'basic' ? 5 : difficulty === 'intermediate' ? 15 : 30};
+            const maxWords = ${difficulty === 'basic' ? 20 : difficulty === 'intermediate' ? 50 : 100};
+            return wordCount >= minWords && wordCount <= maxWords;
+          `,
+        },
+      ],
+    });
+  }
+}
+
+module.exports = {
+  ...baseConfig,
+  tests,
+};
+```
+
+## TypeScript Configuration
+
+Promptfoo configs can be written in TypeScript:
+
+```typescript title="promptfooconfig.ts"
+import type { UnifiedConfig } from 'promptfoo';
+
+const config: UnifiedConfig = {
+  description: 'My evaluation suite',
+  prompts: ['Tell me about {{topic}} in {{style}}'],
+  providers: ['openai:gpt-5.2', 'anthropic:claude-sonnet-4-5-20250929'],
+  tests: [
+    {
+      vars: {
+        topic: 'quantum computing',
+        style: 'simple terms',
+      },
+      assert: [
+        {
+          type: 'contains',
+          value: 'quantum',
+        },
+      ],
+    },
+  ],
+};
+
+export default config;
+```
+
+### Running TypeScript Configs
+
+Install a TypeScript loader:
+
+```bash
+npm install tsx
+```
+
+Run with `NODE_OPTIONS`:
+
+```bash
+NODE_OPTIONS="--import tsx" promptfoo eval -c promptfooconfig.ts
+```
+
+### Dynamic Schema Generation
+
+Share Zod schemas between your application and promptfoo:
+
+```typescript title="src/schemas/response.ts"
+import { z } from 'zod';
+
+export const ResponseSchema = z.object({
+  answer: z.string(),
+  confidence: z.number().min(0).max(1),
+  sources: z.array(z.string()).nullable(),
+});
+```
+
+```typescript title="promptfooconfig.ts"
+import { zodResponseFormat } from 'openai/helpers/zod.mjs';
+import type { UnifiedConfig } from 'promptfoo';
+import { ResponseSchema } from './src/schemas/response';
+
+const responseFormat = zodResponseFormat(ResponseSchema, 'response');
+
+const config: UnifiedConfig = {
+  prompts: ['Answer this question: {{question}}'],
+  providers: [
+    {
+      id: 'openai:gpt-5.2',
+      config: {
+        response_format: responseFormat,
+      },
+    },
+  ],
+  tests: [
+    {
+      vars: { question: 'What is TypeScript?' },
+      assert: [{ type: 'is-json' }],
+    },
+  ],
+};
+
+export default config;
+```
+
+See the [ts-config example](https://github.com/promptfoo/promptfoo/tree/main/examples/config-ts) for a complete implementation.
+
+## Conditional Configuration Loading
+
+Create configurations that adapt based on environment:
+
+```javascript title="promptfooconfig.js"
+const isQuickTest = process.env.TEST_MODE === 'quick';
+const isComprehensive = process.env.TEST_MODE === 'comprehensive';
+
+const baseConfig = {
+  description: 'Test mode adaptive configuration',
+  prompts: ['file://prompts/'],
+};
+
+// Quick test configuration
+if (isQuickTest) {
+  module.exports = {
+    ...baseConfig,
+    providers: [
+      'openai:gpt-5.1-mini', // Faster, cheaper for quick testing
+    ],
+    tests: 'file://tests/quick/', // Smaller test suite
+    env: {
+      LOG_LEVEL: 'debug',
+    },
+  };
+}
+
+// Comprehensive test configuration
+if (isComprehensive) {
+  module.exports = {
+    ...baseConfig,
+    providers: [
+      'openai:gpt-5.2',
+      'anthropic:claude-sonnet-4-5-20250929',
+      'google:gemini-2.5-flash',
+    ],
+    tests: 'file://tests/comprehensive/', // Full test suite
+    env: {
+      LOG_LEVEL: 'info',
+    },
+    writeLatestResults: true,
+  };
+}
+```
+
+## Directory Structure
+
+Organize your configuration files in a logical hierarchy:
+
+```
+project/
+├── promptfooconfig.yaml              # Main configuration
+├── configs/
+│   ├── providers/
+│   │   ├── development.yaml
+│   │   ├── staging.yaml
+│   │   └── production.yaml
+│   ├── prompts/
+│   │   ├── system-prompts.yaml
+│   │   ├── user-prompts.yaml
+│   │   └── templates.yaml
+│   └── defaults/
+│       ├── assertions.yaml
+│       └── test-config.yaml
+├── tests/
+│   ├── accuracy/
+│   ├── safety/
+│   ├── performance/
+│   └── edge-cases/
+├── prompts/
+│   ├── system/
+│   ├── user/
+│   └── templates/
+└── scripts/
+    ├── config-generators/
+    └── utilities/
+```
+
+## See Also
+
+- [Configuration Guide](./guide.md) - Basic configuration concepts
+- [Configuration Reference](./reference.md) - Complete configuration options
+- [Test Cases](./test-cases.md) - Organizing test cases
+- [Prompts](./prompts.md) - Managing prompts and templates
+- [Providers](/docs/providers/) - Configuring LLM providers

site/docs/configuration/outputs.md

@@ -0,0 +1,453 @@
+---
+sidebar_position: 31
+sidebar_label: Output Formats
+title: Output Formats - Results Export and Analysis
+description: Configure output formats for LLM evaluation results. Export to HTML, JSON, CSV, and YAML formats for analysis, reporting, and data processing.
+keywords:
+  [
+    output formats,
+    evaluation results,
+    export options,
+    HTML reports,
+    JSON export,
+    CSV analysis,
+    result visualization,
+  ]
+pagination_prev: configuration/huggingface-datasets
+pagination_next: configuration/chat
+---
+
+# Output Formats
+
+Save and analyze your evaluation results in various formats.
+
+## Quick Start
+
+```bash
+# Interactive web viewer (default)
+promptfoo eval
+
+# Save as HTML report
+promptfoo eval --output results.html
+
+# Export as JSON for further processing
+promptfoo eval --output results.json
+
+# Create CSV for spreadsheet analysis
+promptfoo eval --output results.csv
+
+# Generate JUnit XML for CI test-report integrations
+promptfoo eval --output results.junit.xml
+```
+
+## Available Formats
+
+### HTML Report
+
+Generate a visual, shareable report:
+
+```bash
+promptfoo eval --output report.html
+```
+
+**Features:**
+
+- Interactive table with sorting and filtering
+- Side-by-side output comparison
+- Pass/fail statistics
+- Shareable standalone file
+
+**Use when:** Presenting results to stakeholders or reviewing outputs visually.
+
+### JSON Output
+
+Export complete evaluation data:
+
+```bash
+promptfoo eval --output results.json
+```
+
+**Structure:**
+
+```json
+{
+  "version": 3,
+  "timestamp": "2024-01-15T10:30:00Z",
+  "results": {
+    "prompts": [...],
+    "providers": [...],
+    "outputs": [...],
+    "stats": {...}
+  }
+}
+```
+
+**Use when:** Integrating with other tools or performing custom analysis.
+
+### CSV Export
+
+Create spreadsheet-compatible data:
+
+```bash
+promptfoo eval --output results.csv
+```
+
+**Columns include:**
+
+- Test variables
+- Prompt used
+- Model outputs
+- Pass/fail status
+- Latency
+- Token usage
+
+**Use when:** Analyzing results in Excel, Google Sheets, or data science tools.
+
+### YAML Format
+
+Human-readable structured data:
+
+```bash
+promptfoo eval --output results.yaml
+```
+
+**Use when:** Reviewing results in a text editor or version control.
+
+### JSONL Format
+
+Each line contains one JSON result:
+
+```bash
+promptfoo eval --output results.jsonl
+```
+
+**Use when:** Working with very large evaluations or when JSON export fails with memory errors.
+
+```jsonl
+{"testIdx":0,"promptIdx":0,"success":true,"score":1.0,"response":{"output":"Response 1"},"gradingResult":{"pass":true,"score":1.0,"reason":"All assertions passed","componentResults":[{"pass":true,"score":1.0,"reason":"Expected output to contain \"hello\"","assertion":{"type":"contains","value":"hello"}}]}}
+{"testIdx":1,"promptIdx":0,"success":false,"score":0.0,"response":{"output":"Response 2"},"gradingResult":null}
+```
+
+For assertion-level details, inspect each row's `gradingResult?.componentResults` array when
+present. The top-level `success`, `score`, and `gradingResult` fields describe the aggregate
+result for the row, while each `componentResults[]` entry contains the pass/fail, score,
+reason, and assertion metadata for one evaluated assertion. Both `gradingResult` and
+`componentResults` may be absent on error rows or rows without assertions.
+
+To stream a JSONL file and read each row's component results:
+
+```ts
+import fs from 'node:fs';
+import readline from 'node:readline';
+
+const rl = readline.createInterface({
+  input: fs.createReadStream('results.jsonl', { encoding: 'utf8' }),
+  crlfDelay: Infinity,
+});
+
+for await (const line of rl) {
+  if (!line.trim()) {
+    continue;
+  }
+  const row = JSON.parse(line);
+  for (const component of row.gradingResult?.componentResults ?? []) {
+    console.log({
+      type: component.assertion?.type,
+      pass: component.pass,
+      score: component.score,
+      reason: component.reason,
+    });
+  }
+}
+```
+
+`?.` and `?? []` together cover the `gradingResult: null` case shown above and rows
+where a single top-level assertion produced no nested `componentResults`.
+
+### JUnit XML Format
+
+Compact CI test-report output:
+
+```bash
+promptfoo eval --output results.junit.xml
+```
+
+**Structure:**
+
+```xml
+<testsuites tests="2" failures="1" errors="0" time="0.840">
+  <testsuite name="[openai:gpt-4.1] prompt 1" tests="2" failures="1" errors="0" time="0.840">
+    <testcase name="test 1: greets the customer" classname="[openai:gpt-4.1] prompt 1" time="0.420" />
+    <testcase name="test 2: refuses refunds outside policy" classname="[openai:gpt-4.1] prompt 1" time="0.420">
+      <failure message="Assertion failed">Score: 0
+Reason: Assertion failed
+Failed assertions:
+- contains</failure>
+    </testcase>
+  </testsuite>
+</testsuites>
+```
+
+**Use when:** Publishing eval results into CI systems that already understand JUnit-style test reports, such as GitLab, Azure Pipelines, Bitbucket Pipelines, Jenkins, and other test-report viewers.
+
+JUnit XML intentionally stays compact:
+
+- one `testsuite` per prompt/provider pair so CI groups related cases together
+- one `testcase` per eval result so every promptfoo test appears in CI
+- `failure` for failed assertions and `error` for provider/runtime errors so CI can distinguish incorrect behavior from execution problems
+- concise failure/error summaries only; use JSON, HTML, or Promptfoo XML when you need assertion reasons, provider errors, prompts, variables, raw model outputs, or full config
+
+A JUnit report viewer can render the same file into a compact pass/fail report:
+
+![Rendered JUnit XML report showing one suite, two failures, and two passing tests](/img/docs/configuration/junit-xml-report.png)
+
+### Promptfoo XML Full Export
+
+Full eval data for XML-only consumers:
+
+```bash
+promptfoo eval --output results.xml
+```
+
+**Structure:**
+
+```xml
+<promptfoo>
+  <evalId>abc-123-def</evalId>
+  <results>
+    <version>3</version>
+    <timestamp>2024-01-15T10:30:00Z</timestamp>
+    <prompts>...</prompts>
+    <providers>...</providers>
+    <outputs>...</outputs>
+    <stats>...</stats>
+  </results>
+  <config>...</config>
+  <shareableUrl>...</shareableUrl>
+</promptfoo>
+```
+
+**Use when:** A downstream system specifically requires the full Promptfoo export in XML. This is not a JUnit-compatible CI report format; use JUnit XML for CI dashboards and test-report viewers.
+
+## Configuration Options
+
+### Setting Output Path in Config
+
+```yaml title="promptfooconfig.yaml"
+# Specify default output file
+outputPath: evaluations/latest_results.html
+
+prompts:
+  - '...'
+tests:
+  - '...'
+```
+
+### Multiple Output Formats
+
+Generate multiple formats simultaneously:
+
+```bash
+# Command line
+promptfoo eval --output results.html --output results.json
+
+# Or use shell commands
+promptfoo eval --output results.json && \
+promptfoo eval --output results.csv
+```
+
+## Output Contents
+
+### Structured Output Fields
+
+`json`, `yaml`, `yml`, `txt`, and Promptfoo XML outputs include:
+
+| Field       | Description                  |
+| ----------- | ---------------------------- |
+| `timestamp` | When the evaluation ran      |
+| `prompts`   | Prompts used in evaluation   |
+| `providers` | LLM providers tested         |
+| `tests`     | Test cases with variables    |
+| `outputs`   | Raw LLM responses            |
+| `results`   | Pass/fail for each assertion |
+| `stats`     | Summary statistics           |
+
+:::warning
+
+`json`, `yaml`, `yml`, `txt`, `html`, and Promptfoo XML outputs include the eval `config`. Sensitive fields are redacted using Promptfoo's sanitizer rules on a best-effort basis (not comprehensive). Non-sensitive `config.env` values may still appear in exports.
+
+JUnit XML omits the eval config, prompts, variables, raw model outputs, assertion reasons, and provider error payloads by design so CI test-report viewers stay compact and do not become a second full export surface.
+
+:::
+
+### Detailed Metrics
+
+When available, outputs include:
+
+- **Latency**: Response time in milliseconds
+- **Token Usage**: Input/output token counts
+- **Cost**: Estimated API costs
+- **Error Details**: Failure reasons and stack traces
+
+## Analyzing Results
+
+### JSON Processing Example
+
+```javascript
+const fs = require('fs');
+
+// Load results
+const results = JSON.parse(fs.readFileSync('results.json', 'utf8'));
+
+// Analyze pass rates by provider
+const providerStats = {};
+results.results.outputs.forEach((output) => {
+  const provider = output.provider;
+  if (!providerStats[provider]) {
+    providerStats[provider] = { pass: 0, fail: 0 };
+  }
+
+  if (output.pass) {
+    providerStats[provider].pass++;
+  } else {
+    providerStats[provider].fail++;
+  }
+});
+
+console.log('Pass rates by provider:', providerStats);
+```
+
+### CSV Analysis with Pandas
+
+```python
+import pandas as pd
+
+# Load results
+df = pd.read_csv('results.csv')
+
+# Group by provider and calculate metrics
+summary = df.groupby('provider').agg({
+    'pass': 'mean',
+    'latency': 'mean',
+    'cost': 'sum'
+})
+
+print(summary)
+```
+
+## Best Practices
+
+### 1. Organize Output Files
+
+```text
+project/
+├── promptfooconfig.yaml
+├── evaluations/
+│   ├── 2024-01-15-baseline.html
+│   ├── 2024-01-16-improved.html
+│   └── comparison.json
+```
+
+### 2. Use Descriptive Filenames
+
+```bash
+# Include date and experiment name
+promptfoo eval --output "results/$(date +%Y%m%d)-gpt4-temperature-test.html"
+```
+
+### 3. Version Control Considerations
+
+```gitignore
+# .gitignore
+# Exclude large output files
+evaluations/*.html
+evaluations/*.json
+
+# But keep summary reports
+!evaluations/summary-*.csv
+```
+
+### 4. Automate Report Generation
+
+```bash
+#!/bin/bash
+# run_evaluation.sh
+
+TIMESTAMP=$(date +%Y%m%d-%H%M%S)
+promptfoo eval \
+  --output "reports/${TIMESTAMP}-full.json" \
+  --output "reports/${TIMESTAMP}-summary.html"
+```
+
+## Sharing Results
+
+### Web Viewer
+
+The default web viewer (`promptfoo view`) provides:
+
+- Real-time updates during evaluation
+- Interactive exploration
+- Local-only (no data sent externally)
+
+### Sharing HTML Reports
+
+HTML outputs are self-contained:
+
+```bash
+# Generate report
+promptfoo eval --output team-review.html
+
+# Share via email, Slack, etc.
+# No external dependencies required
+```
+
+### Promptfoo Share
+
+For collaborative review:
+
+```bash
+# Share results with your team
+promptfoo share
+```
+
+Creates a shareable link with:
+
+- Read-only access
+- Commenting capabilities
+- No setup required for viewers
+
+## Troubleshooting
+
+### Large Output Files
+
+For extensive evaluations:
+
+```yaml
+# Limit output size
+outputPath: results.json
+sharing:
+  # Exclude raw outputs from file
+  includeRawOutputs: false
+```
+
+### Encoding Issues
+
+Ensure proper encoding for international content:
+
+```bash
+# Explicitly set encoding
+LANG=en_US.UTF-8 promptfoo eval --output results.csv
+```
+
+### Performance Tips
+
+1. **Use JSONL for large datasets** - avoids memory issues
+2. **Use JSON for standard datasets** - complete data structure
+3. **Generate HTML for presentations** - best visual format
+4. **Use CSV for data analysis** - Excel/Sheets compatible
+
+## Related Documentation
+
+- [Configuration Reference](/docs/configuration/reference) - All output options
+- [Integrations](/docs/category/integrations/) - Using outputs with other tools
+- [Command Line Guide](/docs/usage/command-line) - CLI options

site/docs/configuration/parameters.md

@@ -0,0 +1,251 @@
+---
+displayed_sidebar: promptfoo
+sidebar_label: Overview
+title: Configuration Overview - Prompts, Tests, and Outputs
+description: Quick overview of promptfoo's core configuration concepts including prompts, test cases, outputs, and common patterns for LLM evaluation.
+keywords:
+  [
+    promptfoo overview,
+    configuration basics,
+    prompt setup,
+    test cases,
+    output formats,
+    evaluation workflow,
+  ]
+pagination_prev: configuration/reference
+pagination_next: configuration/prompts
+---
+
+# Prompts, tests, and outputs
+
+Configure how promptfoo evaluates your LLM applications.
+
+:::tip Detailed Documentation
+For comprehensive guides, see the dedicated pages:
+
+- **[Prompts](/docs/configuration/prompts)** - Configure what you send to LLMs
+- **[Test Cases](/docs/configuration/test-cases)** - Set up evaluation scenarios
+- **[Output Formats](/docs/configuration/outputs)** - Save and analyze results
+  :::
+
+## Quick Start
+
+```yaml title="promptfooconfig.yaml"
+# Define your prompts
+prompts:
+  - 'Translate to {{language}}: {{text}}'
+
+# Configure test cases
+tests:
+  - vars:
+      language: French
+      text: Hello world
+    assert:
+      - type: contains
+        value: Bonjour
+# Run evaluation
+# promptfoo eval
+```
+
+## Core Concepts
+
+### 📝 [Prompts](/docs/configuration/prompts)
+
+Define what you send to your LLMs - from simple strings to complex conversations.
+
+<details>
+<summary><strong>Common patterns</strong></summary>
+
+**Text prompts**
+
+```yaml
+prompts:
+  - 'Summarize this: {{content}}'
+  - file://prompts/customer_service.txt
+```
+
+**Chat conversations**
+
+```yaml
+prompts:
+  - file://prompts/chat.json
+```
+
+**Dynamic prompts**
+
+```yaml
+prompts:
+  - file://generate_prompt.js
+  - file://create_prompt.py
+```
+
+</details>
+
+[Learn more about prompts →](/docs/configuration/prompts)
+
+### 🧪 [Test Cases](/docs/configuration/test-cases)
+
+Configure evaluation scenarios with variables and assertions.
+
+<details>
+<summary><strong>Common patterns</strong></summary>
+
+**Inline tests**
+
+```yaml
+tests:
+  - vars:
+      question: "What's 2+2?"
+    assert:
+      - type: equals
+        value: '4'
+```
+
+**CSV test data**
+
+```yaml
+tests: file://test_cases.csv
+```
+
+**HuggingFace datasets**
+
+```yaml
+tests: huggingface://datasets/rajpurkar/squad
+```
+
+**Azure Blob Storage**
+
+```yaml
+tests: az://myaccount/evals/tests.json
+```
+
+**Dynamic generation**
+
+```yaml
+tests: file://generate_tests.js
+```
+
+</details>
+
+[Learn more about test cases →](/docs/configuration/test-cases)
+
+### 📊 [Output Formats](/docs/configuration/outputs)
+
+Save and analyze your evaluation results.
+
+<details>
+<summary><strong>Available formats</strong></summary>
+
+```bash
+# Visual report
+promptfoo eval --output results.html
+
+# Data analysis
+promptfoo eval --output results.json
+
+# Spreadsheet
+promptfoo eval --output results.csv
+```
+
+</details>
+
+[Learn more about outputs →](/docs/configuration/outputs)
+
+## Complete Example
+
+Here's a real-world example that combines multiple features:
+
+```yaml title="promptfooconfig.yaml"
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: Customer service chatbot evaluation
+
+prompts:
+  # Simple text prompt
+  - 'You are a helpful customer service agent. {{query}}'
+
+  # Chat conversation format
+  - file://prompts/chat_conversation.json
+
+  # Dynamic prompt with logic
+  - file://prompts/generate_prompt.js
+
+providers:
+  - openai:gpt-5-mini
+  - anthropic:claude-3-haiku
+
+tests:
+  # Inline test cases
+  - vars:
+      query: 'I need to return a product'
+    assert:
+      - type: contains
+        value: 'return policy'
+      - type: llm-rubric
+        value: 'Response is helpful and professional'
+
+  # Load more tests from CSV
+  - file://test_scenarios.csv
+
+# Save results
+outputPath: evaluations/customer_service_results.html
+```
+
+## Quick Reference
+
+### Supported File Formats
+
+| Format               | Prompts | Tests | Use Case                            |
+| -------------------- | ------- | ----- | ----------------------------------- |
+| `.txt`               | ✅      | ❌    | Simple text prompts                 |
+| `.json`              | ✅      | ✅    | Chat conversations, structured data |
+| `.yaml`              | ✅      | ✅    | Complex configurations              |
+| `.csv`               | ✅      | ✅    | Bulk data, multiple variants        |
+| `.js`/`.ts`          | ✅      | ✅    | Dynamic generation with logic       |
+| `.py`                | ✅      | ✅    | Python-based generation             |
+| `.md`                | ✅      | ❌    | Markdown-formatted prompts          |
+| `.j2`                | ✅      | ❌    | Jinja2 templates                    |
+| HuggingFace datasets | ❌      | ✅    | Import from existing datasets       |
+
+### Variable Syntax
+
+Variables use [Nunjucks](https://mozilla.github.io/nunjucks/) templating:
+
+```yaml
+# Basic substitution
+prompt: 'Hello {{name}}'
+
+# Filters
+prompt: 'URGENT: {{message | upper}}'
+
+# Conditionals
+prompt: '{% if premium %}Premium support: {% endif %}{{query}}'
+```
+
+### File References
+
+All file paths are relative to the config file:
+
+```yaml
+# Single file
+prompts:
+  - file://prompts/main.txt
+
+# Multiple files with glob
+tests:
+  - file://tests/*.yaml
+
+# Specific function
+prompts:
+  - file://generate.js:createPrompt
+```
+
+Wildcards like `path/to/prompts/**/*.py:func_name` are also supported.
+
+## Next Steps
+
+- **[Prompts](/docs/configuration/prompts)** - Deep dive into prompt configuration
+- **[Test Cases](/docs/configuration/test-cases)** - Learn about test scenarios and assertions
+- **[HuggingFace Datasets](/docs/configuration/huggingface-datasets)** - Import test cases from existing datasets
+- **[Output Formats](/docs/configuration/outputs)** - Understand evaluation results
+- **[Expected Outputs](/docs/configuration/expected-outputs)** - Configure assertions
+- **[Configuration Reference](/docs/configuration/reference)** - All configuration options

site/docs/configuration/prompts.md

@@ -0,0 +1,496 @@
+---
+sidebar_position: 11
+sidebar_label: Prompts
+title: Prompt Configuration - Text, Chat, and Dynamic Prompts
+description: Configure prompts for LLM evaluation including text prompts, chat conversations, file-based prompts, and dynamic prompt generation with variables.
+keywords:
+  [
+    prompt configuration,
+    LLM prompts,
+    chat conversations,
+    dynamic prompts,
+    template variables,
+    prompt engineering,
+  ]
+pagination_prev: configuration/reference
+pagination_next: configuration/test-cases
+---
+
+# Prompt Configuration
+
+Define what you send to your LLMs - from simple strings to complex multi-turn conversations.
+
+## Text Prompts
+
+The simplest way to define prompts is with plain text:
+
+```yaml title="promptfooconfig.yaml"
+prompts:
+  - 'Translate the following text to French: "{{text}}"'
+  - 'Summarize this article: {{article}}'
+```
+
+### Multiline Prompts
+
+Use YAML's multiline syntax for longer prompts:
+
+```yaml title="promptfooconfig.yaml"
+prompts:
+  - |-
+    You are a helpful assistant.
+
+    Please answer the following question:
+    {{question}}
+
+    Provide a detailed explanation.
+```
+
+### Variables and Templates
+
+Prompts use [Nunjucks](https://mozilla.github.io/nunjucks/) templating:
+
+```yaml
+prompts:
+  - 'Hello {{name}}, welcome to {{company}}!'
+  - 'Product: {{product | upper}}' # Using filters
+  - '{% if premium %}Priority support: {% endif %}{{issue}}' # Conditionals
+```
+
+## File-Based Prompts
+
+Store prompts in external files for better organization:
+
+```yaml title="promptfooconfig.yaml"
+prompts:
+  - file://prompts/customer_service.txt
+  - file://prompts/technical_support.txt
+```
+
+```txt title="prompts/customer_service.txt"
+You are a friendly customer service representative for {{company}}.
+
+Customer query: {{query}}
+
+Please provide a helpful and professional response.
+```
+
+### Supported File Formats
+
+#### Text Files (.txt)
+
+Simple text prompts with variable substitution.
+
+#### Markdown Files (.md)
+
+```markdown title="prompt.md"
+# System Instructions
+
+You are an AI assistant for {{company}}.
+
+## Your Task
+
+{{task}}
+```
+
+#### Jinja2 Templates (.j2)
+
+```jinja title="prompt.j2"
+You are assisting with {{ topic }}.
+{% if advanced_mode %}
+Provide technical details and code examples.
+{% else %}
+Keep explanations simple and clear.
+{% endif %}
+```
+
+#### CSV Files (.csv)
+
+Define multiple prompts in a CSV file:
+
+```csv title="prompts.csv"
+prompt,label
+"Translate to French: {{text}}","French Translation"
+"Translate to Spanish: {{text}}","Spanish Translation"
+"Translate to German: {{text}}","German Translation"
+```
+
+### Multiple Prompts in One File
+
+Separate multiple prompts with `---`:
+
+```text title="prompts.txt"
+Translate to French: {{text}}
+---
+Translate to Spanish: {{text}}
+---
+Translate to German: {{text}}
+```
+
+### Using Globs
+
+Load multiple files with glob patterns:
+
+```yaml
+prompts:
+  - file://prompts/*.txt
+  - file://scenarios/**/*.json
+```
+
+Wildcards like `path/to/prompts/**/*.py:func_name` are also supported.
+
+## Chat Format (JSON)
+
+For conversation-style interactions, use JSON format:
+
+```yaml title="promptfooconfig.yaml"
+prompts:
+  - file://chat_prompt.json
+```
+
+```json title="chat_prompt.json"
+[
+  {
+    "role": "system",
+    "content": "You are a helpful coding assistant."
+  },
+  {
+    "role": "user",
+    "content": "Write a function to {{task}}"
+  }
+]
+```
+
+### Multi-Turn Conversations
+
+```json title="conversation.json"
+[
+  {
+    "role": "system",
+    "content": "You are a tutoring assistant."
+  },
+  {
+    "role": "user",
+    "content": "What is recursion?"
+  },
+  {
+    "role": "assistant",
+    "content": "Recursion is a programming technique where a function calls itself."
+  },
+  {
+    "role": "user",
+    "content": "Can you show me an example in {{language}}?"
+  }
+]
+```
+
+## Dynamic Prompts (Functions)
+
+Use JavaScript or Python to generate prompts with custom logic:
+
+### JavaScript Functions
+
+```yaml title="promptfooconfig.yaml"
+prompts:
+  - file://generate_prompt.js
+```
+
+```javascript title="generate_prompt.js"
+module.exports = async function ({ vars, provider }) {
+  // Access variables and provider info
+  const topic = vars.topic;
+  const complexity = vars.complexity || 'medium';
+
+  // Build prompt based on logic
+  if (complexity === 'simple') {
+    return `Explain ${topic} in simple terms.`;
+  } else {
+    return `Provide a detailed explanation of ${topic} with examples.`;
+  }
+};
+```
+
+### Python Functions
+
+```yaml title="promptfooconfig.yaml"
+prompts:
+  - file://generate_prompt.py:create_prompt
+```
+
+```python title="generate_prompt.py"
+def create_prompt(context):
+    vars = context['vars']
+    provider = context['provider']
+
+    # Dynamic prompt generation
+    if vars.get('technical_audience'):
+        return f"Provide a technical analysis of {vars['topic']}"
+    else:
+        return f"Explain {vars['topic']} for beginners"
+```
+
+### Function with Configuration
+
+Return both prompt and provider configuration:
+
+```javascript title="prompt_with_config.js"
+module.exports = async function ({ vars }) {
+  const complexity = vars.complexity || 'medium';
+
+  return {
+    prompt: `Analyze ${vars.topic}`,
+    config: {
+      temperature: complexity === 'creative' ? 0.9 : 0.3,
+      max_tokens: complexity === 'detailed' ? 1000 : 200,
+    },
+  };
+};
+```
+
+## Executable Scripts
+
+Run any script or binary to generate prompts dynamically. This lets you use your existing tooling and any programming language.
+
+Your script receives test context as JSON in the first argument and outputs the prompt to stdout.
+
+### Usage
+
+Explicitly mark as executable:
+
+```yaml title="promptfooconfig.yaml"
+prompts:
+  - exec:./generate-prompt.sh
+  - exec:/usr/bin/my-prompt-tool
+```
+
+Or just reference the script directly (auto-detected for `.sh`, `.bash`, `.rb`, `.pl`, and other common script extensions):
+
+```yaml title="promptfooconfig.yaml"
+prompts:
+  - ./generate-prompt.sh
+  - ./prompt_builder.rb
+```
+
+:::note
+Python files (`.py`) are processed as Python prompt templates, not executables. To run a Python script as an executable prompt, use the `exec:` prefix: `exec:./generator.py`
+:::
+
+Pass configuration if needed:
+
+```yaml title="promptfooconfig.yaml"
+prompts:
+  - label: 'Technical Prompt'
+    raw: exec:./generator.sh
+    config:
+      style: technical
+      verbose: true
+```
+
+### Examples
+
+Shell script that reads from a database:
+
+```bash title="fetch-context.sh"
+#!/bin/bash
+CONTEXT=$1
+USER_ID=$(echo "$CONTEXT" | jq -r '.vars.user_id')
+
+# Fetch user history from database
+HISTORY=$(psql -h localhost -U myapp -t -v user_id="$USER_ID" -c \
+  "SELECT prompt_context FROM users WHERE id = :'user_id'")
+
+echo "Based on your previous interactions: $HISTORY
+
+How can I help you today?"
+```
+
+Ruby script:
+
+```ruby title="ab-test.rb"
+#!/usr/bin/env ruby
+require 'json'
+require 'digest'
+
+context = JSON.parse(ARGV[0])
+user_id = context['vars']['user_id']
+
+# Call LLM API here...
+puts "\nUser query: #{context['vars']['query']}"
+```
+
+### Security Considerations
+
+:::warning
+Executable scripts run with full permissions of the promptfoo process. Be mindful of:
+
+- **User Input**: Scripts receive user-controlled `vars` as JSON. Always validate and sanitize inputs before using them in commands.
+- **Untrusted Scripts**: Only run scripts from trusted sources. Scripts can access files, make network calls, and execute commands.
+- **Environment Access**: Scripts can access environment variables, including API keys.
+- **Timeout**: Configure a timeout via `config.timeout` (default: 60 seconds) to prevent hanging scripts.
+  :::
+
+### When to Use
+
+This approach works well when you're already using scripts for prompt generation, need to query external systems (databases, APIs), or want to reuse code written in languages other than JavaScript or Python.
+
+Scripts can be written in any language - Bash, Go, Rust, or even compiled binaries - as long as it reads JSON from argv and prints to stdout.
+
+Note that there are dedicated handlers for Python and Javascript (see above).
+
+## Model-Specific Prompts
+
+Different prompts for different providers:
+
+```yaml title="promptfooconfig.yaml"
+prompts:
+  - id: file://prompts/gpt_prompt.json
+    label: gpt_prompt
+  - id: file://prompts/claude_prompt.txt
+    label: claude_prompt
+
+providers:
+  - id: openai:gpt-4
+    prompts: [gpt_prompt]
+  - id: anthropic:claude-3
+    prompts: [claude_prompt]
+```
+
+Prompt filters match labels exactly, support group prefixes (e.g. `group` matches `group:...`), and allow wildcard prefixes like `group:*`.
+
+The `prompts` field also works when providers are defined in external files (`file://provider.yaml`).
+
+## External Prompt Management Systems
+
+Promptfoo integrates with external prompt management platforms, allowing you to centralize and version control your prompts:
+
+### Langfuse
+
+[Langfuse](/docs/integrations/langfuse) is an open-source LLM engineering platform with collaborative prompt management:
+
+```yaml
+prompts:
+  # Reference by version (numeric values)
+  - langfuse://my-prompt:3:text
+  - langfuse://chat-prompt:1:chat
+
+  # Reference by label using @ syntax (recommended for clarity)
+  - langfuse://my-prompt@production
+  - langfuse://chat-prompt@staging:chat
+  - langfuse://email-template@latest:text
+
+  # Reference by label using : syntax (auto-detected strings)
+  - langfuse://my-prompt:production # String detected as label
+  - langfuse://chat-prompt:staging:chat # String detected as label
+```
+
+### Portkey
+
+[Portkey](/docs/integrations/portkey) provides AI observability with prompt management capabilities:
+
+```yaml
+prompts:
+  - portkey://pp-customer-support-v2
+  - portkey://pp-email-generator-prod
+```
+
+### Helicone
+
+[Helicone](/docs/integrations/helicone) offers prompt management alongside observability features:
+
+```yaml
+prompts:
+  - helicone://greeting-prompt:1.0
+  - helicone://support-chat:2.5
+```
+
+Variables from your test cases are automatically passed to these external prompts.
+
+## Advanced Features
+
+### Custom Nunjucks Filters
+
+Create custom filters for prompt processing:
+
+```js title="uppercase_first.js"
+module.exports = function (str) {
+  return str.charAt(0).toUpperCase() + str.slice(1);
+};
+```
+
+```yaml title="promptfooconfig.yaml"
+nunjucksFilters:
+  uppercaseFirst: ./uppercase_first.js
+
+prompts:
+  - 'Dear {{ name | uppercaseFirst }}, {{ message }}'
+```
+
+### Prompt Labels and IDs
+
+Organize prompts with labels:
+
+```yaml
+prompts:
+  - id: file://customer_prompt.txt
+    label: 'Customer Service'
+  - id: file://technical_prompt.txt
+    label: 'Technical Support'
+```
+
+### Default Prompt
+
+If no prompts are specified, promptfoo uses `{{prompt}}` as a passthrough.
+
+## Best Practices
+
+1. **Start Simple**: Use inline text for basic use cases
+2. **Organize Complex Prompts**: Move longer prompts to files
+3. **Use Version Control**: Track prompt files in Git
+4. **Leverage Templates**: Use variables for reusable prompts
+5. **Test Variations**: Create multiple versions to compare performance
+
+## Common Patterns
+
+### System + User Message
+
+```json
+[
+  { "role": "system", "content": "You are {{role}}" },
+  { "role": "user", "content": "{{query}}" }
+]
+```
+
+### Few-Shot Examples
+
+```yaml
+prompts:
+  - |-
+    Classify the sentiment:
+
+    Text: "I love this!" → Positive
+    Text: "This is terrible" → Negative
+    Text: "{{text}}" →
+```
+
+### Chain of Thought
+
+```yaml
+prompts:
+  - |-
+    Question: {{question}}
+
+    Let's think step by step:
+    1. First, identify what we know
+    2. Then, determine what we need to find
+    3. Finally, solve the problem
+
+    Answer:
+```
+
+## Viewing Final Prompts
+
+To see the final rendered prompts:
+
+1. Run `promptfoo view`
+2. Enable **Table Settings** > **Show full prompt in output cell**
+
+This shows exactly what was sent to each provider after variable substitution.

site/docs/configuration/rate-limits.md

@@ -0,0 +1,210 @@
+---
+title: Rate Limits
+description: Configure automatic rate limit handling with exponential backoff, header-aware delays, and adaptive concurrency for LLM provider APIs.
+sidebar_label: Rate Limits
+sidebar_position: 15
+---
+
+# Rate Limits
+
+Promptfoo automatically handles rate limits from LLM providers. When a provider returns HTTP 429 or similar rate limit errors, requests are automatically retried with exponential backoff.
+
+## Automatic Handling
+
+Rate limit handling is built into the evaluator and requires no configuration:
+
+- **Automatic retry**: Failed requests are retried up to 3 times with exponential backoff by default (overridable per provider via `maxRetries`, including `0` to disable retries)
+- **Header-aware delays**: Respects `retry-after` headers from providers
+- **Adaptive concurrency**: Reduces concurrent requests when rate limits are hit
+- **Per-provider isolation**: Each provider and API key has separate rate limit tracking
+
+### Supported Headers
+
+Promptfoo parses rate limit headers from major providers:
+
+| Provider     | Headers                                                                                                          |
+| ------------ | ---------------------------------------------------------------------------------------------------------------- |
+| OpenAI       | `x-ratelimit-remaining-requests`, `x-ratelimit-limit-requests`, `x-ratelimit-remaining-tokens`, `retry-after-ms` |
+| Anthropic    | `anthropic-ratelimit-requests-remaining`, `anthropic-ratelimit-tokens-remaining`, `retry-after`                  |
+| Azure OpenAI | `x-ratelimit-remaining-requests`, `retry-after-ms`, `retry-after`                                                |
+| Generic      | `retry-after`, `ratelimit-remaining`, `ratelimit-reset`                                                          |
+
+### Transient Error Handling
+
+Promptfoo automatically retries requests that fail with transient server errors:
+
+| Status Code | Description         | Retry Condition                                      |
+| ----------- | ------------------- | ---------------------------------------------------- |
+| 502         | Bad Gateway         | Status text contains "bad gateway"                   |
+| 503         | Service Unavailable | Status text contains "service unavailable"           |
+| 504         | Gateway Timeout     | Status text contains "gateway timeout"               |
+| 524         | A Timeout Occurred  | Status text contains "timeout" (Cloudflare-specific) |
+
+These errors are retried up to 3 times with exponential backoff (1s, 2s, 4s). The status text check ensures that permanent failures (like authentication errors that happen to use 502) are not retried.
+
+### How Adaptive Concurrency Works
+
+The scheduler uses AIMD (Additive Increase, Multiplicative Decrease) to optimize throughput:
+
+1. When a rate limit is hit, concurrency is reduced by 50%
+2. After sustained successful requests, concurrency increases by 1
+3. When remaining quota drops below 10% (from headers), concurrency is proactively reduced
+
+This allows you to set a higher `maxConcurrency` and let promptfoo find the optimal rate automatically.
+
+## Configuration
+
+### Concurrency
+
+Control the maximum number of concurrent requests:
+
+```yaml
+evaluateOptions:
+  maxConcurrency: 10
+```
+
+Or via CLI:
+
+```bash
+promptfoo eval --max-concurrency 10
+```
+
+The adaptive scheduler will reduce this if rate limits are encountered, but cannot exceed your configured maximum.
+
+### Fixed Delay
+
+Add a fixed delay between requests (in addition to any rate limit backoff):
+
+```yaml
+evaluateOptions:
+  delay: 1000 # milliseconds
+```
+
+Or via CLI:
+
+```bash
+promptfoo eval --delay 1000
+```
+
+Or via environment variable:
+
+```bash
+PROMPTFOO_DELAY_MS=1000 promptfoo eval
+```
+
+### Backoff Configuration
+
+Promptfoo has two retry layers:
+
+1. **Provider-level retry** (scheduler): Retries `callApi()` with 1-second base backoff, up to 3 times by default. If a provider config sets `maxRetries`, the scheduler uses that value (including `0` to disable scheduler retries entirely).
+2. **HTTP-level retry**: Retries failed HTTP requests. Defaults to 4 retries, or the provider's `maxRetries` when set.
+
+When a provider config includes `maxRetries`, promptfoo propagates that value to both layers. Explicit per-call overrides (e.g. a provider that passes a specific `maxRetries` to `fetchWithRetries`) still take precedence. For direct `fetchWithProxy` calls, transient retries (502/503/504/524) are disabled when the provider sets `maxRetries: 0`.
+
+Example — disable retries for a provider to fail fast on rate limits:
+
+```yaml
+providers:
+  - id: openai:chat:gpt-4.1-mini
+    config:
+      maxRetries: 0
+```
+
+Environment variables for the scheduler:
+
+| Environment Variable                   | Description                                | Default  |
+| -------------------------------------- | ------------------------------------------ | -------- |
+| `PROMPTFOO_DISABLE_ADAPTIVE_SCHEDULER` | Disable adaptive concurrency (use fixed)   | false    |
+| `PROMPTFOO_MIN_CONCURRENCY`            | Minimum concurrency (floor for adaptive)   | 1        |
+| `PROMPTFOO_SCHEDULER_QUEUE_TIMEOUT_MS` | Timeout for queued requests (0 to disable) | 300000ms |
+
+Environment variables for HTTP-level retry:
+
+| Environment Variable           | Description                       | Default |
+| ------------------------------ | --------------------------------- | ------- |
+| `PROMPTFOO_REQUEST_BACKOFF_MS` | Base delay for HTTP retry backoff | 5000ms  |
+| `PROMPTFOO_RETRY_5XX`          | Retry on HTTP 500 errors          | false   |
+
+Example:
+
+```bash
+PROMPTFOO_REQUEST_BACKOFF_MS=10000 PROMPTFOO_RETRY_5XX=true promptfoo eval
+```
+
+The scheduler's retry handles most rate limiting automatically. The HTTP-level retry provides additional resilience for network issues.
+
+## Provider-Specific Notes
+
+### OpenAI
+
+OpenAI has separate rate limits for requests and tokens. The scheduler tracks both. For high-volume evaluations:
+
+```yaml
+evaluateOptions:
+  maxConcurrency: 20 # Scheduler will adapt down if needed
+```
+
+See [OpenAI troubleshooting](/docs/providers/openai#troubleshooting) for additional options.
+
+### Anthropic
+
+Anthropic rate limits are typically per-minute. The scheduler respects `retry-after` headers from the API.
+
+### Custom Providers
+
+Custom providers trigger automatic retry when errors contain:
+
+- "429"
+- "rate limit"
+- "too many requests"
+
+To provide retry timing, include headers in your response metadata:
+
+```javascript
+return {
+  output: 'response',
+  metadata: {
+    headers: {
+      'retry-after': '60', // seconds
+    },
+  },
+};
+```
+
+## Debugging
+
+To see rate limit events, enable debug logging:
+
+```bash
+LOG_LEVEL=debug promptfoo eval -c config.yaml
+```
+
+Events logged:
+
+- `ratelimit:hit` - Rate limit encountered
+- `ratelimit:learned` - Provider limits discovered from headers
+- `ratelimit:warning` - Approaching rate limit threshold
+- `concurrency:decreased` / `concurrency:increased` - Adaptive concurrency changes
+- `request:retrying` - Retry in progress
+
+## Best Practices
+
+1. **Start with higher concurrency** - Set `maxConcurrency` to your desired throughput; the scheduler will adapt down if needed
+
+2. **Use caching** - Enable [caching](/docs/configuration/caching) to avoid re-running identical requests
+
+3. **Monitor debug logs** - If evaluations are slow, check for frequent `ratelimit:hit` events
+
+4. **Consider provider tiers** - Higher API tiers typically have higher rate limits; the scheduler will automatically use whatever limits the provider allows
+
+## Disabling Automatic Handling
+
+The scheduler is always active but has minimal overhead. For fully deterministic behavior (e.g., in tests), use:
+
+```yaml
+evaluateOptions:
+  maxConcurrency: 1
+  delay: 1000
+```
+
+This ensures sequential execution with fixed delays between requests.

site/docs/configuration/scenarios.md

@@ -0,0 +1,140 @@
+---
+sidebar_position: 13
+sidebar_label: Scenarios
+title: Scenario Configuration - Grouping Tests and Data
+description: Configure scenarios to group test data with evaluation tests. Learn how to organize and run multiple test combinations efficiently in promptfoo.
+keywords:
+  [
+    test scenarios,
+    grouped testing,
+    test organization,
+    data combinations,
+    evaluation scenarios,
+    test management,
+  ]
+pagination_prev: configuration/test-cases
+pagination_next: configuration/datasets
+---
+
+# Scenarios
+
+The `scenarios` configuration lets you group a set of data along with a set of tests that should be run on that data.
+This is useful for when you want to test a wide range of inputs with the same set of tests.
+
+## Example
+
+Let's take the example of a language translation app. We want to test whether the system can accurately translate three phrases ('Hello world', 'Good morning', and 'How are you?') from English to three different languages (Spanish, French, and German).
+
+```text title="prompts.txt"
+You're a translator. Translate this into {{language}}: {{input}}
+---
+Speak in {{language}}: {{input}}
+```
+
+Instead of creating individual `tests` for each combination,
+we can create a `scenarios` that groups this data and the tests/assertions together:
+
+```yaml title="promptfooconfig.yaml"
+scenarios:
+  - config:
+      - vars:
+          language: Spanish
+          expectedHelloWorld: 'Hola mundo'
+          expectedGoodMorning: 'Buenos días'
+          expectedHowAreYou: '¿Cómo estás?'
+      - vars:
+          language: French
+          expectedHelloWorld: 'Bonjour le monde'
+          expectedGoodMorning: 'Bonjour'
+          expectedHowAreYou: 'Comment ça va?'
+      - vars:
+          language: German
+          expectedHelloWorld: 'Hallo Welt'
+          expectedGoodMorning: 'Guten Morgen'
+          expectedHowAreYou: 'Wie geht es dir?'
+    tests:
+      - description: Translated Hello World
+        vars:
+          input: 'Hello world'
+        assert:
+          - type: similar
+            value: '{{expectedHelloWorld}}'
+            threshold: 0.90
+      - description: Translated Good Morning
+        vars:
+          input: 'Good morning'
+        assert:
+          - type: similar
+            value: '{{expectedGoodMorning}}'
+            threshold: 0.90
+      - description: Translated How are you?
+        vars:
+          input: 'How are you?'
+        assert:
+          - type: similar
+            value: '{{expectedHowAreYou}}'
+            threshold: 0.90
+```
+
+This will generate a matrix of tests for each language and input phrase combination, running the same set of assertions on each.
+
+The full source behind this sample is in [`examples/config-multiple-translations`][1].
+
+## Configuration
+
+The `scenarios` configuration is an array of `Scenario` objects. Each `Scenario` has two main parts:
+
+- `config`: an array of `vars` objects. Each `vars` object represents a set of variables that will be passed to the tests.
+- `tests`: an array of `TestCase` objects. These are the tests that will be run for each set of variables in the `config`.
+
+Here is the structure of a `Scenario`:
+
+| Property    | Type                  | Required | Description                                                        |
+| ----------- | --------------------- | -------- | ------------------------------------------------------------------ |
+| description | `string`              | No       | Optional description of what you're testing                        |
+| config      | `Partial<TestCase>[]` | Yes      | An array of variable sets. Each set will be run through the tests. |
+| tests       | `TestCase[]`          | Yes      | The tests to be run on each set of variables.                      |
+
+Scenarios can also be loaded from external files. To reference an external file, use the `file://` prefix:
+
+```yaml
+scenarios:
+  - file://path/to/your/scenario.yaml
+```
+
+The external file should follow the same structure as inline scenarios.
+
+### Using Glob Patterns
+
+You can use glob patterns to load multiple scenario files at once:
+
+```yaml
+scenarios:
+  - file://scenarios/*.yaml # All YAML files in scenarios directory
+  - file://scenarios/unit-*.yaml # All files matching unit-*.yaml
+  - file://scenarios/**/*.yaml # All YAML files in subdirectories
+```
+
+When using glob patterns, all matched files are loaded and their scenarios are automatically flattened into a single array. This is useful for organizing large test suites:
+
+```
+scenarios/
+├── unit/
+│   ├── auth-scenarios.yaml
+│   └── api-scenarios.yaml
+└── integration/
+    ├── workflow-scenarios.yaml
+    └── e2e-scenarios.yaml
+```
+
+You can mix glob patterns with direct file references:
+
+```yaml
+scenarios:
+  - file://scenarios/critical.yaml # Specific file
+  - file://scenarios/unit/*.yaml # All unit test scenarios
+```
+
+This functionality allows you to easily run a wide range of tests without having to manually create each one. It also keeps your configuration file cleaner and easier to read.
+
+[1]: https://github.com/promptfoo/promptfoo/tree/main/examples/config-multiple-translations

site/docs/configuration/telemetry.md

@@ -0,0 +1,47 @@
+---
+sidebar_position: 42
+sidebar_label: Telemetry
+title: Telemetry Configuration - Usage Analytics and Monitoring
+description: Configure telemetry and analytics for promptfoo usage monitoring. Learn data collection settings, privacy controls, and usage tracking options.
+keywords:
+  [
+    telemetry configuration,
+    usage analytics,
+    monitoring,
+    data collection,
+    privacy settings,
+    usage tracking,
+    analytics setup,
+  ]
+pagination_prev: configuration/caching
+pagination_next: null
+---
+
+# Telemetry
+
+`promptfoo` collects basic usage telemetry by default. This telemetry helps us decide how to spend time on development.
+
+An event is recorded when:
+
+- A command is run (e.g. `init`, `eval`, `view`)
+- An assertion is used (along with the type of assertion, e.g. `is-json`, `similar`, `llm-rubric`)
+
+Telemetry events include package version and whether the command is running in CI. When account information is present in the local promptfoo config, hosted telemetry also includes the promptfoo user ID, email address, cloud login status, and authentication method.
+
+Telemetry does not include prompts, model outputs, test cases, provider API keys, or full configuration files.
+
+To disable telemetry, set the following environment variable:
+
+```sh
+PROMPTFOO_DISABLE_TELEMETRY=1
+```
+
+## Updates
+
+The CLI checks NPM's package registry for updates. If there is a newer version available, it will display a banner to the user.
+
+To disable, set:
+
+```sh
+PROMPTFOO_DISABLE_UPDATE=1
+```

site/docs/configuration/test-cases.md

@@ -0,0 +1,773 @@
+---
+sidebar_position: 12
+sidebar_label: Test Cases
+title: Test Case Configuration - Variables, Assertions, and Data
+description: Configure test cases for LLM evaluation with variables, assertions, CSV data, and dynamic generation. Learn inline tests, external files, and media support.
+keywords:
+  [
+    test cases,
+    LLM testing,
+    evaluation data,
+    assertions,
+    CSV tests,
+    variables,
+    dynamic testing,
+    test automation,
+  ]
+pagination_prev: configuration/prompts
+pagination_next: configuration/scenarios
+---
+
+# Test Case Configuration
+
+Define evaluation scenarios with variables, assertions, and test data.
+
+## Inline Tests
+
+The simplest way to define tests is directly in your config:
+
+```yaml title="promptfooconfig.yaml"
+tests:
+  - vars:
+      question: 'What is the capital of France?'
+    assert:
+      - type: contains
+        value: 'Paris'
+
+  - vars:
+      question: 'What is 2 + 2?'
+    assert:
+      - type: equals
+        value: '4'
+```
+
+### Test Structure
+
+Each test case can include:
+
+```yaml
+tests:
+  - description: 'Optional test description'
+    vars:
+      # Variables to substitute in prompts
+      var1: value1
+      var2: value2
+    assert:
+      # Expected outputs and validations
+      - type: contains
+        value: 'expected text'
+    metadata:
+      # Filterable metadata
+      category: math
+      difficulty: easy
+```
+
+### Repeating an Individual Test
+
+Set `options.repeat` to a positive integer to run one test case multiple times:
+
+```yaml title="promptfooconfig.yaml"
+tests:
+  - description: 'Sample a nondeterministic response'
+    vars:
+      question: 'Write a short greeting'
+    options:
+      repeat: 3
+```
+
+The per-test value overrides `--repeat`, `commandLineOptions.repeat`, or
+`evaluateOptions.repeat` for that test. Other tests continue to use the global repeat count.
+Repeat indexes use separate cache entries; add `--no-cache` when every run must call the provider.
+
+### Filtering Tests by Provider
+
+Control which providers run specific tests using the `providers` field. This allows you to run different test suites against different models in a single evaluation:
+
+```yaml
+providers:
+  - id: openai:gpt-3.5-turbo
+    label: fast-model
+  - id: openai:gpt-4
+    label: smart-model
+
+tests:
+  # Only run on fast-model
+  - vars:
+      question: 'What is 2 + 2?'
+    providers:
+      - fast-model
+    assert:
+      - type: equals
+        value: '4'
+
+  # Only run on smart-model
+  - vars:
+      question: 'Explain quantum entanglement'
+    providers:
+      - smart-model
+    assert:
+      - type: llm-rubric
+        value: 'Provides accurate physics explanation'
+```
+
+**Matching syntax:**
+
+| Pattern        | Matches                                                              |
+| -------------- | -------------------------------------------------------------------- |
+| `fast-model`   | Exact label match                                                    |
+| `openai:gpt-4` | Exact provider ID match                                              |
+| `openai:*`     | Wildcard - any provider starting with `openai:`                      |
+| `openai`       | Legacy prefix - matches `openai:gpt-4`, `openai:gpt-3.5-turbo`, etc. |
+
+**Apply to all tests using `defaultTest`:**
+
+```yaml
+defaultTest:
+  providers:
+    - openai:* # All tests default to OpenAI providers only
+
+tests:
+  - vars:
+      question: 'Simple question'
+  - vars:
+      question: 'Complex question'
+    providers:
+      - smart-model # Override default for this test
+```
+
+**Edge cases:**
+
+- **No filter**: Without the `providers` field, the test runs against all providers (cross-product behavior)
+- **Empty array**: `providers: []` means the test runs on no providers and is effectively skipped
+- **Stacking with providerPromptMap**: When both `providers` and `providerPromptMap` are set, they filter together—a provider must match both to run
+- **CLI `--filter-providers`**: If you use `--filter-providers` to filter providers at the CLI level, validation only sees the filtered providers. Tests referencing providers excluded by `--filter-providers` will fail validation
+
+### Filtering Tests by Prompt
+
+By default, each test runs against all prompts (a cartesian product). You can use the `prompts` field to restrict a test to specific prompts:
+
+```yaml
+prompts:
+  - id: prompt-factual
+    label: Factual Assistant
+    raw: 'You are a factual assistant. Answer: {{question}}'
+  - id: prompt-creative
+    label: Creative Writer
+    raw: 'You are a creative writer. Answer: {{question}}'
+
+providers:
+  - openai:gpt-4o-mini
+
+tests:
+  # This test only runs with the Factual Assistant prompt
+  - vars:
+      question: 'What is the capital of France?'
+    prompts:
+      - Factual Assistant
+    assert:
+      - type: contains
+        value: 'Paris'
+
+  # This test only runs with the Creative Writer prompt
+  - vars:
+      question: 'Write a poem about Paris'
+    prompts:
+      - prompt-creative # You can reference by ID or label
+    assert:
+      - type: llm-rubric
+        value: 'Contains poetic language'
+
+  # This test runs with all prompts (default behavior)
+  - vars:
+      question: 'Hello'
+```
+
+The `prompts` field accepts:
+
+- **Exact labels**: `prompts: ['Factual Assistant']`
+- **Exact IDs**: `prompts: ['prompt-factual']`
+- **Wildcard patterns**: `prompts: ['Math:*']` matches `Math:Basic`, `Math:Advanced`, etc.
+- **Prefix patterns**: `prompts: ['Math']` matches `Math:Basic`, `Math:Advanced` (legacy syntax)
+
+:::note
+
+Invalid prompt references will cause an error at config load time. This strict validation catches typos early.
+
+:::
+
+You can also set a default prompt filter in `defaultTest`:
+
+```yaml
+defaultTest:
+  prompts:
+    - Factual Assistant
+
+tests:
+  # Inherits prompts: ['Factual Assistant'] from defaultTest
+  - vars:
+      question: 'What is 2+2?'
+
+  # Override to use a different prompt
+  - vars:
+      question: 'Write a story'
+    prompts:
+      - Creative Writer
+```
+
+## External Test Files
+
+For larger test suites, store tests in separate files:
+
+```yaml title="promptfooconfig.yaml"
+tests: file://tests.yaml
+```
+
+Or load multiple files:
+
+```yaml
+tests:
+  - file://basic_tests.yaml
+  - file://advanced_tests.yaml
+  - file://edge_cases/*.yaml
+```
+
+## CSV Format
+
+CSV or Excel (XLSX) files are ideal for bulk test data:
+
+```yaml title="promptfooconfig.yaml"
+tests: file://test_cases.csv
+```
+
+```yaml title="promptfooconfig.yaml"
+tests: file://test_cases.xlsx
+```
+
+### Basic CSV
+
+```csv title="test_cases.csv"
+question,expectedAnswer
+"What is 2+2?","4"
+"What is the capital of France?","Paris"
+"Who wrote Romeo and Juliet?","Shakespeare"
+```
+
+Variables are automatically mapped from column headers.
+
+### Excel (XLSX/XLS) Support
+
+Excel files (.xlsx and .xls) are supported as an optional feature. To use Excel files:
+
+1. Install the `read-excel-file` package as a peer dependency:
+
+   ```bash
+   npm install read-excel-file
+   ```
+
+2. Use Excel files just like CSV files:
+   ```yaml title="promptfooconfig.yaml"
+   tests: file://test_cases.xlsx
+   ```
+
+**Multi-sheet support:** By default, only the first sheet is used. To specify a different sheet, use the `#` syntax:
+
+- `file://test_cases.xlsx#Sheet2` - Select sheet by name
+- `file://test_cases.xlsx#2` - Select sheet by 1-based index (2 = second sheet)
+
+```yaml title="promptfooconfig.yaml"
+# Use a specific sheet by name
+tests: file://test_cases.xlsx#DataSheet
+
+# Or by index (1-based)
+tests: file://test_cases.xlsx#2
+```
+
+### XLSX Example
+
+Your Excel file should have column headers in the first row, with each subsequent row representing a test case:
+
+| question                       | expectedAnswer |
+| ------------------------------ | -------------- |
+| What is 2+2?                   | 4              |
+| What is the capital of France? | Paris          |
+| Name a primary color           | blue           |
+
+**Tips for Excel files:**
+
+- First row must contain column headers
+- Column names become variable names in your prompts
+- Empty cells are treated as empty strings
+- Use `__expected` columns for assertions (same as CSV)
+
+### CSV with Assertions
+
+Use special `__expected` columns for assertions:
+
+```csv title="test_cases.csv"
+input,__expected
+"Hello world","contains: Hello"
+"Calculate 5 * 6","equals: 30"
+"What's the weather?","llm-rubric: Provides weather information"
+```
+
+Values without a type prefix default to `equals`:
+
+| `__expected` value                | Assertion type               |
+| --------------------------------- | ---------------------------- |
+| `Paris`                           | `equals`                     |
+| `contains:Paris`                  | `contains`                   |
+| `factuality:The capital is Paris` | `factuality`                 |
+| `similar(0.8):Hello there`        | `similar` with 0.8 threshold |
+
+Multiple assertions:
+
+```csv title="test_cases.csv"
+question,__expected1,__expected2,__expected3
+"What is 2+2?","equals: 4","contains: four","javascript: output.length < 10"
+```
+
+:::note
+**contains-any**, **icontains-any**, **contains-all**, and **icontains-all** expect comma-delimited values inside the `__expected` column. Surrounding whitespace around each value is trimmed.
+
+```csv title="test_cases.csv"
+translated_text,__expected
+"<span>Hola</span> <b>mundo</b>","contains-any: <b>,</span>"
+```
+
+If you write `"contains-any: <b> </span>"`, promptfoo treats `<b> </span>` as a single search term rather than two separate tags.
+
+To match a value that itself contains a comma, wrap that value in double quotes. Because the assertion is inside a quoted CSV cell, write each of those wrapping quotes twice:
+
+```csv title="test_cases.csv"
+text,__expected
+"1,000 items in stock","contains-all: ""1,000"",in stock"
+```
+
+Here `"1,000"` is a single search term (the comma is preserved), while `in stock` is a second term. These quoting rules apply to `__expected` columns in CSV, XLSX, and Google Sheets. JSON and JSONL test files should use the structured `assert` object form instead.
+
+At the assertion-string level, escape a literal double quote inside a quoted value as `\"` or `""`. In a CSV file, remember to apply CSV escaping as well by doubling every double quote in the cell.
+:::
+
+### Special CSV Columns
+
+| Column                                                      | Purpose                                                  | Example                                                           |
+| ----------------------------------------------------------- | -------------------------------------------------------- | ----------------------------------------------------------------- |
+| `__expected`                                                | Single assertion                                         | `contains: Paris`                                                 |
+| `__expected1`, `__expected2`, ...                           | Multiple assertions                                      | `equals: 42`                                                      |
+| `__description`                                             | Test description                                         | `Basic math test`                                                 |
+| `__prefix`                                                  | Prepend to prompt                                        | `You must answer: `                                               |
+| `__suffix`                                                  | Append to prompt                                         | ` (be concise)`                                                   |
+| `__metric`                                                  | Display name in reports (does not change assertion type) | `accuracy`                                                        |
+| `__threshold`                                               | Pass threshold (applies to all asserts)                  | `0.8`                                                             |
+| `__metadata:*`                                              | Filterable metadata                                      | See below                                                         |
+| `__config:__expected:<key>` or `__config:__expectedN:<key>` | Set configuration for all or specific assertions         | `__config:__expected:threshold`, `__config:__expected2:threshold` |
+
+Using `__metadata` without a key is not supported. Specify the metadata field like `__metadata:category`.
+If a CSV file includes a `__metadata` column without a key, Promptfoo logs a warning and ignores the column.
+
+### Metadata in CSV
+
+Add filterable metadata:
+
+```csv title="test_cases.csv"
+question,__expected,__metadata:category,__metadata:difficulty
+"What is 2+2?","equals: 4","math","easy"
+"Explain quantum physics","llm-rubric: Accurate explanation","science","hard"
+```
+
+Array metadata with `[]`:
+
+```csv
+topic,__metadata:tags[]
+"Machine learning","ai,technology,data science"
+"Climate change","environment,science,global\,warming"
+```
+
+Filter tests:
+
+```bash
+promptfoo eval --filter-metadata category=math
+promptfoo eval --filter-metadata difficulty=easy
+promptfoo eval --filter-metadata tags=ai
+
+# Multiple filters use AND logic (tests must match ALL conditions)
+promptfoo eval --filter-metadata category=math --filter-metadata difficulty=easy
+```
+
+### JSON in CSV
+
+Include structured data:
+
+```csv title="test_cases.csv"
+query,context,__expected
+"What's the temperature?","{""location"":""NYC"",""units"":""celsius""}","contains: celsius"
+```
+
+Access in prompts:
+
+```yaml
+prompts:
+  - 'Query: {{query}}, Location: {{(context | load).location}}'
+```
+
+### CSV with defaultTest
+
+Apply the same assertions to all tests loaded from a CSV file using [`defaultTest`](/docs/configuration/guide#default-test-cases):
+
+```yaml title="promptfooconfig.yaml"
+defaultTest:
+  assert:
+    - type: factuality
+      value: '{{reference_answer}}'
+  options:
+    provider: openai:gpt-5.2
+
+tests: file://tests.csv
+```
+
+```csv title="tests.csv"
+question,reference_answer
+"What does GPT stand for?","Generative Pre-trained Transformer"
+"What is the capital of France?","Paris is the capital of France"
+```
+
+Use regular column names (like `reference_answer`) instead of `__expected` when referencing values in `defaultTest` assertions. The `__expected` column automatically creates assertions per row.
+
+## Dynamic Test Generation
+
+Generate tests programmatically:
+
+### JavaScript/TypeScript
+
+```yaml title="promptfooconfig.yaml"
+tests: file://generate_tests.js
+```
+
+```javascript title="generate_tests.js"
+module.exports = async function () {
+  // Fetch data, compute test cases, etc.
+  const testCases = [];
+
+  for (let i = 1; i <= 10; i++) {
+    testCases.push({
+      description: `Test case ${i}`,
+      vars: {
+        number: i,
+        squared: i * i,
+      },
+      assert: [
+        {
+          type: 'contains',
+          value: String(i * i),
+        },
+      ],
+    });
+  }
+
+  return testCases;
+};
+```
+
+### Python
+
+```yaml title="promptfooconfig.yaml"
+tests: file://generate_tests.py:create_tests
+```
+
+```python title="generate_tests.py"
+import json
+
+def create_tests():
+    test_cases = []
+
+    # Load test data from database, API, etc.
+    test_data = load_test_data()
+
+    for item in test_data:
+        test_cases.append({
+            "vars": {
+                "input": item["input"],
+                "context": item["context"]
+            },
+            "assert": [{
+                "type": "contains",
+                "value": item["expected"]
+            }]
+        })
+
+    return test_cases
+```
+
+### With Configuration
+
+Pass configuration to generators:
+
+```yaml title="promptfooconfig.yaml"
+tests:
+  - path: file://generate_tests.py:create_tests
+    config:
+      dataset: 'validation'
+      category: 'math'
+      sample_size: 100
+```
+
+```python title="generate_tests.py"
+def create_tests(config):
+    dataset = config.get('dataset', 'train')
+    category = config.get('category', 'all')
+    size = config.get('sample_size', 50)
+
+    # Use configuration to generate tests
+    return generate_test_cases(dataset, category, size)
+```
+
+## JSON/JSONL Format
+
+### JSON Array
+
+```json title="tests.json"
+[
+  {
+    "vars": {
+      "topic": "artificial intelligence"
+    },
+    "assert": [
+      {
+        "type": "contains",
+        "value": "AI"
+      }
+    ]
+  },
+  {
+    "vars": {
+      "topic": "climate change"
+    },
+    "assert": [
+      {
+        "type": "llm-rubric",
+        "value": "Discusses environmental impact"
+      }
+    ]
+  }
+]
+```
+
+### JSONL (One test per line)
+
+```jsonl title="tests.jsonl"
+{"vars": {"x": 5, "y": 3}, "assert": [{"type": "equals", "value": "8"}]}
+{"vars": {"x": 10, "y": 7}, "assert": [{"type": "equals", "value": "17"}]}
+```
+
+## Loading Media Files
+
+Include images, PDFs, and other files as variables:
+
+```yaml title="promptfooconfig.yaml"
+tests:
+  - vars:
+      image: file://images/chart.png
+      document: file://docs/report.pdf
+      data: file://data/config.yaml
+```
+
+### Path Resolution
+
+`file://` paths are resolved relative to your **config file's directory**, not the current working directory. This ensures consistent behavior regardless of where you run `promptfoo` from:
+
+```yaml title="src/tests/promptfooconfig.yaml"
+tests:
+  - vars:
+      # Resolved as src/tests/data/input.json
+      data: file://./data/input.json
+
+      # Also works - resolved as src/tests/data/input.json
+      data2: file://data/input.json
+
+      # Parent directory - resolved as src/shared/context.json
+      shared: file://../shared/context.json
+```
+
+Without the `file://` prefix, values are passed as plain strings to your provider.
+
+### Supported File Types
+
+| Type                    | Handling            | Usage             |
+| ----------------------- | ------------------- | ----------------- |
+| Images (png, jpg, etc.) | Converted to base64 | Vision models     |
+| Videos (mp4, etc.)      | Converted to base64 | Multimodal models |
+| PDFs                    | Text extraction     | Document analysis |
+| Text files              | Loaded as string    | Any use case      |
+| YAML/JSON               | Parsed to object    | Structured data   |
+
+### Example: Vision Model Test
+
+```yaml
+tests:
+  - vars:
+      image: file://test_image.jpg
+      question: 'What objects are in this image?'
+    assert:
+      - type: contains
+        value: 'dog'
+```
+
+In your prompt:
+
+```json
+[
+  {
+    "role": "user",
+    "content": [
+      { "type": "text", "text": "{{question}}" },
+      {
+        "type": "image_url",
+        "image_url": {
+          "url": "data:image/jpeg;base64,{{image}}"
+        }
+      }
+    ]
+  }
+]
+```
+
+## Best Practices
+
+### 1. Organize Test Data
+
+```
+project/
+├── promptfooconfig.yaml
+├── prompts/
+│   └── main_prompt.txt
+└── tests/
+    ├── basic_functionality.csv
+    ├── edge_cases.yaml
+    └── regression_tests.json
+```
+
+### 2. Use Descriptive Names
+
+```yaml
+tests:
+  - description: 'Test French translation with formal tone'
+    vars:
+      text: 'Hello'
+      language: 'French'
+      tone: 'formal'
+```
+
+### 3. Group Related Tests
+
+```yaml
+# Use metadata for organization
+tests:
+  - vars:
+      query: 'Reset password'
+    metadata:
+      feature: authentication
+      priority: high
+```
+
+### 4. Combine Approaches
+
+```yaml
+tests:
+  # Quick smoke tests inline
+  - vars:
+      test: 'quick check'
+
+  # Comprehensive test suite from file
+  - file://tests/full_suite.csv
+
+  # Dynamic edge case generation
+  - file://tests/generate_edge_cases.js
+```
+
+## Common Patterns
+
+### A/B Testing Variables
+
+```csv title="ab_tests.csv"
+message_style,greeting,__expected
+"formal","Good morning","contains: Good morning"
+"casual","Hey there","contains: Hey"
+"friendly","Hello!","contains: Hello"
+```
+
+### Error Handling Tests
+
+```yaml
+tests:
+  - description: 'Handle empty input'
+    vars:
+      input: ''
+    assert:
+      - type: contains
+        value: 'provide more information'
+```
+
+### Performance Tests
+
+```yaml
+tests:
+  - vars:
+      prompt: 'Simple question'
+    assert:
+      - type: latency
+        threshold: 1000 # milliseconds
+```
+
+### Passing Arrays to Assertions
+
+By default, array variables expand into multiple test cases. To pass an array directly to assertions like `contains-any`, disable variable expansion:
+
+```yaml
+defaultTest:
+  options:
+    disableVarExpansion: true
+  assert:
+    - type: contains-any
+      value: '{{expected_values}}'
+
+tests:
+  - description: 'Check for any valid response'
+    vars:
+      expected_values: ['option1', 'option2', 'option3']
+```
+
+## External Data Sources
+
+### Google Sheets
+
+See [Google Sheets integration](/docs/integrations/google-sheets) for details on loading test data directly from spreadsheets.
+
+### SharePoint
+
+See [SharePoint integration](/docs/integrations/sharepoint) for details on loading test data from Microsoft SharePoint document libraries.
+
+### Azure Blob Storage
+
+Promptfoo can read test sets directly from Azure Blob Storage:
+
+```yaml
+tests: az://myaccount/evals/tests.json
+```
+
+Use `az://<account>/<container>/<blob>`. Promptfoo supports CSV, JSON, JSONL, YAML, and YML test-set blobs. Blob names may keep the original extension and append a suffix, such as `tests.json.<sha256>`.
+
+Authentication uses the first available option:
+
+1. A SAS query string on the URI, such as `az://myaccount/evals/tests.json?<sas-token>`
+2. `AZURE_STORAGE_CONNECTION_STRING`
+3. Azure identity credentials through `DefaultAzureCredential`, such as Azure CLI login, managed identity, or service principal environment variables
+
+When using `AZURE_STORAGE_CONNECTION_STRING`, the storage account comes from the connection string. Keep the `az://` account segment aligned with that account so the URI remains self-describing; Promptfoo rejects clearly mismatched `AccountName` values. Query strings are interpreted as SAS tokens and must include `sig`.
+
+SAS query strings and `DefaultAzureCredential` use the standard public Azure Blob endpoint for the named account. For Azure Government, Azure operated by 21Vianet, or custom blob endpoints, use `AZURE_STORAGE_CONNECTION_STRING` with the appropriate `EndpointSuffix` or explicit `BlobEndpoint`.
+
+Blob-hosted YAML, JSON, and JSONL files are treated as remote test-case data. Promptfoo does not expand local file references or provider references found inside those blob contents.
+
+### HuggingFace Datasets
+
+See [HuggingFace Datasets](/docs/configuration/huggingface-datasets) for instructions on importing test cases from existing datasets.

site/docs/configuration/tools.md

@@ -0,0 +1,401 @@
+---
+sidebar_position: 7
+title: Tool Calling
+description: Configure tool definitions that work across OpenAI, Anthropic, AWS Bedrock, Google, and other LLM providers
+---
+
+# Tool Calling
+
+Tool calling (also known as function calling) allows LLMs to invoke functions that you define, rather than only generating text responses.
+
+## Overview
+
+### How It Works
+
+1. **You define tools** - Tell the model what functions are available by providing their names, descriptions, and parameter schemas
+2. **Model requests a tool call** - The model outputs a function name and arguments. This name is an identifier that maps to a function in your code—the model doesn't execute anything itself
+3. **Your code executes the function** - Your application matches the function name to real code and runs it with the provided arguments
+4. **Results go back to the model** - You send the function's output back to the model, which uses it to generate its final response
+
+```
+User: "What's the weather in San Francisco?"
+    ↓
+Model outputs: { tool: "get_weather", args: { location: "San Francisco" } }
+    ↓
+Your code runs: getWeather("San Francisco") → "72°F, sunny"
+    ↓
+You send result back to model
+    ↓
+Model responds: "It's currently 72°F and sunny in San Francisco."
+```
+
+### Configuration
+
+There are two parts to configuring tool calling:
+
+1. **Tool definitions** - Describe the functions available to the model: their names, descriptions, and parameter schemas. The model uses these to decide which tool to call and what arguments to pass.
+
+2. **Tool choice** - Control _when_ the model uses tools: let it decide automatically, force it to use a specific tool, or disable tools entirely.
+
+While many providers have standardized around OpenAI's tool format, some maintain their own syntax:
+
+| Provider                 | Native Format                                          |
+| ------------------------ | ------------------------------------------------------ |
+| OpenAI/Azure/Groq/Ollama | `{ type: 'function', function: { name, parameters } }` |
+| Anthropic                | `{ name, input_schema }`                               |
+| AWS Bedrock              | `{ toolSpec: { name, inputSchema: { json } } }`        |
+| Google                   | `{ functionDeclarations: [{ name, parameters }] }`     |
+
+Promptfoo uses OpenAI's tool format as the standard. For built-in providers (OpenAI, Anthropic, Bedrock, Google, etc.), promptfoo automatically converts tool definitions to the required native format. For the [HTTP provider](/docs/providers/http), set `transformToolsFormat` to tell promptfoo what format the target API expects.
+
+### Reusing tools between providers
+
+Define your tools once in OpenAI format and reuse them across all providers using [YAML anchors and aliases](https://yaml.org/spec/1.2.2/#3222-anchors-and-aliases). An anchor (`&tools`) saves a value, and an alias (`*tools`) references it elsewhere:
+
+```yaml
+providers:
+  - id: openai:gpt-4o
+    config:
+      tools: &tools # Anchor: define tools once
+        - type: function
+          function:
+            name: get_weather
+            description: Get current weather for a location
+            parameters:
+              type: object
+              properties:
+                location: { type: string }
+              required: [location]
+
+  - id: anthropic:claude-sonnet-4-20250514
+    config:
+      tools: *tools # Alias: reuse the same tools
+
+  - id: google:gemini-2.0-flash
+    config:
+      tools: *tools # Alias: works here too
+```
+
+## Defining Tools
+
+Define tools in OpenAI format:
+
+```yaml
+providers:
+  - id: openai:gpt-4
+    config:
+      tools:
+        - type: function
+          function:
+            name: get_weather
+            description: Get the current weather for a location
+            parameters:
+              type: object
+              properties:
+                location:
+                  type: string
+                  description: City name (e.g., "San Francisco, CA")
+                unit:
+                  type: string
+                  enum: [celsius, fahrenheit]
+                  description: Temperature unit
+              required:
+                - location
+```
+
+### Fields
+
+| Field                  | Type    | Required | Description                                             |
+| ---------------------- | ------- | -------- | ------------------------------------------------------- |
+| `type`                 | string  | Yes      | Must be `'function'`                                    |
+| `function.name`        | string  | Yes      | The function name (used by the model to call it)        |
+| `function.description` | string  | No       | Description of what the function does                   |
+| `function.parameters`  | object  | No       | JSON Schema defining the function's parameters          |
+| `function.strict`      | boolean | No       | Enable strict schema validation (OpenAI/Anthropic only) |
+
+### Full JSON Schema Support
+
+The `parameters` field supports full JSON Schema draft-07, including:
+
+```yaml
+tools:
+  - type: function
+    function:
+      name: complex_function
+      parameters:
+        type: object
+        properties:
+          coordinates:
+            $ref: '#/$defs/coordinate'
+          tags:
+            type: array
+            items:
+              type: string
+            minItems: 1
+        required: [coordinates]
+        $defs:
+          coordinate:
+            type: object
+            properties:
+              lat:
+                type: number
+                minimum: -90
+                maximum: 90
+              lon:
+                type: number
+                minimum: -180
+                maximum: 180
+            required: [lat, lon]
+```
+
+### Strict Mode
+
+Enable strict schema validation for providers that support it:
+
+```yaml
+tools:
+  - type: function
+    function:
+      name: get_weather
+      strict: true # Guarantees output matches schema exactly
+      parameters:
+        type: object
+        properties:
+          location:
+            type: string
+        required: [location]
+        additionalProperties: false # Required for strict mode
+```
+
+**Strict mode provider support:**
+
+| Provider       | Support                                         |
+| -------------- | ----------------------------------------------- |
+| OpenAI         | Full support — guarantees output matches schema |
+| Anthropic      | Enables structured outputs beta feature         |
+| Bedrock/Google | Ignored (not supported)                         |
+
+## Tool Choice
+
+Tool choice controls _when_ and _how_ the model uses the tools you've defined. By default, the model decides on its own whether a tool call is appropriate (`auto`). You can override this to force tool usage, disable it, or constrain the model to a specific tool — useful for testing that the model calls the right function or for pipelines where a tool call is always expected.
+
+```yaml
+providers:
+  - id: openai:gpt-4
+    config:
+      tools:
+        - type: function
+          function:
+            name: get_weather
+            parameters: { ... }
+      tool_choice: required # Model must call a tool
+```
+
+### Modes
+
+Tool choice uses OpenAI's native format:
+
+| Value                                                 | Description                                                                                                      |
+| ----------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------- |
+| `auto`                                                | Model decides whether to call a tool based on the prompt (default)                                               |
+| `none`                                                | Model cannot call any tools, even if they are defined — useful for A/B testing tool use vs. plain text responses |
+| `required`                                            | Model must call at least one tool — useful when you always expect a structured tool response                     |
+| `{ type: function, function: { name: get_weather } }` | Model must call the specified tool — useful for testing a particular function                                    |
+
+### Examples
+
+```yaml
+# Let the model decide
+tool_choice: auto
+
+# Force the model to use tools
+tool_choice: required
+
+# Force a specific tool
+tool_choice:
+  type: function
+  function:
+    name: get_weather
+
+# Disable tools for this request
+tool_choice: none
+```
+
+## Provider Transformations
+
+### Tool Definition Mappings
+
+For built-in providers, tool definitions in OpenAI format are automatically converted to the provider's native format. For the [HTTP provider](/docs/providers/http), set `transformToolsFormat` to specify the target format. If you pass tool definitions that don't match OpenAI format, they are passed through directly without transformation.
+
+| OpenAI Field           | Anthropic      | Bedrock                     | Google                               |
+| ---------------------- | -------------- | --------------------------- | ------------------------------------ |
+| `function.name`        | `name`         | `toolSpec.name`             | `functionDeclarations[].name`        |
+| `function.description` | `description`  | `toolSpec.description`      | `functionDeclarations[].description` |
+| `function.parameters`  | `input_schema` | `toolSpec.inputSchema.json` | `functionDeclarations[].parameters`  |
+| `function.strict`      | _(ignored)_    | _(ignored)_                 | _(ignored)_                          |
+
+### Tool Choice Mappings
+
+| OpenAI (default)                           | Anthropic                | Bedrock              | Google                                                                    |
+| ------------------------------------------ | ------------------------ | -------------------- | ------------------------------------------------------------------------- |
+| `'auto'`                                   | `{ type: 'auto' }`       | `{ auto: {} }`       | `{ functionCallingConfig: { mode: 'AUTO' } }`                             |
+| `'none'`                                   | `{ type: 'auto' }`       | _(omitted)_          | `{ functionCallingConfig: { mode: 'NONE' } }`                             |
+| `'required'`                               | `{ type: 'any' }`        | `{ any: {} }`        | `{ functionCallingConfig: { mode: 'ANY' } }`                              |
+| `{ type: 'function', function: { name } }` | `{ type: 'tool', name }` | `{ tool: { name } }` | `{ functionCallingConfig: { mode: 'ANY', allowedFunctionNames: [...] } }` |
+
+## Other Provider Formats
+
+You can also use provider-native formats directly. They pass through unchanged without transformation:
+
+```yaml
+# Anthropic native format - passes through as-is
+providers:
+  - id: anthropic:claude-sonnet-4-20250514
+    config:
+      tools:
+        - name: get_weather
+          description: Get weather
+          input_schema:
+            type: object
+            properties:
+              location: { type: string }
+```
+
+Promptfoo auto-detects the format. If tools are in OpenAI format (`type: 'function'` with `function.name`), they can be transformed. Otherwise, they pass through unchanged.
+
+## Loading Tools from Files
+
+Tools can be loaded from external files:
+
+```yaml
+providers:
+  - id: openai:gpt-4
+    config:
+      tools: file://tools/my-tools.json
+```
+
+**tools/my-tools.json:**
+
+```json
+[
+  {
+    "type": "function",
+    "function": {
+      "name": "get_weather",
+      "description": "Get current weather",
+      "parameters": {
+        "type": "object",
+        "properties": {
+          "location": { "type": "string" }
+        }
+      }
+    }
+  }
+]
+```
+
+## HTTP Provider with Tools
+
+For custom HTTP endpoints, use the `transformToolsFormat` option to automatically convert OpenAI-format tools to the format your endpoint expects.
+
+### OpenAI-Compatible Endpoints
+
+```yaml
+providers:
+  - id: http://localhost:8080/v1/chat/completions
+    config:
+      method: POST
+      headers:
+        Content-Type: application/json
+      transformToolsFormat: openai # Tools already in OpenAI format, pass through
+      body:
+        model: gpt-4
+        messages: '{{ prompt }}'
+        tools: '{{ tools }}'
+        tool_choice: '{{ tool_choice }}'
+      tools:
+        - type: function
+          function:
+            name: get_weather
+            description: Get weather for a location
+            parameters:
+              type: object
+              properties:
+                location: { type: string }
+      tool_choice: required
+```
+
+### Anthropic-Compatible Endpoints
+
+```yaml
+providers:
+  - id: http://localhost:8080/v1/messages
+    config:
+      method: POST
+      headers:
+        Content-Type: application/json
+        x-api-key: '{{ env.ANTHROPIC_API_KEY }}'
+        anthropic-version: '2023-06-01'
+      transformToolsFormat: anthropic # Transforms OpenAI → Anthropic format
+      body:
+        model: claude-sonnet-4-20250514
+        max_tokens: 1024
+        messages: '{{ prompt }}'
+        tools: '{{ tools }}'
+        tool_choice: '{{ tool_choice }}'
+      tools:
+        - type: function
+          function:
+            name: get_weather
+            description: Get weather for a location
+            parameters:
+              type: object
+              properties:
+                location:
+                  type: string
+                  description: City name
+              required:
+                - location
+      tool_choice: required
+```
+
+The `transformToolsFormat` option accepts: `openai`, `anthropic`, `bedrock`, or `google`. The `{{ tools }}` and `{{ tool_choice }}` template variables are automatically serialized as JSON when injected into the request body.
+
+### Native Format Pass-Through
+
+If your endpoint requires a specific format, you can define tools in that format directly and omit `transformToolsFormat`. Tools pass through unchanged:
+
+```yaml
+providers:
+  - id: http://localhost:8080/v1/messages
+    config:
+      method: POST
+      headers:
+        Content-Type: application/json
+      # No transformToolsFormat - tools pass through as-is
+      body:
+        model: claude-sonnet-4-20250514
+        messages: '{{ prompt }}'
+        tools: '{{ tools }}'
+      tools:
+        # Native Anthropic format with input_schema
+        - name: get_weather
+          description: Get weather for a location
+          input_schema:
+            type: object
+            properties:
+              location:
+                type: string
+            required:
+              - location
+```
+
+This is useful when your endpoint expects a custom or non-standard tool format.
+
+## See Also
+
+- [OpenAI Provider](/docs/providers/openai) - OpenAI-specific tool features
+- [Anthropic Provider](/docs/providers/anthropic) - Anthropic tool calling
+- [AWS Bedrock Provider](/docs/providers/aws-bedrock) - Bedrock Converse API tools
+- [Google Provider](/docs/providers/google) - Gemini function calling
+- [Custom HTTP Provider](/docs/providers/custom-api) - Tools with custom endpoints