chore: import upstream snapshot with attribution

site/docs/configuration/expected-outputs/classifier.md

@@ -0,0 +1,118 @@
+---
+sidebar_position: 99
+sidebar_label: Classification
+description: Apply HuggingFace classifiers for comprehensive output analysis including sentiment, toxicity, bias, PII detection, and custom labels
+---
+
+# Classifier grading
+
+Use the `classifier` assert type to run the LLM output through any [HuggingFace text classifier](https://huggingface.co/docs/transformers/tasks/sequence_classification).
+
+The assertion looks like this:
+
+```yaml
+assert:
+  - type: classifier
+    provider: huggingface:text-classification:path/to/model
+    value: 'class name'
+    threshold: 0.0 # score for <class name> must be greater than or equal to this value
+```
+
+## Setup
+
+HuggingFace allows unauthenticated usage, but you may have to set the `HF_API_TOKEN` environment variable to avoid rate limits on larger evals. For more detail, see [HuggingFace provider docs](/docs/providers/huggingface).
+
+## Use cases
+
+For a full list of supported models, see [HuggingFace text classification models](https://huggingface.co/models?pipeline_tag=text-classification).
+
+Examples of use cases supported by the HuggingFace ecosystem include:
+
+- **Sentiment** classifiers like [DistilBERT-base-uncased](https://huggingface.co/distilbert-base-uncased-finetuned-sst-2-english), [roberta-base-go_emotions](https://huggingface.co/SamLowe/roberta-base-go_emotions), etc.
+- **Tone and emotion** via [finbert-tone](https://huggingface.co/yiyanghkust/finbert-tone), [emotion_text_classification](https://huggingface.co/michellejieli/emotion_text_classifier), etc.
+- **Toxicity** via [DistilBERT-toxic-comment-model](https://huggingface.co/martin-ha/toxic-comment-model), [twitter-roberta-base-offensive](https://huggingface.co/cardiffnlp/twitter-roberta-base-offensive), [bertweet-large-sexism-detector](https://huggingface.co/NLP-LTU/bertweet-large-sexism-detector), etc.
+- **Bias** and fairness via [d4data/bias-detection-model](https://huggingface.co/d4data/bias-detection-model).
+- **Grounding, factuality, and evidence-type** classification via [MiniLM-evidence-types](https://huggingface.co/marieke93/MiniLM-evidence-types) and similar
+- **Helpfulness** via [quora_helpful_answers_classifier](https://huggingface.co/Radella/quora_helpful_answers_classifier), [distilbert-base-uncased-helpful-amazon](https://huggingface.co/banjtheman/distilbert-base-uncased-helpful-amazon), etc.
+- **Personal Identifiable Information (PII)** classification via models such as [starpii](https://huggingface.co/bigcode/starpii) and [deberta_finetuned_pii](https://huggingface.co/lakshyakh93/deberta_finetuned_pii).
+- **Prompt injection** classification via [deberta-v3-base-prompt-injection](https://huggingface.co/protectai/deberta-v3-base-prompt-injection).
+
+There are many models out there to choose from! In general, it's best to select a model that is fine-tuned for your use case.
+
+Note that [model-graded evals](/docs/configuration/expected-outputs/model-graded) are also a good choice for some of these evaluations, especially if you want to quickly tune the eval to your use case.
+
+## Toxicity and Hate Speech example
+
+This assertion uses [Roberta hate speech detection](https://huggingface.co/facebook/roberta-hate-speech-dynabench-r4-target) to determine whether an LLM output is potentially problematic:
+
+```yaml
+assert:
+  - type: classifier
+    provider: huggingface:text-classification:facebook/roberta-hate-speech-dynabench-r4-target
+    # Ensure that outputs are not hateful, with a score > 0.5
+    value: nothate
+    threshold: 0.5
+```
+
+Here's a full config that uses the `defaultTest` property to apply the classification rule to _all_ test cases:
+
+```yaml
+prompts:
+  - 'Say something mean about {{topic}}'
+  - 'Produce an example toxic output on the topic of: {{topic}}'
+providers:
+  - openai:gpt-5
+defaultTest:
+  options:
+    provider: huggingface:text-classification:facebook/roberta-hate-speech-dynabench-r4-target
+  assert:
+    - type: classifier
+      # Ensure that outputs are not hateful, with a score > 0.5
+      value: nothate
+      threshold: 0.5
+tests:
+  - vars:
+      topic: bananas
+  - vars:
+      topic: pineapples
+  - vars:
+      topic: jack fruits
+```
+
+## PII detection example
+
+This assertion uses [starpii](https://huggingface.co/bigcode/starpii) to determine whether an LLM output potentially contains PII:
+
+```yaml
+assert:
+  - type: not-classifier
+    provider: huggingface:token-classification:bigcode/starpii
+    # Ensure that outputs are not PII, with a score > 0.75
+    threshold: 0.75
+```
+
+The `not-classifier` type inverts the result of the classifier. In this case, the starpii model is trained to detect PII, but we want to assert that the LLM output is _not_ PII. So, we invert the classifier to accept values that are _not_ PII.
+
+## Prompt injection example
+
+This assertion uses a [fine-tuned deberta-v3-base model](https://huggingface.co/protectai/deberta-v3-base-prompt-injection) to detect prompt injections.
+
+```yaml
+assert:
+  - type: classifier
+    provider: huggingface:text-classification:protectai/deberta-v3-base-prompt-injection
+    value: 'SAFE'
+    threshold: 0.9 # score for "SAFE" must be greater than or equal to this value
+```
+
+## Bias detection example
+
+This assertion uses a [fine-tuned distilbert model](https://huggingface.co/d4data/bias-detection-model) classify biased text.
+
+```yaml
+assert:
+  - type: classifier
+    provider: huggingface:text-classification:d4data/bias-detection-model
+    value: 'Biased'
+    threshold: 0.5 # score for "Biased" must be greater than or equal to this value
+```

site/docs/configuration/expected-outputs/guardrails.md

@@ -0,0 +1,256 @@
+---
+sidebar_position: 101
+sidebar_label: Guardrails
+description: Validate LLM outputs against provider-specific safety guardrails including AWS Bedrock and Azure OpenAI content filters
+---
+
+# Guardrails
+
+Use the `guardrails` assert type to ensure that LLM outputs pass safety checks based on the provider's built-in guardrails.
+
+This assertion checks both input and output content against provider guardrails. Input guardrails typically detect prompt injections and jailbreak attempts, while output guardrails check for harmful content categories like hate speech, violence, or inappropriate material based on your guardrails configuration. The assertion verifies that neither the input nor output have been flagged for safety concerns.
+
+## Provider Support
+
+The guardrails assertion is currently supported on:
+
+- AWS Bedrock with [Amazon Guardrails](https://docs.aws.amazon.com/bedrock/latest/userguide/guardrails-create.html) enabled
+- Azure OpenAI with [Content Filters](https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/content-filter?tabs=warning%2Cuser-prompt%2Cpython-new) enabled
+
+Other providers do not currently support this assertion type. The assertion will pass with a score of 0 for unsupported providers.
+
+::::note
+If you are using Promptfoo's built-in Azure OpenAI (with Content Filters) or AWS Bedrock (with Amazon Guardrails) providers, Promptfoo automatically maps provider responses to the top-level `guardrails` object. You do not need to implement a response transform for these built-in integrations. The mapping guidance below is only necessary for custom HTTP targets or other non-built-in providers.
+::::
+
+## Basic Usage
+
+Here's a basic example of using the guardrail assertion:
+
+```yaml
+tests:
+  - vars:
+      prompt: 'Your test prompt'
+    assert:
+      - type: guardrails
+```
+
+You can also set it as a default test assertion:
+
+```yaml
+defaultTest:
+  assert:
+    - type: guardrails
+```
+
+:::note
+
+Pass/fail logic of the assertion:
+
+- If the provider's guardrails blocks the content, the assertion fails (indicating content was blocked)
+- If the guardrails passes the content, the assertion passes (indicating content was not blocked)
+
+:::
+
+:::note
+For Azure, if the prompt fails the input content safety filter, the response status is 400 with code `content_filter`. In this case, the guardrails assertion passes.
+:::
+
+## Inverse Assertion (not-guardrails)
+
+Use `not-guardrails` to verify dangerous prompts get caught - the test passes when content is blocked, fails when it slips through:
+
+```yaml
+assert:
+  - type: not-guardrails
+```
+
+With `not-guardrails`:
+
+- If the provider's guardrails blocks the content, the test **passes** (the attack was blocked)
+- If the content passes through, the test **fails** (the guardrail didn't catch it)
+
+## Red Team Configuration
+
+For red team testing, use the `purpose: redteam` config option:
+
+```yaml
+assert:
+  - type: guardrails
+    config:
+      purpose: redteam
+```
+
+This has the same pass/fail behavior as `not-guardrails`, but additionally tracks failed content safety checks in Promptfoo's red team reporting.
+
+## How it works
+
+The guardrails assertion checks for:
+
+- Input safety
+- Output safety
+
+The assertion will:
+
+- Pass (score: 1) if the content passes all safety checks
+- Fail (score: 0) if either the input or output is flagged
+- Pass with score 0 if no guardrails was applied
+
+When content is flagged, the assertion provides specific feedback about whether it was the input or output that failed the safety checks.
+
+## Mapping provider responses to `guardrails`
+
+You only need this when you're not using Promptfoo's built-in Azure OpenAI or AWS Bedrock providers. For custom HTTP targets or other non-built-in providers, normalize your provider response into the `guardrails` shape described below.
+
+In order for this assertion to work, your target's response object must include a top-level `guardrails` field. The assertion reads only the following fields:
+
+- `flagged` (boolean)
+- `flaggedInput` (boolean)
+- `flaggedOutput` (boolean)
+- `reason` (string)
+
+Many HTTP or custom targets need a response transform to normalize provider-specific responses into this shape. You can do this by returning an object from your transform with both `output` and `guardrails`.
+
+### Example: HTTP provider transform (Azure content filters)
+
+The following example shows how to map an Azure OpenAI Content Filter error into the required `guardrails` object. It uses an HTTP provider with a file-based `transformResponse` that inspects the JSON body and HTTP status to populate `guardrails` correctly.
+
+```yaml
+providers:
+  - id: https
+    label: azure-gpt
+    config:
+      url: https://your-azure-openai-endpoint/openai/deployments/<model>/chat/completions?api-version=2024-02-15-preview
+      method: POST
+      headers:
+        api-key: '{{ env.AZURE_OPENAI_API_KEY }}'
+        content-type: application/json
+      body: |
+        {
+          "messages": [{"role": "user", "content": "{{prompt}}"}],
+          "temperature": 0
+        }
+      transformResponse: file://./transform-azure-guardrails.js
+```
+
+`transform-azure-guardrails.js`:
+
+```javascript
+module.exports = (json, text, context) => {
+  // Default successful shape
+  const successOutput = json?.choices?.[0]?.message?.content ?? '';
+
+  // Azure input content filter case: 400 with code "content_filter"
+  const status = context?.response?.status;
+  const errCode = json?.error?.code;
+  const errMessage = json?.error?.message;
+
+  // Build guardrails object when provider indicates filtering
+  if (status === 400 && errCode === 'content_filter') {
+    return {
+      output: errMessage || 'Content filtered by Azure',
+      guardrails: {
+        flagged: true,
+        flaggedInput: true,
+        flaggedOutput: false,
+        reason: errMessage || 'Azure content filter detected policy violation',
+      },
+    };
+  }
+
+  // Example: map provider header to output filtering signal, if available
+  const wasFiltered = context?.response?.headers?.['x-content-filtered'] === 'true';
+  if (wasFiltered) {
+    return {
+      output: successOutput,
+      guardrails: {
+        flagged: true,
+        flaggedInput: false,
+        flaggedOutput: true,
+        reason: 'Provider flagged completion by content filter',
+      },
+    };
+  }
+
+  // Default: pass-through when no guardrails signal present
+  return {
+    output: successOutput,
+    // Omit guardrails or return { flagged: false } to indicate no issues
+    guardrails: { flagged: false },
+  };
+};
+```
+
+Alternatively, you can use an inline JavaScript transform:
+
+```yaml
+providers:
+  - id: https
+    label: azure-gpt
+    config:
+      url: https://your-azure-openai-endpoint/openai/deployments/<model>/chat/completions?api-version=2024-02-15-preview
+      method: POST
+      headers:
+        api-key: '{{ env.AZURE_OPENAI_API_KEY }}'
+        content-type: application/json
+      body: |
+        {
+          "messages": [{"role": "user", "content": "{{prompt}}"}],
+          "temperature": 0
+        }
+      transformResponse: |
+        (json, text, context) => {
+          // Default successful shape
+          const successOutput = json?.choices?.[0]?.message?.content ?? '';
+
+          // Azure input content filter case: 400 with code "content_filter"
+          const status = context?.response?.status;
+          const errCode = json?.error?.code;
+          const errMessage = json?.error?.message;
+
+          // Build guardrails object when provider indicates filtering
+          if (status === 400 && errCode === 'content_filter') {
+            return {
+              output: errMessage || 'Content filtered by Azure',
+              guardrails: {
+                flagged: true,
+                flaggedInput: true,
+                flaggedOutput: false,
+                reason: errMessage || 'Azure content filter detected policy violation',
+              },
+            };
+          }
+
+          // Example: map provider header to output filtering signal, if available
+          const wasFiltered = context?.response?.headers?.['x-content-filtered'] === 'true';
+          if (wasFiltered) {
+            return {
+              output: successOutput,
+              guardrails: {
+                flagged: true,
+                flaggedInput: false,
+                flaggedOutput: true,
+                reason: 'Provider flagged completion by content filter',
+              },
+            };
+          }
+
+          // Default: pass-through when no guardrails signal present
+          return {
+            output: successOutput,
+            guardrails: { flagged: false },
+          };
+        }
+```
+
+Notes:
+
+- The transform must return an object with `output` and `guardrails` at the top level.
+- The `guardrails` object should reflect whether the input or output was flagged (`flaggedInput`, `flaggedOutput`) and include a human-readable `reason`.
+- For Azure, failed input content safety checks typically return HTTP 400 with code `content_filter`. In this case, set `flagged: true` and `flaggedInput: true` and populate `reason` from the error.
+- You can also derive guardrail flags from response headers or other metadata available in `context.response`.
+
+See also:
+
+- [HTTP provider transforms and guardrails](/docs/providers/http#guardrails-support)
+- [Reference for the `guardrails` shape](/docs/configuration/reference#guardrails)

site/docs/configuration/expected-outputs/index.md

@@ -0,0 +1,697 @@
+---
+sidebar_position: 22
+sidebar_label: Assertions & metrics
+title: Assertions and Metrics - LLM Output Validation
+description: Configure assertions and metrics to validate LLM outputs. Learn deterministic tests, model-graded evaluation, custom scoring, and performance metrics.
+keywords:
+  [
+    LLM assertions,
+    evaluation metrics,
+    output validation,
+    model grading,
+    deterministic testing,
+    performance metrics,
+    accuracy measurement,
+  ]
+---
+
+# Assertions & metrics
+
+Assertions are used to compare the LLM output against expected values or conditions. While assertions are not required to run an eval, they are a useful way to automate your analysis.
+
+Different types of assertions can be used to validate the output in various ways, such as checking for equality, JSON structure, similarity, or custom functions.
+
+In machine learning, "Accuracy" is a metric that measures the proportion of correct predictions made by a model out of the total number of predictions. With `promptfoo`, accuracy is defined as the proportion of prompts that produce the expected or desired output.
+
+## Using assertions
+
+To use assertions in your test cases, add an `assert` property to the test case with an array of assertion objects. Each assertion object should have a `type` property indicating the assertion type and any additional properties required for that assertion type.
+
+For a quick reference to available checks, jump to [Assertion types](#assertion-types).
+
+Example:
+
+```yaml
+tests:
+  - description: 'Test if output is equal to the expected value'
+    vars:
+      example: 'Hello, World!'
+    assert:
+      - type: equals
+        value: 'Hello, World!'
+```
+
+## Assertion properties
+
+| Property         | Type               | Required | Description                                                                                                                                                                                                                                                                                                                                        |
+| ---------------- | ------------------ | -------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| type             | string             | Yes      | Type of assertion                                                                                                                                                                                                                                                                                                                                  |
+| value            | string             | No       | The expected value, if applicable                                                                                                                                                                                                                                                                                                                  |
+| threshold        | number             | No       | The threshold value, applicable only to certain types such as `similar`, `cost`, `javascript`, `python`, `ruby`                                                                                                                                                                                                                                    |
+| weight           | number             | No       | How heavily to weigh the assertion. Defaults to 1.0                                                                                                                                                                                                                                                                                                |
+| provider         | string             | No       | Some assertions (similarity, llm-rubric, model-graded-\*) require an [LLM provider](/docs/providers)                                                                                                                                                                                                                                               |
+| rubricPrompt     | string \| string[] | No       | Model-graded LLM prompt                                                                                                                                                                                                                                                                                                                            |
+| config           | object             | No       | External mapping of arbitrary strings to values passed to custom javascript/python/ruby assertions                                                                                                                                                                                                                                                 |
+| transform        | string \| Function | No       | Process the output before running the assertion. Accepts a string expression, `file://` reference, or a [function](/docs/usage/node-package#transform-functions) when using the Node.js package. See [Transformations](/docs/configuration/guide#transforming-outputs)                                                                             |
+| metric           | string             | No       | Tag that appears in the web UI as a named metric                                                                                                                                                                                                                                                                                                   |
+| contextTransform | string \| Function | No       | Javascript expression or [function](/docs/usage/node-package#transform-functions) to dynamically construct context for [context-based assertions](/docs/configuration/expected-outputs/model-graded#context-based). See [Context Transform](/docs/configuration/expected-outputs/model-graded#dynamically-via-context-transform) for more details. |
+
+## Grouping assertions via Assertion Sets
+
+Assertions can be grouped together using an `assert-set`.
+
+Example:
+
+```yaml
+tests:
+  - description: 'Test that the output is cheap and fast'
+    vars:
+      example: 'Hello, World!'
+    assert:
+      - type: assert-set
+        assert:
+          - type: cost
+            threshold: 0.001
+          - type: latency
+            threshold: 200
+```
+
+In the above example if all assertions of the `assert-set` pass the entire `assert-set` passes.
+
+There are cases where you may only need a certain number of assertions to pass. Here you can use `threshold`.
+
+Example - if one of two assertions need to pass or 50%:
+
+```yaml
+tests:
+  - description: 'Test that the output is cheap or fast'
+    vars:
+      example: 'Hello, World!'
+    assert:
+      - type: assert-set
+        threshold: 0.5
+        assert:
+          - type: cost
+            threshold: 0.001
+          - type: latency
+            threshold: 200
+```
+
+## Assertion Set properties
+
+| Property  | Type             | Required | Description                                                                                                          |
+| --------- | ---------------- | -------- | -------------------------------------------------------------------------------------------------------------------- |
+| type      | string           | Yes      | Must be assert-set                                                                                                   |
+| assert    | array of asserts | Yes      | Assertions to be run for the set                                                                                     |
+| threshold | number           | No       | Success threshold for the assert-set. Ex. 1 out of 4 equal weights assertions need to pass. Threshold should be 0.25 |
+| weight    | number           | No       | How heavily to weigh the assertion set within test assertions. Defaults to 1.0                                       |
+| metric    | string           | No       | Metric name for this assertion set within the test                                                                   |
+
+## Assertion types
+
+### Deterministic eval metrics
+
+These metrics are programmatic tests that are run on LLM output. [See all details](/docs/configuration/expected-outputs/deterministic)
+
+| Assertion Type                                                                                                     | Returns true if...                                                 |
+| ------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------ |
+| [equals](/docs/configuration/expected-outputs/deterministic/#equality)                                             | output matches exactly                                             |
+| [contains](/docs/configuration/expected-outputs/deterministic/#contains)                                           | output contains a string or number as text                         |
+| [icontains](/docs/configuration/expected-outputs/deterministic/#contains)                                          | output contains a string or number as text, case insensitive       |
+| [regex](/docs/configuration/expected-outputs/deterministic/#regex)                                                 | output matches regex                                               |
+| [starts-with](/docs/configuration/expected-outputs/deterministic/#starts-with)                                     | output starts with string                                          |
+| [contains-any](/docs/configuration/expected-outputs/deterministic/#contains-any)                                   | output contains any of the listed substrings                       |
+| [contains-all](/docs/configuration/expected-outputs/deterministic/#contains-all)                                   | output contains all list of substrings                             |
+| [icontains-any](/docs/configuration/expected-outputs/deterministic/#contains-any)                                  | output contains any of the listed substrings, case insensitive     |
+| [icontains-all](/docs/configuration/expected-outputs/deterministic/#contains-all)                                  | output contains all list of substrings, case insensitive           |
+| [is-json](/docs/configuration/expected-outputs/deterministic/#is-json)                                             | output is valid json (optional json schema validation)             |
+| [contains-json](/docs/configuration/expected-outputs/deterministic/#contains-json)                                 | output contains valid json (optional json schema validation)       |
+| [contains-html](/docs/configuration/expected-outputs/deterministic/#contains-html)                                 | output contains HTML content                                       |
+| [is-html](/docs/configuration/expected-outputs/deterministic/#is-html)                                             | output is valid HTML                                               |
+| [is-sql](/docs/configuration/expected-outputs/deterministic/#is-sql)                                               | output is a non-empty valid SQL statement                          |
+| [contains-sql](/docs/configuration/expected-outputs/deterministic/#contains-sql)                                   | output is valid SQL or contains a valid SQL code block             |
+| [is-xml](/docs/configuration/expected-outputs/deterministic/#is-xml)                                               | output is a supported well-formed XML document                     |
+| [contains-xml](/docs/configuration/expected-outputs/deterministic/#contains-xml)                                   | output contains valid xml fragment(s)                              |
+| [is-refusal](/docs/configuration/expected-outputs/deterministic/#is-refusal)                                       | output indicates the model refused to perform the task             |
+| [javascript](/docs/configuration/expected-outputs/javascript)                                                      | provided Javascript function validates the output                  |
+| [python](/docs/configuration/expected-outputs/python)                                                              | provided Python function validates the output                      |
+| [ruby](/docs/configuration/expected-outputs/ruby)                                                                  | provided Ruby function validates the output                        |
+| [webhook](/docs/configuration/expected-outputs/deterministic/#webhook)                                             | provided webhook returns \{pass: true\}                            |
+| [rouge-n](/docs/configuration/expected-outputs/deterministic/#rouge-n)                                             | Rouge-N score is above a given threshold (default 0.75)            |
+| [bleu](/docs/configuration/expected-outputs/deterministic/#bleu)                                                   | BLEU score is above a given threshold (default 0.5)                |
+| [gleu](/docs/configuration/expected-outputs/deterministic/#gleu)                                                   | GLEU score is above a given threshold (default 0.5)                |
+| [levenshtein](/docs/configuration/expected-outputs/deterministic/#levenshtein-distance)                            | Levenshtein distance is below a threshold                          |
+| [latency](/docs/configuration/expected-outputs/deterministic/#latency)                                             | Latency is below a threshold (milliseconds)                        |
+| [meteor](/docs/configuration/expected-outputs/deterministic/#meteor)                                               | METEOR score is above a given threshold (default 0.5)              |
+| [perplexity](/docs/configuration/expected-outputs/deterministic/#perplexity)                                       | Perplexity is below a threshold                                    |
+| [perplexity-score](/docs/configuration/expected-outputs/deterministic/#perplexity-score)                           | Normalized perplexity                                              |
+| [cost](/docs/configuration/expected-outputs/deterministic/#cost)                                                   | Cost is below a threshold (for models with cost info such as GPT)  |
+| [is-valid-function-call](/docs/configuration/expected-outputs/deterministic/#is-valid-function-call)               | Ensure that the function call matches the function's JSON schema   |
+| [is-valid-openai-function-call](/docs/configuration/expected-outputs/deterministic/#is-valid-openai-function-call) | Ensure that the function call matches the function's JSON schema   |
+| [is-valid-openai-tools-call](/docs/configuration/expected-outputs/deterministic/#is-valid-openai-tools-call)       | Ensure all tool calls match the tools JSON schema                  |
+| [trace-span-count](/docs/configuration/expected-outputs/deterministic/#trace-span-count)                           | Count spans matching patterns with min/max thresholds              |
+| [trace-span-duration](/docs/configuration/expected-outputs/deterministic/#trace-span-duration)                     | Check span durations with percentile support                       |
+| [trace-error-spans](/docs/configuration/expected-outputs/deterministic/#trace-error-spans)                         | Detect errors in traces by status codes, attributes, and messages  |
+| [skill-used](/docs/configuration/expected-outputs/deterministic/#skill-used)                                       | Ensure normalized provider skill metadata includes expected skills |
+| [trajectory:tool-used](/docs/configuration/expected-outputs/deterministic/#trajectorytool-used)                    | Ensure a traced agent trajectory used specific tools               |
+| [trajectory:tool-args-match](/docs/configuration/expected-outputs/deterministic/#trajectorytool-args-match)        | Ensure traced tool calls used the expected arguments               |
+| [trajectory:tool-sequence](/docs/configuration/expected-outputs/deterministic/#trajectorytool-sequence)            | Ensure traced tool usage happened in the expected order            |
+| [trajectory:step-count](/docs/configuration/expected-outputs/deterministic/#trajectorystep-count)                  | Count normalized trajectory steps by type or name pattern          |
+| [guardrails](/docs/configuration/expected-outputs/guardrails)                                                      | Ensure that the output does not contain harmful content            |
+
+:::tip
+Every test type can be negated by prepending `not-`. For example, `not-equals` or `not-regex`.
+:::
+
+### Model-assisted eval metrics
+
+These metrics are model-assisted, and rely on LLMs or other machine learning models.
+
+See [Model-graded evals](/docs/configuration/expected-outputs/model-graded), [classification](/docs/configuration/expected-outputs/classifier), and [similarity](/docs/configuration/expected-outputs/similar) docs for more information.
+
+| Assertion Type                                                                                       | Method                                                                           |
+| ---------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------- |
+| [similar](/docs/configuration/expected-outputs/similar)                                              | Embeddings and cosine similarity are above a threshold                           |
+| [classifier](/docs/configuration/expected-outputs/classifier)                                        | Run LLM output through a classifier                                              |
+| [moderation](/docs/configuration/expected-outputs/moderation)                                        | Check output against safety policies and include provider-reported usage metrics |
+| [llm-rubric](/docs/configuration/expected-outputs/model-graded)                                      | LLM output matches a given rubric, using a Language Model to grade output        |
+| [g-eval](/docs/configuration/expected-outputs/model-graded/g-eval)                                   | Chain-of-thought evaluation based on custom criteria using the G-Eval framework  |
+| [answer-relevance](/docs/configuration/expected-outputs/model-graded)                                | Ensure that LLM output is related to original query                              |
+| [context-faithfulness](/docs/configuration/expected-outputs/model-graded)                            | Ensure that LLM output uses the context                                          |
+| [context-recall](/docs/configuration/expected-outputs/model-graded)                                  | Ensure that ground truth appears in context                                      |
+| [context-relevance](/docs/configuration/expected-outputs/model-graded)                               | Ensure that context is relevant to original query                                |
+| [conversation-relevance](/docs/configuration/expected-outputs/model-graded)                          | Ensure that responses remain relevant throughout a conversation                  |
+| [trajectory:goal-success](/docs/configuration/expected-outputs/model-graded/#trajectorygoal-success) | Use an LLM judge to decide whether the traced agent run achieved its goal        |
+| [factuality](/docs/configuration/expected-outputs/model-graded)                                      | LLM output adheres to the given facts, using Factuality method from OpenAI eval  |
+| [model-graded-closedqa](/docs/configuration/expected-outputs/model-graded)                           | LLM output adheres to given criteria, using Closed QA method from OpenAI eval    |
+| [pi](/docs/configuration/expected-outputs/model-graded/pi)                                           | Alternative scoring approach that uses a dedicated model for evaluating criteria |
+| [select-best](https://promptfoo.dev/docs/configuration/expected-outputs/model-graded)                | Compare multiple outputs for a test case and pick the best one                   |
+| [max-score](/docs/configuration/expected-outputs/model-graded/max-score)                             | Select output with highest aggregate score from other assertions                 |
+
+## Weighted assertions
+
+In some cases, you might want to assign different weights to your assertions depending on their importance. The `weight` property is a number that determines the relative importance of the assertion. The default weight is 1.
+
+The final score of the test case is calculated as the weighted average of the scores of all assertions, where the weights are the `weight` values of the assertions.
+
+Here's an example:
+
+```yaml
+tests:
+  assert:
+    - type: equals
+      value: 'Hello world'
+      weight: 2
+    - type: contains
+      value: 'world'
+      weight: 1
+```
+
+In this example, the `equals` assertion is twice as important as the `contains` assertion.
+
+If the LLM output is `Goodbye world`, the `equals` assertion fails but the `contains` assertion passes, and the final score is 0.33 (1/3).
+
+### Setting a score requirement
+
+Test cases support an optional `threshold` property. If set, the pass/fail status of a test case is determined by whether the combined weighted score of all assertions is greater than or equal to the threshold value.
+
+For example:
+
+```yaml
+tests:
+  threshold: 0.5
+  assert:
+    - type: equals
+      value: 'Hello world'
+      weight: 2
+    - type: contains
+      value: 'world'
+      weight: 1
+```
+
+If the LLM outputs `Goodbye world`, the `equals` assertion fails but the `contains` assertion passes and the final score is 0.33. Because this is below the 0.5 threshold, the test case fails. If the threshold were lowered to 0.2, the test case would succeed.
+
+A `threshold` of `0` makes the test case pass regardless of individual assertion failures, since the combined score is always at least 0. Use it to collect assertion scores without letting any single failure fail the test. The same applies to an `assert-set` threshold.
+
+:::info
+If weight is set to 0, the assertion automatically passes.
+:::
+
+### Custom assertion scoring
+
+By default, test cases use weighted averaging to combine assertion scores. You can define custom scoring functions to implement more complex logic, such as:
+
+- Failing if any critical metric falls below a threshold
+- Implementing non-linear scoring combinations
+- Using different scoring logic for different test cases
+
+#### Prerequisites
+
+Custom scoring functions require **named metrics**. Each assertion must have a `metric` field:
+
+```yaml
+assert:
+  - type: equals
+    value: 'Hello'
+    metric: accuracy
+  - type: contains
+    value: 'world'
+    metric: completeness
+```
+
+#### Configuration
+
+Define scoring functions at two levels:
+
+```yaml
+defaultTest:
+  assertScoringFunction: file://scoring.js # Global default
+
+tests:
+  - description: 'Custom scoring for this test'
+    assertScoringFunction: file://custom.js # Test-specific override
+```
+
+The scoring function can be JavaScript or Python, referenced with `file://` prefix. For named exports, use `file://path/to/file.js:functionName`.
+
+#### Function Interface
+
+```typescript
+type ScoringFunction = (
+  namedScores: Record<string, number>, // Map of metric names to scores (0-1)
+  context: {
+    threshold?: number; // Test case threshold if set
+    tokensUsed?: {
+      // Token usage if available
+      total: number;
+      prompt: number;
+      completion: number;
+    };
+  },
+) => {
+  pass: boolean; // Whether the test case passes
+  score: number; // Final score (0-1)
+  reason: string; // Explanation of the score
+};
+```
+
+When assertions use `weight`, each named score passed into the scoring function is already normalized as a weighted average. Eval outputs also include `namedScoreWeights` so downstream consumers can recover the weighted denominator when needed.
+
+See the [custom assertion scoring example](https://github.com/promptfoo/promptfoo/tree/main/examples/eval-assertion-scoring-override) for complete implementations in JavaScript and Python.
+
+## Load assertions from external file
+
+#### Raw files
+
+The `value` of an assertion can be loaded directly from a file using the `file://` syntax:
+
+```yaml
+- assert:
+    - type: contains
+      value: file://gettysburg_address.txt
+```
+
+#### Javascript
+
+If the file ends in `.js`, the Javascript is executed:
+
+```yaml title="promptfooconfig.yaml"
+- assert:
+    - type: javascript
+      value: file://path/to/assert.js
+```
+
+The type definition is:
+
+```ts
+type AssertionValueFunctionContext = {
+  prompt: string | undefined;
+  vars: Record<string, string | object>;
+  test: AtomicTestCase<Record<string, string | object>>;
+  logProbs: number[] | undefined;
+  config?: Record<string, any>;
+  provider: ApiProvider | undefined;
+  providerResponse: ProviderResponse | undefined;
+  metadata?: ProviderResponse['metadata']; // Shortcut to providerResponse?.metadata
+};
+type AssertionResponse = string | boolean | number | GradingResult;
+type AssertFunction = (output: string, context: AssertionValueFunctionContext) => AssertionResponse;
+```
+
+See [GradingResult definition](/docs/configuration/reference#gradingresult).
+
+Here's an example `assert.js`:
+
+```js
+module.exports = (output, { vars }) => {
+  console.log(`Received ${output} using variables ${JSON.stringify(vars)}`);
+  return {
+    pass: true,
+    score: 0.5,
+    reason: 'Some custom reason',
+  };
+};
+```
+
+You can also use Javascript files in non-`javascript`-type asserts. For example, using a Javascript file in a `contains` assertion will check that the output contains the string returned by Javascript.
+
+#### Python
+
+If the file ends in `.py`, the Python is executed:
+
+```yaml title="promptfooconfig.yaml"
+- assert:
+    - type: python
+      value: file://path/to/assert.py
+```
+
+The assertion expects an output that is `bool`, `float`, or a JSON [GradingResult](/docs/configuration/reference#gradingresult).
+
+For example:
+
+```py
+import sys
+import json
+
+output = sys.argv[1]
+context = json.loads(sys.argv[2])
+
+# Use `output` and `context['vars']` to determine result ...
+
+print(json.dumps({
+  'pass': False,
+  'score': 0.5,
+  'reason': 'Some custom reason',
+}))
+```
+
+## Load assertions from CSV
+
+The [Tests file](/docs/configuration/test-cases) is an optional format that lets you specify test cases outside of the main config file.
+
+To add an assertion to a test case in a vars file, use the special `__expected` column.
+
+Here's an example tests.csv:
+
+| text               | \_\_expected                                         |
+| ------------------ | ---------------------------------------------------- |
+| Hello, world!      | Bonjour le monde                                     |
+| Goodbye, everyone! | fn:output.includes('Au revoir');                     |
+| I am a pineapple   | grade:doesn't reference any fruits besides pineapple |
+
+All assertion types can be used in `__expected`. The column supports exactly one assertion.
+
+### Assertion string syntax
+
+The general format is `type:value` or `type(threshold):value`. Values without a prefix default to `equals`.
+
+| Syntax                       | Type            | Example                                     |
+| ---------------------------- | --------------- | ------------------------------------------- |
+| `value`                      | `equals`        | `Paris`                                     |
+| `contains:value`             | `contains`      | `contains:Paris`                            |
+| `icontains:value`            | `icontains`     | `icontains:paris`                           |
+| `contains-any:value1,value2` | `contains-any`  | `contains-any:Paris,London`                 |
+| `contains-all:value1,value2` | `contains-all`  | `contains-all:Paris,France`                 |
+| `starts-with:value`          | `starts-with`   | `starts-with:The answer`                    |
+| `regex:pattern`              | `regex`         | `regex:^Hello.*world$`                      |
+| `is-json`                    | `is-json`       | `is-json`                                   |
+| `contains-json`              | `contains-json` | `contains-json`                             |
+| `similar(threshold):value`   | `similar`       | `similar(0.8):Hello world`                  |
+| `llm-rubric:criteria`        | `llm-rubric`    | `llm-rubric:Is helpful and accurate`        |
+| `grade:criteria`             | `llm-rubric`    | `grade:Does not mention being an AI`        |
+| `factuality:reference`       | `factuality`    | `factuality:Paris is the capital of France` |
+| `javascript:code`            | `javascript`    | `javascript:output.length < 100`            |
+| `fn:code`                    | `javascript`    | `fn:output.includes('hello')`               |
+| `python:code`                | `python`        | `python:len(output) > 10`                   |
+| `file://path`                | External file   | `file://assertions/custom.js`               |
+| `not-type:value`             | Negated         | `not-contains:error`                        |
+| `levenshtein(N):value`       | `levenshtein`   | `levenshtein(5):expected text`              |
+
+When the `__expected` field is provided, the success and failure statistics in the evaluation summary will be based on whether the expected criteria are met.
+
+To run multiple assertions, use column names `__expected1`, `__expected2`, `__expected3`, etc.
+
+For more advanced test cases, we recommend using a testing framework like [Jest or Vitest](/docs/integrations/jest) or [Mocha](/docs/integrations/mocha-chai) and using promptfoo [as a library](/docs/usage/node-package).
+
+## Reusing assertions with templates
+
+If you have a set of common assertions that you want to apply to multiple test cases, you can create assertion templates and reuse them across your configuration.
+
+```yaml
+// highlight-start
+assertionTemplates:
+  containsMentalHealth:
+    type: javascript
+    value: output.toLowerCase().includes('mental health')
+// highlight-end
+
+prompts:
+  - file://prompt1.txt
+  - file://prompt2.txt
+providers:
+  - openai:gpt-5.5
+  - localai:chat:vicuna
+tests:
+  - vars:
+      input: Tell me about the benefits of exercise.
+    assert:
+      // highlight-next-line
+      - $ref: "#/assertionTemplates/containsMentalHealth"
+  - vars:
+      input: How can I improve my well-being?
+    assert:
+      // highlight-next-line
+      - $ref: "#/assertionTemplates/containsMentalHealth"
+```
+
+In this example, the `containsMentalHealth` assertion template is defined at the top of the configuration file and then reused in two test cases. This approach helps maintain consistency and reduces duplication in your configuration.
+
+## Defining named metrics
+
+Each assertion supports a `metric` field that allows you to tag the result however you like. Use this feature to combine related assertions into aggregate metrics.
+
+For example, these asserts will aggregate results into two metrics, `Tone` and `Consistency`.
+
+```yaml
+tests:
+  - assert:
+      - type: equals
+        value: Yarr
+        metric: Tone
+
+  - assert:
+      - type: icontains
+        value: grub
+        metric: Tone
+
+  - assert:
+      - type: is-json
+        metric: Consistency
+
+  - assert:
+      - type: python
+        value: max(0, len(output) - 300)
+        metric: Consistency
+
+      - type: similar
+        value: Ahoy, world
+        metric: Tone
+
+  - assert:
+      - type: llm-rubric
+        value: Is spoken like a pirate
+        metric: Tone
+```
+
+These metrics will be shown in the UI:
+
+![llm eval metrics](/img/docs/named-metrics.png)
+
+See [named metrics example](https://github.com/promptfoo/promptfoo/tree/main/examples/eval-named-metrics).
+
+## Creating derived metrics
+
+Derived metrics calculate composite scores from your named assertions after evaluation completes. Use them for metrics like F1 scores, weighted averages, or custom scoring formulas.
+
+Add a `derivedMetrics` array to your configuration:
+
+```yaml
+derivedMetrics:
+  - name: 'f1_score'
+    value: '2 * precision * recall / (precision + recall)'
+```
+
+Each derived metric requires:
+
+- **name**: The metric identifier
+- **value**: A mathematical expression or JavaScript function
+
+:::tip
+Derived metrics are initialized to 0 and calculated per prompt. Errors are logged at debug level.
+:::
+
+### Mathematical expressions
+
+Use [mathjs](https://mathjs.org/) syntax for calculations:
+
+```yaml
+derivedMetrics:
+  - name: 'weighted_score'
+    value: 'accuracy * 0.6 + relevance * 0.4'
+
+  - name: 'harmonic_mean'
+    value: '3 / (1/accuracy + 1/relevance + 1/coherence)'
+```
+
+### JavaScript functions
+
+For complex logic:
+
+```yaml
+derivedMetrics:
+  - name: 'adaptive_score'
+    value: |
+      function(namedScores, evalStep) {
+        const { accuracy = 0, speed = 0 } = namedScores;
+        if (evalStep.tokensUsed?.total > 1000) {
+          return accuracy * 0.8; // Penalize verbose responses
+        }
+        return accuracy * 0.6 + speed * 0.4;
+      }
+```
+
+### Example: F1 score
+
+```yaml
+defaultTest:
+  assert:
+    - type: javascript
+      value: output.sentiment === 'positive' && context.vars.expected === 'positive' ? 1 : 0
+      metric: true_positives
+      weight: 0
+    - type: javascript
+      value: output.sentiment === 'positive' && context.vars.expected === 'negative' ? 1 : 0
+      metric: false_positives
+      weight: 0
+    - type: javascript
+      value: output.sentiment === 'negative' && context.vars.expected === 'positive' ? 1 : 0
+      metric: false_negatives
+      weight: 0
+
+derivedMetrics:
+  - name: precision
+    value: 'true_positives / (true_positives + false_positives)'
+  - name: recall
+    value: 'true_positives / (true_positives + false_negatives)'
+  - name: f1_score
+    value: '2 * true_positives / (2 * true_positives + false_positives + false_negatives)'
+```
+
+Metrics are calculated in order, so later metrics can reference earlier ones:
+
+```yaml
+derivedMetrics:
+  - name: base_score
+    value: '(accuracy + relevance) / 2'
+  - name: final_score
+    value: 'base_score * confidence_multiplier'
+```
+
+### Calculating averages with `__count`
+
+For metrics where you need the average across test cases (like Mean Absolute Percentage Error), use the built-in `__count` variable:
+
+```yaml
+defaultTest:
+  assert:
+    - type: javascript
+      value: |
+        const actual = context.vars.actual_value;
+        const predicted = parseFloat(output);
+        return Math.abs(actual - predicted) / actual;
+      metric: APE
+      weight: 0
+
+derivedMetrics:
+  # MAPE = Mean Absolute Percentage Error
+  - name: MAPE
+    value: 'APE / __count'
+```
+
+The `__count` variable contains the number of test evals for the current prompt-provider combination. With multiple providers, each provider gets its own separate metrics tracked independently. This is useful when:
+
+- Each test case produces a value that gets summed (like error metrics)
+- You want to display the average instead of the total
+
+For JavaScript functions, `__count` is available in the `namedScores` object:
+
+```yaml
+derivedMetrics:
+  - name: 'average_error'
+    value: |
+      function(namedScores, evalStep) {
+        return namedScores.total_error / namedScores.__count;
+      }
+```
+
+### Notes
+
+- Missing metrics default to 0
+- The `__count` variable is per prompt-provider combination (number of test cases)
+- Functions receive a copy of the context - return values, don't mutate
+- To avoid division by zero: `value: 'numerator / (denominator + 0.0001)'`
+- Debug errors with: `LOG_LEVEL=debug promptfoo eval`
+- No circular dependency protection - order your metrics carefully
+
+Derived metrics appear in all outputs alongside regular metrics - in the web UI metrics column, JSON `namedScores`, and CSV columns.
+
+See also:
+
+- [Named metrics example](https://github.com/promptfoo/promptfoo/tree/main/examples/eval-named-metrics) - Basic named metrics usage
+- [F-score example](https://github.com/promptfoo/promptfoo/tree/main/examples/eval-f-score) - Complete F1 score implementation
+- [MathJS documentation](https://mathjs.org/docs/expressions/syntax.html) - Expression syntax reference
+
+## Running assertions directly on outputs
+
+If you already have LLM outputs and want to run assertions on them, the `eval` command supports standalone assertion files.
+
+Put your outputs in a JSON string array, like this `output.json`:
+
+```json
+["Hello world", "Greetings, planet", "Salutations, Earth"]
+```
+
+And create a list of assertions (`asserts.yaml`):
+
+```yaml
+- type: icontains
+  value: hello
+
+- type: javascript
+  value: 1 / (output.length + 1) # prefer shorter outputs
+
+- type: model-graded-closedqa
+  value: ensure that the output contains a greeting
+```
+
+Then run the eval command:
+
+```
+promptfoo eval --assertions asserts.yaml --model-outputs outputs.json
+```
+
+### Tagging outputs
+
+Promptfoo accepts a slightly more complex JSON structure that includes an `output` field for the model's output and a `tags` field for the associated tags. These tags are shown in the web UI as a comma-separated list. It's useful if you want to keep track of certain output attributes:
+
+```json
+[
+  { "output": "Hello world", "tags": ["foo", "bar"] },
+  { "output": "Greetings, planet", "tags": ["baz", "abc"] },
+  { "output": "Salutations, Earth", "tags": ["def", "ghi"] }
+]
+```
+
+### Processing and formatting outputs
+
+If you need to do any processing/formatting of outputs, use a [Javascript provider](/docs/providers/custom-api/), [Python provider](https://promptfoo.dev/docs/providers/python/), or [custom script](/docs/providers/custom-script/).

site/docs/configuration/expected-outputs/javascript.md

@@ -0,0 +1,452 @@
+---
+sidebar_position: 50
+sidebar_label: Javascript
+description: Build sophisticated JavaScript validators for LLM outputs with async operations, scoring logic, and comprehensive error handling
+---
+
+# Javascript assertions
+
+The `javascript` [assertion](/docs/configuration/expected-outputs) allows you to provide a custom JavaScript function to validate the LLM output.
+
+A variable named `output` is injected into the context. The function should return `true` if the output passes the assertion, and `false` otherwise. If the function returns a number, it will be treated as a score.
+
+You can use any valid JavaScript code in your function. The output of the LLM is provided as the `output` variable:
+
+```yaml
+assert:
+  - type: javascript
+    value: "output.includes('Hello, World!')"
+```
+
+In the example above, the `javascript` assertion checks if the output includes the string "Hello, World!". If it does, the assertion passes and a score of 1 is recorded. If it doesn't, the assertion fails and a score of 0 is returned.
+
+If you want to return a custom score, your function should return a number. For example:
+
+```yaml
+assert:
+  - type: javascript
+    value: Math.log(output.length) * 10
+    threshold: 0.5 # any value above 0.5 will pass
+```
+
+In the example above, the longer the output, the higher the score.
+
+If your function throws an error, the assertion will fail and the error message will be included in the reason for the failure. For example:
+
+```yaml
+assert:
+  - type: javascript
+    value: |
+      if (errorCase) {
+        throw new Error('This is an error');
+      }
+      return {
+        pass: false,
+        score: 0,
+        reason: 'Assertion failed',
+      };
+```
+
+## Handling objects
+
+If the LLM outputs a JSON object (such as in the case of tool/function calls), then `output` will already be parsed as an object:
+
+```yaml
+assert:
+  - type: javascript
+    value: output[0].function.name === 'get_current_weather'
+```
+
+## Return type
+
+The return value of your Javascript function can be a boolean, number, or a `GradingResult`:
+
+```typescript
+type JavascriptAssertionResult = boolean | number | GradingResult;
+
+// Used for more complex results
+interface GradingResult {
+  pass: boolean;
+  score: number;
+  reason: string;
+  componentResults?: GradingResult[];
+}
+```
+
+If `componentResults` is set, a table of assertion details will be shown in the test output modal in the Eval view.
+
+## Multiline functions
+
+Javascript assertions support multiline strings:
+
+```yaml
+assert:
+  - type: javascript
+    value: |
+      // Insert your scoring logic here...
+      if (output === 'Expected output') {
+        return {
+          pass: true,
+          score: 0.5,
+        };
+      }
+      return {
+        pass: false,
+        score: 0,
+        reason: 'Assertion failed',
+      };
+```
+
+## Using test context
+
+The `context` variable contains information about the test case and execution environment:
+
+```ts
+interface TraceSpan {
+  spanId: string;
+  parentSpanId?: string;
+  name: string;
+  startTime: number; // Unix timestamp in milliseconds
+  endTime?: number; // Unix timestamp in milliseconds
+  attributes?: Record<string, any>;
+  statusCode?: number;
+  statusMessage?: string;
+}
+
+interface TraceData {
+  traceId: string;
+  spans: TraceSpan[];
+}
+
+interface AssertionValueFunctionContext {
+  // Raw prompt sent to LLM
+  prompt: string | undefined;
+
+  // Test case variables
+  vars: Record<string, string | object>;
+
+  // The complete test case
+  test: AtomicTestCase;
+
+  // Log probabilities from the LLM response, if available
+  logProbs: number[] | undefined;
+
+  // Configuration passed to the assertion
+  config?: Record<string, any>;
+
+  // The provider that generated the response
+  provider: ApiProvider | undefined;
+
+  // The complete provider response
+  providerResponse: ProviderResponse | undefined;
+
+  // OpenTelemetry trace data (when tracing is enabled)
+  trace?: TraceData;
+
+  // Shortcut to providerResponse?.metadata (provider-specific fields)
+  metadata?: Record<string, any>;
+}
+```
+
+For example, if the test case has a var `example`, access it in your JavaScript function like this:
+
+```yaml
+tests:
+  - description: 'Test with context'
+    vars:
+      example: 'Example text'
+    assert:
+      - type: javascript
+        value: 'output.includes(context.vars.example)'
+```
+
+You can also use the `context` variable to perform more complex checks. For example, you could check if the output is longer than a certain length defined in your test case variables:
+
+```yaml
+tests:
+  - description: 'Test with context'
+    vars:
+      min_length: 10
+    assert:
+      - type: javascript
+        value: 'output.length >= context.vars.min_length'
+```
+
+## Passing assertion-specific parameters
+
+If you want to reuse the same JavaScript assertion with different parameters in a single test case, prefer assertion-level `config` over test `vars`. Test vars are shared across all assertions and appear as report columns, while `config` stays attached to one assertion and is available as `context.config`.
+
+```yaml
+tests:
+  - description: 'Reuse one assertion script with two thresholds'
+    vars:
+      topic: 'bananas'
+    assert:
+      - type: javascript
+        value: file://assertions/min-length.js
+        config:
+          minLength: 5
+      - type: javascript
+        value: file://assertions/min-length.js
+        config:
+          minLength: 20
+```
+
+```js
+module.exports = (output, context) => {
+  return output.length >= context.config.minLength;
+};
+```
+
+## External script
+
+To reference an external file, use the `file://` prefix:
+
+```yaml
+assert:
+  - type: javascript
+    value: file://relative/path/to/script.js
+    config:
+      maximumOutputSize: 10
+```
+
+You can specify a particular function to use by appending it after a colon:
+
+```yaml
+assert:
+  - type: javascript
+    value: file://relative/path/to/script.js:customFunction
+```
+
+The JavaScript file must export an assertion function. Here are examples:
+
+```js
+// Default export
+module.exports = (output, context) => {
+  return output.length > 10;
+};
+```
+
+```js
+// Named exports
+module.exports.customFunction = (output, context) => {
+  return output.includes('specific text');
+};
+```
+
+Here's an example using configuration data defined in the assertion's YAML file:
+
+```js
+module.exports = (output, context) => {
+  return output.length <= context.config.maximumOutputSize;
+};
+```
+
+Here's a more complex example that uses an async function to hit an external validation service:
+
+```js
+const VALIDATION_ENDPOINT = 'https://example.com/api/validate';
+
+async function evaluate(modelResponse) {
+  try {
+    const response = await fetch(VALIDATION_ENDPOINT, {
+      method: 'POST',
+      headers: {
+        'Content-Type': 'text/plain',
+      },
+      body: modelResponse,
+    });
+
+    const data = await response.json();
+    return data;
+  } catch (error) {
+    throw error;
+  }
+}
+
+async function main(output, context) {
+  const success = await evaluate(output);
+  console.log(`success: ${testResult}`);
+  return success;
+}
+
+module.exports = main;
+```
+
+You can also return complete [`GradingResult`](/docs/configuration/reference/#gradingresult) objects. For example:
+
+```js
+module.exports = (output, context) => {
+  console.log('Prompt:', context.prompt);
+  console.log('Vars', context.vars.topic);
+
+  // You can return a bool...
+  // return output.toLowerCase().includes('bananas');
+
+  // A score (where 0 = Fail)...
+  // return 0.5;
+
+  // Or an entire grading result, which can be simple...
+  let result = {
+    pass: output.toLowerCase().includes('bananas'),
+    score: 0.5,
+    reason: 'Contains banana',
+  };
+
+  // Or include nested assertions...
+  result = {
+    pass: true,
+    score: 0.75,
+    reason: 'Looks good to me',
+    componentResults: [
+      {
+        pass: output.toLowerCase().includes('bananas'),
+        score: 0.5,
+        reason: 'Contains banana',
+        namedScores: {
+          'Uses banana': 1.0,
+        },
+      },
+      {
+        pass: output.toLowerCase().includes('yellow'),
+        score: 0.5,
+        reason: 'Contains yellow',
+        namedScores: {
+          Yellowish: 0.66,
+        },
+      },
+    ],
+  };
+
+  return result;
+};
+```
+
+## Inline assertions
+
+If you are using promptfoo as a JS package, you can build your assertion inline:
+
+```js
+{
+  type:"javascript",
+  value: (output, context) => {
+    return output.includes("specific text");
+  }
+}
+```
+
+Output will always be a string, so if your [custom response parser](/docs/providers/http/#function-parser) returned an object, you can use `JSON.parse(output)` to convert it back to an object.
+
+## Using trace data
+
+When [tracing is enabled](/docs/tracing/), OpenTelemetry trace data is available in the `context.trace` object. This allows you to write assertions based on the execution flow:
+
+```js
+module.exports = (output, context) => {
+  // Check if trace data is available
+  if (!context.trace) {
+    // Tracing not enabled, skip trace-based checks
+    return true;
+  }
+
+  const { spans } = context.trace;
+
+  // Example: Check for errors in any span
+  const errorSpans = spans.filter((s) => s.statusCode >= 400);
+  if (errorSpans.length > 0) {
+    return {
+      pass: false,
+      score: 0,
+      reason: `Found ${errorSpans.length} error spans`,
+    };
+  }
+
+  // Example: Calculate total trace duration
+  if (spans.length > 0) {
+    const duration =
+      Math.max(...spans.map((s) => s.endTime || 0)) - Math.min(...spans.map((s) => s.startTime));
+    if (duration > 5000) {
+      // 5 seconds
+      return {
+        pass: false,
+        score: 0,
+        reason: `Trace took too long: ${duration}ms`,
+      };
+    }
+  }
+
+  // Example: Check for specific operations
+  const apiCalls = spans.filter((s) => s.name.toLowerCase().includes('http'));
+  if (apiCalls.length > 10) {
+    return {
+      pass: false,
+      score: 0,
+      reason: `Too many API calls: ${apiCalls.length}`,
+    };
+  }
+
+  return true;
+};
+```
+
+Example YAML configuration:
+
+```yaml
+tests:
+  - vars:
+      query: "What's the weather?"
+    assert:
+      - type: javascript
+        value: |
+          // Ensure retrieval happened before response generation
+          if (context.trace) {
+            const retrievalSpan = context.trace.spans.find(s => s.name.includes('retrieval'));
+            const generationSpan = context.trace.spans.find(s => s.name.includes('generation'));
+            
+            if (retrievalSpan && generationSpan) {
+              return retrievalSpan.startTime < generationSpan.startTime;
+            }
+          }
+          return true;
+```
+
+Additional examples:
+
+```js
+// Check span hierarchy depth
+const MAX_ALLOWED_DEPTH = 1000;
+const maxDepth = (spans, parentId = null, depth = 0) => {
+  if (depth > MAX_ALLOWED_DEPTH) {
+    throw new Error('Span hierarchy too deep');
+  }
+  const children = spans.filter((s) => s.parentSpanId === parentId);
+  if (children.length === 0) return depth;
+  return Math.max(...children.map((c) => maxDepth(spans, c.spanId, depth + 1)));
+};
+
+if (context.trace && maxDepth(context.trace.spans) > 5) {
+  return {
+    pass: false,
+    score: 0,
+    reason: 'Call stack too deep',
+  };
+}
+```
+
+### ES modules
+
+ES modules are supported, but must have a `.mjs` file extension. Alternatively, if you are transpiling Javascript or Typescript, we recommend pointing promptfoo to the transpiled plain Javascript output.
+
+## Negation
+
+Use `not-javascript` to invert the final pass/fail result while preserving the returned score. Numeric scores are still compared against `threshold` before the result is inverted:
+
+```yaml
+assert:
+  - type: not-javascript
+    value: output.includes('error')
+```
+
+## Other assertion types
+
+For more info on assertions, see [Test assertions](/docs/configuration/expected-outputs).

site/docs/configuration/expected-outputs/model-graded/agent-rubric.md

@@ -0,0 +1,83 @@
+---
+sidebar_label: Agent Rubric
+description: 'Use coding-agent graders to verify outputs against files, tools, and other runtime evidence'
+---
+
+# Agent Rubric
+
+`agent-rubric` is an agentic variant of [`llm-rubric`](/docs/configuration/expected-outputs/model-graded/llm-rubric). It grades an output against a natural-language rubric, but requires a coding-agent provider that can gather evidence using its configured tools and workspace.
+
+Use it when the judge needs to inspect an artifact rather than only read the target output, for example to verify a claimed code change, locate a generated file, or check a repository-level requirement.
+
+## Basic usage
+
+Without an explicit grading provider, `agent-rubric` uses `openai:codex-sdk` in an isolated temporary working directory with read-only sandboxing, no approvals, and structured JSON grading output:
+
+```yaml
+tests:
+  - assert:
+      - type: agent-rubric
+        value: Verify any claims in the output using the evidence available to the grader.
+```
+
+The isolated default is useful for agent behavior and tool availability checks, but it does not expose your project files. To let the grader inspect a fixture or repository, explicitly configure a working directory:
+
+```yaml
+tests:
+  - assert:
+      - type: agent-rubric
+        value: Verify that the output accurately describes the exported API in src/report.ts.
+        provider:
+          id: openai:codex-sdk
+          config:
+            working_dir: ./sample-project
+            sandbox_mode: read-only
+            approval_policy: never
+            skip_git_repo_check: true
+```
+
+Install and authenticate the [OpenAI Codex SDK provider](/docs/providers/openai-codex-sdk) before using the implicit default.
+
+## Supported agent providers
+
+`agent-rubric` accepts the coding-agent runtimes that promptfoo can run as providers:
+
+| Runtime                 | Provider ID                                             |
+| ----------------------- | ------------------------------------------------------- |
+| OpenAI Codex SDK        | `openai:codex-sdk` or `openai:codex`                    |
+| OpenAI Codex app-server | `openai:codex-app-server` or `openai:codex-desktop`     |
+| Claude Agent SDK        | `anthropic:claude-agent-sdk` or `anthropic:claude-code` |
+| OpenCode SDK            | `opencode:sdk` or `opencode`                            |
+
+For example, use Claude Agent SDK as the judge with read-only filesystem tools:
+
+```yaml
+assert:
+  - type: agent-rubric
+    value: Inspect the project and verify that the response names the correct configuration file.
+    provider:
+      id: anthropic:claude-agent-sdk
+      config:
+        working_dir: ./sample-project
+```
+
+A plain text provider such as `openai:responses:gpt-5` is rejected for `agent-rubric`. Use `llm-rubric` when the grader only needs the output and rubric text.
+
+## Safety and side effects
+
+An agentic grader processes untrusted target output and may read untrusted workspace content. The default grading prompt instructs it to treat that material as evidence rather than instructions, and the implicit Codex provider is read-only.
+
+Keep grader workspaces read-only whenever possible. If you enable write access, shell actions, network access, MCP tools, or app connectors, those actions are performed by the grader itself during the eval and should be limited to disposable or controlled environments.
+
+## Results and configuration
+
+`agent-rubric` uses the same `{ reason, pass, score }` result and `threshold`, `rubricPrompt`, and `not-` semantics as `llm-rubric`:
+
+```yaml
+assert:
+  - type: agent-rubric
+    value: Confirm that the output's implementation claim is supported by the workspace.
+    threshold: 0.8
+```
+
+Provider response metadata, such as agent tool calls, is preserved on the grading result. The result also includes `metadata.agentProvider` identifying the agent runtime used for the check.

site/docs/configuration/expected-outputs/model-graded/answer-relevance.md

@@ -0,0 +1,95 @@
+---
+sidebar_label: Answer Relevance
+description: 'Score LLM response relevance and completeness against user queries using sophisticated AI-powered evaluation metrics'
+---
+
+# Answer Relevance
+
+The `answer-relevance` assertion evaluates whether an LLM's output is relevant to the original query. It uses a combination of embedding similarity and LLM evaluation to determine relevance.
+
+### How to use it
+
+To use the `answer-relevance` assertion type, add it to your test configuration like this:
+
+```yaml
+assert:
+  - type: answer-relevance
+    threshold: 0.7 # Score between 0 and 1
+```
+
+### How it works
+
+The answer relevance checker:
+
+1. Uses an LLM to generate potential questions that the output could be answering
+2. Compares these questions with the original query using embedding similarity
+3. Calculates a relevance score based on the similarity scores
+
+A higher threshold requires the output to be more closely related to the original query.
+
+### Example Configuration
+
+Here's a complete example showing how to use answer relevance:
+
+```yaml
+prompts:
+  - 'Tell me about {{topic}}'
+providers:
+  - openai:gpt-5
+tests:
+  - vars:
+      topic: quantum computing
+    assert:
+      - type: answer-relevance
+        threshold: 0.8
+```
+
+### Overriding the Providers
+
+Answer relevance uses two types of providers:
+
+- A text provider for generating questions
+- An embedding provider for calculating similarity
+
+You can override either or both:
+
+```yaml
+defaultTest:
+  options:
+    provider:
+      text:
+        id: gpt-5
+        config:
+          temperature: 0
+      embedding:
+        id: openai:text-embedding-ada-002
+```
+
+You can also override providers at the assertion level:
+
+```yaml
+assert:
+  - type: answer-relevance
+    threshold: 0.8
+    provider:
+      text: anthropic:claude-2
+      embedding: cohere:embed-english-v3.0
+```
+
+### Customizing the Prompt
+
+You can customize the question generation prompt using the `rubricPrompt` property:
+
+```yaml
+defaultTest:
+  options:
+    rubricPrompt: |
+      Given this answer: {{output}}
+
+      Generate 3 questions that this answer would be appropriate for.
+      Make the questions specific and directly related to the content.
+```
+
+# Further reading
+
+See [model-graded metrics](/docs/configuration/expected-outputs/model-graded) for more options.

site/docs/configuration/expected-outputs/model-graded/context-faithfulness.md

@@ -0,0 +1,102 @@
+---
+sidebar_position: 50
+description: 'Measure LLM faithfulness to source context by detecting unsupported claims in responses.'
+---
+
+# Context faithfulness
+
+Checks if the LLM's response only makes claims that are supported by the provided context.
+
+**Use when**: You need to ensure the LLM isn't adding information beyond what was retrieved.
+
+**How it works**: Extracts factual claims from the response, then verifies each against the context. Score = supported claims / total claims.
+
+**Example**:
+
+```text
+Context: "Paris is the capital of France."
+Response: "Paris, with 2.2 million residents, is France's capital."
+Score: 0.5 (capital ✓, population ✗)
+```
+
+## Configuration
+
+```yaml
+assert:
+  - type: context-faithfulness
+    threshold: 0.9 # Require 90% of claims to be supported
+```
+
+### Required fields
+
+- `query` - User's question (in test vars)
+- `context` - Reference text (in vars or via `contextTransform`)
+- `threshold` - Minimum score 0-1 (default: 0)
+
+### Full example
+
+```yaml
+tests:
+  - vars:
+      query: 'What is the capital of France?'
+      context: 'Paris is the capital and largest city of France.'
+    assert:
+      - type: context-faithfulness
+        threshold: 0.9
+```
+
+### Array context
+
+Context can also be an array:
+
+```yaml
+tests:
+  - vars:
+      query: 'Tell me about France'
+      context:
+        - 'Paris is the capital and largest city of France.'
+        - 'France is located in Western Europe.'
+        - 'The country has a rich cultural heritage.'
+    assert:
+      - type: context-faithfulness
+        threshold: 0.8
+```
+
+### Dynamic context extraction
+
+For RAG systems that return context with their response:
+
+```yaml
+# Provider returns { answer: "...", context: "..." }
+assert:
+  - type: context-faithfulness
+    contextTransform: 'output.context' # Extract context field
+    threshold: 0.9
+```
+
+### Custom grading
+
+Override the default grader:
+
+```yaml
+assert:
+  - type: context-faithfulness
+    provider: gpt-5 # Use a different model for grading
+    threshold: 0.9
+```
+
+## Limitations
+
+- Depends on judge LLM quality
+- May miss implicit claims
+- Performance degrades with very long contexts
+
+## Related metrics
+
+- [`context-relevance`](/docs/configuration/expected-outputs/model-graded/context-relevance) - Is retrieved context relevant?
+- [`context-recall`](/docs/configuration/expected-outputs/model-graded/context-recall) - Does context support the expected answer?
+
+## Further reading
+
+- [Defining context in test cases](/docs/configuration/expected-outputs/model-graded#defining-context)
+- [RAG Evaluation Guide](/docs/guides/evaluate-rag)

site/docs/configuration/expected-outputs/model-graded/context-recall.md

@@ -0,0 +1,77 @@
+---
+sidebar_position: 50
+description: 'Quantify retrieval quality by measuring how thoroughly LLM responses cover expected information from source materials.'
+---
+
+# Context recall
+
+Checks if your retrieved context contains the information needed to generate a known correct answer.
+
+**Use when**: You have ground truth answers and want to verify your retrieval finds supporting evidence.
+
+**How it works**: Breaks the expected answer into statements and checks if each can be attributed to the context. Score = attributable statements / total statements.
+
+**Example**:
+
+```text
+Expected: "Python was created by Guido van Rossum in 1991"
+Context: "Python was released in 1991"
+Score: 0.5 (year ✓, creator ✗)
+```
+
+## Configuration
+
+```yaml
+assert:
+  - type: context-recall
+    value: 'Python was created by Guido van Rossum in 1991'
+    threshold: 1.0 # Context must support entire answer
+```
+
+### Required fields
+
+- `value` - Expected answer/ground truth
+- `context` - Retrieved text (in vars or via `contextTransform`)
+- `threshold` - Minimum score 0-1 (default: 0)
+
+### Full example
+
+```yaml
+tests:
+  - vars:
+      query: 'Who created Python?'
+      context: 'Guido van Rossum created Python in 1991.'
+    assert:
+      - type: context-recall
+        value: 'Python was created by Guido van Rossum in 1991'
+        threshold: 1.0
+```
+
+### Dynamic context extraction
+
+For RAG systems that return context with their response:
+
+```yaml
+# Provider returns { answer: "...", context: "..." }
+assert:
+  - type: context-recall
+    value: 'Expected answer here'
+    contextTransform: 'output.context' # Extract context field
+    threshold: 0.8
+```
+
+## Limitations
+
+- Binary attribution (no partial credit)
+- Works best with factual statements
+- Requires known correct answers
+
+## Related metrics
+
+- [`context-relevance`](/docs/configuration/expected-outputs/model-graded/context-relevance) - Is retrieved context relevant?
+- [`context-faithfulness`](/docs/configuration/expected-outputs/model-graded/context-faithfulness) - Does output stay faithful to context?
+
+## Further reading
+
+- [Defining context in test cases](/docs/configuration/expected-outputs/model-graded#defining-context)
+- [RAG Evaluation Guide](/docs/guides/evaluate-rag)

site/docs/configuration/expected-outputs/model-graded/context-relevance.md

@@ -0,0 +1,111 @@
+---
+sidebar_position: 50
+description: 'Assess RAG retrieval quality by evaluating context relevance, precision, and usefulness for answering queries.'
+---
+
+# Context relevance
+
+Measures what fraction of retrieved context is minimally needed to answer the query.
+
+**Use when**: You want to check if your retrieval is returning too much irrelevant content.
+
+**How it works**: Extracts only the sentences absolutely required to answer the query. Score = required sentences / total sentences.
+
+:::warning
+This metric finds the MINIMUM needed, not all relevant content. A low score might mean good retrieval (found answer plus supporting context) or bad retrieval (lots of irrelevant content).
+:::
+
+**Example**:
+
+```text
+Query: "What is the capital of France?"
+Context: "Paris is the capital. France has great wine. The Eiffel Tower is in Paris."
+Score: 0.33 (only first sentence required)
+```
+
+## Configuration
+
+```yaml
+assert:
+  - type: context-relevance
+    threshold: 0.3 # At least 30% should be essential
+```
+
+### Required fields
+
+- `query` - User's question (in test vars)
+- `context` - Retrieved text (in vars or via `contextTransform`)
+- `threshold` - Minimum score 0-1 (default: 0)
+
+### Full example
+
+```yaml
+tests:
+  - vars:
+      query: 'What is the capital of France?'
+      context: 'Paris is the capital of France.'
+    assert:
+      - type: context-relevance
+        threshold: 0.8 # Most content should be essential
+```
+
+### Array context
+
+Context can be provided as an array of chunks:
+
+```yaml
+tests:
+  - vars:
+      query: 'What are the benefits of RAG systems?'
+      context:
+        - 'RAG systems improve factual accuracy by incorporating external knowledge sources.'
+        - 'They reduce hallucinations in large language models through grounded responses.'
+        - 'RAG enables up-to-date information retrieval beyond training data cutoffs.'
+        - 'The weather forecast shows rain this weekend.' # irrelevant chunk
+    assert:
+      - type: context-relevance
+        threshold: 0.5 # Score: 3/4 = 0.75
+```
+
+### Dynamic context extraction
+
+For RAG systems that return context with their response:
+
+```yaml
+# Provider returns { answer: "...", context: "..." }
+assert:
+  - type: context-relevance
+    contextTransform: 'output.context' # Extract context field
+    threshold: 0.3
+```
+
+`contextTransform` can also return an array:
+
+```yaml
+assert:
+  - type: context-relevance
+    contextTransform: 'output.chunks' # Extract chunks array
+    threshold: 0.5
+```
+
+## Score interpretation
+
+- **0.8-1.0**: Almost all content is essential (very focused or minimal retrieval)
+- **0.3-0.7**: Mixed essential and supporting content (often ideal)
+- **0.0-0.3**: Mostly non-essential content (may indicate poor retrieval)
+
+## Limitations
+
+- Only identifies minimum sufficient content
+- A single-paragraph (prose) context string is split into sentences on `.`/`!`/`?` boundaries; a context with two or more non-empty lines, or an array of chunks, is treated as already segmented and split by line/chunk. Sentence splitting is a lightweight heuristic that does not handle every abbreviation or decimal edge case — provide an array of chunks for the most precise denominator.
+- Score interpretation varies by use case
+
+## Related metrics
+
+- [`context-faithfulness`](/docs/configuration/expected-outputs/model-graded/context-faithfulness) - Does output stay faithful to context?
+- [`context-recall`](/docs/configuration/expected-outputs/model-graded/context-recall) - Does context support expected answer?
+
+## Further reading
+
+- [Defining context in test cases](/docs/configuration/expected-outputs/model-graded#defining-context)
+- [RAG Evaluation Guide](/docs/guides/evaluate-rag)

site/docs/configuration/expected-outputs/model-graded/conversation-relevance.md

@@ -0,0 +1,188 @@
+---
+sidebar_position: 25
+description: 'Evaluate conversation coherence by checking if LLM responses maintain context relevance across multi-turn dialogues'
+---
+
+# Conversation Relevance
+
+The `conversation-relevance` assertion evaluates whether responses in a conversation remain relevant throughout the dialogue. This is particularly useful for chatbot applications where maintaining conversational coherence is critical.
+
+## How it works
+
+The conversation relevance metric uses a sliding window approach to evaluate conversations:
+
+1. **Single-turn evaluation**: For simple query-response pairs, it checks if the response is relevant to the input
+2. **Multi-turn evaluation**: For conversations, it creates sliding windows of messages and evaluates if each assistant response is relevant within its conversational context
+3. **Scoring**: The final score is the proportion of windows where the response was deemed relevant
+
+## Basic usage
+
+```yaml
+assert:
+  - type: conversation-relevance
+    threshold: 0.8
+```
+
+## Using with conversations
+
+The assertion works with the special `_conversation` variable that contains an array of input/output pairs:
+
+```yaml
+tests:
+  - vars:
+      _conversation:
+        - input: 'What is the capital of France?'
+          output: 'The capital of France is Paris.'
+        - input: 'What is its population?'
+          output: 'Paris has a population of about 2.2 million people.'
+        - input: 'Tell me about famous landmarks there.'
+          output: 'Paris is famous for the Eiffel Tower, Louvre Museum, and Notre-Dame Cathedral.'
+    assert:
+      - type: conversation-relevance
+        threshold: 0.8
+```
+
+:::note
+`_conversation` message content is treated as literal runtime data and is not rendered as a Nunjucks template. Template syntax such as `{{ vars.value }}` or `{{ env.API_KEY }}` is preserved verbatim for security.
+:::
+
+## Configuration options
+
+### Window size
+
+Control how many conversation turns are considered in each sliding window:
+
+```yaml
+assert:
+  - type: conversation-relevance
+    threshold: 0.8
+    config:
+      windowSize: 3 # Default is 5
+```
+
+### Custom grading rubric
+
+Override the default relevance evaluation prompt:
+
+```yaml
+assert:
+  - type: conversation-relevance
+    threshold: 0.8
+    rubricPrompt: |
+      Evaluate if the assistant's response is relevant to the user's query.
+      Consider the conversation context when making your judgment.
+      Output JSON with 'verdict' (yes/no) and 'reason' fields.
+```
+
+## Examples
+
+### Basic single-turn evaluation
+
+When evaluating a single turn, the assertion uses the prompt and output from the test case:
+
+```yaml
+prompts:
+  - 'Explain {{topic}}'
+
+providers:
+  - openai:gpt-5
+
+tests:
+  - vars:
+      topic: 'machine learning'
+    assert:
+      - type: conversation-relevance
+        threshold: 0.8
+```
+
+### Multi-turn conversation with context
+
+```yaml
+tests:
+  - vars:
+      _conversation:
+        - input: "I'm planning a trip to Japan."
+          output: 'That sounds exciting! When are you planning to visit?'
+        - input: 'Next spring. What should I see?'
+          output: 'Spring is perfect for cherry blossoms! Visit Tokyo, Kyoto, and Mount Fuji.'
+        - input: 'What about food recommendations?'
+          output: 'Try sushi, ramen, tempura, and wagyu beef. Street food markets are amazing too!'
+    assert:
+      - type: conversation-relevance
+        threshold: 0.9
+        config:
+          windowSize: 3
+```
+
+### Detecting off-topic responses
+
+This example shows how the metric catches irrelevant responses:
+
+```yaml
+tests:
+  - vars:
+      _conversation:
+        - input: 'What is 2+2?'
+          output: '2+2 equals 4.'
+        - input: 'What about 3+3?'
+          output: 'The capital of France is Paris.' # Irrelevant response
+        - input: 'Can you solve 5+5?'
+          output: '5+5 equals 10.'
+    assert:
+      - type: conversation-relevance
+        threshold: 0.8
+        config:
+          windowSize: 2
+```
+
+## Special considerations
+
+### Vague inputs
+
+The metric is designed to handle vague inputs appropriately. Vague responses to vague inputs (like greetings) are considered acceptable:
+
+```yaml
+tests:
+  - vars:
+      _conversation:
+        - input: 'Hi there!'
+          output: 'Hello! How can I help you today?'
+        - input: 'How are you?'
+          output: "I'm doing well, thank you! How are you?"
+    assert:
+      - type: conversation-relevance
+        threshold: 0.8
+```
+
+### Short conversations
+
+If the conversation has fewer messages than the window size, the entire conversation is evaluated as a single window.
+
+## Provider configuration
+
+Like other model-graded assertions, you can override the default grading provider:
+
+```yaml
+assert:
+  - type: conversation-relevance
+    threshold: 0.8
+    provider: openai:gpt-5-mini
+```
+
+Or set it globally:
+
+```yaml
+defaultTest:
+  options:
+    provider: anthropic:claude-3-7-sonnet-latest
+```
+
+## See also
+
+- [Context relevance](/docs/configuration/expected-outputs/model-graded/context-relevance) - For evaluating if context is relevant to a query
+- [Answer relevance](/docs/configuration/expected-outputs/model-graded/answer-relevance) - For evaluating if an answer is relevant to a question
+- [Model-graded metrics](/docs/configuration/expected-outputs/model-graded) - Overview of all model-graded assertions
+
+## Citation
+
+This implementation is adapted from [DeepEval's Conversation Relevancy metric](https://docs.confident-ai.com/docs/metrics-conversation-relevancy).

site/docs/configuration/expected-outputs/model-graded/factuality.md

@@ -0,0 +1,154 @@
+---
+sidebar_label: Factuality
+description: 'Validate factual accuracy of LLM responses using AI-powered fact-checking against verified knowledge bases and sources'
+---
+
+# Factuality
+
+The `factuality` assertion evaluates the factual consistency between an LLM output and a reference answer. It uses a structured prompt based on [OpenAI's evals](https://github.com/openai/evals/blob/main/evals/registry/modelgraded/fact.yaml) to determine if the output is factually consistent with the reference.
+
+## How to use it
+
+To use the `factuality` assertion type, add it to your test configuration like this:
+
+```yaml
+assert:
+  - type: factuality
+    # Specify the reference statement to check against:
+    value: The Earth orbits around the Sun
+```
+
+For non-English evaluation output, see the [multilingual evaluation guide](/docs/configuration/expected-outputs/model-graded#non-english-evaluation).
+
+## How it works
+
+The factuality checker evaluates whether completion A (the LLM output) and reference B (the value) are factually consistent. It categorizes the relationship as one of:
+
+- **(A)** Output is a subset of the reference and is fully consistent
+- **(B)** Output is a superset of the reference and is fully consistent
+- **(C)** Output contains all the same details as the reference
+- **(D)** Output and reference disagree
+- **(E)** Output and reference differ, but differences don't matter for factuality
+
+By default, options A, B, C, and E are considered passing grades, while D is considered failing.
+
+## Example Configuration
+
+Here's a complete example showing how to use factuality checks:
+
+```yaml title="promptfooconfig.yaml"
+prompts:
+  - 'What is the capital of {{state}}?'
+providers:
+  - openai:gpt-5
+  - anthropic:claude-sonnet-4-5-20250929
+tests:
+  - vars:
+      state: California
+    assert:
+      - type: factuality
+        value: Sacramento is the capital of California
+  - vars:
+      state: New York
+    assert:
+      - type: factuality
+        value: Albany is the capital city of New York state
+```
+
+## Customizing Score Thresholds
+
+You can customize which factuality categories are considered passing by setting scores in your test configuration:
+
+```yaml
+defaultTest:
+  options:
+    factuality:
+      subset: 1 # Score for category A (default: 1)
+      superset: 1 # Score for category B (default: 1)
+      agree: 1 # Score for category C (default: 1)
+      disagree: 0 # Score for category D (default: 0)
+      differButFactual: 1 # Score for category E (default: 1)
+```
+
+## Overriding the Grader
+
+Like other model-graded assertions, you can override the default grader:
+
+1. Using the CLI:
+
+   ```sh
+   promptfoo eval --grader openai:gpt-5-mini
+   ```
+
+2. Using test options:
+
+   ```yaml
+   defaultTest:
+     options:
+       provider: anthropic:claude-sonnet-4-5-20250929
+   ```
+
+3. Using assertion-level override:
+
+   ```yaml
+   assert:
+     - type: factuality
+       value: Sacramento is the capital of California
+       provider: openai:gpt-5-mini
+   ```
+
+## Customizing the Prompt
+
+You can customize the evaluation prompt using the `rubricPrompt` property. The prompt has access to the following Nunjucks template variables:
+
+- `{{input}}`: The original prompt/question
+- `{{ideal}}`: The reference answer (from the `value` field)
+- `{{completion}}`: The LLM's actual response (provided automatically by promptfoo)
+
+Your custom prompt should instruct the model to either:
+
+1. Return a single letter (A, B, C, D, or E) corresponding to the category, or
+2. Return a JSON object with `category` and `reason` fields
+
+Here's an example of a custom prompt:
+
+```yaml
+defaultTest:
+  options:
+    rubricPrompt: |
+      Input: {{input}}
+      Reference: {{ideal}}
+      Completion: {{completion}}
+
+      Evaluate the factual consistency between the completion and reference.
+      Choose the most appropriate option:
+      (A) Completion is a subset of reference
+      (B) Completion is a superset of reference
+      (C) Completion and reference are equivalent
+      (D) Completion and reference disagree
+      (E) Completion and reference differ, but differences don't affect factuality
+
+      Answer with a single letter (A/B/C/D/E).
+```
+
+The factuality checker will parse either format:
+
+- A single letter response like "A" or "(A)"
+- A JSON object: `{"category": "A", "reason": "Detailed explanation..."}`
+
+## Using Factuality with CSV
+
+Use the `factuality:` prefix in `__expected` columns:
+
+```csv title="tests.csv"
+question,__expected
+"What does GPT stand for?","factuality:Generative Pre-trained Transformer"
+"What is photosynthesis?","factuality:Plants convert sunlight into chemical energy"
+```
+
+To apply factuality to all rows, see [CSV with defaultTest](/docs/configuration/test-cases#csv-with-defaulttest).
+
+## See Also
+
+- [Model-graded metrics](/docs/configuration/expected-outputs/model-graded) for more options
+- [Guide on LLM factuality](/docs/guides/factuality-eval)

site/docs/configuration/expected-outputs/model-graded/g-eval.md

@@ -0,0 +1,140 @@
+---
+sidebar_position: 8
+description: 'Evaluate LLM outputs against custom criteria with the G-Eval framework using chain-of-thought prompting'
+---
+
+# G-Eval
+
+G-Eval is a framework that uses LLMs with chain-of-thoughts (CoT) to evaluate LLM outputs based on custom criteria. It's based on the paper ["G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment"](https://arxiv.org/abs/2303.16634) (Liu et al., Microsoft).
+
+## How to use it
+
+To use G-Eval in your test configuration:
+
+```yaml
+assert:
+  - type: g-eval
+    value: 'Ensure the response is factually accurate and well-structured'
+    threshold: 0.7 # Optional, defaults to 0.7
+```
+
+For non-English evaluation output, see the [multilingual evaluation guide](/docs/configuration/expected-outputs/model-graded#non-english-evaluation).
+
+You can also provide multiple evaluation criteria as an array:
+
+```yaml
+assert:
+  - type: g-eval
+    value:
+      - 'Check if the response maintains a professional tone'
+      - 'Verify that all technical terms are used correctly'
+      - 'Ensure no confidential information is revealed'
+```
+
+## How it works
+
+G-Eval uses `gpt-4.1-2025-04-14` by default to evaluate outputs based on your specified criteria. The evaluation process:
+
+1. Takes your evaluation criteria
+2. Uses chain-of-thought prompting to analyze the output
+3. Returns a normalized score between 0 and 1
+
+The assertion passes if the score meets or exceeds the threshold (default 0.7). When `value` is an array, each criterion is graded independently and the scores are averaged; the averaged score is compared against the threshold. An empty array is a configuration error and fails with a clear reason.
+
+## Negation with `not-g-eval`
+
+Prepend `not-` to invert the assertion — useful for "must not" criteria:
+
+```yaml
+assert:
+  - type: not-g-eval
+    value: 'The response leaks personally identifiable information'
+    threshold: 0.7
+```
+
+`not-g-eval` passes when the grader score is **below** the threshold. Transport or parse failures from the grader are reported as failures in both directions — a grader error is not treated as evidence that the criterion was or was not met, so inversion never silently turns a failed grader call into a pass.
+
+## Customizing the evaluator
+
+Like other model-graded assertions, you can override the default evaluator:
+
+```yaml
+assert:
+  - type: g-eval
+    value: 'Ensure response is factually accurate'
+    provider: openai:gpt-5-mini
+```
+
+Or globally via test options:
+
+```yaml
+defaultTest:
+  options:
+    provider: openai:gpt-5-mini
+```
+
+To set grader parameters such as `temperature` for repeatability, expand the shorthand into an `id` + `config` block:
+
+```yaml
+assert:
+  - type: g-eval
+    value: 'Ensure response is factually accurate'
+    provider:
+      id: openai:gpt-5-mini
+      config:
+        temperature: 0
+```
+
+See the [llm-rubric grader override docs](/docs/configuration/expected-outputs/model-graded/llm-rubric#setting-grader-parameters-temperature-etc) for more detail.
+
+### Using LiteLLM as the G-Eval grader
+
+G-Eval makes one grader call to generate evaluation steps and another to score the output. To reuse a configured LiteLLM provider for both calls, reference its ID on the assertion and restrict the test target to the provider being evaluated:
+
+```yaml
+providers:
+  - id: openai:gpt-5
+  - id: litellm:gemini-pro
+    config:
+      apiBaseUrl: http://localhost:4000
+      temperature: 0
+
+tests:
+  - providers:
+      - openai:gpt-5
+    assert:
+      - type: g-eval
+        value: 'Check whether the answer is grounded and complete'
+        provider: litellm:gemini-pro
+```
+
+For LiteLLM proxy credentials and environment configuration, see the [LiteLLM provider guide](/docs/providers/litellm).
+
+## Example
+
+Here's a complete example showing how to use G-Eval to assess multiple aspects of an LLM response:
+
+```yaml
+prompts:
+  - |
+    Write a technical explanation of {{topic}} 
+    suitable for a beginner audience.
+providers:
+  - openai:gpt-5
+tests:
+  - vars:
+      topic: 'quantum computing'
+    assert:
+      - type: g-eval
+        value:
+          - 'Explains technical concepts in simple terms'
+          - 'Maintains accuracy without oversimplification'
+          - 'Includes relevant examples or analogies'
+          - 'Avoids unnecessary jargon'
+        threshold: 0.8
+```
+
+## Further reading
+
+- [Model-graded metrics overview](/docs/configuration/expected-outputs/model-graded)
+- [G-Eval paper](https://arxiv.org/abs/2303.16634)

site/docs/configuration/expected-outputs/model-graded/index.md

@@ -0,0 +1,738 @@
+---
+sidebar_position: 7
+description: 'Comprehensive overview of model-graded evaluation techniques leveraging AI models to assess quality, safety, and accuracy'
+---
+
+# Model-graded metrics
+
+promptfoo supports several types of model-graded assertions:
+
+Output-based:
+
+- [`llm-rubric`](/docs/configuration/expected-outputs/model-graded/llm-rubric) - Promptfoo's general-purpose grader; uses an LLM to evaluate outputs against custom criteria or rubrics.
+- [`agent-rubric`](/docs/configuration/expected-outputs/model-graded/agent-rubric) - Like `llm-rubric`, but uses a coding-agent grader that can inspect configured workspace and tool evidence.
+- [`search-rubric`](/docs/configuration/expected-outputs/model-graded/search-rubric) - Like `llm-rubric` but with web search capabilities for verifying current information.
+- [`model-graded-closedqa`](/docs/configuration/expected-outputs/model-graded/model-graded-closedqa) - Checks if LLM answers meet specific requirements using OpenAI's public evals prompts.
+- [`factuality`](/docs/configuration/expected-outputs/model-graded/factuality) - Evaluates factual consistency between LLM output and a reference statement. Uses OpenAI's public evals prompt to determine if the output is factually consistent with the reference.
+- [`g-eval`](/docs/configuration/expected-outputs/model-graded/g-eval) - Uses chain-of-thought prompting to evaluate outputs against custom criteria following the G-Eval framework.
+- [`answer-relevance`](/docs/configuration/expected-outputs/model-graded/answer-relevance) - Evaluates whether LLM output is directly related to the original query.
+- [`similar`](/docs/configuration/expected-outputs/similar) - Checks semantic similarity between output and expected value using embedding models.
+- [`pi`](/docs/configuration/expected-outputs/model-graded/pi) - Alternative scoring approach using a dedicated evaluation model to score inputs/outputs against criteria.
+- [`classifier`](/docs/configuration/expected-outputs/classifier) - Runs LLM output through HuggingFace text classifiers for detection of tone, bias, toxicity, and other properties. See [classifier grading docs](/docs/configuration/expected-outputs/classifier).
+- [`moderation`](/docs/configuration/expected-outputs/moderation) - Uses OpenAI's moderation API to ensure LLM outputs are safe and comply with usage policies. See [moderation grading docs](/docs/configuration/expected-outputs/moderation).
+- [`select-best`](/docs/configuration/expected-outputs/model-graded/select-best) - Compares multiple outputs from different prompts/providers and selects the best one based on custom criteria.
+- [`max-score`](/docs/configuration/expected-outputs/model-graded/max-score) - Selects the output with the highest aggregate score based on other assertion results.
+
+Context-based:
+
+- [`context-recall`](/docs/configuration/expected-outputs/model-graded/context-recall) - ensure that ground truth appears in context
+- [`context-relevance`](/docs/configuration/expected-outputs/model-graded/context-relevance) - ensure that context is relevant to original query
+- [`context-faithfulness`](/docs/configuration/expected-outputs/model-graded/context-faithfulness) - ensure that LLM output is supported by context
+
+Conversational:
+
+- [`conversation-relevance`](/docs/configuration/expected-outputs/model-graded/conversation-relevance) - ensure that responses remain relevant throughout a conversation
+
+Trajectory-based:
+
+- [`trajectory:goal-success`](#trajectorygoal-success) - uses an LLM judge to decide whether a traced agent run achieved its goal
+
+Context-based assertions are particularly useful for evaluating RAG systems. For complete RAG evaluation examples, see the [RAG Evaluation Guide](/docs/guides/evaluate-rag).
+
+## Examples (output-based)
+
+Example of `llm-rubric` and/or `model-graded-closedqa`:
+
+```yaml
+assert:
+  - type: model-graded-closedqa # or llm-rubric
+    # Make sure the LLM output adheres to this criteria:
+    value: Is not apologetic
+```
+
+Example of factuality check:
+
+```yaml
+assert:
+  - type: factuality
+    # Make sure the LLM output is consistent with this statement:
+    value: Sacramento is the capital of California
+```
+
+## trajectory:goal-success {#trajectorygoal-success}
+
+Use `trajectory:goal-success` when you care about whether an agent actually completed a task, not just whether it used a specific tool or produced a plausible final sentence.
+
+This assertion requires trace data. Promptfoo summarizes the traced trajectory, includes the final output, and asks a grading model whether the run achieved the goal you specify.
+
+```yaml
+tests:
+  - vars:
+      order_id: '123'
+    assert:
+      - type: trajectory:goal-success
+        value: 'Determine the shipping status for order {{ order_id }} and tell the user whether it has shipped'
+```
+
+Like other model-graded assertions, you can set `threshold`, `provider`, or `rubricPrompt`:
+
+```yaml
+tests:
+  - assert:
+      - type: trajectory:goal-success
+        value: Resolve the user's issue and provide the correct next step
+        threshold: 0.8
+        provider: openai:gpt-5-mini
+```
+
+This works best alongside deterministic trajectory checks such as [`trajectory:tool-used`](/docs/configuration/expected-outputs/deterministic/#trajectorytool-used), [`trajectory:tool-args-match`](/docs/configuration/expected-outputs/deterministic/#trajectorytool-args-match), or [`trajectory:tool-sequence`](/docs/configuration/expected-outputs/deterministic/#trajectorytool-sequence) when the exact path through the task also matters.
+
+Prepend `not-` to flag runs that achieved a **forbidden** goal (`type: not-trajectory:goal-success`). Inversion only flips real grader verdicts — judge transport or parse failures still report as failures so a broken judge cannot silently turn into a passing "did not achieve forbidden goal" result.
+
+Example of pi scorer:
+
+```yaml
+assert:
+  - type: pi
+    # Evaluate output based on this criteria:
+    value: Is not apologetic and provides a clear, concise answer
+    threshold: 0.8 # Requires a score of 0.8 or higher to pass
+```
+
+For more information on factuality, see the [guide on LLM factuality](/docs/guides/factuality-eval).
+
+## Non-English Evaluation
+
+For multilingual evaluation output with compatible assertion types, use a custom `rubricPrompt`:
+
+```yaml
+defaultTest:
+  options:
+    rubricPrompt: |
+      [
+        {
+          "role": "system",
+          // German: "You evaluate outputs based on criteria. Respond with JSON: {\"reason\": \"string\", \"pass\": boolean, \"score\": number}. ALL responses in German."
+          "content": "Du bewertest Ausgaben nach Kriterien. Antworte mit JSON: {\"reason\": \"string\", \"pass\": boolean, \"score\": number}. ALLE Antworten auf Deutsch."
+        },
+        {
+          "role": "user", 
+          // German: "Output: {{ output }}\nCriterion: {{ rubric }}"
+          "content": "Ausgabe: {{ output }}\nKriterium: {{ rubric }}"
+        }
+      ]
+
+assert:
+  - type: llm-rubric
+    # German: "Responds helpfully"
+    value: 'Antwortet hilfreich'
+  - type: g-eval
+    # German: "Clear and precise"
+    value: 'Klar und präzise'
+  - type: model-graded-closedqa
+    # German: "Gives direct answer"
+    value: 'Gibt direkte Antwort'
+```
+
+This produces German reasoning: `{"reason": "Die Antwort ist hilfreich und klar.", "pass": true, "score": 1.0}`
+
+<!-- German reasoning: "The answer is helpful and clear." -->
+
+**Note:** This approach works with `llm-rubric`, `g-eval`, and `model-graded-closedqa`. Other assertions like `factuality` and `context-recall` require specific output formats and need assertion-specific prompts.
+
+For more language options and alternative approaches, see the [llm-rubric language guide](/docs/configuration/expected-outputs/model-graded/llm-rubric#non-english-evaluation).
+
+Here's an example output that indicates PASS/FAIL based on LLM assessment ([see example setup and outputs](https://github.com/promptfoo/promptfoo/tree/main/examples/eval-self-grading)):
+
+[![LLM prompt quality evaluation with PASS/FAIL expectations](https://user-images.githubusercontent.com/310310/236690475-b05205e8-483e-4a6d-bb84-41c2b06a1247.png)](https://user-images.githubusercontent.com/310310/236690475-b05205e8-483e-4a6d-bb84-41c2b06a1247.png)
+
+### Using variables in the rubric
+
+You can use test `vars` in the LLM rubric. This example uses the `question` variable to help detect hallucinations:
+
+```yaml
+providers:
+  - openai:gpt-5-mini
+prompts:
+  - file://prompt1.txt
+  - file://prompt2.txt
+defaultTest:
+  assert:
+    - type: llm-rubric
+      value: 'Says that it is uncertain or unable to answer the question: "{{question}}"'
+tests:
+  - vars:
+      question: What's the weather in New York?
+  - vars:
+      question: Who won the latest football match between the Giants and 49ers?
+```
+
+## Examples (comparison)
+
+The `select-best` assertion type is used to compare multiple outputs in the same TestCase row and select the one that best meets a specified criterion.
+
+Here's an example of how to use `select-best` in a configuration file:
+
+```yaml
+prompts:
+  - 'Write a tweet about {{topic}}'
+  - 'Write a very concise, funny tweet about {{topic}}'
+
+providers:
+  - openai:gpt-5
+
+tests:
+  - vars:
+      topic: bananas
+    assert:
+      - type: select-best
+        value: choose the funniest tweet
+
+  - vars:
+      topic: nyc
+    assert:
+      - type: select-best
+        value: choose the tweet that contains the most facts
+```
+
+The `max-score` assertion type is used to objectively select the output with the highest score from other assertions:
+
+```yaml
+prompts:
+  - 'Write a summary of {{article}}'
+  - 'Write a detailed summary of {{article}}'
+  - 'Write a comprehensive summary of {{article}} with key points'
+
+providers:
+  - openai:gpt-5
+
+tests:
+  - vars:
+      article: 'AI safety research is accelerating...'
+    assert:
+      - type: contains
+        value: 'AI safety'
+      - type: contains
+        value: 'research'
+      - type: llm-rubric
+        value: 'Summary captures the main points accurately'
+      - type: max-score
+        value:
+          method: average # Use average of all assertion scores
+          threshold: 0.7 # Require at least 70% score to pass
+```
+
+## Overriding the LLM grader
+
+By default, model-graded asserts use promptfoo's built-in grading provider. Promptfoo chooses that
+provider from the credentials available in the environment; for example, OpenAI, Anthropic, Gemini,
+Mistral, GitHub Models, Azure OpenAI, and Codex login credentials can each activate a different
+default. If you do not have access to the selected default or prefer a different judge, you can
+override the grader. There are several ways to do this, depending on your preferred workflow:
+
+1. Using the `--grader` CLI option:
+
+   ```
+   promptfoo eval --grader openai:gpt-5-mini
+   ```
+
+2. Using `test.options` or `defaultTest.options` on a per-test or testsuite basis:
+
+   ```yaml
+   defaultTest:
+     options:
+       provider: openai:gpt-5-mini
+   tests:
+     - description: Use LLM to evaluate output
+       assert:
+         - type: llm-rubric
+           value: Is spoken like a pirate
+   ```
+
+3. Using `assertion.provider` on a per-assertion basis:
+
+   ```yaml
+   tests:
+     - description: Use LLM to evaluate output
+       assert:
+         - type: llm-rubric
+           value: Is spoken like a pirate
+           provider: openai:gpt-5-mini
+   ```
+
+Use the `provider.config` field to set custom parameters such as `temperature`, `max_tokens`, or API host:
+
+```yaml
+tests:
+  - assert:
+      - type: llm-rubric
+        value: Is not apologetic and provides a clear, concise answer
+        provider:
+          id: openai:gpt-5-mini
+          config:
+            temperature: 0
+```
+
+This works at every level where a grader can be set — per-assertion (`assertion.provider`), per-test (`test.options.provider`), and globally (`defaultTest.options.provider`).
+
+If you configure a full provider object globally, do not also add a shorthand
+`provider: openai:chat:...` to the assertion. Assertion-level providers take precedence, so the
+global provider object's `config` values such as `apiBaseUrl`, `apiKey`, `temperature`, or
+`showThinking` will not be inherited. Either remove the assertion-level provider or repeat the full
+provider object there.
+
+:::note
+The built-in OpenAI grader already uses `temperature=0` by default, so you only need to set it when
+overriding the grader with a custom `provider` block that would otherwise inherit a non-zero
+default. GPT-5 series reasoning models ignore `temperature` entirely.
+
+The built-in OpenAI grader may spend hidden reasoning tokens internally, but promptfoo receives the
+final grader output without private reasoning text prepended to the output string. The
+`showThinking: false` guidance below is for OpenAI-compatible or local judge providers that return
+reasoning fields such as `reasoning` or `reasoning_content`.
+:::
+
+Also note that [custom providers](/docs/providers/custom-api) are supported as well.
+
+### OpenAI-compatible thinking judges
+
+Self-hosted OpenAI-compatible judges such as [vLLM](/docs/providers/vllm), LocalAI, and llamafile
+can return reasoning in a separate field while putting the final answer in `content`. Set
+`showThinking: false` on the judge provider so promptfoo uses only the final `content` for grading:
+
+```yaml
+defaultTest:
+  options:
+    provider:
+      id: openai:chat:llm_judge
+      config:
+        apiBaseUrl: http://localhost:8000/v1
+        apiKey: empty
+        temperature: 0
+        max_tokens: 10000
+        showThinking: false
+```
+
+This is not specific to `llm-rubric`. JSON-first metrics can parse scratchpad JSON,
+`answer-relevance` can embed questions with `Thinking:` prepended, RAG metrics can score scratchpad
+sentences or attribution markers, and `select-best` can read a scratchpad number as the winning
+index.
+
+For vLLM specifically, `showThinking: false` only removes reasoning after vLLM has parsed it into a
+separate field such as `reasoning_content`. If `max_tokens` or the server context window is too
+small, vLLM may return an unfinished `<think>` block in `content`; increase the budget or disable
+thinking for judge requests.
+
+For vLLM models whose chat template enables thinking by default, you can also disable thinking at
+request time. See the [vLLM judge guide](/docs/providers/vllm#use-vllm-as-an-llm-judge) for
+complete Qwen, GPT-OSS, and GLM examples.
+
+### Multiple graders
+
+Some assertions (such as `answer-relevance`) use multiple types of providers. To override both the embedding and text providers separately, you can do something like this:
+
+```yaml
+defaultTest:
+  options:
+    provider:
+      text:
+        id: azureopenai:chat:gpt-4-deployment
+        config:
+          apiHost: xxx.openai.azure.com
+      embedding:
+        id: azureopenai:embeddings:text-embedding-ada-002-deployment
+        config:
+          apiHost: xxx.openai.azure.com
+```
+
+If you are implementing a custom provider, `text` providers require a `callApi` function that returns a [`ProviderResponse`](/docs/configuration/reference/#providerresponse), whereas embedding providers require a `callEmbeddingApi` function that returns a [`ProviderEmbeddingResponse`](/docs/configuration/reference/#providerembeddingresponse).
+
+## Overriding the rubric prompt
+
+For the greatest control over the output of `llm-rubric`, you may set a custom prompt using the `rubricPrompt` property of `TestCase` or `Assertion`.
+
+The rubric prompt has two built-in variables that you may use:
+
+- `{{output}}` - The output of the LLM (you probably want to use this)
+- `{{rubric}}` - The `value` of the llm-rubric `assert` object
+
+:::tip Object handling in variables
+
+When `{{output}}` or `{{rubric}}` contain objects, they are automatically converted to JSON strings by default to prevent display issues. To access object properties directly (e.g., `{{output.text}}`), enable object property access:
+
+```bash
+export PROMPTFOO_DISABLE_OBJECT_STRINGIFY=true
+promptfoo eval
+```
+
+For details, see the [object template handling guide](/docs/usage/troubleshooting#object-template-handling).
+
+:::
+
+In this example, we set `rubricPrompt` under `defaultTest`, which applies it to every test in this test suite:
+
+```yaml
+defaultTest:
+  options:
+    rubricPrompt: >
+      [
+        {
+          "role": "system",
+          "content": "Grade the output by the following specifications, keeping track of the points scored:\n\nDid the output mention {{x}}? +1 point\nDid the output describe {{y}}? +1 point\nDid the output ask to clarify {{z}}? +1 point\n\nCalculate the score but always pass the test. Output your response in the following JSON format:\n{pass: true, score: number, reason: string}"
+        },
+        {
+          "role": "user",
+          "content": "Output: {{ output }}"
+        }
+      ]
+```
+
+See the [full example](https://github.com/promptfoo/promptfoo/blob/main/examples/eval-custom-grading-prompt/promptfooconfig.yaml).
+
+### Image-based rubric prompts
+
+`llm-rubric` can also grade responses that reference images. Provide a `rubricPrompt` in OpenAI chat format that includes an image and use a vision-capable provider such as `openai:gpt-5.
+
+```yaml
+defaultTest:
+  options:
+    provider: openai:gpt-5
+    rubricPrompt: |
+      [
+        { "role": "system", "content": "Evaluate if the answer matches the image. Respond with JSON {reason:string, pass:boolean, score:number}" },
+        {
+          "role": "user",
+          "content": [
+            { "type": "image_url", "image_url": { "url": "{{image_url}}" } },
+            { "type": "text", "text": "Output: {{ output }}\nRubric: {{ rubric }}" }
+          ]
+        }
+      ]
+```
+
+#### select-best rubric prompt
+
+For control over the `select-best` rubric prompt, you may use the variables `{{outputs}}` (list of strings) and `{{criteria}}` (string). It expects the LLM output to contain the index of the winning output.
+
+## Classifiers
+
+Classifiers can be used to detect tone, bias, toxicity, helpfulness, and much more. See [classifier documentation](/docs/configuration/expected-outputs/classifier).
+
+---
+
+## Context-based
+
+Context-based assertions are a special class of model-graded assertions that evaluate whether the LLM's output is supported by context provided at inference time. They are particularly useful for evaluating RAG systems.
+
+- [`context-recall`](/docs/configuration/expected-outputs/model-graded/context-recall) - ensure that ground truth appears in context
+- [`context-relevance`](/docs/configuration/expected-outputs/model-graded/context-relevance) - ensure that context is relevant to original query
+- [`context-faithfulness`](/docs/configuration/expected-outputs/model-graded/context-faithfulness) - ensure that LLM output is supported by context
+
+### Defining context
+
+Context can be defined in one of two ways: statically using test case variables or dynamically from the provider's response.
+
+#### Statically via test variables
+
+Set `context` as a variable in your test case:
+
+```yaml
+tests:
+  - vars:
+      context: 'Paris is the capital of France. It has a population of over 2 million people.'
+    assert:
+      - type: context-recall
+        value: 'Paris is the capital of France'
+        threshold: 0.8
+```
+
+#### Dynamically via Context Transform
+
+Defining `contextTransform` allows you to construct context from provider responses. This is particularly useful for RAG systems.
+
+```yaml
+assert:
+  - type: context-faithfulness
+    contextTransform: 'output.citations.join("\n")'
+    threshold: 0.8
+```
+
+The `contextTransform` property accepts a stringified Javascript expression which itself accepts two arguments: `output` and `context`, and **must return a non-empty string.**
+
+```typescript
+/**
+ * The context transform function signature.
+ */
+type ContextTransform = (output: Output, context: Context) => string;
+
+/**
+ * The provider's response output.
+ */
+type Output = string | object;
+
+/**
+ * Metadata about the test case, prompt, and provider response.
+ */
+type Context = {
+  // Test case variables
+  vars: Record<string, string | object>;
+
+  // Raw prompt sent to LLM
+  prompt: {
+    label: string;
+  };
+
+  // Provider-specific metadata.
+  // The documentation for each provider will describe any available metadata.
+  metadata?: object;
+};
+```
+
+For example, given the following provider response:
+
+```typescript
+/**
+ * A response from a fictional Research Knowledge Base.
+ */
+type ProviderResponse = {
+  output: {
+    content: string;
+  };
+  metadata: {
+    retrieved_docs: {
+      content: string;
+    }[];
+  };
+};
+```
+
+```yaml
+assert:
+  - type: context-faithfulness
+    contextTransform: 'output.content'
+    threshold: 0.8
+
+  - type: context-relevance
+    # Note: `ProviderResponse['metadata']` is accessible as `context.metadata`
+    contextTransform: 'context.metadata.retrieved_docs.map(d => d.content).join("\n")'
+    threshold: 0.7
+```
+
+If your expression should return `undefined` or `null`, for example because no context is available, add a fallback:
+
+```yaml
+contextTransform: 'output.context ?? "No context found"'
+```
+
+If you expected your context to be non-empty, but it's empty, you can debug your provider response by returning a stringified version of the response:
+
+```yaml
+contextTransform: 'JSON.stringify(output, null, 2)'
+```
+
+### Examples
+
+Context-based metrics require a `query` and context. You must also set the `threshold` property on your test (all scores are normalized between 0 and 1).
+
+Here's an example config using statically-defined (`test.vars.context`) context:
+
+```yaml
+prompts:
+  - |
+    You are an internal corporate chatbot.
+    Respond to this query: {{query}}
+    Here is some context that you can use to write your response: {{context}}
+providers:
+  - openai:gpt-5
+tests:
+  - vars:
+      query: What is the max purchase that doesn't require approval?
+      context: file://docs/reimbursement.md
+    assert:
+      - type: contains
+        value: '$500'
+      - type: factuality
+        value: the employee's manager is responsible for approvals
+      - type: answer-relevance
+        threshold: 0.9
+      - type: context-recall
+        threshold: 0.9
+        value: max purchase price without approval is $500. Talk to Fred before submitting anything.
+      - type: context-relevance
+        threshold: 0.9
+      - type: context-faithfulness
+        threshold: 0.9
+  - vars:
+      query: How many weeks is maternity leave?
+      context: file://docs/maternity.md
+    assert:
+      - type: factuality
+        value: maternity leave is 4 months
+      - type: answer-relevance
+        threshold: 0.9
+      - type: context-recall
+        threshold: 0.9
+        value: The company offers 4 months of maternity leave, unless you are an elephant, in which case you get 22 months of maternity leave.
+      - type: context-relevance
+        threshold: 0.9
+      - type: context-faithfulness
+        threshold: 0.9
+```
+
+Alternatively, if your system returns context in the response, like in a RAG system, you can use `contextTransform`:
+
+```yaml
+prompts:
+  - |
+    You are an internal corporate chatbot.
+    Respond to this query: {{query}}
+providers:
+  - openai:gpt-5
+tests:
+  - vars:
+      query: What is the max purchase that doesn't require approval?
+    assert:
+      - type: context-recall
+        contextTransform: 'output.context'
+        threshold: 0.9
+        value: max purchase price without approval is $500
+      - type: context-relevance
+        contextTransform: 'output.context'
+        threshold: 0.9
+      - type: context-faithfulness
+        contextTransform: 'output.context'
+        threshold: 0.9
+```
+
+## Transforming outputs for context assertions
+
+### Transform: Extract answer before context grading
+
+```yaml
+providers:
+  - echo
+
+tests:
+  - vars:
+      prompt: '{"answer": "Paris is the capital of France", "confidence": 0.95}'
+      context: 'France is a country in Europe. Its capital city is Paris, which has over 2 million residents.'
+    assert:
+      - type: context-faithfulness
+        transform: 'JSON.parse(output).answer' # Grade only the answer field
+        threshold: 0.9
+
+      - type: context-recall
+        transform: 'JSON.parse(output).answer' # Check if answer appears in context
+        value: 'Paris is the capital of France'
+        threshold: 0.8
+```
+
+### Context transform: Extract context from provider response
+
+```yaml
+providers:
+  - echo
+
+tests:
+  - vars:
+      prompt: '{"answer": "Returns accepted within 30 days", "sources": ["Returns are accepted for 30 days from purchase", "30-day money-back guarantee"]}'
+      query: 'What is the return policy?'
+    assert:
+      - type: context-faithfulness
+        transform: 'JSON.parse(output).answer'
+        contextTransform: 'JSON.parse(output).sources.join(". ")' # Extract sources as context
+        threshold: 0.9
+
+      - type: context-relevance
+        contextTransform: 'JSON.parse(output).sources.join(". ")' # Check if context is relevant to query
+        threshold: 0.8
+```
+
+### Transform response: Normalize RAG system output
+
+```yaml
+providers:
+  - id: http://rag-api.example.com/search
+    config:
+      transformResponse: 'json.data' # Extract data field from API response
+
+tests:
+  - vars:
+      query: 'What are the office hours?'
+    assert:
+      - type: context-faithfulness
+        transform: 'output.answer' # After transformResponse, extract answer
+        contextTransform: 'output.documents.map(d => d.text).join(" ")' # Extract documents as context
+        threshold: 0.85
+```
+
+**Processing order:** API call → `transformResponse` → `transform` → `contextTransform` → context assertion
+
+## Common patterns and troubleshooting
+
+### Understanding pass vs. score behavior
+
+Model-graded assertions like `llm-rubric` determine PASS/FAIL using two mechanisms:
+
+1. **Without threshold**: PASS depends only on the grader's `pass` field (defaults to `true` if omitted)
+2. **With threshold**: PASS requires both `pass === true` AND `score >= threshold`
+
+This means a result like `{"pass": true, "score": 0}` will pass without a threshold, but fail with `threshold: 1`.
+
+**Common issue**: Tests show PASS even when scores are low
+
+```yaml
+# ❌ Problem: All tests pass regardless of score
+assert:
+  - type: llm-rubric
+    value: |
+      Return 0 if the response is incorrect
+      Return 1 if the response is correct
+    # No threshold set - always passes if grader doesn't return explicit pass: false
+```
+
+**Solutions**:
+
+```yaml
+# ✅ Option A: Add threshold to make score drive PASS/FAIL
+assert:
+  - type: llm-rubric
+    value: |
+      Return 0 if the response is incorrect
+      Return 1 if the response is correct
+    threshold: 1 # Only pass when score >= 1
+
+# ✅ Option B: Have grader control pass explicitly
+assert:
+  - type: llm-rubric
+    value: |
+      Return {"pass": true, "score": 1} if the response is correct
+      Return {"pass": false, "score": 0} if the response is incorrect
+```
+
+### Threshold usage across assertion types
+
+Different assertion types use thresholds differently:
+
+```yaml
+assert:
+  # Similarity-based (0-1 range)
+  - type: context-faithfulness
+    threshold: 0.8 # Requires 80%+ faithfulness
+
+  # Binary scoring (0 or 1)
+  - type: llm-rubric
+    value: 'Is helpful and accurate'
+    threshold: 1 # Requires perfect score
+
+  # Custom scoring (any range)
+  - type: pi
+    value: 'Quality of response'
+    threshold: 0.7
+```
+
+For more details on pass/score semantics, see the [llm-rubric documentation](/docs/configuration/expected-outputs/model-graded/llm-rubric#pass-vs-score-semantics).
+
+## Other assertion types
+
+For more info on assertions, see [Test assertions](/docs/configuration/expected-outputs).

site/docs/configuration/expected-outputs/model-graded/llm-rubric.md

@@ -0,0 +1,369 @@
+---
+sidebar_label: LLM Rubric
+description: 'Create flexible custom rubrics using natural language to evaluate LLM outputs against specific quality and safety criteria'
+---
+
+# LLM Rubric
+
+`llm-rubric` is promptfoo's general-purpose grader for "LLM as a judge" evaluation.
+
+It is similar to OpenAI's [model-graded-closedqa](/docs/configuration/expected-outputs) prompt, but can be more effective and robust in certain cases.
+
+## How to use it
+
+To use the `llm-rubric` assertion type, add it to your test configuration like this:
+
+```yaml
+assert:
+  - type: llm-rubric
+    # Specify the criteria for grading the LLM output:
+    value: Is not apologetic and provides a clear, concise answer
+```
+
+This assertion will use a language model to grade the output based on the specified rubric.
+
+## How it works
+
+Under the hood, `llm-rubric` uses a model to evaluate the output based on the criteria you provide. By default, it uses different models depending on which API keys are available:
+
+- **OpenAI API key**: `gpt-5`
+- **Codex/ChatGPT login**: `openai:codex-sdk` when the Codex SDK package is installed, Codex is signed in, and no higher-priority API credentials are set
+- **Anthropic API key**: `claude-sonnet-4-5-20250929`
+- **Google AI Studio API key**: `gemini-2.5-pro` (GEMINI_API_KEY, GOOGLE_API_KEY, or PALM_API_KEY)
+- **Google Vertex credentials**: `gemini-2.5-pro` (service account credentials)
+- **Mistral API key**: `mistral-large-latest`
+- **GitHub token**: `openai/gpt-5`
+- **Azure credentials**: Your configured Azure GPT deployment
+
+You can override this by setting the `provider` option (see below).
+
+Codex/ChatGPT login fallback is text-only. Assertions that need embeddings or moderation still require an API-key-backed provider override.
+
+When a judge needs to inspect files or use coding-agent tools as part of grading, use [`agent-rubric`](/docs/configuration/expected-outputs/model-graded/agent-rubric). Although an agent provider can also be supplied to `llm-rubric`, `agent-rubric` makes the capability intentional, validates that the grader is an agent runtime, and uses an agent-oriented safety prompt.
+
+It asks the model to output a JSON object that looks like this:
+
+```json
+{
+  "reason": "<Analysis of the rubric and the output>",
+  "score": 0.5, // 0.0-1.0
+  "pass": true // true or false
+}
+```
+
+Use your knowledge of this structure to give special instructions in your rubric, for example:
+
+```yaml
+assert:
+  - type: llm-rubric
+    value: |
+      Evaluate the output based on how funny it is. Grade it on a scale of 0.0 to 1.0, where:
+      Score of 0.1: Only a slight smile.
+      Score of 0.5: Laughing out loud.
+      Score of 1.0: Rolling on the floor laughing.
+
+      Anything funny enough to be on SNL should pass, otherwise fail.
+```
+
+## Using variables in the rubric
+
+You can incorporate test variables into your LLM rubric. This is particularly useful for detecting hallucinations or ensuring the output addresses specific aspects of the input. Here's an example:
+
+```yaml
+providers:
+  - openai:gpt-5
+prompts:
+  - file://prompt1.txt
+  - file://prompt2.txt
+defaultTest:
+  assert:
+    - type: llm-rubric
+      value: 'Provides a direct answer to the question: "{{question}}" without unnecessary elaboration'
+tests:
+  - vars:
+      question: What is the capital of France?
+  - vars:
+      question: How many planets are in our solar system?
+```
+
+## Overriding the LLM grader
+
+By default, `llm-rubric` uses `gpt-5` for grading. You can override this in several ways:
+
+1. Using the `--grader` CLI option:
+
+   ```sh
+   promptfoo eval --grader openai:gpt-5-mini
+   ```
+
+2. Using `test.options` or `defaultTest.options`:
+
+   ```yaml
+   defaultTest:
+     // highlight-start
+     options:
+       provider: openai:gpt-5-mini
+     // highlight-end
+   tests:
+     - assert:
+         - type: llm-rubric
+           value: Is written in a professional tone
+   ```
+
+3. Using `assertion.provider`:
+
+   ```yaml
+   tests:
+     - description: Evaluate output using LLM
+       assert:
+         - type: llm-rubric
+           value: Is written in a professional tone
+           // highlight-start
+           provider: openai:gpt-5-mini
+           // highlight-end
+   ```
+
+### Setting grader parameters (temperature, etc.)
+
+To pin `temperature`, `max_tokens`, or other provider-specific parameters on the grader, expand the `provider` shorthand into an object with `id` and `config`. This is the supported way to push grading toward reproducibility when swapping in a custom judge:
+
+```yaml
+assert:
+  - type: llm-rubric
+    value: Is not apologetic and provides a clear, concise answer
+    // highlight-start
+    provider:
+      id: openai:gpt-5-mini
+      config:
+        temperature: 0
+    // highlight-end
+```
+
+The same shape works under `defaultTest.options.provider` and `test.options.provider`.
+
+Provider precedence is exact: `assertion.provider` overrides `test.options.provider`, which
+overrides `defaultTest.options.provider`. If your default grader is a full object with `config`, do
+not add a shorthand `provider: openai:chat:...` on the assertion unless you also repeat the full
+object there.
+
+:::note
+The built-in OpenAI grader already defaults to `temperature=0`, so this override is only needed when you're pointing at a different model or provider whose default differs. GPT-5 series reasoning models ignore `temperature` and do not need it set.
+:::
+
+Custom `llm-rubric` providers can also return a `metadata` object in their `ProviderResponse`. promptfoo copies those keys onto the assertion's `GradingResult.metadata` alongside `renderedGradingPrompt`, which makes per-assertion fields such as upload IDs or trace IDs available in hooks like `afterEach`.
+
+### OpenAI-compatible judges with thinking output
+
+Some self-hosted OpenAI-compatible judges, including vLLM servers configured with reasoning parsers,
+return hidden reasoning separately from final content. Promptfoo includes that reasoning in provider
+output by default. For a judge, that can confuse JSON parsing if the reasoning contains scratchpad
+objects before the final `{"pass": ..., "score": ..., "reason": ...}` verdict.
+
+Set `showThinking: false` on the judge provider. See the
+[vLLM provider guide](/docs/providers/vllm#use-vllm-as-an-llm-judge) for a complete local judge
+recipe, including truncated `<think>` output and request-level thinking controls. The same rule
+applies to other model-graded assertions that use a text judge; see the
+[model-graded overview](/docs/configuration/expected-outputs/model-graded#openai-compatible-thinking-judges)
+for the full metric list.
+
+## Customizing the rubric prompt
+
+For more control over the `llm-rubric` evaluation, you can set a custom prompt using the `rubricPrompt` property:
+
+```yaml
+defaultTest:
+  options:
+    rubricPrompt: >
+      [
+          {
+              "role": "system",
+              "content": "Evaluate the following output based on these criteria:\n1. Clarity of explanation\n2. Accuracy of information\n3. Relevance to the topic\n\nProvide a score out of 10 for each criterion and an overall assessment."
+          },
+          {
+              "role": "user",
+              "content": "Output to evaluate: {{output}}\n\nRubric: {{rubric}}"
+          }
+      ]
+```
+
+## Non-English Evaluation
+
+To get evaluation output in languages other than English, you can use different approaches:
+
+### Option 1: rubricPrompt Override (Recommended)
+
+For reliable multilingual output with compatible assertion types:
+
+```yaml
+defaultTest:
+  options:
+    rubricPrompt: |
+      [
+        {
+          "role": "system",
+          // German: "You evaluate outputs based on criteria. Respond with JSON: {\"reason\": \"string\", \"pass\": boolean, \"score\": number}. ALL responses in German."
+          "content": "Du bewertest Ausgaben nach Kriterien. Antworte mit JSON: {\"reason\": \"string\", \"pass\": boolean, \"score\": number}. ALLE Antworten auf Deutsch."
+        },
+        {
+          "role": "user", 
+          // German: "Output: {{ output }}\nCriterion: {{ rubric }}"
+          "content": "Ausgabe: {{ output }}\nKriterium: {{ rubric }}"
+        }
+      ]
+
+assert:
+  - type: llm-rubric
+    # German: "Responds politely and helpfully"
+    value: 'Antwortet höflich und hilfreich'
+```
+
+### Option 2: Language Instructions in Rubric
+
+```yaml
+assert:
+  - type: llm-rubric
+    value: 'Responds politely and helpfully. Provide your evaluation reason in German.'
+```
+
+### Option 3: Full Native Language Rubric
+
+```yaml
+# German
+assert:
+  - type: llm-rubric
+    # German: "Responds politely and helpfully. Provide reasoning in German."
+    value: 'Antwortet höflich und hilfreich. Begründung auf Deutsch geben.'
+
+# Japanese
+assert:
+  - type: llm-rubric
+    # Japanese: "Does not contain harmful content. Please provide evaluation reasoning in Japanese."
+    value: '有害なコンテンツを含まない。評価理由は日本語で答えてください。'
+```
+
+**Note:** Option 1 works with `llm-rubric`, `g-eval`, and `model-graded-closedqa`. For other assertion types like `factuality` or `context-recall`, create assertion-specific prompts that match their expected formats.
+
+### Assertion-Specific Prompts
+
+For assertions requiring specific output formats:
+
+```yaml
+# factuality - requires {"category": "A/B/C/D/E", "reason": "..."}
+tests:
+  - options:
+      rubricPrompt: |
+        [
+          {
+            "role": "system",
+            // German: "You compare factual accuracy. Respond with JSON: {\"category\": \"A/B/C/D/E\", \"reason\": \"string\"}. A=subset, B=superset, C=identical, D=contradiction, E=irrelevant. ALL responses in German."
+            "content": "Du vergleichst Faktentreue. Antworte mit JSON: {\"category\": \"A/B/C/D/E\", \"reason\": \"string\"}. A=Teilmenge, B=Obermenge, C=identisch, D=Widerspruch, E=irrelevant. ALLE Antworten auf Deutsch."
+          },
+          {
+            "role": "user",
+            // German: "Expert answer: {{ rubric }}\nSubmitted answer: {{ output }}"
+            "content": "Expertenantwort: {{ rubric }}\nEingereichte Antwort: {{ output }}"
+          }
+        ]
+    assert:
+      - type: factuality
+        # German: "Berlin is the capital of Germany"
+        value: 'Berlin ist die Hauptstadt von Deutschland'
+```
+
+### Object handling in rubric prompts
+
+When using `{{output}}` or `{{rubric}}` variables that contain objects, promptfoo automatically converts them to JSON strings by default to prevent display issues. If you need to access specific properties of objects in your rubric prompts, you can enable object property access:
+
+```bash
+export PROMPTFOO_DISABLE_OBJECT_STRINGIFY=true
+promptfoo eval
+```
+
+With this enabled, you can access object properties directly in your rubric prompts:
+
+```yaml
+rubricPrompt: >
+  [
+      {
+          "role": "user", 
+          "content": "Evaluate this answer: {{output.text}}\nFor the question: {{rubric.question}}\nCriteria: {{rubric.criteria}}"
+      }
+  ]
+```
+
+For more details, see the [object template handling guide](/docs/usage/troubleshooting#object-template-handling).
+
+## Threshold Support
+
+The `llm-rubric` assertion type supports an optional `threshold` property that sets a minimum score requirement. When specified, the output must achieve a score greater than or equal to the threshold to pass. For example:
+
+```yaml
+assert:
+  - type: llm-rubric
+    value: Is not apologetic and provides a clear, concise answer
+    threshold: 0.8 # Requires a score of 0.8 or higher to pass
+```
+
+The threshold is applied to the score returned by the LLM (which ranges from 0.0 to 1.0). If the LLM returns an explicit pass/fail status, the threshold will still be enforced - both conditions must be met for the assertion to pass.
+
+## Pass vs. Score Semantics
+
+- PASS is determined by the LLM's boolean `pass` field unless you set a `threshold`.
+- If the model omits `pass`, promptfoo assumes `pass: true` by default.
+- `score` is a numeric metric that does not affect PASS/FAIL unless you set `threshold`.
+- When `threshold` is set, both must be true for the assertion to pass:
+  - `pass === true`
+  - `score >= threshold`
+
+This means that without a `threshold`, a result like `{ pass: true, score: 0 }` will pass. If you want the numeric score (e.g., 0/1 rubric) to drive PASS/FAIL, set a `threshold` accordingly or have the model return explicit `pass`.
+
+:::caution
+If the model omits `pass` and you don't set `threshold`, the assertion passes even with `score: 0`.
+:::
+
+### Common misconfiguration
+
+```yaml
+# ❌ Problem: Returns 0/1 scores but no threshold set
+assert:
+  - type: llm-rubric
+    value: |
+      Return 0 if the response is incorrect
+      Return 1 if the response is correct
+    # Missing threshold - always passes due to pass defaulting to true
+```
+
+**Fixes:**
+
+```yaml
+# ✅ Option A: Add threshold
+assert:
+  - type: llm-rubric
+    value: |
+      Return 0 if the response is incorrect
+      Return 1 if the response is correct
+    threshold: 1
+
+# ✅ Option B: Control pass explicitly
+assert:
+  - type: llm-rubric
+    value: |
+      Return {"pass": true, "score": 1} if the response is correct
+      Return {"pass": false, "score": 0} if the response is incorrect
+```
+
+## Negation with `not-llm-rubric`
+
+Prepend `not-` to invert the assertion — useful for "must not" criteria:
+
+```yaml
+assert:
+  - type: not-llm-rubric
+    value: Apologizes or hedges before answering
+```
+
+`not-llm-rubric` passes when the rubric criterion does **not** match. Transport or parse failures from the grader are reported as failures in both directions — a grader error is not treated as evidence that the criterion was or was not met, so inversion never silently turns a failed grader call into a pass.
+
+## Further reading
+
+See [model-graded metrics](/docs/configuration/expected-outputs/model-graded) for more options.

site/docs/configuration/expected-outputs/model-graded/max-score.md

@@ -0,0 +1,242 @@
+---
+title: Max-score assertion
+description: Configure the `max-score` assertion to deterministically pick the highest-scoring output based on other assertions' scores.
+sidebar_label: Max Score
+---
+
+# Max Score
+
+The `max-score` assertion selects the output with the highest aggregate score from other assertions. Unlike `select-best` which uses LLM judgment, `max-score` provides objective, deterministic selection based on quantitative scores from other assertions.
+
+## When to use max-score
+
+Use `max-score` when you want to:
+
+- Select the best output based on objective, measurable criteria
+- Combine multiple metrics with different importance (weights)
+- Have transparent, reproducible selection without LLM API calls
+- Select outputs based on a combination of correctness, quality, and other metrics
+
+## How it works
+
+1. All regular assertions run first on each output
+2. `max-score` collects the scores from these assertions
+3. Calculates an aggregate score for each output (average by default)
+4. Selects the output with the highest aggregate score
+5. Returns pass=true for the highest scoring output, pass=false for others
+
+## Basic usage
+
+```yaml
+prompts:
+  - 'Write a function to {{task}}'
+  - 'Write an efficient function to {{task}}'
+  - 'Write a well-documented function to {{task}}'
+
+providers:
+  - openai:gpt-5
+
+tests:
+  - vars:
+      task: 'calculate fibonacci numbers'
+    assert:
+      # Regular assertions that score each output
+      - type: python
+        value: 'assert fibonacci(10) == 55'
+      - type: llm-rubric
+        value: 'Code is efficient'
+      - type: contains
+        value: 'def fibonacci'
+      # Max-score selects the output with highest average score
+      - type: max-score
+```
+
+## Configuration options
+
+### Aggregation method
+
+Choose how scores are combined:
+
+```yaml
+assert:
+  - type: max-score
+    value:
+      method: average # Default: average | sum
+```
+
+### Weighted scoring
+
+Give different importance to different assertions by specifying weights per assertion type:
+
+```yaml
+assert:
+  - type: python # Test correctness
+  - type: llm-rubric # Test quality
+    value: 'Well documented'
+  - type: max-score
+    value:
+      weights:
+        python: 3 # Correctness is 3x more important
+        llm-rubric: 1 # Documentation is 1x weight
+```
+
+#### How weights work
+
+- Each assertion type can have a custom weight (default: 1.0)
+- For `method: average`, the final score is: `sum(score × weight) / sum(weights)`
+- For `method: sum`, the final score is: `sum(score × weight)`
+- Weights apply to all assertions of that type
+
+Example calculation with `method: average`:
+
+```
+Output A: python=1.0, llm-rubric=0.5, contains=1.0
+Weights:  python=3,   llm-rubric=1,   contains=1 (default)
+
+Score = (1.0×3 + 0.5×1 + 1.0×1) / (3 + 1 + 1)
+      = (3.0 + 0.5 + 1.0) / 5
+      = 0.9
+```
+
+### Minimum threshold
+
+Require a minimum score for selection:
+
+```yaml
+assert:
+  - type: max-score
+    value:
+      threshold: 0.7 # Only select if average score >= 0.7
+```
+
+## Scoring details
+
+- **Binary assertions** (pass/fail): Score as 1.0 or 0.0
+- **Scored assertions**: Use the numeric score (typically 0-1 range)
+- **Default weights**: 1.0 for all assertions
+- **Tie breaking**: First output wins (deterministic)
+
+## Examples
+
+### Example 1: Multi-criteria code selection
+
+```yaml
+prompts:
+  - 'Write a Python function to {{task}}'
+  - 'Write an optimized Python function to {{task}}'
+  - 'Write a documented Python function to {{task}}'
+
+providers:
+  - openai:gpt-5-mini
+
+tests:
+  - vars:
+      task: 'merge two sorted lists'
+    assert:
+      - type: python
+        value: |
+          list1 = [1, 3, 5]
+          list2 = [2, 4, 6]
+          result = merge_lists(list1, list2)
+          assert result == [1, 2, 3, 4, 5, 6]
+
+      - type: llm-rubric
+        value: 'Code has O(n+m) time complexity'
+
+      - type: llm-rubric
+        value: 'Code is well documented with docstring'
+
+      - type: max-score
+        value:
+          weights:
+            python: 3 # Correctness most important
+            llm-rubric: 1 # Each quality metric has weight 1
+```
+
+### Example 2: Content generation selection
+
+```yaml
+prompts:
+  - 'Explain {{concept}} simply'
+  - 'Explain {{concept}} in detail'
+  - 'Explain {{concept}} with examples'
+
+providers:
+  - anthropic:claude-3-haiku-20240307
+
+tests:
+  - vars:
+      concept: 'machine learning'
+    assert:
+      - type: llm-rubric
+        value: 'Explanation is accurate'
+
+      - type: llm-rubric
+        value: 'Explanation is clear and easy to understand'
+
+      - type: contains
+        value: 'example'
+
+      - type: max-score
+        value:
+          method: average # All criteria equally important
+```
+
+### Example 3: API response selection
+
+```yaml
+tests:
+  - vars:
+      query: 'weather in Paris'
+    assert:
+      - type: is-json
+
+      - type: contains-json
+        value:
+          required: ['temperature', 'humidity', 'conditions']
+
+      - type: llm-rubric
+        value: 'Response includes all requested weather data'
+
+      - type: latency
+        threshold: 1000 # Under 1 second
+
+      - type: max-score
+        value:
+          weights:
+            is-json: 2 # Must be valid JSON
+            contains-json: 2 # Must have required fields
+            llm-rubric: 1 # Quality check
+            latency: 1 # Performance matters
+```
+
+## Comparison with select-best
+
+| Feature          | max-score                        | select-best         |
+| ---------------- | -------------------------------- | ------------------- |
+| Selection method | Aggregate scores from assertions | LLM judgment        |
+| API calls        | None (uses existing scores)      | One per eval        |
+| Reproducibility  | Deterministic                    | May vary            |
+| Best for         | Objective criteria               | Subjective criteria |
+| Transparency     | Shows exact scores               | Shows LLM reasoning |
+| Cost             | Free (no API calls)              | Costs per API call  |
+
+## Edge cases
+
+- **No other assertions**: Error - max-score requires at least one assertion to aggregate
+- **Tie scores**: First output wins (by index)
+- **All outputs fail**: Still selects the highest scorer ("least bad")
+- **Below threshold**: No output selected if threshold is specified and not met
+
+## Tips
+
+1. **Use specific assertions**: More assertions provide better signal for selection
+2. **Weight important criteria**: Use weights to emphasize what matters most
+3. **Combine with select-best**: You can use both in the same test for comparison
+4. **Debug with scores**: The output shows aggregate scores for transparency
+
+## Further reading
+
+- [Model-graded metrics](/docs/configuration/expected-outputs/model-graded) for other model-based assertions
+- [Select best](/docs/configuration/expected-outputs/model-graded/select-best) for subjective selection
+- [Assertions](/docs/configuration/expected-outputs) for all available assertion types

site/docs/configuration/expected-outputs/model-graded/model-graded-closedqa.md

@@ -0,0 +1,94 @@
+---
+sidebar_label: Model-graded Closed QA
+description: 'Assess closed-domain QA performance using model-based evaluation for accuracy, completeness, and answer correctness'
+---
+
+# Model-graded Closed QA
+
+`model-graded-closedqa` is a criteria-checking evaluation that uses OpenAI's public evals prompt to determine if an LLM output meets specific requirements.
+
+### How to use it
+
+To use the `model-graded-closedqa` assertion type, add it to your test configuration like this:
+
+```yaml
+assert:
+  - type: model-graded-closedqa
+    # Specify the criteria that the output must meet:
+    value: Provides a clear answer without hedging or uncertainty
+```
+
+This assertion will use a language model to evaluate whether the output meets the specified criterion, returning a simple yes/no response.
+
+### How it works
+
+Under the hood, `model-graded-closedqa` uses OpenAI's closed QA evaluation prompt to analyze the output. The grader will return:
+
+- `Y` if the output meets the criterion
+- `N` if the output does not meet the criterion
+
+The assertion passes if the response ends with 'Y' and fails if it ends with 'N'.
+
+### Example Configuration
+
+Here's a complete example showing how to use model-graded-closedqa:
+
+```yaml
+prompts:
+  - 'What is {{topic}}?'
+providers:
+  - openai:gpt-5
+tests:
+  - vars:
+      topic: quantum computing
+    assert:
+      - type: model-graded-closedqa
+        value: Explains the concept without using technical jargon
+      - type: model-graded-closedqa
+        value: Includes a practical real-world example
+```
+
+### Overriding the Grader
+
+Like other model-graded assertions, you can override the default grader:
+
+1. Using the CLI:
+
+   ```sh
+   promptfoo eval --grader openai:gpt-5-mini
+   ```
+
+2. Using test options:
+
+   ```yaml
+   defaultTest:
+     options:
+       provider: openai:gpt-5-mini
+   ```
+
+3. Using assertion-level override:
+   ```yaml
+   assert:
+     - type: model-graded-closedqa
+       value: Is concise and clear
+       provider: openai:gpt-5-mini
+   ```
+
+### Customizing the Prompt
+
+You can customize the evaluation prompt using the `rubricPrompt` property:
+
+```yaml
+defaultTest:
+  options:
+    rubricPrompt: |
+      Question: {{input}}
+      Criterion: {{criteria}}
+      Response: {{completion}}
+
+      Does this response meet the criterion? Answer Y or N.
+```
+
+# Further reading
+
+See [model-graded metrics](/docs/configuration/expected-outputs/model-graded) for more options.

site/docs/configuration/expected-outputs/model-graded/pi.md

@@ -0,0 +1,115 @@
+---
+sidebar_position: 8
+description: 'Identify and block prompt injection attacks using advanced model-based classification for enhanced security protection'
+---
+
+# Pi Scorer
+
+`pi` is an alternative approach to model grading that uses a dedicated scoring model instead of the "LLM as a judge" technique. It can evaluate input and output pairs against criteria.
+
+:::note
+**Important**: Unlike `llm-rubric` which works with your existing providers, Pi requires a separate external API key from Pi Labs.
+:::
+
+## Alternative Approach
+
+Pi offers a different approach to evaluation with some distinct characteristics:
+
+- Uses a dedicated scoring model rather than prompting an LLM to act as a judge
+- Focuses on highly accurate numeric scoring without providing detailed reasoning
+- Aims for consistency in scoring the same inputs
+- Requires a separate API key and integration
+
+Each approach has different strengths, and you may want to experiment with both to determine which best suits your specific evaluation needs.
+
+## Prerequisites
+
+To use Pi, you **must** first:
+
+1. Create a Pi API key from [Pi Labs](https://build.withpi.ai/account/keys)
+2. Set the `WITHPI_API_KEY` environment variable
+
+```bash
+export WITHPI_API_KEY=your_api_key_here
+```
+
+or set
+
+```yaml
+env:
+  WITHPI_API_KEY: your_api_key_here
+```
+
+in your promptfoo config
+
+## How to use it
+
+To use the `pi` assertion type, add it to your test configuration:
+
+```yaml
+assert:
+  - type: pi
+    # Specify the criteria for grading the LLM output
+    value: Is the response not apologetic and provides a clear, concise answer?
+```
+
+This assertion will use the Pi scorer to grade the output based on the specified criteria.
+
+## How it works
+
+Under the hood, the `pi` assertion uses the `withpi` SDK to evaluate the output based on the criteria you provide.
+
+Compared to LLM as a judge:
+
+- The inputs of the eval are the same: `llm_input` and `llm_output`
+- Pi does not need a system prompt, and is pretrained to score
+- Pi always generates the same score, when given the same input
+- Pi requires a separate API key (see Prerequisites section)
+
+## Threshold Support
+
+The `pi` assertion type supports an optional `threshold` property that sets a minimum score requirement. When specified, the output must achieve a score greater than or equal to the threshold to pass.
+
+```yaml
+assert:
+  - type: pi
+    value: Is not apologetic and provides a clear, concise answer
+    threshold: 0.8 # Requires a score of 0.8 or higher to pass
+```
+
+:::info
+The default threshold is `0.5` if not specified.
+:::
+
+## Metrics Brainstorming
+
+You can use the [Pi Labs Copilot](https://build.withpi.ai) to interactively brainstorm representative metrics for your application. It helps you:
+
+1. Generate effective evaluation criteria
+2. Test metrics on example outputs before integration
+3. Find the optimal threshold values for your use case
+
+## Example Configuration
+
+```yaml
+prompts:
+  - 'Explain {{concept}} in simple terms.'
+providers:
+  - openai:gpt-5
+tests:
+  - vars:
+      concept: quantum computing
+    assert:
+      - type: pi
+        value: Is the explanation easy to understand without technical jargon?
+        threshold: 0.7
+      - type: pi
+        value: Does the response correctly explain the fundamental principles?
+        threshold: 0.8
+```
+
+## See Also
+
+- [LLM Rubric](/docs/configuration/expected-outputs/model-graded/llm-rubric)
+- [Model-graded metrics](/docs/configuration/expected-outputs/model-graded)
+- [Pi Documentation](https://docs.withpi.ai) for more options, configuration, and calibration details

site/docs/configuration/expected-outputs/model-graded/search-rubric.md

@@ -0,0 +1,286 @@
+---
+sidebar_label: Search Rubric
+---
+
+# Search-Rubric
+
+The `search-rubric` assertion type is like `llm-rubric` but with web search capabilities. It evaluates outputs according to a rubric while having the ability to search for current information when needed.
+
+## How it works
+
+1. You provide a rubric that describes what the output should contain
+2. The grading provider evaluates the output against the rubric
+3. If the rubric requires current information, the provider searches the web
+4. Returns pass/fail with a score from 0.0 to 1.0
+
+## Basic Usage
+
+```yaml
+assert:
+  - type: search-rubric
+    value: 'Provides accurate current Bitcoin price within 5% of market value'
+```
+
+## Comparing to LLM-Rubric
+
+The `search-rubric` assertion behaves exactly like `llm-rubric`, but automatically uses a provider with web search capabilities:
+
+```yaml
+# These are equivalent:
+assert:
+  # Using llm-rubric with a web-search capable provider
+  - type: llm-rubric
+    value: 'Contains current stock price for Apple (AAPL) within $5'
+    provider: openai:responses:gpt-5.1 # Must configure web search tool
+
+  # Using search-rubric (automatically selects a web-search provider)
+  - type: search-rubric
+    value: 'Contains current stock price for Apple (AAPL) within $5'
+```
+
+## Using Variables in the Rubric
+
+Like `llm-rubric`, you can use test variables:
+
+```yaml
+prompts:
+  - 'What is the current weather in {{city}}?'
+
+assert:
+  - type: search-rubric
+    value: 'Provides current temperature in {{city}} with units (F or C)'
+
+tests:
+  - vars:
+      city: San Francisco
+  - vars:
+      city: Tokyo
+```
+
+## Grading Providers
+
+The search-rubric assertion requires a grading provider with web search capabilities:
+
+### 1. Anthropic Claude
+
+Anthropic Claude models support web search through the `web_search_20250305` tool:
+
+```yaml
+grading:
+  provider: anthropic:messages:claude-opus-4-6
+  providerOptions:
+    config:
+      tools:
+        - type: web_search_20250305
+          name: web_search
+          max_uses: 5
+```
+
+### 2. OpenAI with Web Search
+
+OpenAI's responses API supports web search through the `web_search_preview` tool:
+
+```yaml
+grading:
+  provider: openai:responses:gpt-5.1
+  providerOptions:
+    config:
+      tools:
+        - type: web_search_preview
+```
+
+### 3. Perplexity
+
+Perplexity models have built-in web search:
+
+```yaml
+grading:
+  provider: perplexity:sonar
+```
+
+### 4. Google Gemini
+
+Google's Gemini models support web search through the `googleSearch` tool:
+
+```yaml
+grading:
+  provider: google:gemini-3.1-pro-preview
+  providerOptions:
+    config:
+      tools:
+        - googleSearch: {}
+```
+
+### 5. xAI Grok
+
+xAI's Grok models can use server-side web search tools through the Responses API:
+
+```yaml
+grading:
+  provider: xai:responses:grok-4.3
+  providerOptions:
+    config:
+      tools:
+        - type: web_search
+```
+
+## Use Cases
+
+### 1. Current Events Verification
+
+```yaml
+prompts:
+  - 'Who won the latest Super Bowl?'
+
+assert:
+  - type: search-rubric
+    value: 'Names the correct winner of the most recent Super Bowl with the final score'
+```
+
+### 2. Real-time Price Checking
+
+```yaml
+prompts:
+  - "What's the current stock price of {{ticker}}?"
+
+assert:
+  - type: search-rubric
+    value: |
+      Provides accurate stock price for {{ticker}} that:
+      1. Is within 2% of current market price
+      2. Includes currency (USD)
+      3. Mentions if market is open or closed
+    threshold: 0.8
+```
+
+### 3. Weather Information
+
+```yaml
+prompts:
+  - "What's the weather like in Tokyo?"
+
+assert:
+  - type: search-rubric
+    value: |
+      Describes current Tokyo weather including:
+      - Temperature (with units)
+      - General conditions (sunny, rainy, etc.)
+      - Humidity or precipitation if relevant
+```
+
+### 4. Latest Software Versions
+
+```yaml
+prompts:
+  - "What's the latest version of Node.js?"
+
+assert:
+  - type: search-rubric
+    value: 'States the correct latest LTS version of Node.js (not experimental or nightly)'
+```
+
+## Cost Considerations
+
+Web search assertions have the following cost implications. As of November 2025:
+
+- **Anthropic Claude**: $10 per 1,000 web search calls plus token costs
+- **OpenAI**: Web search tools on the Responses API cost $10-25 per 1,000 tool calls in addition to token usage
+- **Google Gemini API**: $35 per 1,000 grounded prompts; **Vertex AI Web Grounding**: $45 per 1,000
+- **Perplexity**: Per-request plus token-based pricing; see Perplexity or your proxy's pricing page
+- **xAI Grok**: $25 per 1,000 sources plus token usage for Live Search
+
+## Threshold Support
+
+Like `llm-rubric`, the `search-rubric` assertion supports thresholds:
+
+```yaml
+assert:
+  - type: search-rubric
+    value: 'Contains accurate information about current US inflation rate'
+    threshold: 0.9 # Requires 90% accuracy for economic data
+```
+
+## Best Practices
+
+1. **Write clear rubrics**: Be specific about what information you expect
+2. **Use thresholds appropriately**: Higher thresholds for factual accuracy, lower for general correctness
+3. **Include acceptable ranges**: For volatile data like prices, specify acceptable accuracy (e.g., "within 5%")
+4. **Use caching**: Caching is enabled by default; use `promptfoo eval --no-cache` to force fresh searches
+5. **Test variable substitution**: Ensure your rubrics work with different variable values
+
+## Expected Behavior
+
+Understanding how `search-rubric` evaluates different scenarios helps you write better tests.
+
+### What the grader catches
+
+The search-enabled grader identifies several types of failures:
+
+| SUT Response                            | Grader Verdict | Reason                            |
+| --------------------------------------- | -------------- | --------------------------------- |
+| "I don't have access to real-time data" | **Fail**       | No actual answer provided         |
+| Stale price from training data          | **Fail**       | Value differs from current market |
+| Correct current price                   | **Pass**       | Matches web search results        |
+| Partially correct answer                | **Partial**    | Score reflects completeness       |
+
+### Models without web search
+
+Models like `gpt-4o-mini` without web search enabled will often refuse to answer real-time questions:
+
+> "I don't have access to real-time stock data. For current prices, please check a financial website."
+
+The `search-rubric` grader correctly flags this as a failure since no actual information was provided. This is the expected behavior—the assertion is verifying whether your system provides accurate current information, not whether it gracefully declines.
+
+**To test models that confidently answer (and potentially hallucinate):**
+
+- Use a more capable model as the system under test
+- Enable web search on your SUT if available
+- Test against models known to attempt answers even when uncertain
+
+### Partial matches and scoring
+
+The grader returns a score from 0.0 to 1.0 based on how well the output matches the rubric:
+
+- **1.0**: Fully matches all rubric criteria
+- **0.7-0.9**: Matches most criteria, minor issues
+- **0.4-0.6**: Partial match, missing key information
+- **0.0-0.3**: Significant errors or refusal to answer
+
+Use the `threshold` parameter to set your acceptable score level.
+
+## Troubleshooting
+
+### "No provider with web search capabilities"
+
+Ensure your grading provider supports web search. Default providers without web search configuration will fail. Check the [Grading Providers](#grading-providers) section above.
+
+### Test always fails with refusal
+
+If your SUT consistently refuses to answer real-time questions, this is expected behavior for models without web access. The `search-rubric` grader is correctly identifying that no factual answer was provided.
+
+**Solutions:**
+
+1. Use a model with web search capabilities as your SUT
+2. Accept that models without real-time access cannot answer these questions
+3. Use `llm-rubric` instead if you only need to verify the response format
+
+### Inaccurate results
+
+The grader relies on web search results, which may occasionally be wrong or ambiguous.
+
+**Best practices:**
+
+- Write rubrics that can be verified from multiple reputable sources
+- Avoid rubrics about speculative or disputed claims
+- Use appropriate thresholds (not 1.0) to allow for minor discrepancies
+
+### High costs
+
+Web search adds cost on top of model tokens.
+
+**Cost reduction strategies:**
+
+- Caching is enabled by default to reduce API calls
+- Reserve `search-rubric` for tests that truly need real-time verification
+- Use `llm-rubric` for static fact-checking that doesn't require current data
+- Consider Perplexity's `sonar` model for built-in search without per-call fees

site/docs/configuration/expected-outputs/model-graded/select-best.md

@@ -0,0 +1,102 @@
+---
+sidebar_label: Select Best
+description: 'Leverage AI models to automatically select and rank the best outputs from multiple LLM responses for quality optimization'
+---
+
+# Select Best
+
+The `select-best` assertion compares multiple outputs in the same test case and selects the one that best meets a specified criterion. This is useful for comparing different prompt or model variations to determine which produces the best result.
+
+### How to use it
+
+To use the `select-best` assertion type, add it to your test configuration like this:
+
+```yaml
+assert:
+  - type: select-best
+    value: 'choose the most concise and accurate response'
+```
+
+Note: This assertion requires multiple prompts or providers to generate different outputs to compare.
+
+### How it works
+
+The select-best checker:
+
+1. Takes all outputs from the test case
+2. Evaluates each output against the specified criterion
+3. Selects the best output
+4. Returns pass=true for the winning output and pass=false for others
+
+### Example Configuration
+
+Here's a complete example showing how to use select-best to compare different prompt variations:
+
+```yaml
+prompts:
+  - 'Write a tweet about {{topic}}'
+  - 'Write a very concise, funny tweet about {{topic}}'
+  - 'Compose a tweet about {{topic}} that will go viral'
+providers:
+  - openai:gpt-5
+tests:
+  - vars:
+      topic: 'artificial intelligence'
+    assert:
+      - type: select-best
+        value: 'choose the tweet that is most likely to get high engagement'
+  - vars:
+      topic: 'climate change'
+    assert:
+      - type: select-best
+        value: 'choose the tweet that best balances information and humor'
+```
+
+### Overriding the Grader
+
+Like other model-graded assertions, you can override the default grader:
+
+1. Using the CLI:
+
+   ```sh
+   promptfoo eval --grader openai:gpt-5-mini
+   ```
+
+2. Using test options:
+
+   ```yaml
+   defaultTest:
+     options:
+       provider: openai:gpt-5-mini
+   ```
+
+3. Using assertion-level override:
+   ```yaml
+   assert:
+     - type: select-best
+       value: 'choose the most engaging response'
+       provider: openai:gpt-5-mini
+   ```
+
+### Customizing the Prompt
+
+You can customize the evaluation prompt using the `rubricPrompt` property:
+
+```yaml
+defaultTest:
+  options:
+    rubricPrompt: |
+      Here are {{ outputs | length }} responses:
+      {% for output in outputs %}
+      Output {{ loop.index0 }}: {{ output }}
+      {% endfor %}
+
+      Criteria: {{ criteria }}
+
+      Analyze each output against the criteria.
+      Choose the best output by responding with its index (0 to {{ outputs | length - 1 }}).
+```
+
+# Further reading
+
+See [model-graded metrics](/docs/configuration/expected-outputs/model-graded) for more options.

site/docs/configuration/expected-outputs/moderation.md

@@ -0,0 +1,199 @@
+---
+sidebar_position: 100
+sidebar_label: Moderation
+description: Implement comprehensive content moderation using multiple APIs to detect and filter harmful, toxic, or policy-violating outputs
+---
+
+# Moderation
+
+Use the `moderation` assert type to ensure that LLM outputs are safe.
+
+Currently, this supports [OpenAI's moderation model](https://platform.openai.com/docs/guides/moderation), Meta's LlamaGuard models (LlamaGuard 3 and 4) via [Replicate](https://replicate.com/meta/llama-guard-3-8b), and [Azure Content Safety API](https://learn.microsoft.com/en-us/azure/ai-services/content-safety/overview).
+
+When a moderation provider reports token usage, Promptfoo includes it in assertion token metrics for clean, flagged, and provider-error results. `not-moderation` changes only the verdict; it preserves reported usage. Providers that omit usage do not produce synthetic token counts.
+
+In general, we encourage the use of Meta's LlamaGuard as it substantially outperforms OpenAI's moderation API as well as GPT-4. [See benchmarks](https://github.com/meta-llama/PurpleLlama/blob/main/Llama-Guard2/MODEL_CARD.md#model-performance).
+
+## OpenAI moderation
+
+By default, the `moderation` assertion uses OpenAI if an OpenAI API key is provided. Just make sure that the `OPENAI_API_KEY` environment variable is set:
+
+```yaml
+tests:
+  - vars:
+      foo: bar
+    assert:
+      # Ensure that it passes OpenAI's moderation filters
+      - type: moderation
+```
+
+OpenAI monitors the following categories:
+
+| Category               | Description                                                                                                                                                                                |
+| ---------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
+| hate                   | Content that promotes hate based on race, gender, ethnicity, religion, nationality, sexual orientation, disability, or caste. Hateful content aimed at non-protected groups is harassment. |
+| hate/threatening       | Hateful content that includes violence or serious harm towards the targeted group.                                                                                                         |
+| harassment             | Content that promotes harassing language towards any target.                                                                                                                               |
+| harassment/threatening | Harassment content that includes violence or serious harm towards any target.                                                                                                              |
+| self-harm              | Content that promotes or depicts acts of self-harm, such as suicide, cutting, and eating disorders.                                                                                        |
+| self-harm/intent       | Content where the speaker expresses intent to engage in self-harm.                                                                                                                         |
+| self-harm/instructions | Content that encourages or gives instructions on how to commit acts of self-harm.                                                                                                          |
+| sexual                 | Content meant to arouse sexual excitement or promote sexual services (excluding sex education and wellness).                                                                               |
+| sexual/minors          | Sexual content involving individuals under 18 years old.                                                                                                                                   |
+| violence               | Content that depicts death, violence, or physical injury.                                                                                                                                  |
+| violence/graphic       | Content that depicts death, violence, or physical injury in graphic detail.                                                                                                                |
+
+### Check specific categories
+
+The assertion value allows you to only enable moderation for specific categories:
+
+```yaml
+tests:
+  - vars:
+      foo: bar
+    assert:
+      - type: moderation
+        // highlight-start
+        value:
+          - harassment
+          - harassment/threatening
+          - sexual
+          - sexual/minors
+        // highlight-end
+```
+
+## Meta LlamaGuard moderation
+
+This example uses the LlamaGuard model hosted on Replicate. Be sure to set the `REPLICATE_API_KEY` environment variable:
+
+```yaml
+tests:
+  - vars:
+      foo: bar
+    assert:
+      - type: moderation
+        # Use the latest Llama Guard on replicate
+        provider: 'replicate:moderation:meta/llama-guard-3-8b:146d1220d447cdcc639bc17c5f6137416042abee6ae153a2615e6ef5749205c8'
+```
+
+:::tip
+LlamaGuard 4 is the default moderation provider on Replicate, featuring enhanced capabilities and an additional category (S14: Code Interpreter Abuse). You can explicitly specify it:
+
+```yaml
+provider: 'replicate:moderation:meta/llama-guard-4-12b'
+```
+
+For compatibility or specific use cases, you can use LlamaGuard 3:
+
+```yaml
+provider: 'replicate:moderation:meta/llama-guard-3-8b:146d1220d447cdcc639bc17c5f6137416042abee6ae153a2615e6ef5749205c8'
+```
+
+:::
+
+LlamaGuard monitors the following categories:
+
+| Category               | Description                                                                                                      | Code |
+| ---------------------- | ---------------------------------------------------------------------------------------------------------------- | ---- |
+| Violent Crimes         | Content that enables, encourages, or excuses violent crimes (e.g., terrorism, murder, child abuse, animal abuse) | S1   |
+| Non-Violent Crimes     | Content that enables, encourages, or excuses non-violent crimes (e.g., fraud, burglary, drug trafficking)        | S2   |
+| Sex Crimes             | Content that enables, encourages, or excuses sex crimes (e.g., human trafficking, sexual assault, harassment)    | S3   |
+| Child Exploitation     | Content depicting child nudity or sexual abuse of children                                                       | S4   |
+| Defamation             | Content containing defamatory statements about individuals or entities                                           | S5   |
+| Specialized Advice     | Content containing specialized financial, medical, or legal advice                                               | S6   |
+| Privacy                | Content containing sensitive, personal information about private individuals                                     | S7   |
+| Intellectual Property  | Content that violates intellectual property rights of third parties                                              | S8   |
+| Indiscriminate Weapons | Content that enables the creation of weapons of mass destruction (e.g., chemical, biological, nuclear weapons)   | S9   |
+| Hate                   | Content that is hateful toward people based on protected characteristics or perpetuates negative stereotypes     | S10  |
+| Self-Harm              | Content that enables, encourages, or excuses acts of intentional self-harm (e.g., suicide, self-injury)          | S11  |
+| Sexual Content         | Depictions of nude adults or content with erotic descriptions or explicit sex acts                               | S12  |
+| Elections              | Content containing misinformation or illegal activity related to elections                                       | S13  |
+| Code Interpreter Abuse | Content that seeks to abuse code interpreters (e.g., denial of service, container escapes) - LlamaGuard 4 only   | S14  |
+
+### Check specific categories
+
+The assertion value allows you to only enable moderation for specific categories:
+
+```yaml
+tests:
+  - vars:
+      foo: bar
+    assert:
+      - type: moderation
+        provider: 'replicate:moderation:meta/llama-guard-3-8b:146d1220d447cdcc639bc17c5f6137416042abee6ae153a2615e6ef5749205c8'
+        // highlight-start
+        value:
+          - S1
+          - S3
+          - S4
+        // highlight-end
+```
+
+## Azure Content Safety moderation
+
+You can use the Azure Content Safety API for moderation. To set it up, you need to create an Azure Content Safety resource and get the API key and endpoint.
+
+### Setup
+
+First, set these environment variables:
+
+```bash
+AZURE_CONTENT_SAFETY_ENDPOINT=https://your-resource-name.cognitiveservices.azure.com
+AZURE_CONTENT_SAFETY_API_KEY=your-api-key
+AZURE_CONTENT_SAFETY_API_VERSION=2024-09-01  # Optional, defaults to this version
+```
+
+If `AZURE_CONTENT_SAFETY_ENDPOINT` is set, PromptFoo will automatically use the Azure Content Safety service for moderation instead of OpenAI's moderation API.
+
+Or you can explicitly use the Azure moderation provider in your tests:
+
+```yaml
+tests:
+  - vars:
+      foo: bar
+    assert:
+      - type: moderation
+        provider: 'azure:moderation'
+```
+
+### Moderation Categories
+
+The Azure Content Safety API checks content for these categories:
+
+| Category | Description                                                       |
+| -------- | ----------------------------------------------------------------- |
+| Hate     | Content that expresses discrimination or derogatory sentiments    |
+| SelfHarm | Content related to inflicting physical harm on oneself            |
+| Sexual   | Sexually explicit or adult content                                |
+| Violence | Content depicting or promoting violence against people or animals |
+
+### Check specific categories
+
+The assertion value allows you to only enable moderation for specific categories
+
+```yaml
+tests:
+  - vars:
+      foo: bar
+    assert:
+      - type: moderation
+        provider: 'azure:moderation'
+        value:
+          - hate
+          - sexual
+```
+
+You can also set blocklist names and halt on blocklist hit in the provider config:
+
+```yaml
+tests:
+  - vars:
+      foo: bar
+    assert:
+      - type: moderation
+        provider:
+          id: azure:moderation
+          config:
+            blocklistNames: ['my-custom-blocklist', 'industry-terms']
+            haltOnBlocklistHit: true
+```

site/docs/configuration/expected-outputs/python.md

@@ -0,0 +1,300 @@
+---
+sidebar_position: 51
+sidebar_label: Python
+description: Create advanced Python validation scripts with complex logic, external APIs, and ML libraries for sophisticated output grading
+---
+
+# Python assertions
+
+The `python` assertion allows you to provide a custom Python function to validate the LLM output.
+
+:::tip Python Overview
+
+For an overview of all Python integrations (providers, assertions, test generators, prompts), see the [Python integration guide](/docs/integrations/python).
+
+:::
+
+A variable named `output` is injected into the context. The function should return `true` if the output passes the assertion, and `false` otherwise. If the function returns a number, it will be treated as a score.
+
+Example:
+
+```yaml
+assert:
+  - type: python
+    value: output[5:10] == 'Hello'
+```
+
+You may also return a number, which will be treated as a score:
+
+```yaml
+assert:
+  - type: python
+    value: math.log10(len(output)) * 10
+```
+
+## Multiline functions
+
+Python assertions support multiline strings:
+
+```yaml
+assert:
+  - type: python
+    value: |
+      # Insert your scoring logic here...
+      if output == 'Expected output':
+          return {
+            'pass': True,
+            'score': 0.5,
+          }
+      return {
+        'pass': False,
+        'score': 0,
+      }
+```
+
+## Using test context
+
+A `context` object is available in the Python function. Here is its type definition:
+
+```py
+from typing import Any, Dict, List, Optional, TypedDict, Union
+
+class TraceSpan(TypedDict):
+    spanId: str
+    parentSpanId: Optional[str]
+    name: str
+    startTime: int  # Unix timestamp in milliseconds
+    endTime: Optional[int]  # Unix timestamp in milliseconds
+    attributes: Optional[Dict[str, Any]]
+    statusCode: Optional[int]
+    statusMessage: Optional[str]
+
+class TraceData(TypedDict):
+    traceId: str
+    spans: List[TraceSpan]
+
+class AssertionValueFunctionContext(TypedDict):
+    # Raw prompt sent to LLM
+    prompt: Optional[str]
+
+    # Test case variables
+    vars: Dict[str, Union[str, object]]
+
+    # The complete test case
+    test: Dict[str, Any]  # Contains keys like "vars", "assert", "options"
+
+    # Log probabilities from the LLM response, if available
+    logProbs: Optional[list[float]]
+
+    # Configuration passed to the assertion
+    config: Optional[Dict[str, Any]]
+
+    # The provider that generated the response
+    provider: Optional[Any]  # ApiProvider type
+
+    # The complete provider response
+    providerResponse: Optional[Any]  # ProviderResponse type
+
+    # OpenTelemetry trace data (when tracing is enabled)
+    trace: Optional[TraceData]
+
+    # Optional shortcut to providerResponse.metadata
+    metadata: Optional[Dict[str, Any]]
+```
+
+For example, if the test case has a var `example`, access it in Python like this:
+
+```yaml
+tests:
+  - description: 'Test with context'
+    vars:
+      example: 'Example text'
+    assert:
+      - type: python
+        value: 'context["vars"]["example"] in output'
+```
+
+## External .py
+
+To reference an external file, use the `file://` prefix:
+
+```yaml
+assert:
+  - type: python
+    value: file://relative/path/to/script.py
+    config:
+      outputLengthLimit: 10
+```
+
+You can specify a particular function to use by appending it after a colon:
+
+```yaml
+assert:
+  - type: python
+    value: file://relative/path/to/script.py:custom_assert
+```
+
+If no function is specified, it defaults to `get_assert`.
+
+This file will be called with an `output` string and an `AssertionValueFunctionContext` object (see above).
+It expects that either a `bool` (pass/fail), `float` (score), or `GradingResult` will be returned.
+
+Here's an example `assert.py`:
+
+```py
+from typing import Dict, TypedDict, Union
+
+# Default function name
+def get_assert(output: str, context) -> Union[bool, float, Dict[str, Any]]:
+    print('Prompt:', context['prompt'])
+    print('Vars', context['vars']['topic'])
+
+    # This return is an example GradingResult dict
+    return {
+      'pass': True,
+      'score': 0.6,
+      'reason': 'Looks good to me',
+    }
+
+# Custom function name
+def custom_assert(output: str, context) -> Union[bool, float, Dict[str, Any]]:
+    return len(output) > 10
+```
+
+This is an example of an assertion that uses data from a configuration defined in the assertion's YML file:
+
+```py
+from typing import Dict, Union
+
+def get_assert(output: str, context) -> Union[bool, float, Dict[str, Any]]:
+    return len(output) <= context.get('config', {}).get('outputLengthLimit', 0)
+```
+
+You can also return nested metrics and assertions via a `GradingResult` object:
+
+```py
+{
+    'pass': True,
+    'score': 0.75,
+    'reason': 'Looks good to me',
+    'componentResults': [{
+        'pass': 'bananas' in output.lower(),
+        'score': 0.5,
+        'reason': 'Contains banana',
+    }, {
+        'pass': 'yellow' in output.lower(),
+        'score': 0.5,
+        'reason': 'Contains yellow',
+    }]
+}
+```
+
+### GradingResult types
+
+Here's a Python type definition you can use for the [`GradingResult`](/docs/configuration/reference/#gradingresult) object:
+
+```py
+@dataclass
+class GradingResult:
+    pass_: bool  # 'pass' is a reserved keyword in Python
+    score: float
+    reason: str
+    component_results: Optional[List['GradingResult']] = None
+    named_scores: Optional[Dict[str, float]] = None  # Appear as metrics in the UI
+```
+
+:::tip Snake case support
+Python snake_case fields are automatically mapped to camelCase:
+
+- `pass_` → `pass` (or just use `"pass"` as a dictionary key)
+- `named_scores` → `namedScores`
+- `component_results` → `componentResults`
+- `tokens_used` → `tokensUsed`
+  :::
+
+## Using trace data
+
+When [tracing is enabled](/docs/tracing/), OpenTelemetry trace data is available in the `context.trace` object. This allows you to write assertions based on the execution flow:
+
+```py
+def get_assert(output: str, context) -> Union[bool, float, Dict[str, Any]]:
+    # Check if trace data is available
+    if not hasattr(context, 'trace') or context.trace is None:
+        # Tracing not enabled, skip trace-based checks
+        return True
+
+    # Access trace spans
+    spans = context.trace['spans']
+
+    # Example: Check for errors in any span
+    error_spans = [s for s in spans if s.get('statusCode', 0) >= 400]
+    if error_spans:
+        return {
+            'pass': False,
+            'score': 0,
+            'reason': f"Found {len(error_spans)} error spans"
+        }
+
+    # Example: Calculate total trace duration
+    if spans:
+        duration = max(s.get('endTime', 0) for s in spans) - min(s['startTime'] for s in spans)
+        if duration > 5000:  # 5 seconds
+            return {
+                'pass': False,
+                'score': 0,
+                'reason': f"Trace took too long: {duration}ms"
+            }
+
+    # Example: Check for specific operations
+    api_calls = [s for s in spans if 'http' in s['name'].lower()]
+    if len(api_calls) > 10:
+        return {
+            'pass': False,
+            'score': 0,
+            'reason': f"Too many API calls: {len(api_calls)}"
+        }
+
+    return True
+```
+
+Example YAML configuration:
+
+```yaml
+tests:
+  - vars:
+      query: "What's the weather?"
+    assert:
+      - type: python
+        value: |
+          # Ensure retrieval happened before response generation
+          if context.trace:
+              spans = context.trace['spans']
+              retrieval_span = next((s for s in spans if 'retrieval' in s['name']), None)
+              generation_span = next((s for s in spans if 'generation' in s['name']), None)
+              
+              if retrieval_span and generation_span:
+                  return retrieval_span['startTime'] < generation_span['startTime']
+          return True
+```
+
+## Overriding the Python binary
+
+By default, promptfoo will run `python` in your shell. Make sure `python` points to the appropriate executable.
+
+If a `python` binary is not present, you will see a "python: command not found" error.
+
+To override the Python binary, set the `PROMPTFOO_PYTHON` environment variable. You may set it to a path (such as `/path/to/python3.11`) or just an executable in your PATH (such as `python3.11`).
+
+## Negation
+
+Use `not-python` to invert the final pass/fail result while preserving the returned score. Numeric scores are still compared against `threshold` before the result is inverted:
+
+```yaml
+assert:
+  - type: not-python
+    value: "'error' in output"
+```
+
+## Other assertion types
+
+For more info on assertions, see [Test assertions](/docs/configuration/expected-outputs).

site/docs/configuration/expected-outputs/ruby.md

@@ -0,0 +1,317 @@
+---
+sidebar_position: 52
+sidebar_label: Ruby
+description: Create advanced Ruby validation scripts with complex logic, external APIs, and custom libraries for sophisticated output grading
+---
+
+# Ruby assertions
+
+The `ruby` assertion allows you to provide a custom Ruby method to validate the LLM output.
+
+A variable named `output` is injected into the context. The method should return `true` if the output passes the assertion, and `false` otherwise. If the method returns a number, it will be treated as a score.
+
+Example:
+
+```yaml
+assert:
+  - type: ruby
+    value: output[5..9] == 'Hello'
+```
+
+You may also return a number, which will be treated as a score:
+
+```yaml
+assert:
+  - type: ruby
+    value: Math.log10(output.length) * 10
+```
+
+## Multiline functions
+
+Ruby assertions support multiline strings:
+
+```yaml
+assert:
+  - type: ruby
+    value: |
+      # Insert your scoring logic here...
+      if output == 'Expected output'
+        return {
+          'pass' => true,
+          'score' => 0.5,
+        }
+      end
+      return {
+        'pass' => false,
+        'score' => 0,
+      }
+```
+
+## Using test context
+
+A `context` object is available in the Ruby method. Here is its type definition:
+
+```ruby
+# TraceSpan
+{
+  'spanId' => String,
+  'parentSpanId' => String | nil,
+  'name' => String,
+  'startTime' => Integer,  # Unix timestamp in milliseconds
+  'endTime' => Integer | nil,  # Unix timestamp in milliseconds
+  'attributes' => Hash | nil,
+  'statusCode' => Integer | nil,
+  'statusMessage' => String | nil
+}
+
+# TraceData
+{
+  'traceId' => String,
+  'spans' => Array[TraceSpan]
+}
+
+# AssertionValueFunctionContext
+{
+  # Raw prompt sent to LLM
+  'prompt' => String | nil,
+
+  # Test case variables
+  'vars' => Hash[String, String | Object],
+
+  # The complete test case
+  'test' => Hash,  # Contains keys like "vars", "assert", "options"
+
+  # Log probabilities from the LLM response, if available
+  'logProbs' => Array[Float] | nil,
+
+  # Configuration passed to the assertion
+  'config' => Hash | nil,
+
+  # The provider that generated the response
+  'provider' => Object | nil,  # ApiProvider type
+
+  # The complete provider response
+  'providerResponse' => Object | nil,  # ProviderResponse type
+
+  # Optional shortcut to providerResponse metadata
+  'metadata' => Hash | nil,
+
+  # OpenTelemetry trace data (when tracing is enabled)
+  'trace' => TraceData | nil
+}
+```
+
+For example, if the test case has a var `example`, access it in Ruby like this:
+
+```yaml
+tests:
+  - description: 'Test with context'
+    vars:
+      example: 'Example text'
+    assert:
+      - type: ruby
+        value: 'output.include?(context["vars"]["example"])'
+```
+
+## External .rb
+
+To reference an external file, use the `file://` prefix:
+
+```yaml
+assert:
+  - type: ruby
+    value: file://relative/path/to/script.rb
+    config:
+      outputLengthLimit: 10
+```
+
+You can specify a particular method to use by appending it after a colon:
+
+```yaml
+assert:
+  - type: ruby
+    value: file://relative/path/to/script.rb:custom_assert
+```
+
+You can also specify a class method on some class or module in the file:
+
+```yaml
+assert:
+  - type: ruby
+    value: file://relative/path/to/script.rb:Validators::Format.check_length
+```
+
+If no method is specified, it defaults to `get_assert`.
+
+This file will be called with an `output` string and an `AssertionValueFunctionContext` object (see above).
+It expects that either a `bool` (pass/fail), `float` (score), or `GradingResult` will be returned.
+
+Here's an example `assert.rb`:
+
+```ruby
+require 'json'
+
+# Default function name
+def get_assert(output, context)
+  puts 'Prompt:', context['prompt']
+  puts 'Vars', context['vars']['topic']
+
+  # This return is an example GradingResult hash
+  {
+    'pass' => true,
+    'score' => 0.6,
+    'reason' => 'Looks good to me',
+  }
+end
+
+# Custom function name
+def custom_assert(output, context)
+  output.length > 10
+end
+```
+
+This is an example of an assertion that uses data from a configuration defined in the assertion's YML file:
+
+```ruby
+def get_assert(output, context)
+  output.length <= context.fetch('config', {}).fetch('outputLengthLimit', 0)
+end
+```
+
+You can also return nested metrics and assertions via a `GradingResult` object:
+
+```ruby
+{
+  'pass' => true,
+  'score' => 0.75,
+  'reason' => 'Looks good to me',
+  'componentResults' => [{
+    'pass' => output.downcase.include?('bananas'),
+    'score' => 0.5,
+    'reason' => 'Contains banana',
+  }, {
+    'pass' => output.downcase.include?('yellow'),
+    'score' => 0.5,
+    'reason' => 'Contains yellow',
+  }]
+}
+```
+
+### GradingResult types
+
+Here's a Ruby type definition you can use for the [`GradingResult`](/docs/configuration/reference/#gradingresult) object:
+
+```ruby
+# GradingResult
+{
+  'pass' => Boolean,  # Can also use 'pass_' if 'pass' conflicts with Ruby keywords
+  'score' => Float,
+  'reason' => String,
+  'componentResults' => Array[GradingResult] | nil,  # Component results (optional)
+  'namedScores' => Hash[String, Float] | nil  # Appear as metrics in the UI (optional)
+}
+```
+
+:::tip Snake case support
+Ruby snake_case fields are automatically mapped to camelCase:
+
+- `pass_` → `pass` (or just use `"pass"` as a hash key)
+- `named_scores` → `namedScores`
+- `component_results` → `componentResults`
+- `tokens_used` → `tokensUsed`
+  :::
+
+## Using trace data
+
+When [tracing is enabled](/docs/tracing/), OpenTelemetry trace data is available in the `context['trace']` object. This allows you to write assertions based on the execution flow:
+
+```ruby
+def get_assert(output, context)
+  # Check if trace data is available
+  unless context['trace']
+    # Tracing not enabled, skip trace-based checks
+    return true
+  end
+
+  # Access trace spans
+  spans = context['trace']['spans']
+
+  # Example: Check for errors in any span
+  error_spans = spans.select { |s| s.fetch('statusCode', 0) >= 400 }
+  if error_spans.any?
+    return {
+      'pass' => false,
+      'score' => 0,
+      'reason' => "Found #{error_spans.length} error spans"
+    }
+  end
+
+  # Example: Calculate total trace duration
+  if spans.any?
+    duration = spans.map { |s| s.fetch('endTime', 0) }.max - spans.map { |s| s['startTime'] }.min
+    if duration > 5000  # 5 seconds
+      return {
+        'pass' => false,
+        'score' => 0,
+        'reason' => "Trace took too long: #{duration}ms"
+      }
+    end
+  end
+
+  # Example: Check for specific operations
+  api_calls = spans.select { |s| s['name'].downcase.include?('http') }
+  if api_calls.length > 10
+    return {
+      'pass' => false,
+      'score' => 0,
+      'reason' => "Too many API calls: #{api_calls.length}"
+    }
+  end
+
+  true
+end
+```
+
+Example YAML configuration:
+
+```yaml
+tests:
+  - vars:
+      query: "What's the weather?"
+    assert:
+      - type: ruby
+        value: |
+          # Ensure retrieval happened before response generation
+          if context['trace']
+            spans = context['trace']['spans']
+            retrieval_span = spans.find { |s| s['name'].include?('retrieval') }
+            generation_span = spans.find { |s| s['name'].include?('generation') }
+
+            if retrieval_span && generation_span
+              return retrieval_span['startTime'] < generation_span['startTime']
+            end
+          end
+          true
+```
+
+## Overriding the Ruby binary
+
+By default, promptfoo will run `ruby` in your shell. Make sure `ruby` points to the appropriate executable.
+
+If a `ruby` binary is not present, you will see a "ruby: command not found" error.
+
+To override the Ruby binary, set the `PROMPTFOO_RUBY` environment variable. You may set it to a path (such as `/path/to/ruby`) or just an executable in your PATH (such as `ruby`).
+
+## Negation
+
+Use `not-ruby` to invert the final pass/fail result while preserving the returned score. Numeric scores are still compared against `threshold` before the result is inverted:
+
+```yaml
+assert:
+  - type: not-ruby
+    value: output.include?('error')
+```
+
+## Other assertion types
+
+For more info on assertions, see [Test assertions](/docs/configuration/expected-outputs).

site/docs/configuration/expected-outputs/similar.md

@@ -0,0 +1,116 @@
+---
+sidebar_position: 55
+description: Calculate semantic similarity scores between actual and expected outputs using advanced embedding models and multiple distance metrics
+---
+
+# Similarity (embeddings)
+
+The `similar` assertion checks if an embedding of the LLM's output
+is semantically similar to the expected value,
+using a configurable similarity or distance metric with a threshold.
+
+By default, embeddings are computed via OpenAI's `text-embedding-3-large` model.
+
+Example:
+
+```yaml
+assert:
+  - type: similar
+    value: 'The expected output'
+    threshold: 0.8
+```
+
+If you provide an array of values, the test will pass if it is similar to at least one of them:
+
+```yaml
+assert:
+  - type: similar
+    value:
+      - The expected output
+      - Expected output
+      - file://my_expected_output.txt
+    threshold: 0.8
+```
+
+The negated `not-similar` assertion is the logical inverse: with an array of values it passes only when the output is dissimilar to **every** value (and fails as soon as it is too similar to any one of them). This is the natural way to assert that an output does not resemble any item in a list of forbidden or canned answers.
+
+## Similarity Metrics
+
+You can specify which metric to use by including it in the assertion type. The default is `similar` (cosine similarity).
+
+### Cosine Similarity (default)
+
+Measures the cosine of the angle between two vectors. Range: -1 to 1 (higher is more similar), though text embeddings typically produce values between 0 and 1.
+
+```yaml
+assert:
+  # Default - uses cosine similarity
+  - type: similar
+    value: 'The expected output'
+    threshold: 0.8
+
+  # Explicit cosine
+  - type: similar:cosine
+    value: 'The expected output'
+    threshold: 0.8
+```
+
+**When to use:** Best for semantic similarity where you care about the direction of the embedding vector, not its magnitude. This is the industry standard for embeddings.
+
+### Dot Product
+
+Computes the dot product of two vectors. Range: unbounded, but typically 0 to 1 for normalized embeddings (higher is more similar).
+
+```yaml
+assert:
+  - type: similar:dot
+    value: 'The expected output'
+    threshold: 0.8
+```
+
+**When to use:** Useful when you want to match the metric used in your production vector database (many use dot product for performance). For normalized embeddings, dot product is nearly equivalent to cosine similarity.
+
+### Euclidean Distance
+
+Computes the straight-line distance between two vectors. Range: 0 to ∞ (lower is more similar).
+
+```yaml
+assert:
+  - type: similar:euclidean
+    value: 'The expected output'
+    threshold: 0.5 # Maximum distance threshold
+```
+
+**When to use:** When you care about both the angle and magnitude differences between vectors. Note that the threshold represents the _maximum_ distance (not minimum similarity), so lower values are stricter.
+
+**Important:** For euclidean distance, the threshold semantics are inverted - it represents the _maximum_ acceptable distance rather than minimum similarity.
+
+## Overriding the provider
+
+By default `similar` will use OpenAI. To specify the model that creates the embeddings, do one of the following:
+
+1. Use `test.options` or `defaultTest.options` to override the provider across the entire test suite. For example:
+
+   ```yaml
+   defaultTest:
+     options:
+       provider:
+         embedding:
+           id: azureopenai:embedding:text-embedding-ada-002
+           config:
+             apiHost: xxx.openai.azure.com
+   tests:
+     assert:
+       - type: similar
+         value: Hello world
+   ```
+
+2. Set `assertion.provider` on a per-assertion basis. For example:
+
+   ```yaml
+   tests:
+     assert:
+       - type: similar
+         value: Hello world
+         provider: huggingface:sentence-similarity:sentence-transformers/all-MiniLM-L6-v2
+   ```