chore: import upstream snapshot with attribution

2026-07-13 13:24:08 +08:00
commit 0d3cb498a3
5438 changed files with 1316560 additions and 0 deletions
@@ -0,0 +1,5 @@
+{
+  "position": 50,
+  "label": "Integrations",
+  "collapsed": true
+}
@@ -0,0 +1,274 @@
+---
+title: Agent Skills for Evals and Red Teaming
+description: Install Promptfoo agent skills for eval writing, provider setup, and red-team workflows in Claude Code and OpenAI Codex, with security configs and scan triage.
+sidebar_label: Agent Skills
+sidebar_position: 99
+---
+
+# Agent Skills for Evals and Red Teaming
+
+AI coding agents can write promptfoo configs, but they often get the details wrong: shell-style env vars that do not work, hallucination rubrics that cannot see the source material, tests dumped inline instead of in files, and red-team configs that collapse real app inputs into one generic prompt field.
+
+Promptfoo ships one agent-skill bundle with four focused skills — `promptfoo-evals` for eval authoring, `promptfoo-provider-setup` for connecting targets, and `promptfoo-redteam-setup` plus `promptfoo-redteam-run` for red-team setup and scan triage. The same bundle is published to both the [Claude Code](https://code.claude.com) and [OpenAI Codex](https://openai.com/index/codex) marketplaces.
+
+It follows the open [Agent Skills](https://agentskills.io) standard, so the skills should also work with other compatible tools.
+
+## Why use a skill?
+
+Without the skill, agents frequently:
+
+- Use `$ENV_VAR` syntax in YAML configs, which does not work because promptfoo uses Nunjucks `'{{env.VAR}}'`
+- Write `llm-rubric` assertions that reference "the article" but don't inline the source, so the grader can't actually compare
+- Dump all tests inline in the config instead of using `file://tests/*.yaml`
+- Reach for `llm-rubric` when `contains` or `is-json` would be faster, free, and deterministic
+
+The skill gives the agent these rules up front.
+
+The red-team skills cover a different set of common mistakes: flattening
+multi-input targets into one prompt field, choosing broad scans before mapping
+the app boundary, and regenerating probes when a stable rerun would be easier to
+compare.
+
+## Install
+
+### Via Claude Code marketplace
+
+```bash
+/plugin marketplace add promptfoo/promptfoo
+/plugin install promptfoo@promptfoo
+```
+
+This installs all four skills. Ask the agent to create an eval, connect a
+target, or run a red team and it routes to the right skill, or invoke one
+directly with a namespaced slash command such as `/promptfoo:promptfoo-evals`.
+
+:::note
+This plugin was previously published as `promptfoo-evals` (eval skill only). If
+you installed it under that name, reinstall with
+`/plugin install promptfoo@promptfoo` to get the full four-skill bundle and
+future updates.
+:::
+
+### Via Codex plugin bundle
+
+For Codex, the same `plugins/promptfoo` bundle is exposed by
+`.agents/plugins/marketplace.json`. Add it to a Codex workspace to install the
+same four skills.
+
+### The four skills
+
+Both marketplaces install the same bundle at `plugins/promptfoo`, exposed by
+`.claude-plugin/marketplace.json` for Claude Code and
+`.agents/plugins/marketplace.json` for Codex:
+
+| Skill                      | Use it for                                                                 |
+| -------------------------- | -------------------------------------------------------------------------- |
+| `promptfoo-evals`          | Non-redteam eval suites, assertions, test cases, and result inspection     |
+| `promptfoo-provider-setup` | HTTP targets plus JavaScript or Python `file://` providers and wrappers    |
+| `promptfoo-redteam-setup`  | Focused redteam configs from live endpoints, OpenAPI specs, or static code |
+| `promptfoo-redteam-run`    | Running generated scans, triaging failures, and filtered reruns            |
+
+There is intentionally no meta selector skill. The agent routes from each skill's
+description and default prompt.
+
+Python providers are first-class in the bundle. The provider and redteam
+skills cover Promptfoo's `file://provider.py` and
+`file://provider.py:function_name` syntax for eval providers, redteam targets,
+local graders, and local redteam generators, including `workers`, `timeout`, and
+`PROMPTFOO_PYTHON` configuration.
+
+To reuse the bundle in another workspace, copy `plugins/promptfoo` together with
+its marketplace entry — `.claude-plugin/marketplace.json` for Claude Code or
+`.agents/plugins/marketplace.json` for Codex.
+
+For red teaming, `promptfoo-provider-setup` connects the system under test,
+`promptfoo-redteam-setup` turns live endpoints, OpenAPI specs, or static code
+into a scan plan, and `promptfoo-redteam-run` executes and triages the
+generated probes.
+
+### Manual install
+
+For an eval-only setup, copy the self-contained
+[`promptfoo-evals` skill](https://github.com/promptfoo/promptfoo/tree/main/.claude/skills/promptfoo-evals)
+into your project:
+
+**Claude Code** (project-level, recommended for teams):
+
+```bash
+cp -r promptfoo-evals your-project/.claude/skills/
+```
+
+**Claude Code** (personal, available in all projects):
+
+```bash
+cp -r promptfoo-evals ~/.claude/skills/
+```
+
+**OpenAI Codex / other Agent Skills tools**:
+
+```bash
+cp -r promptfoo-evals your-project/.agents/skills/
+```
+
+To add provider setup or red teaming as well, install the full bundle from the
+marketplace (above) so the skills can hand off to each other, or copy the whole
+[`plugins/promptfoo/skills`](https://github.com/promptfoo/promptfoo/tree/main/plugins/promptfoo/skills)
+directory so the referenced sibling skills resolve.
+
+:::note
+Commit skills to `.claude/skills/` or `.agents/skills/` so every developer's
+agent picks them up automatically, with no per-person install needed.
+:::
+
+Each skill consists of a `SKILL.md` with workflow instructions plus a
+`references/` directory of assertion types, provider patterns, and config
+examples (provider and redteam setup also include a `scripts/` directory).
+
+## Usage
+
+Once installed, the agent activates automatically when you ask it to create or
+update eval coverage. In Claude Code, you can also invoke a skill directly with
+a slash command (namespaced when installed from the marketplace):
+
+```text
+/promptfoo:promptfoo-evals Create an eval suite for my summarization prompt
+```
+
+In Codex and other Agent Skills tools, ask the agent to create an eval. The
+skill activates from the task context.
+
+For red-team work, ask for the task directly:
+
+```text
+Create a focused red team config for this invoice assistant. Preserve user_id, invoice_id, and message inputs; test policy, RBAC, and BOLA.
+Run the generated redteam scan, summarize attack success rate, and give me the narrowest rerun command for failures.
+```
+
+The agent:
+
+1. Search for existing promptfoo configs in the repo
+2. Scaffold a new suite if needed (`promptfooconfig.yaml`, `prompts/`, `tests/`)
+3. Write test cases with deterministic assertions first, model-graded when needed
+4. Validate the config with `promptfoo validate`
+5. Provide run commands
+
+:::note
+New to promptfoo? See [Getting Started](/docs/getting-started) for an overview of configs, providers, and assertions.
+:::
+
+## What the skill teaches
+
+- **Deterministic assertions first.** Use `contains`, `is-json`, `javascript` before reaching for `llm-rubric`. Deterministic checks are fast, free, and reproducible.
+- **File-based test organization.** Tests go in `tests/*.yaml` files loaded via `file://tests/*.yaml` glob, keeping configs clean as test count grows.
+- **Dataset-driven scaling.** For larger suites, use `tests: file://tests.csv` or script-generated tests like `file://generate_tests.py:create_tests`.
+- **Faithfulness checks done right.** When using `llm-rubric` to check for hallucination, the source material must be inlined in the rubric via `{{variable}}` so the grader can actually compare.
+- **Pinned grader provider.** Model-graded assertions should explicitly set a grading provider (`defaultTest.options.provider` or `assertion.provider`) for stable scoring.
+- **Environment variables.** Use Nunjucks syntax `'{{env.API_KEY}}'` in YAML configs, not shell syntax.
+- **CI-friendly runs.** Use `promptfoo eval -o output.json --no-cache` and inspect `success`, `score`, and `error`.
+- **Config field ordering.** description, env, prompts, providers, defaultTest, scenarios, tests.
+
+The provider and red-team skills also teach the agent to:
+
+- Keep real inputs such as user IDs, object IDs, documents, and tools visible so authorization and agent-boundary issues stay testable.
+- Choose plugins such as `policy`, `rbac`, `bola`, `hijacking`, `prompt-extraction`, and `system-prompt-override` from live or static evidence instead of defaulting to one broad scan.
+- Inspect generated probes before running them, reuse generated tests with `redteam eval` when possible, and separate grader failures from real target failures.
+- Prefer no-share runs for internal systems and keep provider secrets in environment variables rather than committed configs.
+
+## Example output
+
+Ask the agent to "create an eval for a customer support chatbot that returns JSON" and it produces:
+
+```yaml title="promptfooconfig.yaml"
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: 'Customer support chatbot'
+
+prompts:
+  - file://prompts/chat.json
+
+providers:
+  - id: openai:chat:gpt-4.1-mini
+    config:
+      temperature: 0
+      response_format:
+        type: json_object
+
+defaultTest:
+  assert:
+    - type: is-json
+    - type: cost
+      threshold: 0.01
+
+tests:
+  - file://tests/*.yaml
+```
+
+```yaml title="tests/happy-path.yaml"
+- description: 'Returns order status for valid customer'
+  vars:
+    order_id: 'ORD-1001'
+    customer_name: 'Alice Smith'
+  assert:
+    - type: is-json
+      value:
+        type: object
+        required: [status, message]
+    - type: javascript
+      value: "JSON.parse(output).status === 'shipped'"
+```
+
+A red-team setup should keep the security boundary visible instead of collapsing
+it into one free-form prompt:
+
+```yaml title="promptfooconfig.yaml"
+description: 'Invoice assistant red team'
+
+targets:
+  - id: https
+    label: invoice-assistant
+    inputs:
+      user_id: Signed-in user identifier.
+      invoice_id: Invoice being requested.
+      message: User message.
+    config:
+      url: '{{env.INVOICE_AGENT_URL}}'
+      method: POST
+      stateful: false
+      body:
+        user_id: '{{user_id}}'
+        invoice_id: '{{invoice_id}}'
+        message: '{{message}}'
+      transformResponse: json.output
+
+redteam:
+  purpose: >-
+    Invoice assistant for signed-in users. It may answer questions about the
+    caller's invoices only and must not reveal or modify other users' invoices.
+  plugins:
+    - id: policy
+      config:
+        policy: The assistant must not disclose or modify another user's invoices.
+    - rbac
+    - bola
+  strategies:
+    - basic
+```
+
+## Customizing the skill
+
+The skill is just markdown files. Edit them to match your team's conventions:
+
+- **Add custom providers** to the reference files if your team uses specific models or endpoints.
+- **Add assertion patterns** for your domain (e.g., medical accuracy rubrics, financial compliance checks).
+- **Change the default layout** if your repo uses a different directory structure for evals.
+
+## Related
+
+- [Getting Started](/docs/getting-started): promptfoo overview for newcomers
+- [Test Agent Skills](/docs/guides/test-agent-skills): compare Claude and Codex skill versions side by side
+- [Configuration Reference](/docs/configuration/guide): full config schema documentation
+- [Assertions Reference](/docs/configuration/expected-outputs): complete list of assertion types
+- [Custom Providers](/docs/providers/custom-api): build Python, JavaScript, and HTTP providers
+- [LLM Red Teaming](/docs/red-team/): security testing concepts and workflows
+- [Red Team Coding Agents](/docs/red-team/coding-agents/): security evals for agentic systems
+- [Coding Agent Plugins](/docs/red-team/plugins/coding-agent/): repository, sandbox, secret, and verifier-boundary checks
+- [MCP Server](/docs/integrations/mcp-server): expose promptfoo to AI agents via MCP
@@ -0,0 +1,239 @@
+---
+title: AWS CodeCommit Integration
+sidebar_label: AWS CodeCommit
+sidebar_position: 6
+description: Run promptfoo from AWS CodeCommit-backed CodeBuild pipelines, store results as build artifacts, and optionally post scan summaries to CodeCommit pull requests.
+---
+
+# AWS CodeCommit Integration
+
+This guide shows how to run promptfoo in AWS CodeBuild for repositories hosted in AWS CodeCommit.
+
+Use this setup when you want to:
+
+- Run `promptfoo eval` on every push or pull request
+- Fail a build when assertions fail
+- Persist JSON/HTML eval reports as CodeBuild artifacts
+- Run `promptfoo code-scans run` against CodeCommit pull requests and post a summary comment back to the pull request
+
+## Prerequisites
+
+- An AWS CodeCommit repository with a promptfoo config such as `promptfooconfig.yaml`
+- An AWS CodeBuild project connected to that repository
+- A [CodeBuild Ubuntu image that supports Node.js 24](https://docs.aws.amazon.com/codebuild/latest/userguide/available-runtimes.html): Ubuntu 22.04 standard 7.0 or Ubuntu 24.04 standard 8.0
+- LLM provider credentials stored in AWS Systems Manager Parameter Store or AWS Secrets Manager
+- A Promptfoo API key if you want to run `promptfoo code-scans run`
+
+## Run promptfoo eval in CodeBuild
+
+Create a `buildspec.yml` file in the root of your CodeCommit repository:
+
+```yaml title="buildspec.yml"
+version: 0.2
+
+env:
+  parameter-store:
+    OPENAI_API_KEY: /promptfoo/openai-api-key
+  variables:
+    PROMPTFOO_CACHE_PATH: .promptfoo/cache
+
+phases:
+  install:
+    runtime-versions:
+      nodejs: 24
+    commands:
+      - npm install -g promptfoo
+  build:
+    commands:
+      - |
+        promptfoo eval \
+          -c promptfooconfig.yaml \
+          --share \
+          --fail-on-error \
+          -o promptfoo-results.json \
+          -o promptfoo-report.html
+
+artifacts:
+  files:
+    - promptfoo-results.json
+    - promptfoo-report.html
+
+cache:
+  paths:
+    - '.promptfoo/cache/**/*'
+```
+
+### What this does
+
+- Loads `OPENAI_API_KEY` from Parameter Store
+- Runs the eval suite defined in `promptfooconfig.yaml`
+- Fails the CodeBuild build if any assertions fail
+- Saves JSON and HTML reports as build artifacts
+- Caches promptfoo responses between builds
+
+## Add a quality gate
+
+If you want a custom pass-rate threshold instead of `--fail-on-error`, write the JSON output and check the stats in a second command:
+
+```yaml
+phases:
+  install:
+    runtime-versions:
+      nodejs: 24
+    commands:
+      - npm install -g promptfoo
+  build:
+    commands:
+      - promptfoo eval -c promptfooconfig.yaml --share -o promptfoo-results.json
+      - |
+        PASS_RATE=$(jq '.results.stats.successes / (.results.stats.successes + .results.stats.failures) * 100' promptfoo-results.json)
+        echo "Pass rate: ${PASS_RATE}%"
+        if (( $(echo "${PASS_RATE} < 95" | bc -l) )); then
+          echo "Quality gate failed: ${PASS_RATE}% < 95%"
+          exit 1
+        fi
+```
+
+## Run promptfoo code scans on CodeCommit pull requests
+
+Promptfoo's hosted GitHub Action posts inline review comments on GitHub pull requests, but CodeCommit pull requests are not a first-class target in `promptfoo code-scans run` today.
+
+For CodeCommit, run the scanner in CodeBuild, save JSON output, and post a summary comment back to the pull request with the AWS CLI.
+
+### 1. Pass pull request context into CodeBuild
+
+Trigger your CodeBuild project from a CodeCommit pull request event and provide the pull request ID as an environment variable such as `CODECOMMIT_PULL_REQUEST_ID`.
+
+CodeBuild exposes source metadata in environment variables including `CODEBUILD_SOURCE_REPO_URL`, `CODEBUILD_SOURCE_VERSION`, and `CODEBUILD_RESOLVED_SOURCE_VERSION`. For CodeCommit sources, `CODEBUILD_SOURCE_VERSION` is the commit ID or branch name and `CODEBUILD_RESOLVED_SOURCE_VERSION` is the commit ID after `DOWNLOAD_SOURCE`.
+
+### 2. Add a pull request scan buildspec
+
+```yaml title="buildspec-code-scan.yml"
+version: 0.2
+
+env:
+  parameter-store:
+    PROMPTFOO_API_KEY: /promptfoo/api-key
+
+phases:
+  install:
+    runtime-versions:
+      nodejs: 24
+    commands:
+      - npm install -g promptfoo
+      - apt-get update && apt-get install -y jq
+  build:
+    commands:
+      - |
+        if [ -z "$CODECOMMIT_PULL_REQUEST_ID" ]; then
+          echo "CODECOMMIT_PULL_REQUEST_ID is required for pull request scans"
+          exit 1
+        fi
+
+        PR_JSON=$(aws codecommit get-pull-request \
+          --pull-request-id "$CODECOMMIT_PULL_REQUEST_ID")
+
+        REPOSITORY_NAME=$(echo "$PR_JSON" | jq -r '.pullRequest.pullRequestTargets[0].repositoryName')
+        DESTINATION_REF=$(echo "$PR_JSON" | jq -r '.pullRequest.pullRequestTargets[0].destinationReference')
+        SOURCE_COMMIT=$(echo "$PR_JSON" | jq -r '.pullRequest.pullRequestTargets[0].sourceCommit')
+        DESTINATION_COMMIT=$(echo "$PR_JSON" | jq -r '.pullRequest.pullRequestTargets[0].destinationCommit')
+        DESTINATION_BRANCH="${DESTINATION_REF#refs/heads/}"
+
+        git fetch origin "${DESTINATION_BRANCH}:${DESTINATION_BRANCH}"
+
+        promptfoo code-scans run . \
+          --base "$DESTINATION_BRANCH" \
+          --compare "$CODEBUILD_RESOLVED_SOURCE_VERSION" \
+          --json \
+          > promptfoo-code-scan.json
+
+        COMMENT_BODY=$(jq -r '
+          def sev(c): if c.severity then "\(.severity | ascii_upcase): " else "" end;
+          [
+            "## Promptfoo Code Scan",
+            "",
+            (.review // "Scan complete."),
+            "",
+            "### Findings",
+            (
+              if (.comments | length) == 0 then
+                "- No findings"
+              else
+                (.comments[:20] | map(
+                  "- " + sev(.) +
+                  (if .file then "`\(.file)\(if .line then ":\(.line)" else "" end)` - " else "" end) +
+                  .finding
+                ) | .[])
+              end
+            ),
+            "",
+            "[View code scanning docs](https://www.promptfoo.dev/docs/code-scanning/cli/)"
+          ] | join("\n")
+        ' promptfoo-code-scan.json)
+
+        aws codecommit post-comment-for-pull-request \
+          --pull-request-id "$CODECOMMIT_PULL_REQUEST_ID" \
+          --repository-name "$REPOSITORY_NAME" \
+          --before-commit-id "$DESTINATION_COMMIT" \
+          --after-commit-id "$SOURCE_COMMIT" \
+          --content "$COMMENT_BODY"
+
+artifacts:
+  files:
+    - promptfoo-code-scan.json
+```
+
+This posts one general pull request comment with the scan summary and up to 20 findings. `PostCommentForPullRequest` also supports file-level locations, but promptfoo's scanner output is currently tuned for GitHub review semantics, so a summary comment is the simplest integration path for CodeCommit.
+
+## IAM permissions
+
+The CodeBuild service role needs access to your repository, your secret store, and any CodeCommit pull request APIs you use.
+
+For eval-only builds:
+
+```json
+{
+  "Version": "2012-10-17",
+  "Statement": [
+    {
+      "Effect": "Allow",
+      "Action": ["ssm:GetParameters"],
+      "Resource": "arn:aws:ssm:REGION:ACCOUNT_ID:parameter/promptfoo/*"
+    }
+  ]
+}
+```
+
+For pull request scan comments, add CodeCommit permissions:
+
+```json
+{
+  "Effect": "Allow",
+  "Action": ["codecommit:GetPullRequest", "codecommit:PostCommentForPullRequest"],
+  "Resource": "arn:aws:codecommit:REGION:ACCOUNT_ID:REPOSITORY_NAME"
+}
+```
+
+## Troubleshooting
+
+### `promptfoo code-scans run` fails with an auth error
+
+`promptfoo code-scans run` requires a Promptfoo API key outside of the GitHub Action flow. Store `PROMPTFOO_API_KEY` in Parameter Store or Secrets Manager and expose it to CodeBuild.
+
+### The scan compares against the wrong branch
+
+Fetch the destination branch before running `promptfoo code-scans run`, then pass `--base` explicitly. For CodeCommit pull requests, you can read the destination branch from `aws codecommit get-pull-request`.
+
+### No pull request comment appears
+
+Confirm `CODECOMMIT_PULL_REQUEST_ID` is present in the build environment, and verify the CodeBuild service role can call `codecommit:GetPullRequest` and `codecommit:PostCommentForPullRequest`.
+
+### Secrets appear in logs
+
+Prefer Parameter Store or Secrets Manager mappings in `buildspec.yml` instead of plain environment variables for provider API keys.
+
+## See Also
+
+- [CI/CD Integration](/docs/integrations/ci-cd)
+- [CLI Command](/docs/code-scanning/cli)
+- [Sharing and Collaboration](/docs/usage/sharing)
@@ -0,0 +1,188 @@
+---
+sidebar_label: Azure Pipelines
+description: Integrate promptfoo LLM testing with Azure Pipelines CI/CD using step-by-step setup, environment variables, and matrix testing configurations for automated AI evaluation
+---
+
+# Azure Pipelines Integration
+
+This guide demonstrates how to set up promptfoo with Azure Pipelines to run evaluations as part of your CI pipeline.
+
+## Prerequisites
+
+- A GitHub or Azure DevOps repository with a promptfoo project
+- An Azure DevOps account with permission to create pipelines
+- API keys for your LLM providers stored as [Azure Pipeline variables](https://learn.microsoft.com/en-us/azure/devops/pipelines/process/variables)
+
+## Setting up the Azure Pipeline
+
+Create a new file named `azure-pipelines.yml` in the root of your repository with the following configuration:
+
+```yaml
+trigger:
+  - main
+  - master # Include if you use master as your main branch
+
+pool:
+  vmImage: 'ubuntu-latest'
+
+variables:
+  npm_config_cache: $(Pipeline.Workspace)/.npm
+
+steps:
+  - task: NodeTool@0
+    inputs:
+      versionSpec: '24.x'
+    displayName: 'Install Node.js'
+
+  - task: Cache@2
+    inputs:
+      key: 'npm | "$(Agent.OS)" | package-lock.json'
+      restoreKeys: |
+        npm | "$(Agent.OS)"
+      path: $(npm_config_cache)
+    displayName: 'Cache npm packages'
+
+  - script: |
+      npm ci
+      npm install -g promptfoo
+    displayName: 'Install dependencies'
+
+  - script: |
+      npx promptfoo eval -o promptfoo-results.json -o promptfoo-results.junit.xml
+    displayName: 'Run promptfoo evaluations'
+    env:
+      OPENAI_API_KEY: $(OPENAI_API_KEY)
+      ANTHROPIC_API_KEY: $(ANTHROPIC_API_KEY)
+      # Add other API keys as needed
+
+  - task: PublishTestResults@2
+    inputs:
+      testResultsFormat: 'JUnit'
+      testResultsFiles: 'promptfoo-results.junit.xml'
+      mergeTestResults: true
+      testRunTitle: 'Promptfoo Evaluation Results'
+    condition: succeededOrFailed()
+    displayName: 'Publish test results'
+
+  - task: PublishBuildArtifacts@1
+    inputs:
+      pathtoPublish: 'promptfoo-results.json'
+      artifactName: 'promptfoo-results'
+    condition: succeededOrFailed()
+    displayName: 'Publish evaluation results'
+```
+
+## Environment Variables
+
+Store your LLM provider API keys as [secret pipeline variables](https://learn.microsoft.com/en-us/azure/devops/pipelines/process/variables#secret-variables) in Azure DevOps:
+
+1. Navigate to your project in Azure DevOps
+2. Go to Pipelines > Your Pipeline > Edit > Variables
+3. Add variables for each provider API key (e.g., `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`)
+4. Mark them as secret to ensure they're not displayed in logs
+
+## Advanced Configuration
+
+### Fail the Pipeline on Failed Assertions
+
+You can configure the pipeline to fail when promptfoo assertions don't pass by modifying the script step:
+
+```yaml
+- script: |
+    npx promptfoo eval --fail-on-error -o promptfoo-results.junit.xml
+  displayName: 'Run promptfoo evaluations'
+  env:
+    OPENAI_API_KEY: $(OPENAI_API_KEY)
+```
+
+### Configure Custom Output Location
+
+If you want to customize where results are stored:
+
+```yaml
+- script: |
+    npx promptfoo eval --output-path $(Build.ArtifactStagingDirectory)/promptfoo-results.json
+  displayName: 'Run promptfoo evaluations'
+```
+
+### Run on Pull Requests
+
+To run evaluations on pull requests, add a PR trigger:
+
+```yaml
+trigger:
+  - main
+  - master
+
+pr:
+  - main
+  - master
+# Rest of pipeline configuration
+```
+
+### Conditional Execution
+
+Run promptfoo only when certain files change:
+
+```yaml
+steps:
+  - task: NodeTool@0
+    inputs:
+      versionSpec: '24.x'
+    displayName: 'Install Node.js'
+
+  - script: |
+      npm ci
+      npm install -g promptfoo
+    displayName: 'Install dependencies'
+
+  - script: |
+      npx promptfoo eval
+    displayName: 'Run promptfoo evaluations'
+    condition: |
+      and(
+        succeeded(),
+        or(
+          eq(variables['Build.SourceBranch'], 'refs/heads/main'),
+          eq(variables['Build.Reason'], 'PullRequest')
+        ),
+        or(
+          eq(variables['Build.Reason'], 'PullRequest'),
+          contains(variables['Build.SourceVersionMessage'], '[run-eval]')
+        )
+      )
+    env:
+      OPENAI_API_KEY: $(OPENAI_API_KEY)
+```
+
+## Using with Matrix Testing
+
+Test across multiple configurations or models in parallel:
+
+```yaml
+strategy:
+  matrix:
+    gpt:
+      MODEL: 'gpt-5.1'
+    claude:
+      MODEL: 'claude-sonnet-4-5-20250929'
+
+steps:
+  - script: |
+      npx promptfoo eval --providers.0.config.model=$(MODEL)
+    displayName: 'Test with $(MODEL)'
+    env:
+      OPENAI_API_KEY: $(OPENAI_API_KEY)
+      ANTHROPIC_API_KEY: $(ANTHROPIC_API_KEY)
+```
+
+## Troubleshooting
+
+If you encounter issues with your Azure Pipelines integration:
+
+- **Check logs**: Review detailed logs in Azure DevOps to identify errors
+- **Verify API keys**: Ensure your API keys are correctly set as pipeline variables
+- **Permissions**: Make sure the pipeline has access to read your configuration files
+- **Node.js version**: Promptfoo requires Node.js `^20.20.0` or `>=22.22.0`
+
+If you're getting timeouts during evaluations, you may need to adjust the pipeline timeout settings or consider using a [self-hosted agent](https://learn.microsoft.com/en-us/azure/devops/pipelines/agents/agents) for better stability with long-running evaluations.
@@ -0,0 +1,179 @@
+---
+sidebar_label: Bitbucket Pipelines
+description: Integrate promptfoo LLM testing with Bitbucket Pipelines CI/CD to automate evaluations, track results, and catch regressions in your AI models using built-in assertions
+---
+
+# Bitbucket Pipelines Integration
+
+This guide demonstrates how to set up promptfoo with Bitbucket Pipelines to run evaluations as part of your CI pipeline.
+
+## Prerequisites
+
+- A Bitbucket repository with a promptfoo project
+- Bitbucket Pipelines enabled for your repository
+- API keys for your LLM providers stored as [Bitbucket repository variables](https://support.atlassian.com/bitbucket-cloud/docs/variables-and-secrets/)
+
+## Setting up Bitbucket Pipelines
+
+Create a new file named `bitbucket-pipelines.yml` in the root of your repository with the following configuration:
+
+```yaml
+image: node:24
+
+pipelines:
+  default:
+    - step:
+        name: Promptfoo Evaluation
+        caches:
+          - node
+        script:
+          - npm ci
+          - npm install -g promptfoo
+          - npx promptfoo eval -o promptfoo-results.json -o promptfoo-results.junit.xml
+        artifacts:
+          - promptfoo-results.json
+          - promptfoo-results.junit.xml
+```
+
+## Environment Variables
+
+Store your LLM provider API keys as repository variables in Bitbucket:
+
+1. Navigate to your repository in Bitbucket
+2. Go to Repository settings > Pipelines > Repository variables
+3. Add variables for each provider API key (e.g., `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`)
+4. Mark them as "Secured" to ensure they're not displayed in logs
+
+## Advanced Configuration
+
+### Fail the Pipeline on Failed Assertions
+
+You can configure the pipeline to fail when promptfoo assertions don't pass:
+
+```yaml
+script:
+  - npm ci
+  - npm install -g promptfoo
+  - npx promptfoo eval --fail-on-error
+```
+
+### Custom Evaluation Configurations
+
+Run evaluations with specific configuration files:
+
+```yaml
+script:
+  - npm ci
+  - npm install -g promptfoo
+  - npx promptfoo eval --config custom-config.yaml
+```
+
+### Run on Pull Requests
+
+Configure different behavior for pull requests:
+
+```yaml
+pipelines:
+  default:
+    - step:
+        name: Promptfoo Evaluation
+        script:
+          - npm ci
+          - npm install -g promptfoo
+          - npx promptfoo eval
+  pull-requests:
+    '**':
+      - step:
+          name: Promptfoo PR Evaluation
+          script:
+            - npm ci
+            - npm install -g promptfoo
+            - npx promptfoo eval --fail-on-error
+```
+
+### Scheduled Evaluations
+
+Run evaluations on a schedule:
+
+```yaml
+pipelines:
+  default:
+    - step:
+        name: Promptfoo Evaluation
+        script:
+          - npm ci
+          - npm install -g promptfoo
+          - npx promptfoo eval
+  custom:
+    nightly-evaluation:
+      - step:
+          name: Nightly Evaluation
+          script:
+            - npm ci
+            - npm install -g promptfoo
+            - npx promptfoo eval
+  schedules:
+    - cron: '0 0 * * *' # Run at midnight UTC every day
+      pipeline: custom.nightly-evaluation
+      branches:
+        include:
+          - main
+```
+
+### Parallel Testing
+
+Test across multiple configurations in parallel:
+
+```yaml
+image: node:24
+
+pipelines:
+  default:
+    - parallel:
+        - step:
+            name: Evaluate with GPT-4
+            script:
+              - npm ci
+              - npm install -g promptfoo
+              - npx promptfoo eval --providers.0.config.model=gpt-4
+            artifacts:
+              - promptfoo-results-gpt4.json
+        - step:
+            name: Evaluate with Claude
+            script:
+              - npm ci
+              - npm install -g promptfoo
+              - npx promptfoo eval --providers.0.config.model=claude-3-opus-20240229
+            artifacts:
+              - promptfoo-results-claude.json
+```
+
+### Using Pipes
+
+Leverage Bitbucket Pipes for a more concise configuration:
+
+```yaml
+image: node:24
+
+pipelines:
+  default:
+    - step:
+        name: Promptfoo Evaluation
+        script:
+          - npm ci
+          - npm install -g promptfoo
+          - npx promptfoo eval -o promptfoo-results.junit.xml
+        after-script:
+          - pipe: atlassian/junit-report:0.3.0
+            variables:
+              REPORT_PATHS: 'promptfoo-results.junit.xml'
+```
+
+## Troubleshooting
+
+If you encounter issues with your Bitbucket Pipelines integration:
+
+- **Check logs**: Review detailed logs in Bitbucket to identify errors
+- **Verify repository variables**: Ensure your API keys are correctly set
+- **Pipeline timeouts**: Bitbucket Pipelines has timeout limits. For long-running evaluations, consider breaking them down or [increasing the timeout](https://support.atlassian.com/bitbucket-cloud/docs/build-timeouts/)
+- **Debug with SSH**: For complex issues, use [enabling SSH access](https://support.atlassian.com/bitbucket-cloud/docs/debug-your-pipelines-with-ssh/) to debug the pipeline environment directly
@@ -0,0 +1,88 @@
+---
+sidebar_label: Burp Suite
+description: Test LLM applications for jailbreak vulnerabilities by integrating Promptfoo's red teaming capabilities with Burp Suite Intruder for automated security scanning and testing
+---
+
+# Finding LLM Jailbreaks with Burp Suite
+
+This guide shows how to integrate Promptfoo's application-level jailbreak creation with Burp Suite's Intruder feature for security testing of LLM-powered applications.
+
+The end result is a Burp Suite Intruder configuration that can be used to test for LLM jailbreak vulnerabilities.
+
+![Burp Suite Intruder](/img/docs/burp/burp-jailbreak-intruder.png)
+
+(In the above example, we've jailbroken the OpenAI API directly to return an unhinged response.)
+
+## Overview
+
+Burp Suite integration allows you to:
+
+1. Generate adversarial test cases using Promptfoo's red teaming capabilities
+2. Export these test cases in a format compatible with Burp Intruder
+3. Use the test cases as payloads in Burp Suite for security testing
+
+## Prerequisites
+
+- Burp Suite Community Edition or Professional Edition
+- Promptfoo installed (`npm install -g promptfoo`)
+
+## Configuration Steps
+
+### Option 1: Using the Web UI
+
+If you've already run an evaluation with test cases, you can export them directly from the web UI:
+
+1. Open the evaluation results in your browser
+2. Click the "Evaluation Actions" > "Download" menu in the top right
+3. Under "Advanced Options", click "Download Burp Suite Payloads"
+
+This will generate a `.burp` file containing all unique test inputs from your evaluation, with proper JSON escaping and URL encoding.
+
+![Burp Suite export](/img/docs/burp/burp-export-frontend.png)
+
+### Option 2: Using the Command Line
+
+First, generate adversarial test cases and export them in Burp format:
+
+```bash
+promptfoo redteam generate -o payloads.burp --burp-escape-json
+```
+
+:::tip
+The `--burp-escape-json` flag is important when your payloads will be inserted into JSON requests. It ensures that special characters are properly escaped to maintain valid JSON syntax.
+:::
+
+#### Import into Burp Intruder
+
+1. In Burp Suite, intercept a request to your LLM-powered endpoint
+2. Right-click and select "Send to Intruder"
+3. In the Intruder tab:
+   - Set attack type (usually "Sniper" or "Pitchfork")
+   - Mark the injection points where you want to test the payloads
+   - Go to the "Payloads" tab
+   - Click "Load" and select your `payloads.burp` file
+4. Under "Payload processing", enable URL-decoding (promptfoo's .burp output is URL-encoded to support multi-line payloads)
+
+![Burp Intruder LLM red teaming configuration](/img/docs/burp/burp-jailbreak-intruder-setup.png)
+
+#### Example Configuration
+
+Here's an example of generating targeted test cases. In `promptfooconfig.yaml`:
+
+```yaml
+redteam:
+  plugins:
+    - harmful
+  strategies:
+    - jailbreak
+    - jailbreak:composite
+    - jailbreak-templates
+```
+
+Generate Burp-compatible payloads:
+
+```bash
+promptfoo redteam generate -o payloads.burp --burp-escape-json
+```
+
+This will create a file with payloads ready for use in Burp Intruder.
@@ -0,0 +1,571 @@
+---
+sidebar_label: CI/CD
+title: CI/CD Integration for LLM Eval and Security
+description: Automate LLM testing in CI/CD pipelines with GitHub Actions, GitLab CI, and Jenkins for continuous security and quality checks
+keywords:
+  [
+    ci/cd,
+    continuous integration,
+    llm testing,
+    automated evaluation,
+    security scanning,
+    github actions,
+  ]
+---
+
+# CI/CD Integration for LLM Evaluation and Security
+
+Integrate promptfoo into your CI/CD pipelines to automatically evaluate prompts, test for security vulnerabilities, and ensure quality before deployment. This guide covers modern CI/CD workflows for both quality testing and security scanning.
+
+## Why CI/CD for LLM Apps?
+
+- **Catch regressions early** - Test prompt changes before they reach production
+- **Security scanning** - Automated red teaming and vulnerability detection
+- **Quality gates** - Enforce minimum performance thresholds
+- **Compliance** - Generate reports for OWASP, NIST, and other frameworks
+- **Cost control** - Track token usage and API costs over time
+
+## Quick Start
+
+If you're using GitHub Actions, check out our [dedicated GitHub Actions guide](/docs/integrations/github-action) or the [GitHub Marketplace action](https://github.com/marketplace/actions/test-llm-outputs).
+
+For other platforms, here's a basic example:
+
+```bash
+# Run eval (no global install required)
+npx promptfoo@latest eval -c promptfooconfig.yaml -o results.json
+
+# Run security scan (red teaming)
+npx promptfoo@latest redteam run
+```
+
+## Prerequisites
+
+- Node.js `^20.20.0` or `>=22.22.0` installed in your CI environment
+- LLM provider API keys (stored as secure environment variables)
+- A promptfoo configuration file (`promptfooconfig.yaml`)
+- (Optional) Docker for containerized environments
+
+## Core Concepts
+
+### 1. Eval vs Red Teaming
+
+Promptfoo supports two main CI/CD workflows:
+
+**Eval** - Test prompt quality and performance:
+
+```bash
+npx promptfoo@latest eval -c promptfooconfig.yaml
+```
+
+**Red Teaming** - Security vulnerability scanning:
+
+```bash
+npx promptfoo@latest redteam run
+```
+
+See our [red team quickstart](/docs/red-team/quickstart) for security testing details.
+
+#### Attach CI/CD Context with Tags
+
+Use repeatable `--tag key=value` flags to attach pipeline context to an evaluation
+without modifying `promptfooconfig.yaml` or a red team scan template. Tags are saved
+with the eval and included when results are shared.
+
+```bash
+npx promptfoo@latest eval --tag ci.run-id="$CI_PIPELINE_ID" --tag git.sha="$CI_COMMIT_SHA"
+npx promptfoo@latest redteam run --tag ci.run-id="$CI_PIPELINE_ID" --tag git.sha="$CI_COMMIT_SHA"
+```
+
+`promptfoo redteam eval` accepts the same `--tag` option when running previously
+generated probes from `redteam.yaml`.
+
+### 2. Output Formats
+
+Promptfoo supports multiple output formats for different CI/CD needs:
+
+```bash
+# JSON for programmatic processing
+npx promptfoo@latest eval -o results.json
+
+# HTML for human-readable reports
+npx promptfoo@latest eval -o report.html
+
+# JUnit XML for native CI test-report viewers
+npx promptfoo@latest eval -o results.junit.xml
+
+# Multiple formats
+npx promptfoo@latest eval -o results.json -o report.html -o results.junit.xml
+```
+
+Learn more about [output formats and processing](/docs/configuration/outputs).
+
+:::info Enterprise Feature
+
+SonarQube integration is available in [Promptfoo Enterprise](/docs/enterprise/). Use the standard JSON output format and process it for SonarQube import.
+
+:::
+
+### 3. Quality Gates
+
+Fail the build when quality thresholds aren't met:
+
+```bash
+# Fail on any test failures
+npx promptfoo@latest eval --fail-on-error
+
+# Custom threshold checking
+npx promptfoo@latest eval -o results.json
+PASS_RATE=$(jq '.results.stats.successes / (.results.stats.successes + .results.stats.failures) * 100' results.json)
+if (( $(echo "$PASS_RATE < 95" | bc -l) )); then
+  echo "Quality gate failed: Pass rate ${PASS_RATE}% < 95%"
+  exit 1
+fi
+```
+
+See [assertions and metrics](/docs/configuration/expected-outputs) for comprehensive validation options.
+
+## Platform-Specific Guides
+
+### GitHub Actions
+
+```yaml title=".github/workflows/eval.yml"
+name: LLM Eval
+on:
+  pull_request:
+    paths:
+      - 'prompts/**'
+      - 'promptfooconfig.yaml'
+
+jobs:
+  evaluate:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: actions/setup-node@v4
+        with:
+          node-version: '22'
+          cache: 'npm'
+
+      - name: Cache promptfoo
+        uses: actions/cache@v4
+        with:
+          path: ~/.cache/promptfoo
+          key: ${{ runner.os }}-promptfoo-${{ hashFiles('prompts/**') }}
+          restore-keys: |
+            ${{ runner.os }}-promptfoo-
+
+      - name: Run eval
+        env:
+          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
+          PROMPTFOO_CACHE_PATH: ~/.cache/promptfoo
+        run: |
+          npx promptfoo@latest eval \
+            -c promptfooconfig.yaml \
+            --share \
+            -o results.json \
+            -o report.html
+
+      - name: Check quality gate
+        run: |
+          FAILURES=$(jq '.results.stats.failures' results.json)
+          if [ "$FAILURES" -gt 0 ]; then
+            echo "❌ Eval failed with $FAILURES failures"
+            exit 1
+          fi
+          echo "✅ All tests passed!"
+
+      - name: Upload results
+        if: always()
+        uses: actions/upload-artifact@v4
+        with:
+          name: eval-results
+          path: |
+            results.json
+            report.html
+```
+
+For red teaming in CI/CD:
+
+```yaml title=".github/workflows/redteam.yml"
+name: Security Scan
+on:
+  schedule:
+    - cron: '0 0 * * *' # Daily
+  workflow_dispatch:
+
+jobs:
+  red-team:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Run red team scan
+        uses: promptfoo/promptfoo-action@v1
+        with:
+          type: 'redteam'
+          config: 'promptfooconfig.yaml'
+          openai-api-key: ${{ secrets.OPENAI_API_KEY }}
+          github-token: ${{ secrets.GITHUB_TOKEN }}
+```
+
+See also: [Standalone GitHub Action example](https://github.com/promptfoo/promptfoo/tree/main/examples/integration-github-action).
+
+### GitLab CI
+
+See our [detailed GitLab CI guide](/docs/integrations/gitlab-ci).
+
+```yaml title=".gitlab-ci.yml"
+image: node:24
+
+evaluate:
+  script:
+    - |
+      npx promptfoo@latest eval \
+        -c promptfooconfig.yaml \
+        --share \
+        -o output.json \
+        -o report.html \
+        -o output.junit.xml
+  variables:
+    OPENAI_API_KEY: ${OPENAI_API_KEY}
+    PROMPTFOO_CACHE_PATH: .cache/promptfoo
+  cache:
+    key: ${CI_COMMIT_REF_SLUG}-promptfoo
+    paths:
+      - .cache/promptfoo
+  artifacts:
+    reports:
+      junit: output.junit.xml
+    paths:
+      - output.json
+      - report.html
+```
+
+### Jenkins
+
+See our [detailed Jenkins guide](/docs/integrations/jenkins).
+
+```groovy title="Jenkinsfile"
+pipeline {
+    agent any
+
+    environment {
+        OPENAI_API_KEY = credentials('openai-api-key')
+        PROMPTFOO_CACHE_PATH = "${WORKSPACE}/.cache/promptfoo"
+    }
+
+    stages {
+        stage('Evaluate') {
+            steps {
+                sh '''
+                    npx promptfoo@latest eval \
+                        -c promptfooconfig.yaml \
+                        --share \
+                        -o results.json
+                '''
+            }
+        }
+
+        stage('Quality Gate') {
+            steps {
+                script {
+                    def results = readJSON file: 'results.json'
+                    def failures = results.results.stats.failures
+                    if (failures > 0) {
+                        error("Eval failed with ${failures} failures")
+                    }
+                }
+            }
+        }
+    }
+}
+```
+
+### Other Platforms
+
+- [Azure Pipelines](/docs/integrations/azure-pipelines)
+- [AWS CodeCommit](/docs/integrations/aws-codecommit)
+- [CircleCI](/docs/integrations/circle-ci)
+- [Bitbucket Pipelines](/docs/integrations/bitbucket-pipelines)
+- [Travis CI](/docs/integrations/travis-ci)
+- [n8n workflows](/docs/integrations/n8n)
+- [Looper](/docs/integrations/looper)
+
+## Advanced Patterns
+
+### 1. Docker-based CI/CD
+
+Create a custom Docker image with promptfoo pre-installed:
+
+```dockerfile title="Dockerfile"
+FROM node:24-slim
+WORKDIR /app
+COPY . .
+CMD ["npx", "promptfoo@latest", "eval"]
+```
+
+### 2. Parallel Testing
+
+Test multiple models or configurations in parallel:
+
+```yaml
+# GitHub Actions example
+strategy:
+  matrix:
+    model: [gpt-4, gpt-3.5-turbo, claude-3-opus]
+steps:
+  - name: Test ${{ matrix.model }}
+    run: |
+      npx promptfoo@latest eval \
+        --providers.0.config.model=${{ matrix.model }} \
+        -o results-${{ matrix.model }}.json
+```
+
+### 3. Scheduled Security Scans
+
+Run comprehensive security scans on a schedule:
+
+```yaml
+# GitHub Actions
+on:
+  schedule:
+    - cron: '0 2 * * *' # 2 AM daily
+
+jobs:
+  security-scan:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Full red team scan
+        run: |
+          npx promptfoo@latest redteam generate \
+            --plugins harmful,pii,contracts \
+            --strategies jailbreak,jailbreak-templates
+          npx promptfoo@latest redteam run
+```
+
+### 4. SonarQube Integration
+
+:::info Enterprise Feature
+
+Direct SonarQube output format is available in [Promptfoo Enterprise](/docs/enterprise/). For open-source users, export to JSON and transform the results.
+
+:::
+
+For enterprise environments, integrate with SonarQube:
+
+```yaml
+# Export results for SonarQube processing
+- name: Run promptfoo security scan
+  run: |
+    npx promptfoo@latest eval \
+      --config promptfooconfig.yaml \
+      -o results.json
+
+# Transform results for SonarQube (custom script required)
+- name: Transform for SonarQube
+  run: |
+    node transform-to-sonarqube.js results.json > sonar-report.json
+
+- name: SonarQube scan
+  env:
+    SONAR_TOKEN: ${{ secrets.SONAR_TOKEN }}
+  run: |
+    sonar-scanner \
+      -Dsonar.externalIssuesReportPaths=sonar-report.json
+```
+
+See our [SonarQube integration guide](/docs/integrations/sonarqube) for detailed setup.
+
+## Processing Results
+
+### Parsing JSON Output
+
+The output JSON follows this schema:
+
+```typescript
+interface OutputFile {
+  evalId: string | null;
+  results: {
+    version: 3;
+    timestamp: string;
+    stats: {
+      successes: number;
+      failures: number;
+      errors: number;
+    };
+    prompts: Array<unknown>;
+    results: Array<{
+      success: boolean;
+      score: number;
+      error?: string;
+      // ... other fields
+    }>;
+  };
+  config: Partial<UnifiedConfig>;
+  shareableUrl: string | null;
+  metadata?: OutputMetadata;
+  vars?: string[];
+  runtimeOptions?: Partial<EvaluateOptions>;
+  traces?: TraceData[];
+  blobAssets?: ExportedBlobAsset[];
+}
+```
+
+`promptfoo eval -o results.json` and `promptfoo export eval <evalId>` use the
+same eval output envelope. Portable exports created with
+`promptfoo export eval <evalId> --include-media` may add embedded `blobAssets`.
+
+Example processing script:
+
+```javascript title="process-results.js"
+const fs = require('fs');
+const evalOutput = JSON.parse(fs.readFileSync('results.json', 'utf8'));
+const { stats, results: evalResults } = evalOutput.results;
+
+// Calculate metrics
+const passRate = (stats.successes / (stats.successes + stats.failures)) * 100;
+
+console.log(`Pass rate: ${passRate.toFixed(2)}%`);
+console.log(`Shareable URL: ${evalOutput.shareableUrl}`);
+
+// Check for specific failures
+const criticalFailures = evalResults.filter(
+  (result) => result.error?.includes('security') || result.error?.includes('injection'),
+);
+
+if (criticalFailures.length > 0) {
+  console.error('Critical security failures detected!');
+  process.exit(1);
+}
+```
+
+### Posting Results
+
+Post eval results to PR comments, Slack, or other channels:
+
+```bash
+# Extract and post results
+SHARE_URL=$(jq -r '.shareableUrl' results.json)
+PASS_RATE=$(jq '.results.stats.successes / (.results.stats.successes + .results.stats.failures) * 100' results.json)
+
+# Post to GitHub PR
+gh pr comment --body "
+## Promptfoo Eval Results
+- Pass rate: ${PASS_RATE}%
+- [View detailed results](${SHARE_URL})
+"
+```
+
+## Caching Strategies
+
+<!-- prettier-ignore -->
+Optimize CI/CD performance with proper caching [[memory:3455374]]:
+
+```yaml
+# Set cache location
+env:
+  PROMPTFOO_CACHE_PATH: ~/.cache/promptfoo
+  PROMPTFOO_CACHE_TTL: 86400 # 24 hours
+
+# Cache configuration
+cache:
+  key: promptfoo-${{ hashFiles('prompts/**', 'promptfooconfig.yaml') }}
+  paths:
+    - ~/.cache/promptfoo
+```
+
+## Security Best Practices
+
+1. **API Key Management**
+   - Store API keys as encrypted secrets
+   - Use least-privilege access controls
+   - Rotate keys regularly
+
+2. **Network Security**
+   - Use private runners for sensitive data
+   - Restrict outbound network access
+   - Consider on-premise deployments for enterprise
+
+3. **Data Privacy**
+   - Enable output stripping for sensitive data:
+
+   ```bash
+   export PROMPTFOO_STRIP_RESPONSE_OUTPUT=true
+   export PROMPTFOO_STRIP_TEST_VARS=true
+   ```
+
+4. **Audit Logging**
+   - Keep eval history
+   - Track who triggered security scans
+   - Monitor for anomalous patterns
+
+## Troubleshooting
+
+### Common Issues
+
+| Issue         | Solution                                         |
+| ------------- | ------------------------------------------------ |
+| Rate limits   | Enable caching, reduce concurrency with `-j 1`   |
+| Timeouts      | Increase timeout values, use `--max-concurrency` |
+| Memory issues | Use streaming mode, process results in batches   |
+| Cache misses  | Check cache key includes all relevant files      |
+
+### Debug Mode
+
+Enable detailed logging:
+
+```bash
+LOG_LEVEL=debug npx promptfoo@latest eval -c config.yaml
+```
+
+## Real-World Examples
+
+### Automated Testing Examples
+
+- [Self-grading example](https://github.com/promptfoo/promptfoo/tree/main/examples/eval-self-grading) - Automated LLM evaluation
+- [Custom grading prompts](https://github.com/promptfoo/promptfoo/tree/main/examples/eval-custom-grading-prompt) - Complex evaluation logic
+- [Store and reuse outputs](https://github.com/promptfoo/promptfoo/tree/main/examples/config-store-and-reuse-outputs) - Multi-step testing
+
+### Security Examples
+
+- [Red team starter](https://github.com/promptfoo/promptfoo/tree/main/examples/redteam-starter) - Basic security testing
+- [RAG red team tests](https://github.com/promptfoo/promptfoo/tree/main/examples/redteam-rag) - Customer service red team testing (RBAC, competitors, harmful content)
+- [DoNotAnswer dataset](https://github.com/promptfoo/promptfoo/tree/main/examples/redteam-donotanswer) - Harmful content detection
+
+### Integration Examples
+
+- [GitHub Action standalone](https://github.com/promptfoo/promptfoo/tree/main/examples/integration-github-action) - Custom GitHub workflows
+- [JSON output processing](https://github.com/promptfoo/promptfoo/tree/main/examples/eval-json-output) - Result parsing patterns
+- [CSV test data](https://github.com/promptfoo/promptfoo/tree/main/examples/simple-test) - Bulk test management
+
+## Related Documentation
+
+### Configuration & Testing
+
+- [Configuration Guide](/docs/configuration/guide) - Complete setup instructions
+- [Test Cases](/docs/configuration/test-cases) - Writing effective tests
+- [Assertions & Metrics](/docs/configuration/expected-outputs) - Validation strategies
+- [Python Assertions](/docs/configuration/expected-outputs/python) - Custom Python validators
+- [JavaScript Assertions](/docs/configuration/expected-outputs/javascript) - Custom JS validators
+
+### Security & Red Teaming
+
+- [Red Team Architecture](/docs/red-team/architecture) - Security testing framework
+- [OWASP Top 10 for LLMs](/docs/red-team/owasp-llm-top-10) - Security compliance
+- [RAG Security Testing](/docs/red-team/rag) - Testing retrieval systems
+- [MCP Security Testing](/docs/red-team/mcp-security-testing) - Model Context Protocol security
+
+### Enterprise & Scaling
+
+- [Enterprise Features](/docs/enterprise/) - Team collaboration and compliance
+- [Red Teams in Enterprise](/docs/enterprise/red-teams) - Organization-wide security
+- [Service Accounts](/docs/enterprise/service-accounts) - Automated access
+
+## See Also
+
+- [GitHub Actions Integration](/docs/integrations/github-action)
+- [Red Team Quickstart](/docs/red-team/quickstart)
+- [Enterprise Features](/docs/enterprise/)
+- [Configuration Reference](/docs/configuration/reference)
@@ -0,0 +1,149 @@
+---
+sidebar_label: CircleCI
+description: Automate LLM testing in CircleCI pipelines with promptfoo. Configure caching, API keys, and evaluation workflows to validate prompts and models in CI/CD environments.
+---
+
+# Setting up Promptfoo with CircleCI
+
+This guide shows how to integrate promptfoo's LLM evaluation into your CircleCI pipeline. This allows you to automatically test your prompts and models whenever changes are made to your repository.
+
+## Prerequisites
+
+- A CircleCI account connected to your repository
+- Your LLM provider's API keys (e.g., OpenAI API key)
+- Basic familiarity with CircleCI configuration
+
+## Configuration Steps
+
+### 1. Create CircleCI Configuration
+
+Create a `.circleci/config.yml` file in your repository. Here's a basic configuration that installs promptfoo and runs evaluations:
+
+    ```yaml
+    version: 2.1
+    jobs:
+      evaluate_prompts:
+        docker:
+          - image: cimg/node:lts
+        steps:
+          - checkout
+
+          - restore_cache:
+              keys:
+                - promptfoo-cache-v1-{{ .Branch }}-{{ checksum "prompts/**/*" }}
+                - promptfoo-cache-v1-{{ .Branch }}
+                - promptfoo-cache-v1-
+
+          - run:
+              name: Install promptfoo
+              command: npm install -g promptfoo
+
+          - run:
+              name: Run prompt evaluation
+              command: promptfoo eval -c promptfooconfig.yaml --prompts prompts/**/*.json --share -o output.json
+              environment:
+                OPENAI_API_KEY: ${OPENAI_API_KEY}
+                PROMPTFOO_CACHE_PATH: ~/.promptfoo/cache
+
+          - save_cache:
+              key: promptfoo-cache-v1-{{ .Branch }}-{{ checksum "prompts/**/*" }}
+              paths:
+                - ~/.promptfoo/cache
+
+          - store_artifacts:
+              path: output.json
+              destination: evaluation-results
+
+    workflows:
+      version: 2
+      evaluate:
+        jobs:
+          - evaluate_prompts:
+              filters:
+                paths:
+                  - prompts/**/*
+    ```
+
+### 2. Set Up Environment Variables
+
+1. Go to your project settings in CircleCI
+2. Navigate to Environment Variables
+3. Add your LLM provider's API keys:
+   - e.g. Add `OPENAI_API_KEY` if you're using OpenAI
+
+### 3. Configure Caching (Optional but Recommended)
+
+The configuration above includes caching to save time and API costs. The cache:
+
+- Stores LLM API responses
+- Is keyed by branch and content hash
+- Is saved in `~/.promptfoo/cache`
+
+### 4. Storing Results
+
+The configuration stores the evaluation results as artifacts:
+
+- Results are saved to `output.json`
+- CircleCI makes these available in the Artifacts tab
+- The `--share` flag creates a shareable web URL for results
+
+## Advanced Configuration
+
+### Adding Custom Test Steps
+
+You can add custom steps to process the evaluation results:
+
+    ```yaml
+    - run:
+        name: Check evaluation results
+        command: |
+          if jq -e '.results.stats.failures > 0' output.json; then
+            echo "Evaluation had failures"
+            exit 1
+          fi
+    ```
+
+### Parallel Evaluation
+
+For large test suites, you can parallelize evaluations:
+
+    ```yaml
+    jobs:
+      evaluate_prompts:
+        parallelism: 3
+        steps:
+          - run:
+              name: Split tests
+              command: |
+                prompts=$(find prompts -name "*.json" | circleci tests split)
+                promptfoo eval -c promptfooconfig.yaml --prompts $prompts
+    ```
+
+## Example Output
+
+After the evaluation runs, you'll see:
+
+- Test results in the CircleCI UI
+- Artifacts containing the full evaluation data
+- A shareable link to view results in the promptfoo web viewer
+- Any test failures will cause the CircleCI job to fail
+
+## Troubleshooting
+
+Common issues and solutions:
+
+1. **Cache not working:**
+   - Verify the cache key matches your configuration
+   - Check that the cache path exists
+   - Ensure file permissions are correct
+
+2. **API key errors:**
+   - Confirm environment variables are set in CircleCI
+   - Check for typos in variable names
+   - Verify API key permissions
+
+3. **Evaluation timeout:**
+   - Adjust the `no_output_timeout` setting in your job
+   - Consider splitting tests into smaller batches
+
+For more details on promptfoo configuration, see the [configuration reference](/docs/configuration/reference).
@@ -0,0 +1,159 @@
+---
+sidebar_label: GitHub Actions
+description: Automate LLM prompt testing in CI/CD with GitHub Actions integration. Compare prompt changes, view diffs, and analyze results directly in pull requests using promptfoo.
+---
+
+# Testing Prompts with GitHub Actions
+
+This guide describes how to automatically run a before vs. after evaluation of edited prompts using the [promptfoo GitHub Action](https://github.com/promptfoo/promptfoo-action/).
+
+On every pull request that modifies a prompt, the action will automatically run a full comparison:
+
+![GitHub Action comment on modified LLM prompt](/img/docs/github-action-comment.png)
+
+The provided link opens the [web viewer](/docs/usage/web-ui) interface, which allows you to interactively explore the before vs. after:
+
+![promptfoo web viewer](https://user-images.githubusercontent.com/310310/244891219-2b79e8f8-9b79-49e7-bffb-24cba18352f2.png)
+
+## Using the GitHub Action
+
+Here's an example action that watches a PR for modifications. If any file in the `prompts/` directory is modified, we automatically run the eval and post a link to the results using the `promptfoo/promptfoo-action@v1`:
+
+```yml
+name: 'Prompt Evaluation'
+
+on:
+  pull_request:
+    paths:
+      - 'prompts/**'
+
+jobs:
+  evaluate:
+    runs-on: ubuntu-latest
+    permissions:
+      # This permission is used to post comments on Pull Requests
+      pull-requests: write
+    steps:
+      # This cache is optional, but you'll save money and time by setting it up!
+      - name: Set up promptfoo cache
+        uses: actions/cache@v4
+        with:
+          path: ~/.cache/promptfoo
+          key: ${{ runner.os }}-promptfoo-v1
+          restore-keys: |
+            ${{ runner.os }}-promptfoo-
+
+      # This step will actually run the before/after evaluation
+      - name: Run promptfoo evaluation
+        uses: promptfoo/promptfoo-action@v1
+        with:
+          openai-api-key: ${{ secrets.OPENAI_API_KEY }}
+          github-token: ${{ secrets.GITHUB_TOKEN }}
+          prompts: 'prompts/**/*.json'
+          config: 'prompts/promptfooconfig.yaml'
+          cache-path: ~/.cache/promptfoo
+```
+
+## Configuration
+
+To make this GitHub Action work for your project, you'll need to do a few things:
+
+1. **Set paths**: Replace `'prompts/**'` with the path to the files you want to monitor for changes. This could either be a list of paths to single files or a directory where your prompts are stored.
+
+   Don't forget to also update the paths in the "Run promptfoo evaluation" step to point to your prompts and `promptfooconfig.yaml` configuration file.
+
+2. **Set OpenAI API key**: If you're using an OpenAI API, you need to set the `OPENAI_API_KEY` secret in your GitHub repository.
+
+   To do this, go to your repository's Settings > Secrets and variables > Actions > New repository secret and create one named `OPENAI_API_KEY`.
+
+3. **Set environment variables**: The action uses `PROMPTFOO_CONFIG_DIR` and `PROMPTFOO_CACHE_PATH` to record state on the filesystem.
+
+4. **Add it to your project**: GitHub automatically runs workflows in the `.github/workflows` directory, so save it as something like `.github/workflows/prompt-eval.yml`.
+
+Here are the supported parameters:
+
+| Parameter        | Description                                                                                                               | Required |
+| ---------------- | ------------------------------------------------------------------------------------------------------------------------- | -------- |
+| `github-token`   | The GitHub token. Used to authenticate requests to the GitHub API.                                                        | Yes      |
+| `prompts`        | The glob patterns for the prompt files. These patterns are used to find the prompt files that the action should evaluate. | Yes      |
+| `config`         | The path to the configuration file. This file contains settings for the action.                                           | Yes      |
+| `openai-api-key` | The API key for OpenAI. Used to authenticate requests to the OpenAI API.                                                  | No       |
+| `cache-path`     | The path to the cache. This is where the action stores temporary data.                                                    | No       |
+
+## How It Works
+
+1. **Caching**: We use caching to speed up subsequent runs. The cache stores LLM requests and outputs, which can be reused in future runs to save cost.
+
+2. **Run Promptfoo Evaluation**: This is where the magic happens. We run the evaluation, passing in the configuration file and the prompts we want to evaluate. The results of this step are automatically posted to the pull request.
+
+For more information on how to set up the promptfoo config, see the [Getting Started](/docs/getting-started) docs.
+
+## For red teaming
+
+For red teaming integrations, we recommend embedding more detailed reporting in your workflow.
+
+Here's an example:
+
+```yaml
+- name: Run Promptfoo redteam
+  run: |
+    start=$(date +%s)
+    npx promptfoo@latest redteam run \
+      -c 9d32de26-7926-44f1-af13-bd06cb86f691 \
+      -t 213c2235-865c-4aa4-90cc-a002256e0a94 \
+      -j 5 \
+      -o output.json || true
+    end=$(date +%s)
+    echo "DURATION_SECONDS=$((end-start))" >> $GITHUB_ENV
+    test -f output.json || { echo 'output.json not found'; exit 1; }
+
+- name: Build redteam summary (JS)
+  run: |
+    node .github/scripts/redteam-summary.js --input output.json --out comment.md --print
+```
+
+> **💡 Tip**: To speed up builds, you can cache promptfoo instead of downloading it every run:
+>
+> **Getting started**: If you don't have a package.json file yet, create one first:
+>
+> ```bash
+> npm init -y
+> ```
+>
+> **With package.json**: Add promptfoo as a dependency, then use setup-node with caching:
+>
+> ```bash
+> npm install --save-dev promptfoo
+> ```
+>
+> ```yaml
+> steps:
+>   - uses: actions/checkout@v5
+>   - uses: actions/setup-node@v4
+>     with:
+>       node-version: '22'
+>       cache: 'npm'
+>   - run: npm ci
+>   - name: Run Promptfoo redteam
+>     run: npx promptfoo redteam run -c config.yaml -o output.json
+> ```
+>
+> **Without package.json**: Cache npx downloads using workflow files as the cache key:
+>
+> ```yaml
+> steps:
+>   - uses: actions/checkout@v5
+>   - uses: actions/setup-node@v4
+>     with:
+>       node-version: '22'
+>       cache: 'npm'
+>       cache-dependency-path: '**/.github/workflows/*.yml'
+>   - name: Run Promptfoo redteam
+>     run: npx promptfoo@latest redteam run -c config.yaml -o output.json
+> ```
+
+In this example, the [redteam-summary.js](https://gist.github.com/MrFlounder/2d9c719873ad9f221db5f87efb13ece9) file parses the red team results and produces a summary:
+
+![llm red team github action](/img/docs/github-action-redteam.png)
+
+The results posted on the PR include more detailed information about plugin, strategy, and target performance.
@@ -0,0 +1,161 @@
+---
+sidebar_label: GitLab CI
+description: Automate LLM testing in GitLab CI pipelines with Promptfoo. Configure caching, API keys, and evaluation workflows to validate prompts and models in your CI/CD process.
+---
+
+# Setting up Promptfoo with GitLab CI
+
+This guide shows how to integrate Promptfoo's LLM evaluation into your GitLab CI pipeline. This allows you to automatically test your prompts and models whenever changes are made to your repository.
+
+## Prerequisites
+
+- A GitLab repository
+- Your LLM provider's API keys (e.g., OpenAI API key)
+- Basic familiarity with GitLab CI/CD configuration
+
+## Configuration Steps
+
+### 1. Create GitLab CI Configuration
+
+Create a `.gitlab-ci.yml` file in your repository root. Here's a basic configuration that installs Promptfoo and runs evaluations:
+
+```yaml
+image: node:24
+
+evaluate_prompts:
+  script:
+    - npm install -g promptfoo
+    - promptfoo eval -c promptfooconfig.yaml --prompts prompts/**/*.json --share -o output.json -o output.junit.xml
+  variables:
+    OPENAI_API_KEY: ${OPENAI_API_KEY}
+    PROMPTFOO_CACHE_PATH: .promptfoo/cache
+  cache:
+    key:
+      files:
+        - prompts/**/*
+    paths:
+      - .promptfoo/cache
+  artifacts:
+    paths:
+      - output.json
+      - output.junit.xml
+    reports:
+      junit: output.junit.xml
+  rules:
+    - changes:
+        - prompts/**/*
+```
+
+### 2. Set Up Environment Variables
+
+1. Go to Settings > CI/CD in your GitLab project
+2. Expand the Variables section
+3. Add your LLM provider's API keys:
+   - Click "Add Variable"
+   - Add `OPENAI_API_KEY` (or other provider keys) as masked and protected variables
+
+### 3. Configure Caching (Optional but Recommended)
+
+The configuration above includes caching to save time and API costs. The cache:
+
+- Stores LLM API responses
+- Is keyed based on the content of your prompt files
+- Is saved in `.promptfoo/cache`
+
+### 4. Storing Results
+
+The configuration stores the evaluation results as artifacts:
+
+- Results are saved to `output.json`
+- GitLab makes these available in the job artifacts
+- The `--share` flag creates a shareable web URL for results
+
+## Advanced Configuration
+
+### Adding Custom Test Steps
+
+You can add custom steps to process the evaluation results:
+
+```yaml
+evaluate_prompts:
+  script:
+    - npm install -g promptfoo
+    - promptfoo eval -c promptfooconfig.yaml --prompts prompts/**/*.json --share -o output.json -o output.junit.xml
+    - |
+      if jq -e '.results.stats.failures > 0' output.json; then
+        echo "Evaluation had failures"
+        exit 1
+      fi
+```
+
+### Parallel Evaluation
+
+For large test suites, you can use GitLab's parallel feature:
+
+```yaml
+evaluate_prompts:
+  parallel: 3
+  script:
+    - |
+      prompts=$(find prompts -name "*.json" | awk "NR % $CI_NODE_TOTAL == $CI_NODE_INDEX")
+      promptfoo eval -c promptfooconfig.yaml --prompts $prompts
+```
+
+### Integration with GitLab Merge Requests
+
+You can configure the job to post results as merge request comments:
+
+```yaml
+evaluate_prompts:
+  script:
+    - npm install -g promptfoo
+    - |
+      OUTPUT=$(promptfoo eval -c promptfooconfig.yaml --prompts prompts/**/*.json --share -o output.junit.xml)
+      SHARE_URL=$(echo "$OUTPUT" | grep "View results:" | cut -d' ' -f3)
+      echo "Evaluation Results: $SHARE_URL" | tee merge_request_comment.txt
+  artifacts:
+    reports:
+      junit: output.junit.xml
+    paths:
+      - merge_request_comment.txt
+  after_script:
+    - |
+      if [ -n "$CI_MERGE_REQUEST_IID" ]; then
+        curl --request POST \
+          --header "PRIVATE-TOKEN: ${GITLAB_API_TOKEN}" \
+          --data-urlencode "body=$(cat merge_request_comment.txt)" \
+          "${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/merge_requests/${CI_MERGE_REQUEST_IID}/notes"
+      fi
+```
+
+## Example Output
+
+After the evaluation runs, you'll see:
+
+- Test results in the GitLab CI/CD pipeline interface
+- Artifacts containing the full evaluation data
+- A shareable link to view results in the promptfoo web viewer
+- Any test failures will cause the GitLab job to fail
+
+## Troubleshooting
+
+Common issues and solutions:
+
+1. **Cache not working:**
+   - Verify the cache key and paths in your configuration
+   - Check that the cache path exists
+   - Ensure file permissions are correct
+
+2. **API key errors:**
+   - Confirm variables are set in GitLab CI/CD settings
+   - Check that variables are properly masked
+   - Verify API key permissions
+
+3. **Job timing out:**
+   - Add a timeout override to your job configuration:
+     ```yaml
+     evaluate_prompts:
+       timeout: 2 hours
+     ```
+
+For more details on Promptfoo configuration, see the [configuration reference](/docs/configuration/reference).
@@ -0,0 +1,137 @@
+---
+sidebar_label: Google Sheets
+description: Import and export LLM test cases with Google Sheets integration. Configure public or authenticated access, write evaluation results, and run model-graded metrics.
+---
+
+# Google Sheets Integration
+
+promptfoo allows you to import eval test cases directly from Google Sheets. This can be done either unauthenticated (if the sheet is public) or authenticated using Google's Default Application Credentials, typically with a service account for programmatic access.
+
+## Importing Test Cases from Google Sheets
+
+### Public Sheets (Unauthenticated)
+
+For sheets that are accessible via "anyone with the link", simply specify the share URL in your configuration:
+
+```yaml title="promptfooconfig.yaml"
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: 'Public Google Sheet Example'
+prompts:
+  - 'Please translate the following text to {{language}}: {{input}}'
+providers:
+  - anthropic:messages:claude-3-5-sonnet-20241022
+  - openai:chat:gpt-5
+// highlight-start
+tests: https://docs.google.com/spreadsheets/d/1eqFnv1vzkPvS7zG-mYsqNDwOzvSaiIAsKB3zKg9H18c/edit?usp=sharing
+// highlight-end
+```
+
+The Google Sheet above is structured with columns that define the test cases. Here's a copy of the sheet:
+
+```csv title="Google Sheet"
+language,input,__expected
+French,Hello world,icontains: bonjour
+German,I'm hungry,llm-rubric: is german
+Swahili,Hello world,similar(0.8):hello world
+```
+
+> 💡 See our [example sheet](https://docs.google.com/spreadsheets/d/1eqFnv1vzkPvS7zG-mYsqNDwOzvSaiIAsKB3zKg9H18c/edit#gid=0) for the expected format. For details on sheet structure, refer to [loading assertions from CSV](/docs/configuration/expected-outputs/#load-assertions-from-csv).
+
+### Private Sheets (Authenticated)
+
+For private sheets, you'll need to set up Google's Default Application Credentials:
+
+1. **Install the authenticated Sheets package**
+
+   ```bash
+   npm install @googleapis/sheets
+   ```
+
+   `@googleapis/sheets` is an optional runtime dependency. Public-sheet CSV imports do not need it, but private-sheet reads and Google Sheets output writes do.
+
+2. **Set Up Authentication**
+   - Create a [service account](https://console.cloud.google.com/iam-admin/serviceaccounts) in Google Cloud
+   - Download the JSON key file
+   - Enable the [Google Sheets API](https://console.cloud.google.com/apis/library/sheets.googleapis.com) (`sheets.googleapis.com`)
+   - Share your sheet with the service account email (`your-service-account@project-name.iam.gserviceaccount.com`) with at least viewer permissions
+
+3. **Configure Credentials**
+
+   ```bash
+   export GOOGLE_APPLICATION_CREDENTIALS="/path/to/your/service-account-file.json"
+   ```
+
+4. **Use the Same URL Format**
+   ```yaml
+   tests: https://docs.google.com/spreadsheets/d/1eqFnv1vzkPvS7zG-mYsqNDwOzvSaiIAsKB3zKg9H18c/edit?usp=sharing
+   ```
+   The system will automatically use authenticated access when the sheet is not public.
+
+## Writing Evaluation Results to Google Sheets
+
+The `outputPath` parameter (`--output` or `-o` on the command line) supports writing evaluation results directly to Google Sheets. This requires Default Application Credentials with write access configured.
+
+### Basic Usage
+
+```yaml
+prompts:
+  - ...
+providers:
+  - ...
+tests:
+  - ...
+// highlight-start
+outputPath: https://docs.google.com/spreadsheets/d/1eqFnv1vzkPvS7zG-mYsqNDwOzvSaiIAsKB3zKg9H18c/edit?usp=sharing
+// highlight-end
+```
+
+### Targeting Specific Sheets
+
+You have two options when writing results to a Google Sheet:
+
+1. **Write to an existing sheet** by including the sheet's `gid` parameter in the URL:
+
+   ```yaml
+   outputPath: https://docs.google.com/spreadsheets/d/1eqFnv1vzkPvS7zG-mYsqNDwOzvSaiIAsKB3zKg9H18c/edit#gid=123456789
+   ```
+
+   > 💡 To find a sheet's `gid`, open the sheet in your browser and look at the URL - the `gid` appears after the `#gid=` portion.
+
+2. **Create a new sheet automatically** by omitting the `gid` parameter. The system will:
+   - Create a new sheet with a timestamp-based name (e.g., "Sheet1234567890")
+   - Write results to this new sheet
+   - Preserve existing sheets and their data
+
+This behavior helps prevent accidental data overwrites while keeping your evaluation results organized within the same Google Sheets document.
+
+### Output Format
+
+Results are written with columns for test variables followed by prompt outputs. Prompt columns include the provider in the header using the format `[provider] prompt-label`. For example, with two providers testing the same prompt:
+
+| language | input       | [openai:gpt-5] Translate | [anthropic:claude-4.5-sonnet] Translate |
+| -------- | ----------- | ------------------------ | --------------------------------------- |
+| French   | Hello world | Bonjour le monde         | Bonjour monde                           |
+
+## Using Custom Providers for Model-Graded Metrics
+
+When using Google Sheets for test cases, you can still use custom providers for model-graded metrics
+like `llm-rubric` or `similar`. To do this, override the default LLM grader by adding a `defaultTest` property to your configuration:
+
+```yaml
+prompts:
+  - file://prompt1.txt
+  - file://prompt2.txt
+providers:
+  - anthropic:messages:claude-3-5-sonnet-20241022
+  - openai:chat:gpt-5-mini
+tests: https://docs.google.com/spreadsheets/d/1eqFnv1vzkPvS7zG-mYsqNDwOzvSaiIAsKB3zKg9H18c/edit?usp=sharing
+defaultTest:
+  options:
+    provider:
+      text:
+        id: ollama:chat:llama3.3:70b
+      embedding:
+        id: ollama:embeddings:mxbai-embed-large
+```
+
+For more details on customizing the LLM grader, see the [model-graded metrics documentation](/docs/configuration/expected-outputs/model-graded/#overriding-the-llm-grader).
@@ -0,0 +1,32 @@
+---
+sidebar_label: Helicone
+description: Monitor and optimize LLM testing with Helicone integration in Promptfoo. Track usage, costs, and latency while managing prompts through an open-source observability platform.
+---
+
+# Helicone integration
+
+[Helicone](https://helicone.ai/) is an open source observability platform that proxies your LLM requests and provides key insights into your usage, spend, latency and more.
+
+To reference prompts in Helicone:
+
+1. Log into [Helicone](https://www.helicone.ai) or create an account. Once you have an account, you can generate an [API key](https://helicone.ai/developer).
+
+2. Set the `HELICONE_API_KEY` and environment variables as desired.
+
+3. Use the `helicone://` prefix for your prompts, followed by the Helicone prompt id and version. For example:
+
+   ```yaml
+   prompts:
+     - 'helicone://my-cool-prompt:5.2'
+
+   providers:
+     - openai:gpt-5-mini
+
+   tests:
+     - vars:
+         # ...
+   ```
+
+Variables from your promptfoo test cases will be automatically plugged into the Helicone prompt as variables.
+
+You can follow [this guide](https://docs.helicone.ai/features/prompts#prompts-and-experiments) to create a Prompt using Helicone
@@ -0,0 +1,245 @@
+---
+sidebar_label: Jenkins
+description: Integrate Promptfoo's LLM testing into Jenkins pipelines with automated evaluation, credential management, and CI/CD workflows for production AI deployments
+---
+
+# Setting up Promptfoo with Jenkins
+
+This guide demonstrates how to integrate Promptfoo's LLM evaluation into your Jenkins pipeline. This setup enables automatic testing of your prompts and models whenever changes are made to your repository.
+
+## Prerequisites
+
+- Jenkins server with pipeline support
+- Node.js installed on the Jenkins agent
+- Your LLM provider's API keys (e.g., OpenAI API key)
+- Basic familiarity with Jenkins Pipeline syntax
+
+## Configuration Steps
+
+### 1. Create Jenkinsfile
+
+Create a `Jenkinsfile` in your repository root. Here's a basic configuration that installs Promptfoo and runs evaluations:
+
+```groovy:Jenkinsfile
+pipeline {
+    agent any
+
+    environment {
+        OPENAI_API_KEY = credentials('openai-api-key')
+        PROMPTFOO_CACHE_PATH = '~/.promptfoo/cache'
+    }
+
+    stages {
+        stage('Setup') {
+            steps {
+                sh 'npm install -g promptfoo'
+            }
+        }
+
+        stage('Evaluate Prompts') {
+            steps {
+                script {
+                    try {
+                        sh 'promptfoo eval -c promptfooconfig.yaml --prompts prompts/**/*.json --share -o output.json'
+                    } catch (Exception e) {
+                        currentBuild.result = 'FAILURE'
+                        error("Prompt evaluation failed: ${e.message}")
+                    }
+                }
+            }
+        }
+
+        stage('Process Results') {
+            steps {
+                script {
+                    def output = readJSON file: 'output.json'
+                    echo "Evaluation Results:"
+                    echo "Successes: ${output.results.stats.successes}"
+                    echo "Failures: ${output.results.stats.failures}"
+
+                    if (output.shareableUrl) {
+                        echo "View detailed results at: ${output.shareableUrl}"
+                    }
+
+                    if (output.results.stats.failures > 0) {
+                        currentBuild.result = 'UNSTABLE'
+                    }
+                }
+            }
+        }
+    }
+
+    post {
+        always {
+            archiveArtifacts artifacts: 'output.json', fingerprint: true
+        }
+    }
+}
+```
+
+### 2. Configure Jenkins Credentials
+
+You'll need to add the API keys for any LLM providers you're using. For example, if you're using OpenAI, you'll need to add the OpenAI API key.
+
+1. Navigate to Jenkins Dashboard → Manage Jenkins → Credentials
+2. Add a new credential:
+   - Kind: Secret text
+   - Scope: Global
+   - ID: openai-api-key
+   - Description: OpenAI API Key
+   - Secret: Your API key value
+
+### 3. Set Up Caching
+
+To implement caching for better performance and reduced API costs:
+
+1. Create a cache directory on your Jenkins agent:
+
+```bash
+mkdir -p ~/.promptfoo/cache
+```
+
+2. Ensure the Jenkins user has write permissions:
+
+```bash
+chown -R jenkins:jenkins ~/.promptfoo/cache
+```
+
+### 4. Advanced Pipeline Configuration
+
+Here's an example of a more advanced pipeline with additional features:
+
+The advanced configuration includes several important improvements:
+
+- **Build timeouts**: The `timeout` option ensures builds don't run indefinitely (1 hour limit)
+- **Timestamps**: Adds timestamps to console output for better debugging
+- **SCM polling**: Automatically checks for changes every 15 minutes using `pollSCM`
+- **Conditional execution**: Only runs evaluations when files in `prompts/` directory change
+- **Email notifications**: Sends emails to developers on pipeline failures
+- **Workspace cleanup**: Automatically cleans up workspace after each run
+- **Artifact management**: Archives both JSON and HTML reports with fingerprinting
+- **Better error handling**: More robust error catching and build status management
+
+```groovy:Jenkinsfile
+pipeline {
+    agent any
+
+    environment {
+        OPENAI_API_KEY = credentials('openai-api-key')
+        PROMPTFOO_CACHE_PATH = '~/.promptfoo/cache'
+    }
+
+    options {
+        timeout(time: 1, unit: 'HOURS')
+        timestamps()
+    }
+
+    triggers {
+        pollSCM('H/15 * * * *')
+    }
+
+    stages {
+        stage('Setup') {
+            steps {
+                sh 'npm install -g promptfoo'
+            }
+        }
+
+        stage('Evaluate Prompts') {
+            when {
+                changeset 'prompts/**'
+            }
+            steps {
+                script {
+                    try {
+                        sh '''
+                            promptfoo eval \
+                                -c promptfooconfig.yaml \
+                                --prompts prompts/**/*.json \
+                                --share \
+                                -o output.json
+                        '''
+                    } catch (Exception e) {
+                        currentBuild.result = 'FAILURE'
+                        error("Prompt evaluation failed: ${e.message}")
+                    }
+                }
+            }
+        }
+
+        stage('Process Results') {
+            steps {
+                script {
+                    def output = readJSON file: 'output.json'
+
+                    // Create HTML report
+                    writeFile file: 'evaluation-report.html', text: """
+                        <html>
+                            <body>
+                                <h1>Prompt Evaluation Results</h1>
+                                <p>Successes: ${output.results.stats.successes}</p>
+                                <p>Failures: ${output.results.stats.failures}</p>
+                                <p>View detailed results: <a href="${output.shareableUrl}">${output.shareableUrl}</a></p>
+                            </body>
+                        </html>
+                    """
+
+                    // Publish HTML report
+                    publishHTML([
+                        allowMissing: false,
+                        alwaysLinkToLastBuild: true,
+                        keepAll: true,
+                        reportDir: '.',
+                        reportFiles: 'evaluation-report.html',
+                        reportName: 'Prompt Evaluation Report'
+                    ])
+
+                    if (output.results.stats.failures > 0) {
+                        currentBuild.result = 'UNSTABLE'
+                    }
+                }
+            }
+        }
+    }
+
+    post {
+        always {
+            archiveArtifacts artifacts: 'output.json,evaluation-report.html', fingerprint: true
+            cleanWs()
+        }
+        failure {
+            emailext (
+                subject: "Failed Pipeline: ${currentBuild.fullDisplayName}",
+                body: "Prompt evaluation failed. Check console output at ${env.BUILD_URL}",
+                recipientProviders: [[$class: 'DevelopersRecipientProvider']]
+            )
+        }
+    }
+}
+```
+
+## Troubleshooting
+
+Common issues and solutions:
+
+1. **Permission issues:**
+   - Ensure Jenkins has appropriate permissions to install global npm packages
+   - Verify cache directory permissions
+   - Check API key credential permissions
+
+2. **Pipeline timeout:**
+   - Adjust the timeout in pipeline options
+   - Consider splitting evaluations into smaller batches
+   - Monitor API rate limits
+
+3. **Cache problems:**
+   - Verify cache path exists and is writable
+   - Check disk space availability
+   - Clear cache if needed: `rm -rf ~/.promptfoo/cache/*`
+
+4. **Node.js issues:**
+   - Ensure Node.js is installed on the Jenkins agent
+   - Verify npm is available in PATH
+   - Consider using `nodejs` tool installer in Jenkins
+
+For more information on Promptfoo configuration and usage, refer to the [configuration reference](/docs/configuration/guide/).
@@ -0,0 +1,254 @@
+---
+sidebar_label: Jest & Vitest
+description: Integrate LLM testing into Jest and Vitest workflows with custom matchers for semantic similarity, factuality checks, and automated prompt quality validation
+---
+
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+
+import JestExampleImage from '../assets/jest-example.png';
+
+# Testing prompts with Jest and Vitest
+
+`promptfoo` can be integrated with test frameworks like [Jest](https://jestjs.io/) and [Vitest](https://vitest.dev/) to evaluate prompts as part of existing testing and CI workflows.
+
+This guide includes examples that show how to create test cases for desired prompt quality using semantic similarity and LLM grading. You can also skip to the [full example code](https://github.com/promptfoo/promptfoo/tree/main/examples/integration-jest).
+
+For more information on supported checks, see the [Assertions & Metrics documentation](/docs/configuration/expected-outputs/).
+
+## Prerequisites
+
+Before you begin, make sure you have the following node packages installed:
+
+- [jest](https://jestjs.io/docs/getting-started): `npm install --save-dev jest`
+- [vitest](https://vitest.dev/guide/): `npm install --save-dev vitest`
+- promptfoo: `npm install --save-dev promptfoo`
+
+## Creating custom matchers
+
+First, we'll create custom matchers:
+
+- `toMatchSemanticSimilarity`: Compares two strings for semantic similarity.
+- `toPassLLMRubric`: Checks if a string meets the specified LLM Rubric criteria.
+- `toMatchFactuality`: Checks if a string meets the specified factuality criteria.
+- `toMatchClosedQA`: Checks if a string meets the specified question-answering criteria.
+
+Create a new file called `matchers.js` and add the following:
+
+<Tabs>
+  <TabItem value="Javascript" label="Javascript" default>
+
+```javascript
+import { assertions } from 'promptfoo';
+
+const { matchesSimilarity, matchesLlmRubric } = assertions;
+
+export function installMatchers() {
+  expect.extend({
+    async toMatchSemanticSimilarity(received, expected, threshold = 0.8) {
+      const result = await matchesSimilarity(received, expected, threshold);
+      const pass = received === expected || result.pass;
+      if (pass) {
+        return {
+          message: () => `expected ${received} not to match semantic similarity with ${expected}`,
+          pass: true,
+        };
+      } else {
+        return {
+          message: () =>
+            `expected ${received} to match semantic similarity with ${expected}, but it did not. Reason: ${result.reason}`,
+          pass: false,
+        };
+      }
+    },
+
+    async toPassLLMRubric(received, expected, gradingConfig) {
+      const gradingResult = await matchesLlmRubric(expected, received, gradingConfig);
+      if (gradingResult.pass) {
+        return {
+          message: () => `expected ${received} not to pass LLM Rubric with ${expected}`,
+          pass: true,
+        };
+      } else {
+        return {
+          message: () =>
+            `expected ${received} to pass LLM Rubric with ${expected}, but it did not. Reason: ${gradingResult.reason}`,
+          pass: false,
+        };
+      }
+    },
+
+    async toMatchFactuality(input, expected, received, gradingConfig) {
+      const gradingResult = await matchesFactuality(input, expected, received, gradingConfig);
+      if (gradingResult.pass) {
+        return {
+          message: () => `expected ${received} not to match factuality with ${expected}`,
+          pass: true,
+        };
+      } else {
+        return {
+          message: () =>
+            `expected ${received} to match factuality with ${expected}, but it did not. Reason: ${gradingResult.reason}`,
+          pass: false,
+        };
+      }
+    },
+
+    async toMatchClosedQA(input, expected, received, gradingConfig) {
+      const gradingResult = await matchesClosedQa(input, expected, received, gradingConfig);
+      if (gradingResult.pass) {
+        return {
+          message: () => `expected ${received} not to match ClosedQA with ${expected}`,
+          pass: true,
+        };
+      } else {
+        return {
+          message: () =>
+            `expected ${received} to match ClosedQA with ${expected}, but it did not. Reason: ${gradingResult.reason}`,
+          pass: false,
+        };
+      }
+    },
+  });
+}
+```
+
+  </TabItem>
+  <TabItem value="Typescript" label="Typescript" default>
+
+```typescript
+import { assertions } from 'promptfoo';
+import type { GradingConfig } from 'promptfoo';
+
+const { matchesSimilarity, matchesLlmRubric } = assertions;
+
+declare global {
+  namespace jest {
+    interface Matchers<R> {
+      toMatchSemanticSimilarity(expected: string, threshold?: number): R;
+      toPassLLMRubric(expected: string, gradingConfig: GradingConfig): R;
+    }
+  }
+}
+
+export function installMatchers() {
+  expect.extend({
+    async toMatchSemanticSimilarity(
+      received: string,
+      expected: string,
+      threshold: number = 0.8,
+    ): Promise<jest.CustomMatcherResult> {
+      const result = await matchesSimilarity(received, expected, threshold);
+      const pass = received === expected || result.pass;
+      if (pass) {
+        return {
+          message: () => `expected ${received} not to match semantic similarity with ${expected}`,
+          pass: true,
+        };
+      } else {
+        return {
+          message: () =>
+            `expected ${received} to match semantic similarity with ${expected}, but it did not. Reason: ${result.reason}`,
+          pass: false,
+        };
+      }
+    },
+
+    async toPassLLMRubric(
+      received: string,
+      expected: string,
+      gradingConfig: GradingConfig,
+    ): Promise<jest.CustomMatcherResult> {
+      const gradingResult = await matchesLlmRubric(expected, received, gradingConfig);
+      if (gradingResult.pass) {
+        return {
+          message: () => `expected ${received} not to pass LLM Rubric with ${expected}`,
+          pass: true,
+        };
+      } else {
+        return {
+          message: () =>
+            `expected ${received} to pass LLM Rubric with ${expected}, but it did not. Reason: ${gradingResult.reason}`,
+          pass: false,
+        };
+      }
+    },
+  });
+}
+```
+
+  </TabItem>
+</Tabs>
+
+## Writing tests
+
+Our test code will use the custom matchers to run a few test cases.
+
+Create a new file called `index.test.js` and add the following code:
+
+```javascript
+import { installMatchers } from './matchers';
+
+installMatchers();
+
+const gradingConfig = {
+  provider: 'openai:chat:gpt-5-mini',
+};
+
+describe('semantic similarity tests', () => {
+  test('should pass when strings are semantically similar', async () => {
+    await expect('The quick brown fox').toMatchSemanticSimilarity('A fast brown fox');
+  });
+
+  test('should fail when strings are not semantically similar', async () => {
+    await expect('The quick brown fox').not.toMatchSemanticSimilarity('The weather is nice today');
+  });
+
+  test('should pass when strings are semantically similar with custom threshold', async () => {
+    await expect('The quick brown fox').toMatchSemanticSimilarity('A fast brown fox', 0.7);
+  });
+
+  test('should fail when strings are not semantically similar with custom threshold', async () => {
+    await expect('The quick brown fox').not.toMatchSemanticSimilarity(
+      'The weather is nice today',
+      0.9,
+    );
+  });
+});
+
+describe('LLM evaluation tests', () => {
+  test('should pass when strings meet the LLM Rubric criteria', async () => {
+    await expect('Four score and seven years ago').toPassLLMRubric(
+      'Contains part of a famous speech',
+      gradingConfig,
+    );
+  });
+
+  test('should fail when strings do not meet the LLM Rubric criteria', async () => {
+    await expect('It is time to do laundry').not.toPassLLMRubric(
+      'Contains part of a famous speech',
+      gradingConfig,
+    );
+  });
+});
+```
+
+## Final setup
+
+Add the following line to the `scripts` section in your `package.json`:
+
+```json
+"test": "jest"
+```
+
+Now, you can run the tests with the following command:
+
+```sh
+npm test
+```
+
+This will execute the tests and display the results in the terminal.
+
+Note that if you're using the default providers, you will need to set the `OPENAI_API_KEY` environment variable.
+
+<img src={JestExampleImage} />
@@ -0,0 +1,121 @@
+---
+sidebar_label: Langfuse
+description: Integrate Langfuse prompts with Promptfoo for LLM testing. Configure version control, labels, and collaborative prompt management using environment variables and SDK setup.
+---
+
+# Langfuse integration
+
+[Langfuse](https://langfuse.com) is an open-source LLM engineering platform that includes collaborative prompt management, tracing, and evaluation capabilities.
+
+## Setup
+
+1. Install the langfuse SDK:
+
+   ```bash
+   npm install langfuse
+   ```
+
+2. Set the required environment variables:
+   ```bash
+   export LANGFUSE_PUBLIC_KEY="your-public-key"
+   export LANGFUSE_SECRET_KEY="your-secret-key"
+   export LANGFUSE_HOST="https://cloud.langfuse.com"  # or your self-hosted URL
+   ```
+
+## Using Langfuse prompts
+
+Use the `langfuse://` prefix in your promptfoo configuration to reference prompts managed in Langfuse.
+
+### Prompt formats
+
+You can reference prompts by version or label using two different syntaxes:
+
+#### 1. Explicit @ syntax (recommended for clarity)
+
+```yaml
+# By label
+langfuse://prompt-name@label:type
+
+# Examples
+langfuse://my-prompt@production        # Text prompt with production label
+langfuse://chat-prompt@staging:chat    # Chat prompt with staging label
+```
+
+#### 2. Auto-detection with : syntax
+
+```yaml
+# By version or label (auto-detected)
+langfuse://prompt-name:version-or-label:type
+```
+
+The parser automatically detects:
+
+- **Numeric values** → treated as versions (e.g., `1`, `2`, `3`)
+- **String values** → treated as labels (e.g., `production`, `staging`, `latest`)
+
+Where:
+
+- `prompt-name`: The name of your prompt in Langfuse
+- `version`: Specific version number (e.g., `1`, `2`, `3`)
+- `label`: Label assigned to a prompt version (e.g., `production`, `staging`, `latest`)
+- `type`: Either `text` or `chat` (defaults to `text` if omitted)
+
+### Examples
+
+```yaml
+prompts:
+  # Explicit @ syntax for labels (recommended)
+  - 'langfuse://my-prompt@production' # Production label, text prompt
+  - 'langfuse://chat-prompt@staging:chat' # Staging label, chat prompt
+  - 'langfuse://my-prompt@latest:text' # Latest label, text prompt
+
+  # Auto-detection with : syntax
+  - 'langfuse://my-prompt:production' # String → treated as label
+  - 'langfuse://chat-prompt:staging:chat' # String → treated as label
+  - 'langfuse://my-prompt:latest' # "latest" → treated as label
+
+  # Version references (numeric values only)
+  - 'langfuse://my-prompt:3:text' # Numeric → version 3
+  - 'langfuse://chat-prompt:2:chat' # Numeric → version 2
+
+providers:
+  - openai:gpt-5-mini
+
+tests:
+  - vars:
+      user_query: 'What is the capital of France?'
+      context: 'European geography'
+```
+
+### Variable substitution
+
+Variables from your promptfoo test cases are automatically passed to Langfuse prompts. If your Langfuse prompt contains variables like `{{user_query}}` or `{{context}}`, they will be replaced with the corresponding values from your test cases.
+
+### Label-based deployment
+
+Using labels is recommended for production scenarios as it allows you to:
+
+- Deploy new prompt versions without changing your promptfoo configuration
+- Use different prompts for different environments (production, staging, development)
+- A/B test different prompt versions
+- Roll back to previous versions quickly in Langfuse
+
+Common label patterns:
+
+- `production` - Current production version
+- `staging` - Testing before production
+- `latest` - Most recently created version
+- `experiment-a`, `experiment-b` - A/B testing
+- `tenant-xyz` - Multi-tenant scenarios
+
+### Best practices
+
+1. **Use labels instead of version numbers** for production deployments to avoid hardcoding version numbers in your config
+2. **Use descriptive prompt names** that clearly indicate their purpose
+3. **Test prompts in staging** before promoting them to production
+4. **Version control your promptfoo configs** even though prompts are managed in Langfuse
+
+### Limitations
+
+- While prompt IDs containing `@` symbols are supported, we recommend avoiding them for clarity. The parser looks for the last `@` followed by a label pattern to distinguish between the prompt ID and label.
+- If you need to use `@` in your label names, consider using a different naming convention.
@@ -0,0 +1,143 @@
+---
+sidebar_label: Looper
+description: Automate LLM testing in CI/CD by integrating Promptfoo with Looper workflows. Configure quality gates, caching, and multi-environment evaluations for production AI pipelines.
+---
+
+# Setting up Promptfoo with Looper
+
+This guide shows you how to integrate **Promptfoo** evaluations into a Looper CI/CD workflow so that every pull‑request (and optional nightly job) automatically runs your prompt tests.
+
+## Prerequisites
+
+- A working Looperinstallation with workflow execution enabled
+- A build image (or declared tools) that provides **Node 22+** and **jq 1.6+**
+- `promptfooconfig.yaml` and your prompt fixtures (`prompts/**/*.json`) committed to the repository
+
+## Create `.looper.yml`
+
+Add the following file to the root of your repo:
+
+```yaml
+language: workflow # optional but common
+
+tools:
+  nodejs: 22 # Looper provisions Node.js
+  jq: 1.7
+
+envs:
+  global:
+    variables:
+      PROMPTFOO_CACHE_PATH: '${HOME}/.promptfoo/cache'
+
+triggers:
+  - pr # run on every pull‑request
+  - manual: 'Nightly Prompt Tests' # manual button in UI
+    call: nightly # invokes the nightly flow below
+
+flows:
+  # ---------- default PR flow ----------
+  default:
+    - (name Install Promptfoo) npm install -g promptfoo
+
+    - (name Evaluate Prompts) |
+      promptfoo eval \
+      -c promptfooconfig.yaml \
+      --prompts "prompts/**/*.json" \
+      --share \
+      -o output.json
+
+    - (name Quality gate) |
+      SUCC=$(jq -r '.results.stats.successes' output.json)
+      FAIL=$(jq -r '.results.stats.failures' output.json)
+      echo "✅ $SUCC  ❌ $FAIL"
+      test "$FAIL" -eq 0 # non‑zero exit fails the build
+
+  # ---------- nightly scheduled flow ----------
+  nightly:
+    - call: default # reuse the logic above
+    - (name Upload artefacts) |
+      aws s3 cp output.json s3://your-bucket/promptfoo/output.json
+```
+
+### How it works
+
+| Section                 | Purpose                                                             |
+| ----------------------- | ------------------------------------------------------------------- |
+| `tools`                 | Declares tool versions Looper should provision.                     |
+| `envs.global.variables` | Environment variables available to every step.                      |
+| `triggers`              | Determines when the workflow runs (`pr`, `manual`, `cron`, etc.).   |
+| `flows`                 | Ordered shell commands; execution stops on the first non‑zero exit. |
+
+## Caching Promptfoo results
+
+Looper lacks a first‑class cache API. Two common approaches:
+
+1. **Persistent volume** – mount `${HOME}/.promptfoo/cache` on a reusable volume.
+2. **Persistence tasks** – pull/push the cache at the start and end of the flow:
+
+## Setting quality thresholds
+
+```yaml
+    - (name Pass‑rate gate) |
+        TOTAL=$(jq '.results.stats.successes + .results.stats.failures' output.json)
+        PASS=$(jq '.results.stats.successes' output.json)
+        RATE=$(echo "scale=2; 100*$PASS/$TOTAL" | bc)
+        echo "Pass rate: $RATE%"
+        test $(echo "$RATE >= 95" | bc) -eq 1   # fail if <95 %
+```
+
+## Multi‑environment evaluations
+
+Evaluate both staging and production configs and compare failures:
+
+```yaml
+flows:
+  compare-envs:
+    - (name Eval‑prod) |
+      promptfoo eval \
+      -c promptfooconfig.prod.yaml \
+      --prompts "prompts/**/*.json" \
+      -o output-prod.json
+
+    - (name Eval‑staging) |
+      promptfoo eval \
+      -c promptfooconfig.staging.yaml \
+      --prompts "prompts/**/*.json" \
+      -o output-staging.json
+
+    - (name Compare) |
+      PROD_FAIL=$(jq '.results.stats.failures' output-prod.json)
+      STAGE_FAIL=$(jq '.results.stats.failures' output-staging.json)
+      if [ "$STAGE_FAIL" -gt "$PROD_FAIL" ]; then
+      echo "⚠️  Staging has more failures than production!"
+      fi
+```
+
+## Posting evaluation results to GitHub/GitLab
+
+In order to send evaluation results elsewhere, use:
+
+- **GitHub task**
+  ```yaml
+  - github --add-comment \
+    --repository "$CI_REPOSITORY" \
+    --issue "$PR_NUMBER" \
+    --body "$(cat comment.md)" # set comment as appropriate
+  ```
+- **cURL** with a Personal Access Token (PAT) against the REST API.
+
+## Troubleshooting
+
+| Problem                  | Remedy                                                                                  |
+| ------------------------ | --------------------------------------------------------------------------------------- |
+| `npm: command not found` | Add `nodejs:` under `tools` or use an image with Node pre‑installed.                    |
+| Cache not restored       | Verify the path and that the `files pull` task succeeds.                                |
+| Long‑running jobs        | Split prompt sets into separate flows or raise `timeoutMillis` in the build definition. |
+| API rate limits          | Enable Promptfoo cache and/or rotate API keys.                                          |
+
+## Best practices
+
+1. **Incremental testing** – feed `looper diff --name-only prompts/` into `promptfoo eval` to test only changed prompts.
+2. **Semantic version tags** – tag prompt sets/configs so you can roll back easily.
+3. **Secret management** – store API keys in a secret store and inject them as environment variables.
+4. **Reusable library flows** – if multiple repos need the same evaluation, host the flow definition in a central repo and `import` it.
@@ -0,0 +1,310 @@
+---
+title: Promptfoo MCP Server
+description: Deploy promptfoo as Model Context Protocol server enabling external AI agents to access evaluation and red teaming capabilities
+sidebar_label: MCP Server
+sidebar_position: 21
+---
+
+# Promptfoo MCP Server
+
+Expose promptfoo's eval tools to AI agents via Model Context Protocol (MCP).
+
+:::info Prerequisites
+
+- Node.js installed on your system
+- A promptfoo project with some evaluations (for testing the connection)
+- Cursor IDE, Claude Desktop, or another MCP-compatible AI tool
+- The optional `@modelcontextprotocol/sdk` runtime package when your promptfoo install omits optional dependencies
+
+:::
+
+For slim installs created with omitted optional dependencies, use a local install that includes
+both promptfoo and the MCP SDK:
+
+```bash
+npm install promptfoo @modelcontextprotocol/sdk
+```
+
+When using that local slim-install setup, run the MCP commands below as `npx promptfoo ...`
+instead of `npx promptfoo@latest ...` so both packages resolve from the same project.
+
+## Quick Start
+
+### 1. Start the Server
+
+```bash
+# For Cursor, Claude Desktop (STDIO transport)
+npx promptfoo@latest mcp --transport stdio
+
+# For web tools (HTTP transport)
+npx promptfoo@latest mcp --transport http --port 3100
+```
+
+### 2. Configure Your AI Tool
+
+**Cursor**: Create `.cursor/mcp.json` in your project root
+
+```json title=".cursor/mcp.json"
+{
+  "mcpServers": {
+    "promptfoo": {
+      "command": "npx",
+      "args": ["promptfoo@latest", "mcp", "--transport", "stdio"],
+      "description": "Promptfoo MCP server for LLM evaluation and testing"
+    }
+  }
+}
+```
+
+:::warning Development vs Production Configuration
+
+**For regular usage:** Always use `npx promptfoo@latest` as shown above.
+
+**For promptfoo contributors:** The repository's `.cursor/mcp.json` runs from source code for development. It requires the repo's dev dependencies and won't work elsewhere.
+
+:::
+
+**Claude Desktop**: Add to config file
+
+Config file locations:
+
+- **macOS:** `~/Library/Application Support/Claude/claude_desktop_config.json`
+- **Windows:** `%APPDATA%\Claude\claude_desktop_config.json`
+- **Linux:** `~/.config/Claude/claude_desktop_config.json`
+
+```json title="claude_desktop_config.json"
+{
+  "mcpServers": {
+    "promptfoo": {
+      "command": "npx",
+      "args": ["promptfoo@latest", "mcp", "--transport", "stdio"],
+      "description": "Promptfoo MCP server for LLM evaluation and testing"
+    }
+  }
+}
+```
+
+**Restart your AI tool** after adding the configuration.
+
+### 3. Test the Connection
+
+After restarting your AI tool, you should see promptfoo tools available. Try asking:
+
+> "List my recent evaluations using the promptfoo tools"
+
+## Available Tools
+
+### Core Evaluation Tools
+
+- **`list_evaluations`** - Browse your evaluation runs with optional dataset filtering
+- **`get_evaluation_details`** - Get comprehensive results, metrics, and test cases for a specific evaluation
+- **`run_evaluation`** - Execute evaluations with custom parameters, test case filtering, and concurrency control
+- **`share_evaluation`** - Generate publicly shareable URLs for evaluation results
+
+### Generation Tools
+
+- **`generate_dataset`** - Generate test datasets using AI for comprehensive evaluation coverage
+- **`generate_test_cases`** - Generate test cases with assertions for existing prompts
+- **`compare_providers`** - Compare multiple AI providers side-by-side for performance and quality
+
+### Redteam Security Tools
+
+- **`redteam_run`** - Execute comprehensive security testing against AI applications with dynamic attack probes
+- **`redteam_generate`** - Generate adversarial test cases for redteam security testing with configurable plugins and strategies
+
+### Configuration & Testing
+
+- **`validate_promptfoo_config`** - Validate configuration files using the same logic as the CLI
+- **`test_provider`** - Test AI provider connectivity, credentials, and response quality
+- **`run_assertion`** - Test individual assertion rules against outputs for debugging
+
+## Example Workflows
+
+### 1. Basic Evaluation Workflow
+
+Ask your AI assistant:
+
+> "Help me run an evaluation. First, validate my config, then list recent evaluations, and finally run a new evaluation with just the first 5 test cases."
+
+The AI will use these tools in sequence:
+
+1. `validate_promptfoo_config` - Check your configuration
+2. `list_evaluations` - Show recent runs
+3. `run_evaluation` - Execute with test case filtering, such as `{"start": 0, "end": 5}` for the first five zero-based test indices
+
+### 2. Provider Comparison
+
+> "Compare the performance of GPT-4, Claude 3, and Gemini Pro on my customer support prompt."
+
+The AI will:
+
+1. `test_provider` - Verify each provider works
+2. `compare_providers` - Run side-by-side comparison
+3. Analyze results and provide recommendations
+
+### 3. Security Testing
+
+> "Run a security audit on my chatbot prompt to check for jailbreak vulnerabilities."
+
+The AI will:
+
+1. `redteam_generate` - Create adversarial test cases
+2. `redteam_run` - Execute security tests
+3. `get_evaluation_details` - Analyze vulnerabilities found
+
+### 4. Dataset Generation
+
+> "Generate 20 diverse test cases for my email classification prompt, including edge cases."
+
+The AI will:
+
+1. `generate_dataset` - Create test data with AI
+2. `generate_test_cases` - Add appropriate assertions
+3. `run_evaluation` - Test the generated cases
+
+## Transport Types
+
+Choose the appropriate transport based on your use case:
+
+- **STDIO (`--transport stdio`)**: For desktop AI tools (Cursor, Claude Desktop) that communicate via stdin/stdout
+- **HTTP (`--transport http`)**: For web applications, APIs, and remote integrations that need HTTP endpoints
+
+## Best Practices
+
+### 1. Start Small
+
+Begin with simple tools like `list_evaluations` and `validate_promptfoo_config` before moving to more complex operations.
+
+### 2. Use Filtering
+
+When working with large datasets:
+
+- Filter evaluations by dataset ID
+- Use test case indices to run partial evaluations
+- Apply prompt/provider filters for focused testing
+
+### 3. Iterative Testing
+
+1. Validate configuration first
+2. Test providers individually before comparisons
+3. Run small evaluation subsets before full runs
+4. Review results with `get_evaluation_details`
+
+### 4. Security First
+
+When using redteam tools:
+
+- Start with basic plugins before advanced attacks
+- Review generated test cases before running
+- Always analyze results thoroughly
+
+## Troubleshooting
+
+### Server Issues
+
+**Server won't start:**
+
+```bash
+# Verify promptfoo installation
+npx promptfoo@latest --version
+
+# Check if you have a valid promptfoo project
+npx promptfoo@latest validate
+
+# Test the MCP server manually
+npx promptfoo@latest mcp --transport stdio
+```
+
+If the error says `@modelcontextprotocol/sdk` is required, install that optional runtime package
+alongside the promptfoo installation that runs the server. For example, add both packages to the
+same project before using a local `npx promptfoo ...` command:
+
+```bash
+npm install promptfoo @modelcontextprotocol/sdk
+```
+
+**Port conflicts (HTTP mode):**
+
+```bash
+# Use a different port
+npx promptfoo@latest mcp --transport http --port 8080
+
+# Check what's using port 3100
+lsof -i :3100  # macOS/Linux
+netstat -ano | findstr :3100  # Windows
+```
+
+### AI Tool Connection Issues
+
+**AI tool can't connect:**
+
+1. **Verify config syntax:** Ensure your JSON configuration exactly matches the examples above
+2. **Check file paths:** Confirm config files are in the correct locations
+3. **Restart completely:** Close your AI tool entirely and reopen it
+4. **Test HTTP endpoint:** For HTTP transport, verify with `curl http://localhost:3100/health`
+
+**Tools not appearing:**
+
+1. Look for MCP or "tools" indicators in your AI tool's interface
+2. Try asking explicitly: "What promptfoo tools do you have access to?"
+3. Check your AI tool's logs for MCP connection errors
+
+### Tool-Specific Errors
+
+**"Eval not found":**
+
+- Use `list_evaluations` first to see available evaluation IDs
+- Ensure you're in a directory with promptfoo evaluation data
+
+**"Config error":**
+
+- Run `validate_promptfoo_config` to check your configuration
+- Verify `promptfooconfig.yaml` exists and is valid
+
+**"Provider error":**
+
+- Use `test_provider` to diagnose connectivity and authentication issues
+- Check your API keys and provider configurations
+
+## Advanced Usage
+
+### Custom HTTP Integrations
+
+For HTTP transport, you can integrate with any system that supports HTTP:
+
+```javascript
+// Example: Call MCP server from Node.js
+const response = await fetch('http://localhost:3100/mcp', {
+  method: 'POST',
+  headers: { 'Content-Type': 'application/json' },
+  body: JSON.stringify({
+    method: 'tools/call',
+    params: {
+      name: 'list_evaluations',
+      arguments: { datasetId: 'my-dataset' },
+    },
+  }),
+});
+```
+
+### Environment Variables
+
+The MCP server respects all promptfoo environment variables:
+
+```bash
+# Set provider API keys
+export OPENAI_API_KEY=sk-...
+export ANTHROPIC_API_KEY=sk-ant-...
+
+# Configure promptfoo behavior
+export PROMPTFOO_CONFIG_DIR=/path/to/configs
+
+# Start server with environment
+npx promptfoo@latest mcp --transport stdio
+```
+
+## Resources
+
+- [MCP Protocol Documentation](https://modelcontextprotocol.io)
+- [Promptfoo Documentation](https://promptfoo.dev)
+- [Example Configurations](https://github.com/promptfoo/promptfoo/tree/main/examples)
@@ -0,0 +1,247 @@
+---
+title: Using MCP (Model Context Protocol) in Promptfoo
+description: Enable Model Context Protocol (MCP) integration for enhanced tool use, persistent memory, and agentic workflows across providers
+sidebar_label: Model Context Protocol (MCP)
+sidebar_position: 20
+---
+
+# Using MCP (Model Context Protocol) in Promptfoo
+
+Promptfoo supports the Model Context Protocol (MCP) for advanced tool use, and agentic workflows. MCP allows you to connect your Promptfoo providers to an external MCP server, such as the [modelcontextprotocol/server-memory](https://github.com/modelcontextprotocol/server-memory), to enable tool orchestration, and more.
+
+:::note MCP SDK dependency
+
+Promptfoo's general MCP integration uses the optional `@modelcontextprotocol/sdk` runtime package. Standard npm installs include optional dependencies, but installs that omit them need:
+
+```bash
+npm install @modelcontextprotocol/sdk
+```
+
+:::
+
+## Basic Configuration
+
+To enable MCP for a provider, add the `mcp` block to your provider's `config` in your `promptfooconfig.yaml`:
+
+```yaml title="promptfooconfig.yaml"
+description: Testing MCP memory server integration with Google AI Studio
+providers:
+  - id: google:gemini-2.0-flash
+    config:
+      mcp:
+        enabled: true
+        server:
+          command: npx
+          args: ['-y', '@modelcontextprotocol/server-memory']
+          name: memory
+```
+
+### MCP Config Options
+
+- `enabled`: Set to `true` to enable MCP for this provider.
+- `timeout`: (Optional) Request timeout in milliseconds for MCP tool calls. Defaults to 60000 (60 seconds). Set higher for long-running tools.
+- `resetTimeoutOnProgress`: (Optional) Reset timeout when progress notifications are received. Useful for long-running operations. Default: false.
+- `maxTotalTimeout`: (Optional) Absolute maximum timeout in milliseconds regardless of progress notifications.
+- `pingOnConnect`: (Optional) Ping the server after connecting to verify it's responsive. Default: false.
+- `server`: (Optional) Configuration for launching or connecting to an MCP server.
+  - `command`: The command to launch the MCP server (e.g., `npx`).
+  - `args`: Arguments to pass to the command (e.g., `['-y', '@modelcontextprotocol/server-memory']`).
+  - `name`: (Optional) A name for the server instance.
+  - `url`: URL for connecting to a remote MCP server.
+  - `headers`: (Optional) Custom HTTP headers to send when connecting to a remote MCP server (only applies to `url`-based connections).
+  - `auth`: (Optional) Authentication configuration for the server. Can be used to automatically set auth headers for all connection types.
+    - `type`: Authentication type, either `'bearer'` or `'api_key'`.
+    - `token`: Token for bearer authentication.
+    - `api_key`: API key for api_key authentication.
+- You can also connect to a remote MCP server by specifying a `url` instead of `command`/`args`.
+
+MCP servers can be run locally or accessed remotely. For development and testing, a local server is often simplest, while production environments may use a centralized remote server.
+
+#### Example: Connecting to a Remote MCP Server
+
+```yaml
+providers:
+  - id: openai:responses:gpt-5.1
+    config:
+      apiKey: <your-api-key>
+      mcp:
+        enabled: true
+        server:
+          url: http://localhost:8000
+```
+
+#### Example: Using Custom Headers with a Remote MCP Server
+
+```yaml
+providers:
+  - id: openai:responses:gpt-5.1
+    config:
+      apiKey: <your-api-key>
+      mcp:
+        enabled: true
+        server:
+          url: http://localhost:8000
+          headers:
+            X-API-Key: your-custom-api-key
+            Authorization: Bearer your-token
+            X-Custom-Header: custom-value
+```
+
+This can be useful when:
+
+- The MCP server requires an API key or authentication token
+- You need to provide custom identifiers or session information
+- The server needs specific headers for configuration or tracking
+
+## Connecting a Single Provider to Multiple MCP Servers
+
+Promptfoo allows a single provider to connect to multiple MCP servers by using the `servers` array in your provider's MCP config. All tools from all connected servers will be available to the provider.
+
+### Example: One Provider, Multiple MCP Servers
+
+```yaml title="promptfooconfig.yaml"
+providers:
+  - id: openai:responses:gpt-5.1
+    config:
+      mcp:
+        enabled: true
+        servers:
+          - command: npx
+            args: ['-y', '@modelcontextprotocol/server-memory']
+            name: server_a
+          - url: http://localhost:8001
+            name: server_b
+            headers:
+              X-API-Key: your-api-key
+```
+
+- Use the `servers:` array (not just `server:`) to specify multiple MCP servers.
+- Each entry can be a local launch or a remote URL (if supported).
+- All tools from all servers will be available to the provider.
+- You can specify different headers for each server when using URL connections.
+- You can also connect to the same server multiple times if needed:
+
+```yaml
+providers:
+  - id: anthropic:claude-sonnet-4-5-20250929
+    config:
+      mcp:
+        enabled: true
+        servers:
+          - command: npx
+            args: ['-y', '@modelcontextprotocol/server-memory']
+            name: memory
+          - command: npx
+            args: ['-y', '@modelcontextprotocol/server-filesystem']
+            name: filesystem
+          - command: npx
+            args: ['-y', '@modelcontextprotocol/server-github']
+            name: github
+```
+
+This configuration connects a single provider to multiple MCP servers, giving it access to memory storage, filesystem operations, and GitHub integration simultaneously.
+
+## Using Multiple MCP Servers
+
+You can configure multiple MCP servers by assigning different MCP server configurations to different providers in your `promptfooconfig.yaml`. Each provider can have its own `mcp.server` block, allowing you to run separate memory/tool servers for different models or use cases.
+
+```yaml title="promptfooconfig.yaml"
+description: Using multiple MCP servers
+providers:
+  - id: google:gemini-2.0-flash
+    config:
+      mcp:
+        enabled: true
+        server:
+          command: npx
+          args: ['-y', '@modelcontextprotocol/server-memory']
+          name: gemini-memory
+
+  - id: openai:responses:gpt-5.1
+    config:
+      apiKey: <your-api-key>
+      mcp:
+        enabled: true
+        server:
+          url: http://localhost:8001
+          name: openai-memory
+          headers:
+            X-API-Key: openai-server-api-key
+
+  - id: anthropic:claude-sonnet-4-5-20250929
+    config:
+      mcp:
+        enabled: true
+        server:
+          url: http://localhost:8002
+          name: anthropic-memory
+          headers:
+            Authorization: Bearer anthropic-server-token
+```
+
+In this example:
+
+- The Gemini provider launches a local MCP server using `npx`.
+- The OpenAI and Anthropic providers connect to different remote MCP servers running on different ports.
+- Each provider can have its own memory, tool set, and context, isolated from the others.
+- Custom headers are specified for the remote servers to handle authentication or other requirements.
+
+This setup is useful for testing, benchmarking, or running isolated agentic workflows in parallel.
+
+## Supported Providers
+
+MCP is supported by most major providers in Promptfoo, including:
+
+- Google Gemini (AI Studio, Vertex)
+- OpenAI (and compatible providers like Groq, Together, etc.)
+- Anthropic
+
+## OpenAI Responses API MCP Integration
+
+In addition to the general MCP integration described above, OpenAI's Responses API has native MCP support that allows direct connection to remote MCP servers without running local MCP servers. This approach is specific to OpenAI's Responses API and offers:
+
+- Direct connection to remote MCP servers (like DeepWiki, Stripe, etc.)
+- Built-in approval workflows for data sharing
+- Authentication header support for secured MCP servers
+- Tool filtering capabilities
+
+For detailed information about using MCP with OpenAI's Responses API, see the [OpenAI Provider MCP documentation](../providers/openai.md#mcp-model-context-protocol-support).
+
+## Tool Schema Compatibility
+
+Promptfoo automatically handles JSON Schema compatibility between MCP servers and LLM providers by removing provider-incompatible metadata fields (like `$schema`) while preserving supported features. Tools with no input parameters work without modification.
+
+## Timeout Configuration
+
+MCP tool calls have a default timeout of 60 seconds. For long-running tools, increase the timeout:
+
+```yaml
+providers:
+  - id: openai:responses:gpt-5.1
+    config:
+      mcp:
+        enabled: true
+        timeout: 900000 # 15 minutes in milliseconds
+        server:
+          url: https://api.example.com/mcp
+```
+
+You can also set a global default via environment variable:
+
+```bash
+export MCP_REQUEST_TIMEOUT_MS=900000  # 15 minutes
+```
+
+Priority: `config.timeout` > `MCP_REQUEST_TIMEOUT_MS` env var > SDK default (60s).
+
+## Troubleshooting
+
+- Ensure your MCP server is running and accessible.
+- Check your provider logs for MCP connection errors.
+- Verify that your custom headers are correctly formatted if you're having authentication issues.
+- If tool calls timeout, increase the `timeout` config option or set `MCP_REQUEST_TIMEOUT_MS`.
+
+## See Also
+
+- [Configuration Reference](../configuration/reference.md)
+- [Provider Configuration](../providers/index.md)
@@ -0,0 +1,208 @@
+---
+sidebar_label: Mocha/Chai
+description: Integrate Promptfoo LLM testing with Mocha and Chai to automate prompt quality checks using semantic similarity, factuality, and custom assertions in your test suite.
+---
+
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+
+# Testing prompts with Mocha/Chai
+
+`promptfoo` can be integrated with test frameworks like [Mocha](https://mochajs.org/) and assertion libraries like [Chai](https://www.chaijs.com/) in order to evaluate prompts as part of existing testing and CI workflows.
+
+This guide includes examples that show how to create Mocha test cases for desired prompt quality using semantic similarity and LLM grading.
+
+For more information on supported checks, see [Assertions & Metrics documentation](/docs/configuration/expected-outputs/).
+
+## Prerequisites
+
+Before you begin, make sure you have the following node packages installed:
+
+- [mocha](https://mochajs.org/#installation): `npm install --save-dev mocha`
+- [chai](https://www.chaijs.com/guide/installation/): `npm install --save-dev chai`
+- promptfoo: `npm install --save-dev promptfoo`
+
+## Creating custom chai assertions
+
+First, we'll create custom chai assertions:
+
+- `toMatchSemanticSimilarity`: Compares two strings for semantic similarity.
+- `toPassLLMRubric`: Checks if a string meets the specified LLM Rubric criteria.
+- `toMatchFactuality`: Checks if a string meets the specified factuality criteria.
+- `toMatchClosedQA`: Checks if a string meets the specified question-answering criteria.
+
+Create a new file called `assertions.js` and add the following:
+
+<Tabs>
+  <TabItem value="Javascript" label="Javascript" default>
+
+```javascript
+import { Assertion } from 'chai';
+import { assertions } from 'promptfoo';
+
+const { matchesSimilarity, matchesLlmRubric } = assertions;
+
+Assertion.addAsyncMethod('toMatchSemanticSimilarity', async function (expected, threshold = 0.8) {
+  const received = this._obj;
+  const result = await matchesSimilarity(received, expected, threshold);
+  const pass = received === expected || result.pass;
+
+  this.assert(
+    pass,
+    `expected #{this} to match semantic similarity with #{exp}, but it did not. Reason: ${result.reason}`,
+    `expected #{this} not to match semantic similarity with #{exp}`,
+    expected,
+  );
+});
+
+Assertion.addAsyncMethod('toPassLLMRubric', async function (expected, gradingConfig) {
+  const received = this._obj;
+  const gradingResult = await matchesLlmRubric(expected, received, gradingConfig);
+
+  this.assert(
+    gradingResult.pass,
+    `expected #{this} to pass LLM Rubric with #{exp}, but it did not. Reason: ${gradingResult.reason}`,
+    `expected #{this} not to pass LLM Rubric with #{exp}`,
+    expected,
+  );
+});
+
+Assertion.addAsyncMethod('toMatchFactuality', async function (input, expected, gradingConfig) {
+  const received = this._obj;
+  const gradingResult = await matchesFactuality(input, expected, received, gradingConfig);
+
+  this.assert(
+    gradingResult.pass,
+    `expected #{this} to match factuality with #{exp}, but it did not. Reason: ${gradingResult.reason}`,
+    `expected #{this} not to match factuality with #{exp}`,
+    expected,
+  );
+});
+
+Assertion.addAsyncMethod('toMatchClosedQA', async function (input, expected, gradingConfig) {
+  const received = this._obj;
+  const gradingResult = await matchesClosedQa(input, expected, received, gradingConfig);
+
+  this.assert(
+    gradingResult.pass,
+    `expected #{this} to match ClosedQA with #{exp}, but it did not. Reason: ${gradingResult.reason}`,
+    `expected #{this} not to match ClosedQA with #{exp}`,
+    expected,
+  );
+});
+```
+
+  </TabItem>
+  <TabItem value="Typescript" label="Typescript" default>
+
+```typescript
+import { Assertion } from 'chai';
+import { assertions } from 'promptfoo';
+import type { GradingConfig } from 'promptfoo';
+
+const { matchesSimilarity, matchesLlmRubric } = assertions;
+
+Assertion.addAsyncMethod(
+  'toMatchSemanticSimilarity',
+  async function (this: Assertion, expected: string, threshold: number = 0.8) {
+    const received = this._obj;
+    const result = await matchesSimilarity(received, expected, threshold);
+    const pass = received === expected || result.pass;
+
+    this.assert(
+      pass,
+      `expected #{this} to match semantic similarity with #{exp}, but it did not. Reason: ${result.reason}`,
+      `expected #{this} not to match semantic similarity with #{exp}`,
+      expected,
+    );
+  },
+);
+
+Assertion.addAsyncMethod(
+  'toPassLLMRubric',
+  async function (this: Assertion, expected: string, gradingConfig: GradingConfig) {
+    const received = this._obj;
+    const gradingResult = await matchesLlmRubric(expected, received, gradingConfig);
+
+    this.assert(
+      gradingResult.pass,
+      `expected #{this} to pass LLM Rubric with #{exp}, but it did not. Reason: ${gradingResult.reason}`,
+      `expected #{this} not to pass LLM Rubric with #{exp}`,
+      expected,
+    );
+  },
+);
+```
+
+  </TabItem>
+</Tabs>
+
+## Writing tests
+
+Our test code will use the custom chai assertions in order to run a few test cases.
+
+Create a new file called `index.test.js` and add the following code:
+
+```javascript
+import { expect } from 'chai';
+import './assertions';
+
+const gradingConfig = {
+  provider: 'openai:chat:gpt-5-mini',
+};
+
+describe('semantic similarity tests', () => {
+  it('should pass when strings are semantically similar', async () => {
+    await expect('The quick brown fox').toMatchSemanticSimilarity('A fast brown fox');
+  });
+
+  it('should fail when strings are not semantically similar', async () => {
+    await expect('The quick brown fox').not.toMatchSemanticSimilarity('The weather is nice today');
+  });
+
+  it('should pass when strings are semantically similar with custom threshold', async () => {
+    await expect('The quick brown fox').toMatchSemanticSimilarity('A fast brown fox', 0.7);
+  });
+
+  it('should fail when strings are not semantically similar with custom threshold', async () => {
+    await expect('The quick brown fox').not.toMatchSemanticSimilarity(
+      'The weather is nice today',
+      0.9,
+    );
+  });
+});
+
+describe('LLM evaluation tests', () => {
+  it('should pass when strings meet the LLM Rubric criteria', async () => {
+    await expect('Four score and seven years ago').toPassLLMRubric(
+      'Contains part of a famous speech',
+      gradingConfig,
+    );
+  });
+
+  it('should fail when strings do not meet the LLM Rubric criteria', async () => {
+    await expect('It is time to do laundry').not.toPassLLMRubric(
+      'Contains part of a famous speech',
+      gradingConfig,
+    );
+  });
+});
+```
+
+## Final setup
+
+Add the following line to the `scripts` section in your `package.json`:
+
+```json
+"test": "mocha"
+```
+
+Now, you can run the tests with the following command:
+
+```sh
+npm test
+```
+
+This will execute the tests and display the results in the terminal.
+
+Note that if you're using the default providers, you will need to set the `OPENAI_API_KEY` environment variable.
@@ -0,0 +1,225 @@
+---
+sidebar_label: n8n
+title: Using Promptfoo in n8n Workflows
+description: Learn how to integrate Promptfoo's LLM evaluation into your n8n workflows for automated testing, security and quality gates, and result sharing
+---
+
+# Using Promptfoo in n8n Workflows
+
+This guide shows how to run Promptfoo evaluations from an **n8n** workflow so you can:
+
+- schedule nightly or ad‑hoc LLM tests,
+- gate downstream steps (Slack/Teams alerts, merge approvals, etc.) on pass‑rates, and
+- publish rich results links generated by Promptfoo.
+
+## Prerequisites
+
+| What                                                                                 | Why                                                   |
+| ------------------------------------------------------------------------------------ | ----------------------------------------------------- |
+| **Self‑hosted n8n ≥ v1** (Docker or bare‑metal)                                      | Gives access to the “Execute Command” node.           |
+| **Promptfoo CLI** available in the container/host                                    | Needed to run `promptfoo eval`.                       |
+| (Optional) **LLM provider API keys** set as environment variables or n8n credentials | Example: `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, …     |
+| (Optional) **Slack / email / GitHub nodes** in the same workflow                     | For notifications or comments once the eval finishes. |
+
+### Shipping a custom Docker image (recommended)
+
+The easiest way is to bake Promptfoo into your n8n image so every workflow run already has the CLI:
+
+```dockerfile
+# Dockerfile
+FROM n8nio/n8n:latest          # or a fixed tag
+USER root                      # gain perms to install packages
+RUN npm install -g promptfoo   # installs CLI system‑wide
+ENV PROMPTFOO_RUNNING_IN_DOCKER=1
+USER node                      # drop back to non‑root
+```
+
+Update **`docker‑compose.yml`**:
+
+```yaml
+services:
+  n8n:
+    build: .
+    env_file: .env # where your OPENAI_API_KEY lives
+    volumes:
+      - ./data:/data # prompts & configs live here
+```
+
+If you prefer not to rebuild the image you _can_ install Promptfoo on the fly inside the
+**Execute Command** node, but that adds 10‑15 s to every execution.
+
+## Basic “Run & Alert” workflow
+
+Below is the minimal pattern most teams start with:
+
+| #   | Node                          | Purpose                                                           |
+| --- | ----------------------------- | ----------------------------------------------------------------- |
+| 1   | **Trigger** (Cron or Webhook) | Decide _when_ to evaluate (nightly, on Git push webhook …).       |
+| 2   | **Execute Command**           | Runs Promptfoo and emits raw stdout / stderr.                     |
+| 3   | **Code / Set** node           | Parses the resulting JSON, extracts pass/fail counts & share‑URL. |
+| 4   | **IF** node                   | Branches on “failures > 0”.                                       |
+| 5   | **Slack / Email / GitHub**    | Sends alert or PR comment when the gate fails.                    |
+
+### Execute Command node configuration
+
+```sh
+promptfoo eval \
+ -c /data/promptfooconfig.yaml \
+ --prompts "/data/prompts/**/*.json" \
+ --output /tmp/pf-results.json \
+ --share --fail-on-error
+cat /tmp/pf-results.json
+```
+
+Set the working directory to `/data` (mount it with Docker volume) and set it to execute once (one run per trigger).
+
+The node writes a machine‑readable results file **and** prints it to stdout,
+so the next node can simply `JSON.parse($json["stdout"])`.
+
+:::info
+The **Execute Command** node that we rely on is only available in **self‑hosted** n8n. n8n Cloud does **not** expose it yet.
+:::
+
+### Sample “Parse & alert” snippet (Code node, TypeScript)
+
+```ts
+// Input: raw JSON string from previous node
+const output = JSON.parse(items[0].json.stdout as string);
+
+const { successes, failures } = output.results.stats;
+items[0].json.passRate = successes / (successes + failures);
+items[0].json.failures = failures;
+items[0].json.shareUrl = output.shareableUrl;
+
+return items;
+```
+
+An **IF** node can then route execution:
+
+- **failures = 0** → take _green_ path (maybe just archive the results).
+- **failures > 0** → post to Slack or comment on the pull request.
+
+## Evaluating n8n AI Agent prompts and outputs
+
+If your goal is to test the **prompt inside an n8n AI Agent / OpenAI node** (not just run Promptfoo from a workflow), treat the n8n node like any other app contract:
+
+1. Put the agent prompt in a file,
+2. Map incoming n8n fields to `tests.vars`, and
+3. Assert on the exact JSON or tool-call shape that downstream n8n nodes expect.
+
+This works well when you want to regression-test an agent before wiring it into a larger workflow.
+
+### Validate JSON that downstream n8n nodes consume
+
+If your agent is supposed to emit structured data for a **Set**, **Code**, **Switch**, or **HTTP Request** node, validate the payload directly.
+
+```yaml title="promptfooconfig.yaml"
+prompts:
+  - file://./prompts/n8n-support-router.txt
+
+providers:
+  - openai:gpt-5-mini
+
+tests:
+  - vars:
+      customer_message: 'Customer wants to cancel order #4815 and asks for a refund'
+    assert:
+      - type: contains-json
+        value:
+          type: object
+          required: [route, priority, reply]
+          properties:
+            route:
+              type: string
+              enum: [billing, support, sales]
+            priority:
+              type: string
+              enum: [low, medium, high]
+            reply:
+              type: string
+```
+
+Use `contains-json` when the model may wrap JSON in prose or a markdown code block. If your node must return **only** JSON, use [`is-json`](/docs/guides/evaluate-json) instead.
+
+### Validate tool calls for agent workflows
+
+If your n8n setup uses an OpenAI-compatible agent that should call tools before continuing, validate that Promptfoo sees a real tool call and that it matches your schema.
+
+```yaml title="promptfooconfig.yaml"
+prompts:
+  - file://./prompts/n8n-calendar-agent.txt
+
+providers:
+  - id: openai:gpt-5-mini
+    config:
+      tools: file://./tools/calendar-tools.yaml
+
+tests:
+  - vars:
+      user_request: "Move tomorrow's standup to 3pm and notify the team"
+    assert:
+      - type: finish-reason
+        value: tool_calls
+      - type: is-valid-openai-tools-call
+```
+
+That pattern is especially useful when your n8n workflow branches on whether the LLM produced a tool invocation versus a final answer.
+
+### Useful building blocks
+
+- [`/docs/configuration/tools`](/docs/configuration/tools) for defining tool schemas
+- [`/docs/guides/evaluate-json`](/docs/guides/evaluate-json) for JSON and schema assertions
+- [`examples/openai-tools-call`](https://github.com/promptfoo/promptfoo/tree/main/examples/openai-tools-call) for a concrete OpenAI tool-calling config
+- [`examples/eval-tool-use`](https://github.com/promptfoo/promptfoo/tree/main/examples/eval-tool-use) for finish-reason and tool-use checks across providers
+
+## Advanced patterns
+
+### Run different configs in parallel
+
+Make the first **Execute Command** node loop over an array of model IDs or config
+files and push each run as a separate item.
+Downstream nodes will automatically fan‑out and handle each result independently.
+
+### Version‑controlled prompts
+
+Mount your prompts directory and config file into the container at
+`/data`. When you commit new prompts to Git, your CI/CD system can call the
+**n8n REST API** or a **Webhook trigger** to re‑evaluate immediately.
+
+### Auto‑fail the whole workflow
+
+If you run n8n **headless** via `n8n start --tunnel`, you can call this workflow
+from CI pipelines (GitHub Actions, GitLab, …) with the [CLI `n8n execute`
+command](https://docs.n8n.io/hosting/cli-commands/) and then check the HTTP
+response code; returning `exit 1` from the Execute Command node will propagate
+the failure.
+
+## Security & best practices
+
+- **Keep API keys secret** – store them in the n8n credential store or inject as
+  environment variables from Docker secrets, not hard‑coded in workflows.
+- **Resource usage** – Promptfoo supports caching via
+  `PROMPTFOO_CACHE_PATH`; mount that directory to persist across runs.
+- **Timeouts** – wrap `promptfoo eval` with `timeout --signal=SIGKILL 15m …`
+  (Linux) if you need hard execution limits.
+- **Logging** – route the `stderr` field of Execute Command to a dedicated log
+  channel so you don’t miss stack traces.
+
+## Troubleshooting
+
+| Symptom                                     | Likely cause / fix                                                                                    |
+| ------------------------------------------- | ----------------------------------------------------------------------------------------------------- |
+| **`Execute Command node not available`**    | You’re on n8n Cloud; switch to self‑hosted.                                                           |
+| **`promptfoo: command not found`**          | Promptfoo not installed inside the container. Rebuild your Docker image or add an install step.       |
+| **Run fails with `ENOENT` on config paths** | Make sure the prompts/config volume is mounted at the same path you reference in the command.         |
+| **Large evals time‑out**                    | Increase the node’s “Timeout (s)” setting _or_ chunk your test cases and iterate inside the workflow. |
+
+## Next steps
+
+1. Combine Promptfoo with the **n8n AI Transform** node to chain evaluations
+   into multi‑step RAG pipelines.
+2. Use **n8n Insights** (self‑hosted EE) to monitor historical pass‑rates and
+   surface regressions.
+3. Check out the other [CI integrations](/docs/integrations/ci-cd) ([GitHub Actions](/docs/integrations/github-action), [CircleCI](/docs/integrations/circle-ci), etc) for inspiration.
+
+Happy automating!
@@ -0,0 +1,96 @@
+---
+sidebar_label: Portkey AI
+description: Integrate Portkey AI gateway with promptfoo for LLM testing, including prompt management, observability, and custom configurations with OpenAI models and APIs.
+---
+
+# Portkey AI integration
+
+Portkey is an AI observability suite that includes prompt management capabilities.
+
+To reference prompts in Portkey:
+
+1. Set the `PORTKEY_API_KEY` environment variable.
+
+2. Use the `portkey://` prefix for your prompts, followed by the Portkey prompt ID. For example:
+
+   ```yaml
+   prompts:
+     - 'portkey://pp-test-promp-669f48'
+
+   providers:
+     - openai:gpt-5-mini
+
+   tests:
+     - vars:
+         topic: ...
+   ```
+
+Variables from your promptfoo test cases will be automatically plugged into the Portkey prompt as variables. The resulting prompt will be rendered and returned to promptfoo, and used as the prompt for the test case.
+
+Note that promptfoo does not follow the temperature, model, and other parameters set in Portkey. You must set them in the `providers` configuration yourself.
+
+## Using Portkey gateway
+
+The Portkey AI gateway is directly supported by promptfoo. See also:
+
+- [Portkey's documentation on integrating promptfoo](https://portkey.ai/docs/integrations/libraries/promptfoo)
+
+Example:
+
+```yaml
+providers:
+  id: portkey:gpt-5-mini
+  config:
+    portkeyProvider: openai
+```
+
+More complex portkey configurations are also supported.
+
+```yaml
+providers:
+  id: portkey:gpt-5-mini
+  config:
+    # Can alternatively set environment variable, e.g. PORTKEY_API_KEY
+    portkeyApiKey: xxx
+
+    # Other configuration options
+    portkeyVirtualKey: xxx
+    portkeyMetadata:
+      team: xxx
+    portkeyConfig: xxx
+    portkeyProvider: xxx
+    portkeyApiBaseUrl: xxx
+```
+
+## Portkey MCP Gateway
+
+Promptfoo can connect to [Portkey's MCP Gateway](https://portkey.ai/docs/product/mcp-gateway/) in two ways. Use the [`mcp` provider](/docs/providers/mcp/) to test or red team the MCP server directly. To test an LLM application that uses the server, add the same server block to the model provider's [`mcp` config](/docs/integrations/mcp/).
+
+`PORTKEY_API_BASE_URL` does not configure the MCP connection. It sets the OpenAI-compatible chat-completions endpoint used by the `portkey:` provider and defaults to `https://api.portkey.ai/v1`. Put the MCP Gateway URL in `server.url` instead.
+
+The gateway exposes each registered server at `https://mcp.portkey.ai/<server-slug>/mcp`, where `<server-slug>` is the slug from Portkey's MCP Registry. For non-interactive CLI and CI runs, send a workspace user API key with `mcp invoke` permission in the `x-portkey-api-key` header. Without an API key, Portkey starts an interactive OAuth flow intended for browser-based clients. See [Portkey's authentication guide](https://portkey.ai/docs/product/mcp-gateway/authentication).
+
+```yaml title="promptfooconfig.yaml"
+prompts:
+  - '{{prompt}}'
+
+providers:
+  - id: mcp
+    config:
+      enabled: true
+      server:
+        name: portkey-gateway
+        url: https://mcp.portkey.ai/<your-server-slug>/mcp
+        headers:
+          x-portkey-api-key: '{{env.PORTKEY_API_KEY}}'
+
+tests:
+  # Each test calls one tool. The prompt is a JSON tool-call payload.
+  - vars:
+      prompt: '{"tool": "your_tool_name", "args": {"param1": "value1"}}'
+    assert:
+      - type: contains
+        value: 'expected result'
+```
+
+Each functional test sends one JSON tool call in the form `{"tool": "tool_name", "args": {...}}`. For a red-team run, use the same `id: mcp` target with the [`mcp` plugin](/docs/red-team/plugins/mcp/); Promptfoo converts generated attacks into valid calls to the server's tools.
@@ -0,0 +1,314 @@
+---
+title: Python Integration
+sidebar_label: Python
+sidebar_position: 1
+description: Use Python for promptfoo evals - providers, assertions, test generators, and prompts. Integrates with LangChain, LangGraph, CrewAI, and more.
+keywords:
+  [
+    promptfoo python,
+    python llm testing,
+    python eval,
+    python provider,
+    langchain testing,
+    langgraph testing,
+    python llm eval,
+    test llm python,
+    crewai testing,
+    pydantic ai testing,
+    openai agents sdk,
+    google adk,
+    strands agents,
+    python agent framework,
+  ]
+---
+
+import PythonFileViewer from '@site/src/components/PythonFileViewer';
+
+# Python
+
+Promptfoo is written in TypeScript and runs via Node.js, but it has first-class Python support. You can use Python for any part of your eval pipeline without writing JavaScript.
+
+**Use Python for:**
+
+- [**Providers**](#providers): call custom models, wrap APIs, run Hugging Face/PyTorch
+- [**Assertions**](#assertions): validate outputs with custom scoring logic
+- [**Test generators**](#test-generators): load test cases from databases, APIs, or generate them programmatically
+- [**Prompts**](#prompts): build prompts dynamically based on test variables
+- [**Framework integrations**](#framework-integrations): test LangChain, LangGraph, CrewAI, and other agent frameworks
+
+The `file://` prefix tells promptfoo to execute a Python function. Promptfoo automatically detects your Python installation.
+
+```yaml title="promptfooconfig.yaml"
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+prompts:
+  - file://prompts.py:create_prompt # Python generates the prompt
+
+providers:
+  - file://provider.py # Python calls the model
+
+tests:
+  - file://tests.py:generate_tests # Python generates test cases
+
+defaultTest:
+  assert:
+    - type: python # Python validates the output
+      value: file://assert.py:check
+```
+
+<!-- prettier-ignore-start -->
+<PythonFileViewer
+  defaultOpen="provider.py"
+  files={[
+    {
+      name: 'prompts.py',
+      description: 'Generate prompts',
+      content: `def create_prompt(context):
+    return [
+        {"role": "system", "content": "You are a helpful assistant."},
+        {"role": "user", "content": f"Explain {context['vars']['topic']}"},
+    ]`,
+    },
+    {
+      name: 'provider.py',
+      description: 'Call any model',
+      content: `from openai import OpenAI
+
+client = OpenAI()
+
+def call_api(prompt, options, context):
+    # The Responses API exposes combined text through output_text.
+    response = client.responses.create(
+        model="gpt-5.1-mini",
+        input=prompt,
+    )
+    return {"output": response.output_text}`,
+    },
+    {
+      name: 'tests.py',
+      description: 'Generate tests',
+      content: `def generate_tests(config=None):
+    return [
+        {"vars": {"topic": "decorators"}},
+        {"vars": {"topic": "async/await"}},
+    ]`,
+    },
+    {
+      name: 'assert.py',
+      description: 'Validate output',
+      content: `def check(output, context):
+    topic = context["vars"]["topic"]
+    if topic.lower() not in output.lower():
+        return {"pass": False, "score": 0, "reason": f"Missing: {topic}"}
+    return {"pass": True, "score": 1.0}`,
+    },
+  ]}
+/>
+<!-- prettier-ignore-end -->
+
+```bash
+npx promptfoo@latest init --example provider-python
+```
+
+---
+
+## Providers
+
+Use `file://` to reference a Python file:
+
+```yaml
+providers:
+  - file://provider.py # Uses call_api() by default
+  - file://provider.py:custom_function # Specify a function name
+```
+
+Your function receives three arguments and returns a dict:
+
+```python title="provider.py"
+def call_api(prompt, options, context):  # or: async def call_api(...)
+    # prompt: string or JSON-encoded messages
+    # options: {"config": {...}} from YAML
+    # context: {"vars": {...}} from test case
+
+    return {
+        "output": "response text",
+        # Optional:
+        "tokenUsage": {"total": 100, "prompt": 20, "completion": 80},
+        "cost": 0.001,
+    }
+```
+
+→ [Provider documentation](/docs/providers/python)
+
+---
+
+## Assertions
+
+Use `type: python` to run custom validation:
+
+```yaml
+assert:
+  # Inline expression (returns bool or float 0-1)
+  - type: python
+    value: "'keyword' in output.lower()"
+
+  # External file
+  - type: python
+    value: file://assert.py
+```
+
+For external files, define a `get_assert` function:
+
+```python title="assert.py"
+def get_assert(output, context):
+    # Return bool, float (0-1), or detailed result
+    return {
+        "pass": True,
+        "score": 0.9,
+        "reason": "Meets criteria",
+    }
+```
+
+→ [Assertions documentation](/docs/configuration/expected-outputs/python)
+
+---
+
+## Test Generators
+
+Load or generate test cases from Python:
+
+```yaml
+tests:
+  - file://tests.py:generate_tests
+```
+
+```python title="tests.py"
+def generate_tests(config=None):
+    # Load from database, API, files, etc.
+    return [
+        {"vars": {"input": "test 1"}, "assert": [{"type": "contains", "value": "expected"}]},
+        {"vars": {"input": "test 2"}},
+    ]
+```
+
+Pass configuration from YAML:
+
+```yaml
+tests:
+  - path: file://tests.py:generate_tests
+    config:
+      max_cases: 100
+      category: 'safety'
+```
+
+→ [Test case documentation](/docs/configuration/test-cases#dynamic-test-generation)
+
+---
+
+## Prompts
+
+Build prompts dynamically:
+
+```yaml
+prompts:
+  - file://prompts.py:create_prompt
+```
+
+```python title="prompts.py"
+def create_prompt(context):
+    # Return string or chat messages
+    return [
+        {"role": "system", "content": "You are an expert."},
+        {"role": "user", "content": f"Explain {context['vars']['topic']}"},
+    ]
+```
+
+→ [Prompts documentation](/docs/configuration/prompts)
+
+---
+
+## Framework Integrations
+
+Test Python agent frameworks by wrapping them as providers:
+
+| Framework          | Example                                                                                                      | Guide                                                                  |
+| ------------------ | ------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------- |
+| **LangGraph**      | [`langgraph`](https://github.com/promptfoo/promptfoo/tree/main/examples/integration-langgraph)               | [Evaluate LangGraph agents](/docs/guides/evaluate-langgraph)           |
+| **LangChain**      | [`langchain-python`](https://github.com/promptfoo/promptfoo/tree/main/examples/integration-langchain)        | [Test LLM chains](/docs/configuration/testing-llm-chains)              |
+| **CrewAI**         | [`crewai`](https://github.com/promptfoo/promptfoo/tree/main/examples/integration-crewai)                     | [Evaluate CrewAI agents](/docs/guides/evaluate-crewai)                 |
+| **OpenAI Agents**  | [`openai-agents`](https://github.com/promptfoo/promptfoo/tree/main/examples/openai-agents)                   | [OpenAI Agents Python SDK](/docs/guides/evaluate-openai-agents-python) |
+| **PydanticAI**     | [`pydantic-ai`](https://github.com/promptfoo/promptfoo/tree/main/examples/integration-pydantic-ai)           | Type-safe agents with Pydantic                                         |
+| **Google ADK**     | [`integration-google-adk`](https://github.com/promptfoo/promptfoo/tree/main/examples/integration-google-adk) | [Evaluate Google ADK agents](/docs/guides/evaluate-google-adk)         |
+| **Strands Agents** | [`strands-agents`](https://github.com/promptfoo/promptfoo/tree/main/examples/integration-strands-agents)     | AWS open-source agent framework                                        |
+
+To get started with any example:
+
+```bash
+npx promptfoo@latest init --example integration-langgraph
+```
+
+---
+
+## Jupyter / Colab
+
+```python
+# Install
+!npm install -g promptfoo
+
+# Create config
+%%writefile promptfooconfig.yaml
+prompts:
+  - "Explain {{topic}}"
+providers:
+  - openai:gpt-4.1-mini
+tests:
+  - vars:
+      topic: machine learning
+
+# Run
+!npx promptfoo eval
+```
+
+**[Open in Google Colab](https://colab.research.google.com/gist/typpo/734a5f53eb1922f90198538dbe17aa27/promptfoo-example-1.ipynb)**
+
+---
+
+## Configuration
+
+### Python Path
+
+Set a custom Python executable:
+
+```bash
+export PROMPTFOO_PYTHON=/path/to/python3
+```
+
+Or configure per-provider in YAML:
+
+```yaml
+providers:
+  - id: file://provider.py
+    config:
+      pythonExecutable: ./venv/bin/python
+```
+
+### Module Paths
+
+Add directories to the Python path:
+
+```bash
+export PYTHONPATH=/path/to/modules:$PYTHONPATH
+```
+
+### Debugging
+
+Enable debug output to see Python execution details:
+
+```bash
+LOG_LEVEL=debug npx promptfoo eval
+```
+
+---
+
+## Troubleshooting
+
+See [Python provider troubleshooting](/docs/providers/python#troubleshooting) for common issues like `Python not found`, module import errors, and timeout problems.
@@ -0,0 +1,109 @@
+---
+title: SharePoint Integration
+sidebar_label: SharePoint
+description: Import LLM test cases from Microsoft SharePoint. Configure certificate-based authentication and load test data from SharePoint CSV files.
+---
+
+# SharePoint Integration
+
+promptfoo allows you to import eval test cases directly from Microsoft SharePoint CSV files using certificate-based authentication with Azure AD.
+
+## Prerequisites
+
+1. **Install Peer Dependencies**
+
+   ```bash
+   npm install @azure/msal-node
+   ```
+
+2. **Set Up Azure AD Application**
+   - Register an application in [Azure Portal](https://portal.azure.com/) under "Azure Active Directory" > "App registrations"
+   - Configure API permissions with SharePoint `Sites.Read.All` permission
+   - Set up certificate-based authentication by generating a PEM certificate (containing both private key and certificate) and uploading it to your application
+   - Ensure your application has the necessary permissions to access SharePoint sites
+
+   Consult your IT/DevOps team or [Microsoft documentation](https://learn.microsoft.com/en-us/sharepoint/dev/sp-add-ins/register-sharepoint-add-ins) for detailed setup steps.
+
+3. **Configure Environment Variables**
+
+   Set the following environment variables:
+
+   ```bash
+   export SHAREPOINT_CLIENT_ID="your-application-client-id"
+   export SHAREPOINT_TENANT_ID="your-azure-tenant-id"
+   export SHAREPOINT_CERT_PATH="/path/to/sharepoint-certificate.pem"
+   export SHAREPOINT_BASE_URL="https://yourcompany.sharepoint.com"
+   ```
+
+   Or create a `.env` file in your project root:
+
+   ```bash title=".env"
+   SHAREPOINT_CLIENT_ID=your-application-client-id
+   SHAREPOINT_TENANT_ID=your-azure-tenant-id
+   SHAREPOINT_CERT_PATH=/path/to/sharepoint-certificate.pem
+   SHAREPOINT_BASE_URL=https://yourcompany.sharepoint.com
+   ```
+
+   :::caution
+   Remember to add `.env` to your `.gitignore` file!
+   :::
+
+## Importing Test Cases from SharePoint
+
+Once authentication is configured, specify the SharePoint CSV file URL in your configuration:
+
+```yaml title="promptfooconfig.yaml"
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: 'SharePoint CSV Import Example'
+prompts:
+  - 'Please translate the following text to {{language}}: {{input}}'
+providers:
+  - openai:gpt-5
+  - anthropic:claude-sonnet-4-5-20250929
+# highlight-start
+tests: https://yourcompany.sharepoint.com/sites/yoursite/Shared%20Documents/test-cases.csv
+# highlight-end
+```
+
+The SharePoint CSV file should be structured with columns that define the test cases:
+
+```csv title="test-cases.csv"
+language,input,__expected
+French,Hello world,icontains: bonjour
+German,I'm hungry,llm-rubric: is german
+Swahili,Hello world,similar(0.8):hello world
+```
+
+> 💡 For details on CSV structure, refer to [loading assertions from CSV](/docs/configuration/expected-outputs/#load-assertions-from-csv).
+
+## Environment Variables
+
+| Variable               | Description                                             | Required |
+| ---------------------- | ------------------------------------------------------- | -------- |
+| `SHAREPOINT_CLIENT_ID` | Azure AD application (client) ID from app registration  | Yes      |
+| `SHAREPOINT_TENANT_ID` | Azure AD tenant (directory) ID                          | Yes      |
+| `SHAREPOINT_CERT_PATH` | Path to PEM file containing private key and certificate | Yes      |
+| `SHAREPOINT_BASE_URL`  | Base URL of your SharePoint site                        | Yes      |
+
+## Using Custom Providers for Model-Graded Metrics
+
+When using SharePoint for test cases, you can still use custom providers for model-graded metrics like `llm-rubric` or `similar`. To do this, override the default LLM grader by adding a `defaultTest` property to your configuration:
+
+```yaml
+prompts:
+  - file://prompt1.txt
+  - file://prompt2.txt
+providers:
+  - openai:gpt-5
+  - anthropic:claude-sonnet-4-5-20250929
+tests: https://yourcompany.sharepoint.com/sites/yoursite/Shared%20Documents/test-cases.csv
+defaultTest:
+  options:
+    provider:
+      text:
+        id: ollama:chat:llama3.3:70b
+      embedding:
+        id: ollama:embeddings:mxbai-embed-large
+```
+
+For more details on customizing the LLM grader, see the [model-graded metrics documentation](/docs/configuration/expected-outputs/model-graded/#overriding-the-llm-grader).
@@ -0,0 +1,306 @@
+---
+title: Integrate Promptfoo with SonarQube
+description: Integrate Promptfoo security findings into SonarQube for centralized vulnerability tracking and CI/CD quality gates
+sidebar_label: SonarQube
+---
+
+This guide demonstrates how to integrate Promptfoo's scanning results into SonarQube, allowing red team findings to appear in your normal "Issues" view, participate in Quality Gates, and block pipelines when they breach security policies.
+
+:::info
+
+This feature is available in [Promptfoo Enterprise](/docs/enterprise/).
+
+:::
+
+## Overview
+
+The integration uses SonarQube's Generic Issue Import feature to import Promptfoo findings without requiring any custom plugins. This approach:
+
+- Surfaces LLM security issues alongside traditional code quality metrics
+- Enables Quality Gate enforcement for prompt injection and other LLM vulnerabilities
+- Provides a familiar developer experience within the existing SonarQube UI
+- Works with any CI/CD system that supports SonarQube
+
+## Prerequisites
+
+- SonarQube server (Community Edition or higher)
+- SonarQube Scanner installed in your CI/CD environment
+- Node.js installed in your CI/CD environment
+- A Promptfoo configuration file
+
+## Configuration Steps
+
+### 1. Basic CI/CD Integration
+
+Here's an example GitHub Actions workflow that runs Promptfoo and imports results into SonarQube:
+
+```yaml
+name: SonarQube Analysis with Promptfoo
+
+on:
+  push:
+    branches: [main, develop]
+  pull_request:
+    branches: [main]
+
+jobs:
+  analysis:
+    runs-on: ubuntu-latest
+
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          fetch-depth: 0 # Shallow clones should be disabled for better analysis
+
+      - name: Setup Node.js
+        uses: actions/setup-node@v4
+        with:
+          node-version: '22'
+
+      - name: Install Promptfoo
+        run: npm install -g promptfoo
+
+      - name: Run Promptfoo scan
+        run: |
+          promptfoo eval \
+            --config promptfooconfig.yaml \
+            --output pf-sonar.json \
+            --output-format sonarqube
+
+      - name: SonarQube Scan
+        env:
+          SONAR_TOKEN: ${{ secrets.SONAR_TOKEN }}
+          SONAR_HOST_URL: ${{ secrets.SONAR_HOST_URL }}
+        run: |
+          sonar-scanner \
+            -Dsonar.projectKey=${{ github.event.repository.name }} \
+            -Dsonar.sources=. \
+            -Dsonar.externalIssuesReportPaths=pf-sonar.json
+```
+
+### 2. Advanced Pipeline Configuration
+
+For enterprise environments, here's a more comprehensive setup with caching, conditional execution, and detailed reporting:
+
+```yaml
+name: Advanced SonarQube Integration
+
+on:
+  push:
+    branches: [main, develop]
+  pull_request:
+    branches: [main]
+  schedule:
+    - cron: '0 2 * * *' # Daily security scan
+
+jobs:
+  promptfoo-security-scan:
+    runs-on: ubuntu-latest
+
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          fetch-depth: 0
+
+      - name: Setup Node.js
+        uses: actions/setup-node@v4
+        with:
+          node-version: '22'
+
+      - name: Cache promptfoo
+        uses: actions/cache@v4
+        with:
+          path: ~/.cache/promptfoo
+          key: ${{ runner.os }}-promptfoo-${{ hashFiles('**/promptfooconfig.yaml') }}
+          restore-keys: |
+            ${{ runner.os }}-promptfoo-
+
+      - name: Install dependencies
+        run: |
+          npm install -g promptfoo
+          npm install -g jsonschema
+
+      - name: Validate promptfoo config
+        run: |
+          # Validate configuration before running
+          promptfoo validate --config promptfooconfig.yaml
+
+      - name: Run red team evaluation
+        id: redteam
+        env:
+          PROMPTFOO_CACHE_PATH: ~/.cache/promptfoo
+        run: |
+          # Run with failure threshold
+          promptfoo eval \
+            --config promptfooconfig.yaml \
+            --output pf-results.json \
+            --output-format json \
+            --max-concurrency 5 \
+            --share || echo "EVAL_FAILED=true" >> $GITHUB_OUTPUT
+
+      - name: Generate multiple report formats
+        if: always()
+        run: |
+          # Generate SonarQube format
+          promptfoo eval \
+            --config promptfooconfig.yaml \
+            --output pf-sonar.json \
+            --output-format sonarqube \
+            --no-cache
+
+          # Also generate HTML report for artifacts
+          promptfoo eval \
+            --config promptfooconfig.yaml \
+            --output pf-results.html \
+            --output-format html \
+            --no-cache
+
+      - name: SonarQube Scan
+        if: always()
+        uses: SonarSource/sonarqube-scan-action@master
+        env:
+          SONAR_TOKEN: ${{ secrets.SONAR_TOKEN }}
+          SONAR_HOST_URL: ${{ secrets.SONAR_HOST_URL }}
+        with:
+          args: >
+            -Dsonar.projectKey=${{ github.event.repository.name }}
+            -Dsonar.externalIssuesReportPaths=pf-sonar.json
+            -Dsonar.pullrequest.key=${{ github.event.pull_request.number }}
+            -Dsonar.pullrequest.branch=${{ github.head_ref }}
+            -Dsonar.pullrequest.base=${{ github.base_ref }}
+
+      - name: Check Quality Gate
+        uses: SonarSource/sonarqube-quality-gate-action@master
+        timeout-minutes: 5
+        env:
+          SONAR_TOKEN: ${{ secrets.SONAR_TOKEN }}
+
+      - name: Upload artifacts
+        if: always()
+        uses: actions/upload-artifact@v4
+        with:
+          name: promptfoo-reports
+          path: |
+            pf-results.json
+            pf-results.html
+            pf-sonar.json
+          retention-days: 30
+
+      - name: Comment PR with results
+        if: github.event_name == 'pull_request' && always()
+        uses: actions/github-script@v7
+        with:
+          script: |
+            const fs = require('fs');
+            const results = JSON.parse(fs.readFileSync('pf-results.json', 'utf8'));
+            const stats = results.results.stats;
+
+            const comment = `## 🔒 Promptfoo Security Scan Results
+
+            - **Total Tests**: ${stats.successes + stats.failures}
+            - **Passed**: ${stats.successes} ✅
+            - **Failed**: ${stats.failures} ❌
+
+            ${results.shareableUrl ? `[View detailed results](${results.shareableUrl})` : ''}
+
+            Issues have been imported to SonarQube for tracking.`;
+
+            github.rest.issues.createComment({
+              issue_number: context.issue.number,
+              owner: context.repo.owner,
+              repo: context.repo.repo,
+              body: comment
+            });
+```
+
+### 3. Configure SonarQube
+
+To properly display and track promptfoo findings in SonarQube:
+
+1. **Create Custom Rules** (optional):
+
+   ```bash
+   # Example API call to create a custom rule
+   curl -u admin:$SONAR_PASSWORD -X POST \
+     "$SONAR_HOST/api/rules/create" \
+     -d "custom_key=PF-Prompt-Injection" \
+     -d "name=Prompt Injection Vulnerability" \
+     -d "markdown_description=Potential prompt injection vulnerability detected" \
+     -d "severity=CRITICAL" \
+     -d "type=VULNERABILITY"
+   ```
+
+2. **Configure Quality Gate**:
+   - Navigate to Quality Gates in SonarQube
+   - Add condition: "Security Rating is worse than A"
+   - Add condition: "Security Hotspots Reviewed is less than 100%"
+   - Add custom condition: "Issues from promptfoo > 0" (for critical findings)
+
+3. **Set Up Notifications**:
+   - Configure webhooks to notify on Quality Gate failures
+   - Set up email notifications for security findings
+
+### 4. Jenkins Integration
+
+If using Jenkins instead of GitHub Actions:
+
+```groovy:Jenkinsfile
+pipeline {
+    agent any
+
+    environment {
+        SONAR_TOKEN = credentials('sonar-token')
+    }
+
+    stages {
+        stage('Checkout') {
+            steps {
+                checkout scm
+            }
+        }
+
+        stage('Run Promptfoo') {
+            steps {
+                sh '''
+                    npm install -g promptfoo
+                    promptfoo eval \
+                        --config promptfooconfig.yaml \
+                        --output pf-sonar.json \
+                        --output-format sonarqube
+                '''
+            }
+        }
+
+        stage('SonarQube Analysis') {
+            steps {
+                withSonarQubeEnv('SonarQube') {
+                    sh '''
+                        sonar-scanner \
+                            -Dsonar.projectKey=${JOB_NAME} \
+                            -Dsonar.sources=. \
+                            -Dsonar.externalIssuesReportPaths=pf-sonar.json
+                    '''
+                }
+            }
+        }
+
+        stage('Quality Gate') {
+            steps {
+                timeout(time: 1, unit: 'HOURS') {
+                    waitForQualityGate abortPipeline: true
+                }
+            }
+        }
+    }
+
+    post {
+        always {
+            archiveArtifacts artifacts: '*.json,*.html', fingerprint: true
+        }
+    }
+}
+```
+
+## Next Steps
+
+For more information on Promptfoo configuration and red team testing, refer to the [red team documentation](/docs/red-team/).
@@ -0,0 +1,156 @@
+---
+sidebar_label: Splunk
+sidebar_position: 78
+title: Integrate Promptfoo with Splunk
+description: Send Promptfoo red team findings to Splunk through HTTP Event Collector for SIEM alerting, dashboards, and incident workflows.
+---
+
+# Integrate Promptfoo with Splunk
+
+Use Splunk's [HTTP Event Collector](https://docs.splunk.com/Documentation/Splunk/latest/Data/UsetheHTTPEventCollector) (HEC) to ingest failed Promptfoo red team results as SIEM events. This works with Splunk Cloud Platform and Splunk Enterprise without a Promptfoo-specific Splunk app.
+
+## Overview
+
+The flow is:
+
+1. Run a Promptfoo red team scan.
+2. Export the latest eval to JSON.
+3. Transform failed red team results into Splunk HEC events.
+4. Search, alert, or dashboard on those events in Splunk.
+
+## Prerequisites
+
+- A working [`promptfooconfig.yaml` red team config](/docs/red-team/quickstart/).
+- A Splunk HEC token with access to an index for Promptfoo findings, such as `security` or `promptfoo`.
+- A HEC endpoint:
+  - Splunk Cloud Platform: `https://http-inputs-<stack>.splunkcloud.com/services/collector/event`
+  - Splunk Enterprise: `https://<splunk-host>:8088/services/collector/event`
+
+Use the `/services/collector/event` endpoint for JSON events. Splunk's HEC event format supports top-level metadata such as `time`, `host`, `source`, `sourcetype`, and `index`.
+
+## Export red team results
+
+Run the scan and export the latest eval:
+
+```bash
+promptfoo redteam run -c promptfooconfig.yaml --no-cache
+promptfoo export eval latest -o redteam-results.json
+```
+
+If you already have a specific eval ID, export that instead of `latest`:
+
+```bash
+promptfoo export eval <eval-id> -o redteam-results.json
+```
+
+In CI, set `PROMPTFOO_PASS_RATE_THRESHOLD=0` on the scan step if you want the pipeline to continue to the Splunk upload even when the scan finds failed tests.
+
+## Send findings to Splunk HEC
+
+Use the public [`promptfoo-to-splunk.mjs` transformer gist](https://gist.github.com/ianw-oai/418470829f7feff8e52a24ad6d028402). It reads `redteam-results.json`, filters failed red team assertions, maps Promptfoo fields like plugin ID, strategy ID, score, attack prompt, target output, and share URL into a Splunk HEC event, then sends each finding with `source: promptfoo` and `sourcetype: promptfoo:redteam`.
+
+After saving the gist as `promptfoo-to-splunk.mjs`, run it with your Splunk HEC settings:
+
+```bash
+export SPLUNK_HEC_URL="https://http-inputs-<stack>.splunkcloud.com/services/collector/event"
+export SPLUNK_HEC_TOKEN="<hec-token>"
+export SPLUNK_INDEX="security"
+
+# Run the transformer on the exported red team results and send findings to Splunk HEC.
+node promptfoo-to-splunk.mjs redteam-results.json
+```
+
+For Splunk Enterprise with a self-signed certificate, fix certificate trust rather than disabling TLS verification in CI.
+
+## GitHub Actions example
+
+You can run the export and Splunk upload manually, but most teams put this in CI/CD so SIEM data stays current. The same pattern works in any CI/CD system; this example uses GitHub Actions.
+
+```yaml title=".github/workflows/redteam-splunk.yml"
+name: Promptfoo Red Team to Splunk
+
+on:
+  workflow_dispatch:
+  schedule:
+    - cron: '0 8 * * *'
+
+jobs:
+  redteam:
+    runs-on: ubuntu-latest
+
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: actions/setup-node@v4
+        with:
+          node-version: '22'
+
+      - name: Install promptfoo
+        run: npm install -g promptfoo
+
+      - name: Run red team scan
+        env:
+          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
+          PROMPTFOO_PASS_RATE_THRESHOLD: '0'
+        run: promptfoo redteam run -c promptfooconfig.yaml --no-cache
+
+      - name: Export latest eval
+        run: promptfoo export eval latest -o redteam-results.json
+
+      - name: Download Splunk transformer
+        run: |
+          curl -L \
+            https://gist.githubusercontent.com/ianw-oai/418470829f7feff8e52a24ad6d028402/raw/promptfoo-to-splunk.mjs \
+            -o promptfoo-to-splunk.mjs
+
+      - name: Send findings to Splunk
+        env:
+          SPLUNK_HEC_URL: ${{ secrets.SPLUNK_HEC_URL }}
+          SPLUNK_HEC_TOKEN: ${{ secrets.SPLUNK_HEC_TOKEN }}
+          SPLUNK_INDEX: security
+        run: node promptfoo-to-splunk.mjs redteam-results.json
+```
+
+Store `SPLUNK_HEC_URL`, `SPLUNK_HEC_TOKEN`, and LLM provider API keys as CI secrets.
+
+## Search in Splunk
+
+Use the configured `sourcetype` to find Promptfoo red team findings:
+
+```text
+index=security sourcetype="promptfoo:redteam"
+| stats count as findings values(strategy_id) as strategies by plugin_id provider severity
+| sort - findings
+```
+
+To alert on new high-priority findings:
+
+```text
+index=security sourcetype="promptfoo:redteam" severity IN ("high", "critical")
+| table _time eval_id provider plugin_id strategy_id score attack_prompt failure_reason shareable_url
+```
+
+Tune severity mapping to your internal policy. The open-source eval export contains per-result pass/fail and score fields; [Promptfoo Enterprise findings](/docs/enterprise/findings) include vulnerability status and severity that are better suited for long-lived SIEM case management.
+
+## Enterprise webhooks
+
+For continuous synchronization from Promptfoo Enterprise, configure [webhooks](/docs/enterprise/webhooks) for `issue.created` and `issue.updated`, then forward those webhook payloads to Splunk HEC. Webhook payloads include issue ID, plugin ID, status, severity, target ID, provider ID, and change information.
+
+Forward webhook payloads to HEC with metadata like this:
+
+```json
+{
+  "source": "promptfoo",
+  "sourcetype": "promptfoo:redteam:issue",
+  "event": {
+    "event_type": "promptfoo_redteam_issue",
+    "issue": "<Promptfoo webhook issue payload>"
+  }
+}
+```
+
+## Security notes
+
+- Treat exported red team results as sensitive. Attack prompts and target outputs can contain harmful content or application data.
+- Restrict the Splunk index and HEC token to the teams that need access.
+- Use `PROMPTFOO_STRIP_RESPONSE_OUTPUT=true` or `PROMPTFOO_STRIP_TEST_VARS=true` if you need to reduce what leaves the CI environment.
@@ -0,0 +1,159 @@
+---
+sidebar_label: Travis CI
+description: Set up automated LLM testing in Travis CI pipelines with promptfoo. Configure environment variables, artifacts storage, and continuous evaluation of AI prompts and outputs.
+---
+
+# Travis CI Integration
+
+This guide demonstrates how to set up promptfoo with Travis CI to run evaluations as part of your CI pipeline.
+
+## Prerequisites
+
+- A GitHub repository with a promptfoo project
+- A Travis CI account connected to your repository
+- API keys for your LLM providers stored as [Travis CI environment variables](https://docs.travis-ci.com/user/environment-variables/)
+
+## Setting up Travis CI
+
+Create a new file named `.travis.yml` in the root of your repository with the following configuration:
+
+```yaml
+language: node_js
+node_js:
+  - 18
+
+cache:
+  directories:
+    - node_modules
+
+before_install:
+  - npm install -g promptfoo
+
+install:
+  - npm ci
+
+script:
+  - npx promptfoo eval
+
+after_success:
+  - echo "Prompt evaluation completed successfully"
+
+after_failure:
+  - echo "Prompt evaluation failed"
+
+# Save evaluation results as artifacts
+before_deploy:
+  - mkdir -p artifacts
+  - cp promptfoo-results.json artifacts/
+
+deploy:
+  provider: s3
+  bucket: 'your-bucket-name' # Replace with your bucket name
+  skip_cleanup: true
+  local_dir: artifacts
+  on:
+    branch: main
+```
+
+## Environment Variables
+
+Store your LLM provider API keys as environment variables in Travis CI:
+
+1. Navigate to your repository in Travis CI
+2. Go to More options > Settings > Environment Variables
+3. Add variables for each provider API key (e.g., `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`)
+4. Make sure to mark them as secure to prevent them from being displayed in logs
+
+## Advanced Configuration
+
+### Fail the Build on Failed Assertions
+
+You can configure the pipeline to fail when promptfoo assertions don't pass:
+
+```yaml
+script:
+  - npx promptfoo eval --fail-on-error
+```
+
+### Testing on Multiple Node.js Versions
+
+Test your evaluations across different Node.js versions:
+
+```yaml
+language: node_js
+node_js:
+  - 18
+  - 20
+
+script:
+  - npx promptfoo eval
+```
+
+### Running on Different Platforms
+
+Run evaluations on multiple operating systems:
+
+```yaml
+language: node_js
+node_js:
+  - 18
+
+os:
+  - linux
+  - osx
+
+script:
+  - npx promptfoo eval
+```
+
+### Conditional Builds
+
+Run evaluations only on specific branches or conditions:
+
+```yaml
+language: node_js
+node_js:
+  - 18
+
+# Run evaluations only on main branch and pull requests
+if: branch = main OR type = pull_request
+
+script:
+  - npx promptfoo eval
+```
+
+### Custom Build Stages
+
+Set up different stages for your build process:
+
+```yaml
+language: node_js
+node_js:
+  - 18
+
+stages:
+  - test
+  - evaluate
+
+jobs:
+  include:
+    - stage: test
+      script: npm test
+    - stage: evaluate
+      script: npx promptfoo eval
+      env:
+        - MODEL=gpt-4
+    - stage: evaluate
+      script: npx promptfoo eval
+      env:
+        - MODEL=claude-3-opus-20240229
+```
+
+## Troubleshooting
+
+If you encounter issues with your Travis CI integration:
+
+- **Check logs**: Review detailed logs in Travis CI to identify errors
+- **Verify environment variables**: Ensure your API keys are correctly set
+- **Build timeouts**: Travis CI has a default timeout of 50 minutes for jobs. For long-running evaluations, you may need to configure [job timeouts](https://docs.travis-ci.com/user/customizing-the-build/#build-timeouts)
+- **Resource constraints**: Consider breaking down large evaluations into smaller chunks if you're hitting resource limits