chore: import upstream snapshot with attribution

2026-07-13 13:24:08 +08:00
commit 0d3cb498a3
5438 changed files with 1316560 additions and 0 deletions
@@ -0,0 +1,188 @@
+---
+sidebar_label: Azure Pipelines
+description: Integrate promptfoo LLM testing with Azure Pipelines CI/CD using step-by-step setup, environment variables, and matrix testing configurations for automated AI evaluation
+---
+
+# Azure Pipelines Integration
+
+This guide demonstrates how to set up promptfoo with Azure Pipelines to run evaluations as part of your CI pipeline.
+
+## Prerequisites
+
+- A GitHub or Azure DevOps repository with a promptfoo project
+- An Azure DevOps account with permission to create pipelines
+- API keys for your LLM providers stored as [Azure Pipeline variables](https://learn.microsoft.com/en-us/azure/devops/pipelines/process/variables)
+
+## Setting up the Azure Pipeline
+
+Create a new file named `azure-pipelines.yml` in the root of your repository with the following configuration:
+
+```yaml
+trigger:
+  - main
+  - master # Include if you use master as your main branch
+
+pool:
+  vmImage: 'ubuntu-latest'
+
+variables:
+  npm_config_cache: $(Pipeline.Workspace)/.npm
+
+steps:
+  - task: NodeTool@0
+    inputs:
+      versionSpec: '24.x'
+    displayName: 'Install Node.js'
+
+  - task: Cache@2
+    inputs:
+      key: 'npm | "$(Agent.OS)" | package-lock.json'
+      restoreKeys: |
+        npm | "$(Agent.OS)"
+      path: $(npm_config_cache)
+    displayName: 'Cache npm packages'
+
+  - script: |
+      npm ci
+      npm install -g promptfoo
+    displayName: 'Install dependencies'
+
+  - script: |
+      npx promptfoo eval -o promptfoo-results.json -o promptfoo-results.junit.xml
+    displayName: 'Run promptfoo evaluations'
+    env:
+      OPENAI_API_KEY: $(OPENAI_API_KEY)
+      ANTHROPIC_API_KEY: $(ANTHROPIC_API_KEY)
+      # Add other API keys as needed
+
+  - task: PublishTestResults@2
+    inputs:
+      testResultsFormat: 'JUnit'
+      testResultsFiles: 'promptfoo-results.junit.xml'
+      mergeTestResults: true
+      testRunTitle: 'Promptfoo Evaluation Results'
+    condition: succeededOrFailed()
+    displayName: 'Publish test results'
+
+  - task: PublishBuildArtifacts@1
+    inputs:
+      pathtoPublish: 'promptfoo-results.json'
+      artifactName: 'promptfoo-results'
+    condition: succeededOrFailed()
+    displayName: 'Publish evaluation results'
+```
+
+## Environment Variables
+
+Store your LLM provider API keys as [secret pipeline variables](https://learn.microsoft.com/en-us/azure/devops/pipelines/process/variables#secret-variables) in Azure DevOps:
+
+1. Navigate to your project in Azure DevOps
+2. Go to Pipelines > Your Pipeline > Edit > Variables
+3. Add variables for each provider API key (e.g., `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`)
+4. Mark them as secret to ensure they're not displayed in logs
+
+## Advanced Configuration
+
+### Fail the Pipeline on Failed Assertions
+
+You can configure the pipeline to fail when promptfoo assertions don't pass by modifying the script step:
+
+```yaml
+- script: |
+    npx promptfoo eval --fail-on-error -o promptfoo-results.junit.xml
+  displayName: 'Run promptfoo evaluations'
+  env:
+    OPENAI_API_KEY: $(OPENAI_API_KEY)
+```
+
+### Configure Custom Output Location
+
+If you want to customize where results are stored:
+
+```yaml
+- script: |
+    npx promptfoo eval --output-path $(Build.ArtifactStagingDirectory)/promptfoo-results.json
+  displayName: 'Run promptfoo evaluations'
+```
+
+### Run on Pull Requests
+
+To run evaluations on pull requests, add a PR trigger:
+
+```yaml
+trigger:
+  - main
+  - master
+
+pr:
+  - main
+  - master
+# Rest of pipeline configuration
+```
+
+### Conditional Execution
+
+Run promptfoo only when certain files change:
+
+```yaml
+steps:
+  - task: NodeTool@0
+    inputs:
+      versionSpec: '24.x'
+    displayName: 'Install Node.js'
+
+  - script: |
+      npm ci
+      npm install -g promptfoo
+    displayName: 'Install dependencies'
+
+  - script: |
+      npx promptfoo eval
+    displayName: 'Run promptfoo evaluations'
+    condition: |
+      and(
+        succeeded(),
+        or(
+          eq(variables['Build.SourceBranch'], 'refs/heads/main'),
+          eq(variables['Build.Reason'], 'PullRequest')
+        ),
+        or(
+          eq(variables['Build.Reason'], 'PullRequest'),
+          contains(variables['Build.SourceVersionMessage'], '[run-eval]')
+        )
+      )
+    env:
+      OPENAI_API_KEY: $(OPENAI_API_KEY)
+```
+
+## Using with Matrix Testing
+
+Test across multiple configurations or models in parallel:
+
+```yaml
+strategy:
+  matrix:
+    gpt:
+      MODEL: 'gpt-5.1'
+    claude:
+      MODEL: 'claude-sonnet-4-5-20250929'
+
+steps:
+  - script: |
+      npx promptfoo eval --providers.0.config.model=$(MODEL)
+    displayName: 'Test with $(MODEL)'
+    env:
+      OPENAI_API_KEY: $(OPENAI_API_KEY)
+      ANTHROPIC_API_KEY: $(ANTHROPIC_API_KEY)
+```
+
+## Troubleshooting
+
+If you encounter issues with your Azure Pipelines integration:
+
+- **Check logs**: Review detailed logs in Azure DevOps to identify errors
+- **Verify API keys**: Ensure your API keys are correctly set as pipeline variables
+- **Permissions**: Make sure the pipeline has access to read your configuration files
+- **Node.js version**: Promptfoo requires Node.js `^20.20.0` or `>=22.22.0`
+
+If you're getting timeouts during evaluations, you may need to adjust the pipeline timeout settings or consider using a [self-hosted agent](https://learn.microsoft.com/en-us/azure/devops/pipelines/agents/agents) for better stability with long-running evaluations.