chore: import upstream snapshot with attribution
CI / Shell Format Check (push) Has been cancelled
CI / Check Ruby (3.4) (push) Has been cancelled
CI / CI Config (push) Has been cancelled
CI / Test on Node ${{ matrix.node }} and ${{ matrix.os }}${{ matrix.shard && format(' (shard {0}/3)', matrix.shard) || '' }} (push) Has been cancelled
CI / Build on Node ${{ matrix.node }} (push) Has been cancelled
CI / Style Check (push) Has been cancelled
CI / Generate Assets (push) Has been cancelled
CI / Check Python (3.14) (push) Has been cancelled
CI / Check Python (3.9) (push) Has been cancelled
CI / Build Docs (push) Has been cancelled
CI / Code Scan Action (push) Has been cancelled
CI / Site tests (push) Has been cancelled
CI / webui tests (push) Has been cancelled
CI / Run Integration Tests (push) Has been cancelled
CI / Run Smoke Tests (push) Has been cancelled
CI / Go Tests (push) Has been cancelled
CI / Share Test (push) Has been cancelled
CI / Redteam (Production API) (push) Has been cancelled
CI / Redteam (Staging API) (push) Has been cancelled
CI / GitHub Actions Lint (push) Has been cancelled
CI / Check Ruby (3.0) (push) Has been cancelled
release-please / release-please (push) Has been cancelled
release-please / build (push) Has been cancelled
release-please / publish-npm (push) Has been cancelled
release-please / publish-npm-backfill (push) Has been cancelled
release-please / docker (push) Has been cancelled
release-please / publish-code-scan-action (push) Has been cancelled
release-please / attest-code-scan-action (push) Has been cancelled
Deploy local.promptfoo.app / Deploy to Cloudflare Pages (push) Has been cancelled
Test and Publish Multi-arch Docker Image / test (push) Has been cancelled
Test and Publish Multi-arch Docker Image / build-docker-and-push-digests (map[digest-suffix:linux-amd64 platform:linux/amd64 runner:ubuntu-latest]) (push) Has been cancelled
Test and Publish Multi-arch Docker Image / build-docker-and-push-digests (map[digest-suffix:linux-arm64 platform:linux/arm64 runner:ubuntu-24.04-arm]) (push) Has been cancelled
Test and Publish Multi-arch Docker Image / merge-docker-digests (push) Has been cancelled
Test and Publish Multi-arch Docker Image / Attest Multi-arch Image (push) Has been cancelled
Validate Renovate Config / Validate Renovate Configuration (push) Has been cancelled
CI / Shell Format Check (push) Has been cancelled
CI / Check Ruby (3.4) (push) Has been cancelled
CI / CI Config (push) Has been cancelled
CI / Test on Node ${{ matrix.node }} and ${{ matrix.os }}${{ matrix.shard && format(' (shard {0}/3)', matrix.shard) || '' }} (push) Has been cancelled
CI / Build on Node ${{ matrix.node }} (push) Has been cancelled
CI / Style Check (push) Has been cancelled
CI / Generate Assets (push) Has been cancelled
CI / Check Python (3.14) (push) Has been cancelled
CI / Check Python (3.9) (push) Has been cancelled
CI / Build Docs (push) Has been cancelled
CI / Code Scan Action (push) Has been cancelled
CI / Site tests (push) Has been cancelled
CI / webui tests (push) Has been cancelled
CI / Run Integration Tests (push) Has been cancelled
CI / Run Smoke Tests (push) Has been cancelled
CI / Go Tests (push) Has been cancelled
CI / Share Test (push) Has been cancelled
CI / Redteam (Production API) (push) Has been cancelled
CI / Redteam (Staging API) (push) Has been cancelled
CI / GitHub Actions Lint (push) Has been cancelled
CI / Check Ruby (3.0) (push) Has been cancelled
release-please / release-please (push) Has been cancelled
release-please / build (push) Has been cancelled
release-please / publish-npm (push) Has been cancelled
release-please / publish-npm-backfill (push) Has been cancelled
release-please / docker (push) Has been cancelled
release-please / publish-code-scan-action (push) Has been cancelled
release-please / attest-code-scan-action (push) Has been cancelled
Deploy local.promptfoo.app / Deploy to Cloudflare Pages (push) Has been cancelled
Test and Publish Multi-arch Docker Image / test (push) Has been cancelled
Test and Publish Multi-arch Docker Image / build-docker-and-push-digests (map[digest-suffix:linux-amd64 platform:linux/amd64 runner:ubuntu-latest]) (push) Has been cancelled
Test and Publish Multi-arch Docker Image / build-docker-and-push-digests (map[digest-suffix:linux-arm64 platform:linux/arm64 runner:ubuntu-24.04-arm]) (push) Has been cancelled
Test and Publish Multi-arch Docker Image / merge-docker-digests (push) Has been cancelled
Test and Publish Multi-arch Docker Image / Attest Multi-arch Image (push) Has been cancelled
Validate Renovate Config / Validate Renovate Configuration (push) Has been cancelled
This commit is contained in:
@@ -0,0 +1,179 @@
|
||||
---
|
||||
sidebar_label: Bitbucket Pipelines
|
||||
description: Integrate promptfoo LLM testing with Bitbucket Pipelines CI/CD to automate evaluations, track results, and catch regressions in your AI models using built-in assertions
|
||||
---
|
||||
|
||||
# Bitbucket Pipelines Integration
|
||||
|
||||
This guide demonstrates how to set up promptfoo with Bitbucket Pipelines to run evaluations as part of your CI pipeline.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- A Bitbucket repository with a promptfoo project
|
||||
- Bitbucket Pipelines enabled for your repository
|
||||
- API keys for your LLM providers stored as [Bitbucket repository variables](https://support.atlassian.com/bitbucket-cloud/docs/variables-and-secrets/)
|
||||
|
||||
## Setting up Bitbucket Pipelines
|
||||
|
||||
Create a new file named `bitbucket-pipelines.yml` in the root of your repository with the following configuration:
|
||||
|
||||
```yaml
|
||||
image: node:24
|
||||
|
||||
pipelines:
|
||||
default:
|
||||
- step:
|
||||
name: Promptfoo Evaluation
|
||||
caches:
|
||||
- node
|
||||
script:
|
||||
- npm ci
|
||||
- npm install -g promptfoo
|
||||
- npx promptfoo eval -o promptfoo-results.json -o promptfoo-results.junit.xml
|
||||
artifacts:
|
||||
- promptfoo-results.json
|
||||
- promptfoo-results.junit.xml
|
||||
```
|
||||
|
||||
## Environment Variables
|
||||
|
||||
Store your LLM provider API keys as repository variables in Bitbucket:
|
||||
|
||||
1. Navigate to your repository in Bitbucket
|
||||
2. Go to Repository settings > Pipelines > Repository variables
|
||||
3. Add variables for each provider API key (e.g., `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`)
|
||||
4. Mark them as "Secured" to ensure they're not displayed in logs
|
||||
|
||||
## Advanced Configuration
|
||||
|
||||
### Fail the Pipeline on Failed Assertions
|
||||
|
||||
You can configure the pipeline to fail when promptfoo assertions don't pass:
|
||||
|
||||
```yaml
|
||||
script:
|
||||
- npm ci
|
||||
- npm install -g promptfoo
|
||||
- npx promptfoo eval --fail-on-error
|
||||
```
|
||||
|
||||
### Custom Evaluation Configurations
|
||||
|
||||
Run evaluations with specific configuration files:
|
||||
|
||||
```yaml
|
||||
script:
|
||||
- npm ci
|
||||
- npm install -g promptfoo
|
||||
- npx promptfoo eval --config custom-config.yaml
|
||||
```
|
||||
|
||||
### Run on Pull Requests
|
||||
|
||||
Configure different behavior for pull requests:
|
||||
|
||||
```yaml
|
||||
pipelines:
|
||||
default:
|
||||
- step:
|
||||
name: Promptfoo Evaluation
|
||||
script:
|
||||
- npm ci
|
||||
- npm install -g promptfoo
|
||||
- npx promptfoo eval
|
||||
pull-requests:
|
||||
'**':
|
||||
- step:
|
||||
name: Promptfoo PR Evaluation
|
||||
script:
|
||||
- npm ci
|
||||
- npm install -g promptfoo
|
||||
- npx promptfoo eval --fail-on-error
|
||||
```
|
||||
|
||||
### Scheduled Evaluations
|
||||
|
||||
Run evaluations on a schedule:
|
||||
|
||||
```yaml
|
||||
pipelines:
|
||||
default:
|
||||
- step:
|
||||
name: Promptfoo Evaluation
|
||||
script:
|
||||
- npm ci
|
||||
- npm install -g promptfoo
|
||||
- npx promptfoo eval
|
||||
custom:
|
||||
nightly-evaluation:
|
||||
- step:
|
||||
name: Nightly Evaluation
|
||||
script:
|
||||
- npm ci
|
||||
- npm install -g promptfoo
|
||||
- npx promptfoo eval
|
||||
schedules:
|
||||
- cron: '0 0 * * *' # Run at midnight UTC every day
|
||||
pipeline: custom.nightly-evaluation
|
||||
branches:
|
||||
include:
|
||||
- main
|
||||
```
|
||||
|
||||
### Parallel Testing
|
||||
|
||||
Test across multiple configurations in parallel:
|
||||
|
||||
```yaml
|
||||
image: node:24
|
||||
|
||||
pipelines:
|
||||
default:
|
||||
- parallel:
|
||||
- step:
|
||||
name: Evaluate with GPT-4
|
||||
script:
|
||||
- npm ci
|
||||
- npm install -g promptfoo
|
||||
- npx promptfoo eval --providers.0.config.model=gpt-4
|
||||
artifacts:
|
||||
- promptfoo-results-gpt4.json
|
||||
- step:
|
||||
name: Evaluate with Claude
|
||||
script:
|
||||
- npm ci
|
||||
- npm install -g promptfoo
|
||||
- npx promptfoo eval --providers.0.config.model=claude-3-opus-20240229
|
||||
artifacts:
|
||||
- promptfoo-results-claude.json
|
||||
```
|
||||
|
||||
### Using Pipes
|
||||
|
||||
Leverage Bitbucket Pipes for a more concise configuration:
|
||||
|
||||
```yaml
|
||||
image: node:24
|
||||
|
||||
pipelines:
|
||||
default:
|
||||
- step:
|
||||
name: Promptfoo Evaluation
|
||||
script:
|
||||
- npm ci
|
||||
- npm install -g promptfoo
|
||||
- npx promptfoo eval -o promptfoo-results.junit.xml
|
||||
after-script:
|
||||
- pipe: atlassian/junit-report:0.3.0
|
||||
variables:
|
||||
REPORT_PATHS: 'promptfoo-results.junit.xml'
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
If you encounter issues with your Bitbucket Pipelines integration:
|
||||
|
||||
- **Check logs**: Review detailed logs in Bitbucket to identify errors
|
||||
- **Verify repository variables**: Ensure your API keys are correctly set
|
||||
- **Pipeline timeouts**: Bitbucket Pipelines has timeout limits. For long-running evaluations, consider breaking them down or [increasing the timeout](https://support.atlassian.com/bitbucket-cloud/docs/build-timeouts/)
|
||||
- **Debug with SSH**: For complex issues, use [enabling SSH access](https://support.atlassian.com/bitbucket-cloud/docs/debug-your-pipelines-with-ssh/) to debug the pipeline environment directly
|
||||
Reference in New Issue
Block a user