chore: import upstream snapshot with attribution

2026-07-13 13:24:08 +08:00
commit 0d3cb498a3
5438 changed files with 1316560 additions and 0 deletions
@@ -0,0 +1,37 @@
+# integration-github-action (GitHub Action)
+
+You can run this example with:
+
+```bash
+npx promptfoo@latest init --example integration-github-action
+cd integration-github-action
+```
+
+This folder contains a standalone GitHub Action that runs **promptfoo** when a pull request modifies prompts.
+
+## Prerequisites
+
+- GitHub repository with Actions enabled
+- API keys for LLM providers set as repository secrets:
+  - `OPENAI_API_KEY` - Get from [OpenAI API keys page](https://platform.openai.com/api-keys)
+  - `ANTHROPIC_API_KEY` - Get from [Anthropic Console](https://console.anthropic.com/) (optional)
+
+### Setting up GitHub Secrets:
+
+1. Go to your repository → Settings → Secrets and variables → Actions
+2. Click "New repository secret"
+3. Add `OPENAI_API_KEY` with your OpenAI API key as the value
+4. Repeat for other provider keys as needed
+
+## Usage
+
+Edit [standalone-action.yaml](./standalone-action.yaml) as needed and add it to your repository's `.github/workflows/` directory. The action will automatically run when pull requests modify prompt files.
+
+## Expected Results
+
+When triggered, the action will:
+
+- Install promptfoo in the GitHub runner environment
+- Run evaluations on the modified prompts
+- Post results as PR comments or check results
+- Fail the check if evaluations don't meet configured thresholds
@@ -0,0 +1,70 @@
+#
+# This is a standalone github action that runs promptfoo against a change.
+# You'll have to edit it to use the appropriate branches and prompts.
+#
+# For an easier to use packaged action, see:
+#
+# https://promptfoo.dev/docs/integrations/github-action
+# https://github.com/promptfoo/promptfoo-action
+#
+name: LLM Prompt Evaluation
+
+on:
+  pull_request:
+    paths:
+      - 'prompts/**'
+
+jobs:
+  evaluate:
+    runs-on: ubuntu-latest
+
+    steps:
+      - name: Checkout base ref (original)
+        uses: actions/checkout@v4
+        with:
+          ref: ${{ github.base_ref }}
+          path: base
+
+      - name: Checkout head ref (modified)
+        uses: actions/checkout@v4
+        with:
+          ref: ${{ github.head_ref }}
+          path: head
+
+      - name: Cache promptfoo data
+        id: cache
+        uses: actions/cache@v4
+        with:
+          path: ~/.cache/promptfoo
+          key: ${{ runner.os }}-promptfoo-${{ hashFiles('**/package-lock.json') }}
+          restore-keys: |
+            ${{ runner.os }}-promptfoo-
+
+      - name: Set up Node.js
+        uses: actions/setup-node@v4
+        with:
+          node-version: 22
+
+      - name: Run promptfoo evaluation
+        id: eval
+        env:
+          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
+          PROMPTFOO_CACHE_PATH: ~/.cache/promptfoo
+        run: |
+          npx promptfoo eval -c head/prompts/promptfooconfig.yaml --prompts base/prompts/prompt1.json head/prompts/prompt1.json -o output.json --share
+          echo "OUTPUT_JSON_PATH=$GITHUB_WORKSPACE/output.json" >> $GITHUB_ENV
+
+      - name: Comment PR
+        uses: actions/github-script@v6
+        with:
+          github-token: ${{secrets.GITHUB_TOKEN}}
+          script: |
+            const fs = require('fs');
+            const output = JSON.parse(fs.readFileSync(process.env.OUTPUT_JSON_PATH, 'utf8'));
+            const body = `⚠️ LLM prompt was modified.\n\n| Success | Failure |\n|---------|---------|\n| ${output.results.stats.successes}      | ${output.results.stats.failures}       |\n\n**» [View eval results](${output.shareableUrl}) «**`;
+            github.rest.issues.createComment({
+              issue_number: context.issue.number,
+              owner: context.repo.owner,
+              repo: context.repo.repo,
+              body: body
+            });