chore: import upstream snapshot with attribution

2026-07-13 13:24:08 +08:00
commit 0d3cb498a3
5438 changed files with 1316560 additions and 0 deletions
@@ -0,0 +1,156 @@
+# openai-codex-sdk (OpenAI Codex SDK Examples)
+
+The OpenAI Codex SDK provider enables agentic code analysis and generation evals with thread-based conversations and Git-aware operations.
+
+You can run this example with:
+
+```bash
+npx promptfoo@latest init --example openai-codex-sdk
+cd openai-codex-sdk
+```
+
+## Setup
+
+Install the OpenAI Codex SDK:
+
+```bash
+npm install @openai/codex-sdk
+```
+
+**Requirements**: Node.js `^20.20.0` or `>=22.22.0`
+
+Authenticate with Codex using one of these options:
+
+1. Sign in with ChatGPT through the Codex CLI:
+
+```bash
+codex
+```
+
+2. Or set your OpenAI API key:
+
+```bash
+export OPENAI_API_KEY=your_api_key_here
+# or
+export CODEX_API_KEY=your_api_key_here
+```
+
+When no `apiKey`, `OPENAI_API_KEY`, or `CODEX_API_KEY` is set, promptfoo will let the Codex SDK reuse an existing Codex login.
+
+## Examples
+
+### Basic Usage
+
+Simple code generation with `sandbox_mode: read-only` so Codex can answer from the prompt without writing files. The example also sets `skip_git_repo_check: true` so it works in a standalone example directory that is not a Git repo.
+
+This basic example uses only deterministic string assertions, so it can run with either a Codex login or an API key without needing a separate grader model credential.
+
+**Location**: `./basic/`
+
+**Usage**:
+
+```bash
+(cd basic && promptfoo eval)
+```
+
+### Skills Testing
+
+This example demonstrates evaluating a local Codex skill stored under `.agents/skills/`.
+
+- **Local skill discovery**: Codex discovers `SKILL.md` from the sample project's `.agents/skills/` directory
+- **Skill assertions**: Verifies confirmed skill usage with the `skill-used` assertion over normalized `metadata.skillCalls`
+- **Trace assertions**: `promptfooconfig.tracing.yaml` enables OTEL deep tracing and asserts on the traced command that reads `SKILL.md`
+- **Isolated Codex home**: Uses a project-local `CODEX_HOME` so personal skills and config do not leak into the eval
+- **Controlled shell environment**: Promptfoo now passes a minimal shell environment by default, so the tracing config can override `CODEX_HOME` without inheriting unrelated process secrets while still preserving a usable `PATH`
+
+`metadata.skillCalls` only includes confirmed successful skill reads. When Promptfoo sees more candidate `SKILL.md` paths than confirmed successful reads, it also emits `metadata.attemptedSkillCalls` for debugging.
+
+`metadata.skillCalls` and `metadata.attemptedSkillCalls` are heuristic: Promptfoo infers them from direct command references to `SKILL.md`. Wildcard paths are ignored, and absolute `.agents/...` paths outside the active repo are ignored.
+
+**Location**: `./skills/`
+
+**Usage**:
+
+```bash
+(cd skills && promptfoo eval)
+
+# Trace the skill's internal command activity
+(cd skills && promptfoo eval -c promptfooconfig.tracing.yaml)
+```
+
+Relative `working_dir` values resolve from the config file's directory, so the sample project path stays stable regardless of where you invoke `promptfoo eval`. Codex resolves `CODEX_HOME` itself, so set `CODEX_HOME_OVERRIDE` to an absolute path when you run these configs from another working directory or need Codex to use a different home directory.
+The checked-in relative paths stay local to the config directory so the examples remain self-contained.
+
+The checked-in `sample-codex-home` fixture is intentionally empty of auth state. Use it with `OPENAI_API_KEY`/`CODEX_API_KEY`, or point `CODEX_HOME_OVERRIDE` at `$HOME/.codex` when you want to reuse a local Codex login.
+
+### Skill Comparison
+
+This example compares two versions of the same local Codex skill against identical review tasks.
+
+- **Versioned fixtures**: Each provider points at a different `working_dir` with its own `review-standards` skill
+- **Outcome scoring**: A JavaScript assertion scores issue recall and precision for each response
+- **Winner selection**: `max-score` picks the strongest skill version for each task after combining routing, correctness, cost, and latency signals
+
+**Location**: `./skill-comparison/`
+
+**Usage**:
+
+```bash
+(cd skill-comparison && promptfoo eval --no-cache)
+```
+
+If you run this config from the repo root, set `CODEX_SKILL_COMPARE_V1_DIR` and `CODEX_SKILL_COMPARE_V2_DIR` to the absolute fixture paths first.
+
+If your network requires proxy or custom certificate environment variables such as `HTTP_PROXY`, `HTTPS_PROXY`, `ALL_PROXY`, `NO_PROXY`, `SSL_CERT_FILE`, or `NODE_EXTRA_CA_CERTS`, pass them through `config.cli_env` or set `inherit_process_env: true` in the provider config. Promptfoo intentionally does not forward the full process environment to Codex by default.
+
+### Thread Persistence
+
+This example demonstrates `persist_threads: true` with one prompt template and multiple tests. It checks that Codex can remember a marker from the first test when answering the second test.
+
+**Location**: `./thread-persistence/`
+
+**Usage**:
+
+```bash
+(cd thread-persistence && promptfoo eval)
+```
+
+### Sandbox Enforcement
+
+This example runs Codex in `read-only` mode and asks it to create a file. The assertion checks that the model reports a write denial, and you can also inspect the sample workspace after the eval to confirm no file was created.
+
+**Location**: `./sandbox/`
+
+**Usage**:
+
+```bash
+(cd sandbox && promptfoo eval)
+```
+
+If you run this config from the repo root, set `CODEX_SANDBOX_WORKING_DIR="$PWD/examples/openai-codex-sdk/sandbox/sample-workspace"`.
+
+### Run on Amazon Bedrock
+
+This example runs Codex against OpenAI frontier models hosted on Amazon Bedrock by setting `model_provider: amazon-bedrock` with a Bedrock model id (`openai.gpt-5.5`). It requires AWS credentials and Bedrock model access in a supported Region (`us-east-2` for GPT-5.5).
+
+**Location**: `./bedrock/`
+
+**Usage**:
+
+```bash
+(cd bedrock && promptfoo eval)
+```
+
+Export `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` first; they are forwarded to the Codex CLI via `config.cli_env`.
+
+## Key Features
+
+- **Thread Persistence**: Conversations saved to `~/.codex/sessions`
+- **Git Integration**: Automatic repository detection (can be disabled)
+- **Structured Output**: Native JSON schema support with Zod
+- **Streaming Events**: Real-time progress updates
+- **Custom Binary**: Override Codex binary path with `codex_path_override`
+
+## Configuration Options
+
+See [documentation](https://www.promptfoo.dev/docs/providers/openai-codex-sdk/) for full details.
@@ -0,0 +1,21 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: 'Simple code generation with Codex SDK'
+
+prompts:
+  - 'Write a Python function that calculates the factorial of a number. Output only the code.'
+
+providers:
+  - id: openai:codex-sdk
+    config:
+      model: gpt-5.2-codex
+      model_reasoning_effort: high
+      cli_config:
+        model_verbosity: medium
+      sandbox_mode: read-only
+      skip_git_repo_check: true
+
+tests:
+  - vars: {}
+    assert:
+      - type: contains
+        value: 'def factorial'
@@ -0,0 +1,59 @@
+# openai-codex-sdk/bedrock (Codex SDK on Amazon Bedrock)
+
+Runs OpenAI's Codex coding agent against OpenAI frontier models hosted on **Amazon Bedrock**
+(`openai.gpt-5.5` / `openai.gpt-5.4`) instead of the OpenAI Platform.
+
+You can run this example with:
+
+```bash
+npx promptfoo@latest init --example openai-codex-sdk/bedrock
+```
+
+## Prerequisites
+
+1. Install the Codex SDK:
+
+   ```bash
+   npm install @openai/codex-sdk
+   ```
+
+2. Request access to the OpenAI frontier models in a supported AWS Region:
+   - GPT-5.5: `us-east-2`
+   - GPT-5.4: `us-east-2`, `us-west-2`
+
+3. Export AWS credentials (the Codex CLI reads them from its environment):
+
+   ```bash
+   export AWS_ACCESS_KEY_ID="your_access_key"
+   export AWS_SECRET_ACCESS_KEY="your_secret_key"
+   ```
+
+## How it works
+
+- `model_provider: amazon-bedrock` routes Codex through Bedrock's OpenAI-compatible
+  endpoint (`https://bedrock-mantle.<region>.api.aws/openai/v1`).
+- `model: openai.gpt-5.5` uses the Bedrock model id (note the `openai.` prefix).
+- AWS credentials and `AWS_REGION` are forwarded to the Codex CLI via `cli_env` because
+  promptfoo runs the CLI with a minimal environment by default. You may instead set
+  `AWS_BEARER_TOKEN_BEDROCK` (a Bedrock API key), `AWS_PROFILE`, or
+  `inherit_process_env: true`. If you use temporary credentials (SSO / STS / assumed
+  roles / MFA), also forward `AWS_SESSION_TOKEN` (uncomment it in the config).
+
+> **Security note:** values placed in `cli_env` are exposed to the Codex agent's shell
+> environment. Scope the IAM permissions to Bedrock inference and prefer short-lived
+> credentials.
+
+> **Heads up:** export the AWS credentials before running. If `AWS_ACCESS_KEY_ID` /
+> `AWS_SECRET_ACCESS_KEY` are unset, the unresolved `{{env.*}}` template is forwarded to the
+> Codex CLI verbatim, and you'll get an opaque AWS auth error (e.g. `UnrecognizedClientException`)
+> rather than a clear "credentials not set" message.
+
+For direct (non-agentic) inference against the same models, use the
+[`bedrock:` provider](https://www.promptfoo.dev/docs/providers/aws-bedrock/#openai-models)
+(`bedrock:openai.gpt-5.5`).
+
+## Run it
+
+```bash
+promptfoo eval -c examples/openai-codex-sdk/bedrock/promptfooconfig.yaml
+```
@@ -0,0 +1,34 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: 'Codex SDK running OpenAI frontier models on Amazon Bedrock'
+
+prompts:
+  - 'Write a Python function that calculates the factorial of a number. Output only the code.'
+
+providers:
+  # GPT-5.5 on Bedrock (us-east-2). Codex routes through Bedrock's OpenAI-compatible
+  # endpoint, so the model id uses the openai. prefix.
+  - id: openai:codex-sdk
+    label: codex-gpt-5.5-bedrock
+    config:
+      model: openai.gpt-5.5
+      model_provider: amazon-bedrock
+      model_reasoning_effort: low
+      sandbox_mode: read-only
+      skip_git_repo_check: true
+      # The Codex CLI reads AWS credentials from its own environment. promptfoo runs
+      # the CLI with a minimal env by default, so forward credentials + region here.
+      cli_env:
+        AWS_REGION: us-east-2
+        AWS_ACCESS_KEY_ID: '{{env.AWS_ACCESS_KEY_ID}}'
+        AWS_SECRET_ACCESS_KEY: '{{env.AWS_SECRET_ACCESS_KEY}}'
+        # Using temporary credentials (SSO / STS / assumed role / MFA)? Also forward the
+        # session token (leave it out for long-lived IAM keys):
+        # AWS_SESSION_TOKEN: '{{env.AWS_SESSION_TOKEN}}'
+        # Or authenticate another way instead of static keys — forward one of:
+        # AWS_BEARER_TOKEN_BEDROCK: '{{env.AWS_BEARER_TOKEN_BEDROCK}}' # a Bedrock API key
+        # AWS_PROFILE: '{{env.AWS_PROFILE}}'                           # a named profile
+
+tests:
+  - assert:
+      - type: contains
+        value: 'def factorial'
@@ -0,0 +1,29 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: 'Read-only Codex sandbox'
+
+prompts:
+  - |
+    Try to create hello.txt in the working directory with the text WRITTEN_BY_CODEX.
+    Then reply with one short sentence describing whether the write succeeded.
+
+providers:
+  - id: openai:codex-sdk
+    config:
+      model: gpt-5.2-codex
+      cli_config:
+        model_verbosity: medium
+      working_dir: '{{ env.CODEX_SANDBOX_WORKING_DIR | default("./sample-workspace") }}'
+      sandbox_mode: read-only
+      approval_policy: never
+      skip_git_repo_check: true
+
+tests:
+  - assert:
+      - type: icontains-any
+        value:
+          - 'did not succeed'
+          - 'read-only'
+          - 'permission'
+          - 'not allowed'
+          - 'failed'
+          - "couldn't"
@@ -0,0 +1 @@
+
@@ -0,0 +1,34 @@
+# openai-codex-sdk/skill-comparison (Codex Skill Comparison)
+
+You can run this example with:
+
+```bash
+npx promptfoo@latest init --example openai-codex-sdk/skill-comparison
+cd openai-codex-sdk/skill-comparison
+```
+
+## Overview
+
+This example compares two versions of the same Codex skill against identical review tasks.
+
+- `fixtures/v1` contains a narrower `review-standards` skill that only calls out weak password hashing.
+- `fixtures/v2` contains a stronger version that also checks timing-safe secret comparison.
+- Both providers share an `output_schema` (declared once via a YAML anchor) so each response is guaranteed to match the review JSON shape.
+- The eval verifies `skill-used`, scores issue recall and precision, and uses `max-score` to select the best output for each test case.
+
+Run it from this directory with:
+
+```bash
+promptfoo eval --no-cache
+```
+
+If you run it from another directory, set these environment variables first:
+
+```bash
+export CODEX_SKILL_COMPARE_V1_DIR=/absolute/path/to/fixtures/v1
+export CODEX_SKILL_COMPARE_V2_DIR=/absolute/path/to/fixtures/v2
+```
+
+The checked-in `sample-codex-home` directory is intentionally empty of auth state. Use an API key, or set `CODEX_HOME_OVERRIDE="$HOME/.codex"` to reuse a local Codex login.
+
+Because this example uses `max-score`, the weaker candidate is expected to fail when Promptfoo marks the stronger output as the winner for a test case.
@@ -0,0 +1,10 @@
+---
+name: review-standards
+description: Use this skill when asked to review authentication code for security issues.
+---
+
+When reviewing authentication code:
+
+1. Check password hashing.
+2. Use the issue id `weak-password-hash` when passwords use SHA-1 or MD5.
+3. Return no more than one issue.
@@ -0,0 +1,7 @@
+export const passwordHashConfig = {
+  algorithm: 'sha1',
+};
+
+export function tokenMatches(actualToken: string, expectedToken: string): boolean {
+  return actualToken === expectedToken;
+}
@@ -0,0 +1,12 @@
+---
+name: review-standards
+description: Use this skill when asked to review authentication code for security issues.
+---
+
+When reviewing authentication code:
+
+1. Check password hashing.
+2. Check whether secrets or tokens are compared with `===`.
+3. Use the issue id `weak-password-hash` when passwords use SHA-1 or MD5.
+4. Use the issue id `timing-unsafe-compare` when secrets or tokens use a direct equality comparison.
+5. Report only issues that match the user's requested scope.
@@ -0,0 +1,7 @@
+export const passwordHashConfig = {
+  algorithm: 'sha1',
+};
+
+export function tokenMatches(actualToken: string, expectedToken: string): boolean {
+  return actualToken === expectedToken;
+}
@@ -0,0 +1,119 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: Compare Codex skill versions
+
+prompts:
+  - |
+    Use the review-standards skill.
+    {{ request }}
+
+    Return JSON with this shape:
+    {
+      "summary": "one sentence",
+      "issues": [{"id": "stable-id", "severity": "high|medium|low"}]
+    }
+
+# YAML anchor for the review-output schema. Both providers must enforce the
+# same shape so the JS assertion can compare issue ids without per-provider
+# parsing branches.
+x-review-schema: &reviewSchema
+  type: object
+  required: [summary, issues]
+  additionalProperties: false
+  properties:
+    summary:
+      type: string
+    issues:
+      type: array
+      items:
+        type: object
+        required: [id, severity]
+        additionalProperties: false
+        properties:
+          id:
+            type: string
+          severity:
+            type: string
+            enum: [high, medium, low]
+
+# YAML anchor for the shared Codex config. Each provider overrides
+# `working_dir` so v1/v2 read different `SKILL.md` files but everything else
+# (model, sandbox, schema) stays identical.
+x-codex-config: &codexConfig
+  model: gpt-5.5
+  skip_git_repo_check: true
+  sandbox_mode: read-only
+  enable_streaming: true
+  output_schema: *reviewSchema
+  cli_env:
+    CODEX_HOME: '{{ env.CODEX_HOME_OVERRIDE | default("./sample-codex-home") }}'
+
+providers:
+  - id: openai:codex-sdk
+    label: review-standards-v1
+    config:
+      <<: *codexConfig
+      working_dir: '{{ env.CODEX_SKILL_COMPARE_V1_DIR | default("./fixtures/v1") }}'
+
+  - id: openai:codex-sdk
+    label: review-standards-v2
+    config:
+      <<: *codexConfig
+      working_dir: '{{ env.CODEX_SKILL_COMPARE_V2_DIR | default("./fixtures/v2") }}'
+
+defaultTest:
+  # Without `disableVarExpansion`, Promptfoo would fan each YAML-list var into
+  # one test case per element (src/evaluator.ts:generateVarCombinations), which
+  # would split each comparison test in two and break max-score selection.
+  options:
+    disableVarExpansion: true
+  assert:
+    - type: skill-used
+      value: review-standards
+
+    - type: javascript
+      threshold: 0.7
+      value: |
+        const result = JSON.parse(output);
+        const expected = context.vars.expectedIssues;
+        const found = (result.issues || []).map((issue) => issue.id);
+        const hits = expected.filter((id) => found.includes(id));
+        const extras = found.filter((id) => !expected.includes(id));
+        const recall = hits.length / expected.length;
+        const precision = found.length ? hits.length / found.length : 0;
+        const score = 0.7 * recall + 0.3 * precision;
+
+        return {
+          pass: recall >= 0.75 && precision >= 0.5,
+          score,
+          reason: `matched ${hits.length}/${expected.length} expected issues; ${extras.length} unexpected issues`,
+        };
+
+    - type: cost
+      threshold: 1
+
+    - type: latency
+      threshold: 180000
+
+    - type: max-score
+      value:
+        method: average
+        threshold: 0.7
+        weights:
+          javascript: 4
+          skill-used: 2
+          cost: 0.5
+          latency: 0.5
+
+tests:
+  - description: Finds both auth issues
+    vars:
+      request: Review src/auth.ts for password handling and token comparison issues.
+      expectedIssues:
+        - weak-password-hash
+        - timing-unsafe-compare
+
+  - description: Focuses on token comparison
+    vars:
+      request: Review src/auth.ts only for token comparison issues.
+      expectedIssues:
+        - timing-unsafe-compare
@@ -0,0 +1,5 @@
+# Intentionally empty so the example ships without Codex login state. The
+# README explains how to point CODEX_HOME_OVERRIDE at $HOME/.codex if you
+# want to reuse a real login.
+*
+!.gitignore
@@ -0,0 +1,40 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: Codex skill trace eval
+
+prompts:
+  - 'Use the token-skill skill. Return only the token.'
+
+providers:
+  - id: openai:codex-sdk
+    config:
+      model: gpt-5.2
+      # Relative working_dir values resolve from this config file's directory.
+      # Codex resolves CODEX_HOME itself; set CODEX_HOME_OVERRIDE when running this config from another cwd.
+      # sample-codex-home has no auth state; use an API key or point CODEX_HOME_OVERRIDE at ~/.codex.
+      working_dir: '{{ env.CODEX_SKILLS_WORKING_DIR | default("./sample-project") }}'
+      skip_git_repo_check: true
+      enable_streaming: true
+      deep_tracing: true
+      cli_env:
+        CODEX_HOME: '{{ env.CODEX_HOME_OVERRIDE | default("./sample-codex-home") }}'
+
+tests:
+  - assert:
+      - type: contains
+        value: 'CERULEAN-FALCON-SKILL'
+      - type: trajectory:step-count
+        value:
+          type: command
+          pattern: '*token-skill/SKILL.md*'
+          min: 1
+      - type: skill-used
+        value: token-skill
+
+tracing:
+  enabled: true
+  otlp:
+    http:
+      enabled: true
+      port: 4318
+      host: '127.0.0.1'
+      acceptFormats: ['json']
@@ -0,0 +1,25 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: Codex skill eval
+
+prompts:
+  - 'Use the token-skill skill. Return only the token.'
+
+providers:
+  - id: openai:codex-sdk
+    config:
+      model: gpt-5.2
+      # Relative working_dir values resolve from this config file's directory.
+      # Codex resolves CODEX_HOME itself; set CODEX_HOME_OVERRIDE when running this config from another cwd.
+      # sample-codex-home has no auth state; use an API key or point CODEX_HOME_OVERRIDE at ~/.codex.
+      working_dir: '{{ env.CODEX_SKILLS_WORKING_DIR | default("./sample-project") }}'
+      skip_git_repo_check: true
+      enable_streaming: true
+      cli_env:
+        CODEX_HOME: '{{ env.CODEX_HOME_OVERRIDE | default("./sample-codex-home") }}'
+
+tests:
+  - assert:
+      - type: equals
+        value: 'CERULEAN-FALCON-SKILL'
+      - type: skill-used
+        value: token-skill
@@ -0,0 +1,10 @@
+logs_*.sqlite
+logs_*.sqlite-shm
+logs_*.sqlite-wal
+sessions/
+skills/
+state_*.sqlite
+state_*.sqlite-shm
+state_*.sqlite-wal
+.tmp/
+tmp/
@@ -0,0 +1 @@
+
@@ -0,0 +1,8 @@
+---
+name: token-skill
+description: Use this skill when the user explicitly asks to use token-skill and wants the special token.
+---
+
+When this skill is used, respond with exactly `CERULEAN-FALCON-SKILL`.
+
+Do not add extra words, punctuation, or explanation.
@@ -0,0 +1,32 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: 'Persistent Codex thread memory'
+
+prompts:
+  - '{{request}}'
+
+providers:
+  - id: openai:codex-sdk
+    config:
+      model: gpt-5.2-codex
+      cli_config:
+        model_verbosity: medium
+      sandbox_mode: read-only
+      skip_git_repo_check: true
+      persist_threads: true
+      thread_pool_size: 1
+
+defaultTest:
+  options:
+    runSerially: true
+
+tests:
+  - vars:
+      request: 'Remember this marker: BLUE-OTTER-19. Reply with exactly STORED.'
+    assert:
+      - type: contains
+        value: 'STORED'
+  - vars:
+      request: 'What marker did I ask you to remember? Reply with the marker only.'
+    assert:
+      - type: contains
+        value: 'BLUE-OTTER-19'