chore: import upstream snapshot with attribution
CI / Shell Format Check (push) Has been cancelled
CI / Check Ruby (3.4) (push) Has been cancelled
CI / CI Config (push) Has been cancelled
CI / Test on Node ${{ matrix.node }} and ${{ matrix.os }}${{ matrix.shard && format(' (shard {0}/3)', matrix.shard) || '' }} (push) Has been cancelled
CI / Build on Node ${{ matrix.node }} (push) Has been cancelled
CI / Style Check (push) Has been cancelled
CI / Generate Assets (push) Has been cancelled
CI / Check Python (3.14) (push) Has been cancelled
CI / Check Python (3.9) (push) Has been cancelled
CI / Build Docs (push) Has been cancelled
CI / Code Scan Action (push) Has been cancelled
CI / Site tests (push) Has been cancelled
CI / webui tests (push) Has been cancelled
CI / Run Integration Tests (push) Has been cancelled
CI / Run Smoke Tests (push) Has been cancelled
CI / Go Tests (push) Has been cancelled
CI / Share Test (push) Has been cancelled
CI / Redteam (Production API) (push) Has been cancelled
CI / Redteam (Staging API) (push) Has been cancelled
CI / GitHub Actions Lint (push) Has been cancelled
CI / Check Ruby (3.0) (push) Has been cancelled
release-please / release-please (push) Has been cancelled
release-please / build (push) Has been cancelled
release-please / publish-npm (push) Has been cancelled
release-please / publish-npm-backfill (push) Has been cancelled
release-please / docker (push) Has been cancelled
release-please / publish-code-scan-action (push) Has been cancelled
release-please / attest-code-scan-action (push) Has been cancelled
Deploy local.promptfoo.app / Deploy to Cloudflare Pages (push) Has been cancelled
Test and Publish Multi-arch Docker Image / test (push) Has been cancelled
Test and Publish Multi-arch Docker Image / build-docker-and-push-digests (map[digest-suffix:linux-amd64 platform:linux/amd64 runner:ubuntu-latest]) (push) Has been cancelled
Test and Publish Multi-arch Docker Image / build-docker-and-push-digests (map[digest-suffix:linux-arm64 platform:linux/arm64 runner:ubuntu-24.04-arm]) (push) Has been cancelled
Test and Publish Multi-arch Docker Image / merge-docker-digests (push) Has been cancelled
Test and Publish Multi-arch Docker Image / Attest Multi-arch Image (push) Has been cancelled
Validate Renovate Config / Validate Renovate Configuration (push) Has been cancelled
CI / Shell Format Check (push) Has been cancelled
CI / Check Ruby (3.4) (push) Has been cancelled
CI / CI Config (push) Has been cancelled
CI / Test on Node ${{ matrix.node }} and ${{ matrix.os }}${{ matrix.shard && format(' (shard {0}/3)', matrix.shard) || '' }} (push) Has been cancelled
CI / Build on Node ${{ matrix.node }} (push) Has been cancelled
CI / Style Check (push) Has been cancelled
CI / Generate Assets (push) Has been cancelled
CI / Check Python (3.14) (push) Has been cancelled
CI / Check Python (3.9) (push) Has been cancelled
CI / Build Docs (push) Has been cancelled
CI / Code Scan Action (push) Has been cancelled
CI / Site tests (push) Has been cancelled
CI / webui tests (push) Has been cancelled
CI / Run Integration Tests (push) Has been cancelled
CI / Run Smoke Tests (push) Has been cancelled
CI / Go Tests (push) Has been cancelled
CI / Share Test (push) Has been cancelled
CI / Redteam (Production API) (push) Has been cancelled
CI / Redteam (Staging API) (push) Has been cancelled
CI / GitHub Actions Lint (push) Has been cancelled
CI / Check Ruby (3.0) (push) Has been cancelled
release-please / release-please (push) Has been cancelled
release-please / build (push) Has been cancelled
release-please / publish-npm (push) Has been cancelled
release-please / publish-npm-backfill (push) Has been cancelled
release-please / docker (push) Has been cancelled
release-please / publish-code-scan-action (push) Has been cancelled
release-please / attest-code-scan-action (push) Has been cancelled
Deploy local.promptfoo.app / Deploy to Cloudflare Pages (push) Has been cancelled
Test and Publish Multi-arch Docker Image / test (push) Has been cancelled
Test and Publish Multi-arch Docker Image / build-docker-and-push-digests (map[digest-suffix:linux-amd64 platform:linux/amd64 runner:ubuntu-latest]) (push) Has been cancelled
Test and Publish Multi-arch Docker Image / build-docker-and-push-digests (map[digest-suffix:linux-arm64 platform:linux/arm64 runner:ubuntu-24.04-arm]) (push) Has been cancelled
Test and Publish Multi-arch Docker Image / merge-docker-digests (push) Has been cancelled
Test and Publish Multi-arch Docker Image / Attest Multi-arch Image (push) Has been cancelled
Validate Renovate Config / Validate Renovate Configuration (push) Has been cancelled
This commit is contained in:
wehub-resource-sync
@@ -0,0 +1,27 @@
|
||||
# Agent Documentation
|
||||
|
||||
Guidance for AI agents editing the reusable docs in `docs/agents/`.
|
||||
|
||||
## Rules
|
||||
|
||||
- When changing PR workflow guidance, update both `docs/agents/pr-conventions.md` and
|
||||
the root `AGENTS.md` "Pull Request Creation" section in the same PR.
|
||||
- When changing dependency workflow guidance, keep
|
||||
`docs/agents/dependency-management.md` aligned with the root npm/audit guidance.
|
||||
- Do not introduce hard-coded local env-file requirements into generic eval commands.
|
||||
Use `--env-file .env` only when credentials are needed and the file exists.
|
||||
- Do not claim that eval exit code 0 means all test cases passed unless the text
|
||||
accounts for `PROMPTFOO_PASS_RATE_THRESHOLD`.
|
||||
- Verify implementation claims against source before documenting paths, defaults, or
|
||||
environment variables, and avoid machine-specific paths.
|
||||
- Prefer concrete command examples over broad advice, and avoid duplicating long
|
||||
guidance from root docs.
|
||||
|
||||
## Validation
|
||||
|
||||
For docs-only edits here, run:
|
||||
|
||||
```bash
|
||||
npm run f
|
||||
git diff --check
|
||||
```
|
||||
@@ -0,0 +1,576 @@
|
||||
# Codex App Server Provider Notes
|
||||
|
||||
These notes track the planned Promptfoo integration for the Codex app-server protocol.
|
||||
They are intentionally implementation-facing: keep them current as the provider, docs,
|
||||
examples, and verification expand.
|
||||
|
||||
For the broader coding-agent provider taxonomy, see
|
||||
[`coding-agent-provider-taxonomy.md`](./coding-agent-provider-taxonomy.md).
|
||||
|
||||
## Objective
|
||||
|
||||
Add an experimental Promptfoo provider that drives `codex app-server` directly. The
|
||||
provider should complement, not replace, the existing OpenAI Codex SDK provider:
|
||||
|
||||
- Codex SDK provider: best default for CI and automation.
|
||||
- Codex app-server provider: best for evaluating rich-client behavior exposed by the
|
||||
Codex app-server protocol, including streamed item events, approvals, skills,
|
||||
plugins, apps, filesystem requests, and thread lifecycle primitives.
|
||||
|
||||
Primary provider IDs:
|
||||
|
||||
- `openai:codex-app-server`
|
||||
- `openai:codex-app-server:<model>`
|
||||
- `openai:codex-desktop`
|
||||
- `openai:codex-desktop:<model>`
|
||||
|
||||
Optional top-level aliases may be added after the OpenAI-scoped provider is stable:
|
||||
|
||||
- `codex:app-server`
|
||||
- `codex:desktop`
|
||||
|
||||
## Source Material
|
||||
|
||||
- Official docs: https://developers.openai.com/codex/app-server
|
||||
- Local CLI: `/Applications/Codex.app/Contents/Resources/codex app-server --help`
|
||||
- Local generated schema command:
|
||||
|
||||
```bash
|
||||
codex app-server generate-ts --out /tmp/codex-app-server-schema/ts
|
||||
codex app-server generate-json-schema --out /tmp/codex-app-server-schema/json
|
||||
```
|
||||
|
||||
Current local schema inspection was generated from `codex-cli 0.118.0`.
|
||||
|
||||
## Protocol Shape
|
||||
|
||||
Transport:
|
||||
|
||||
- `stdio://` default, JSONL messages.
|
||||
- `ws://IP:PORT` experimental, one JSON-RPC message per WebSocket text frame.
|
||||
|
||||
Handshake:
|
||||
|
||||
1. Send `initialize` with Promptfoo client metadata.
|
||||
2. Send `initialized` notification.
|
||||
3. Start or resume a thread.
|
||||
4. Start a turn.
|
||||
5. Read notifications until `turn/completed`.
|
||||
|
||||
Core client requests:
|
||||
|
||||
- `initialize`
|
||||
- `thread/start`
|
||||
- `thread/resume`
|
||||
- `thread/archive`
|
||||
- `thread/unsubscribe`
|
||||
- `thread/read`
|
||||
- `turn/start`
|
||||
- `turn/steer`
|
||||
- `turn/interrupt`
|
||||
- `review/start`
|
||||
- `model/list`
|
||||
- `skills/list`
|
||||
- `plugin/list`
|
||||
- `plugin/read`
|
||||
- `app/list`
|
||||
|
||||
High-risk client requests that should not be exposed casually:
|
||||
|
||||
- `fs/writeFile`
|
||||
- `fs/remove`
|
||||
- `fs/copy`
|
||||
- `config/value/write`
|
||||
- `config/batchWrite`
|
||||
- `plugin/install`
|
||||
- `plugin/uninstall`
|
||||
- `command/exec`
|
||||
|
||||
Core server notifications:
|
||||
|
||||
- `thread/started`
|
||||
- `thread/status/changed`
|
||||
- `turn/started`
|
||||
- `turn/completed`
|
||||
- `item/started`
|
||||
- `item/completed`
|
||||
- `item/agentMessage/delta`
|
||||
- `item/commandExecution/outputDelta`
|
||||
- `item/fileChange/outputDelta`
|
||||
- `item/mcpToolCall/progress`
|
||||
- `serverRequest/resolved`
|
||||
- `thread/tokenUsage/updated`
|
||||
- `error`
|
||||
|
||||
Core server requests requiring deterministic Promptfoo responses:
|
||||
|
||||
- `item/commandExecution/requestApproval`
|
||||
- `item/fileChange/requestApproval`
|
||||
- `item/permissions/requestApproval`
|
||||
- `item/tool/requestUserInput`
|
||||
- `mcpServer/elicitation/request`
|
||||
- `item/tool/call`
|
||||
|
||||
## Provider Contract
|
||||
|
||||
### Inputs
|
||||
|
||||
Promptfoo prompt strings remain the default. The provider should also accept a JSON
|
||||
array of Codex input items:
|
||||
|
||||
```json
|
||||
[
|
||||
{ "type": "text", "text": "Review this project" },
|
||||
{ "type": "local_image", "path": "/absolute/path/to/screenshot.png" },
|
||||
{ "type": "skill", "name": "skill-creator", "path": "/absolute/path/SKILL.md" }
|
||||
]
|
||||
```
|
||||
|
||||
Supported input item types for the first implementation:
|
||||
|
||||
- `text`
|
||||
- `local_image`, mapped to app-server `inputImage`
|
||||
- `skill`
|
||||
|
||||
Unknown prompt JSON should be treated as plain text instead of throwing.
|
||||
|
||||
### Output
|
||||
|
||||
The provider response should include:
|
||||
|
||||
- `output`: final assistant text, assembled from `item/agentMessage/delta` and
|
||||
completed `agentMessage` items.
|
||||
- `sessionId`: thread id.
|
||||
- `raw`: serialized protocol-level turn summary and selected notifications.
|
||||
- `metadata.codexAppServer`: thread id, turn id, model, cwd, sandbox, approvals,
|
||||
server requests, item counts, command/file/tool trajectories, and app-server
|
||||
protocol data useful for debugging.
|
||||
- `metadata.skillCalls` / `metadata.attemptedSkillCalls`: heuristic skill usage
|
||||
where available.
|
||||
- `tokenUsage`: from `thread/tokenUsage/updated` if emitted.
|
||||
- `cost`: estimated from Promptfoo's Codex pricing table when model is known.
|
||||
|
||||
### Config
|
||||
|
||||
Provider-level config should be strict. Prompt-level merged config should strip unknown
|
||||
keys so generic Promptfoo prompt config does not break rows.
|
||||
|
||||
Core config:
|
||||
|
||||
- `apiKey`
|
||||
- `base_url`
|
||||
- `working_dir`
|
||||
- `additional_directories`
|
||||
- `skip_git_repo_check`
|
||||
- `codex_path_override`
|
||||
- `model`
|
||||
- `model_provider`
|
||||
- `service_tier`
|
||||
- `sandbox_mode`
|
||||
- `sandbox_policy`
|
||||
- `approval_policy`
|
||||
- `approvals_reviewer`
|
||||
- `model_reasoning_effort`
|
||||
- `reasoning_summary`
|
||||
- `personality`
|
||||
- `output_schema`
|
||||
- `thread_id`
|
||||
- `persist_threads`
|
||||
- `thread_pool_size`
|
||||
- `ephemeral`
|
||||
- `persist_extended_history`
|
||||
- `experimental_raw_events`
|
||||
- `experimental_api`
|
||||
- `cli_config`
|
||||
- `cli_env`
|
||||
- `inherit_process_env`
|
||||
- `reuse_server`
|
||||
- `deep_tracing`
|
||||
- `request_timeout_ms`
|
||||
- `startup_timeout_ms`
|
||||
- `server_request_policy`
|
||||
|
||||
### Safety Defaults
|
||||
|
||||
Default stance should favor repeatable evals over convenience:
|
||||
|
||||
- `approval_policy`: `never`
|
||||
- `sandbox_mode`: `read-only`
|
||||
- `network_access_enabled`: `false`
|
||||
- `ephemeral`: `true`
|
||||
- `reuse_server`: `true` unless `deep_tracing` is enabled
|
||||
- `inherit_process_env`: `false`
|
||||
- Server-side approval requests: decline/cancel or empty grants unless explicitly
|
||||
configured.
|
||||
|
||||
Rationale:
|
||||
|
||||
- The app-server exposes shell, filesystem, app connector, plugin, and config surfaces.
|
||||
- Promptfoo evals should be deterministic and should not block on human approval.
|
||||
- Eval prompts and target behavior can be adversarial.
|
||||
|
||||
## Implementation Phases
|
||||
|
||||
1. Stdio JSON-RPC client
|
||||
- Spawn `codex app-server`.
|
||||
- Parse JSONL stdout.
|
||||
- Route responses, notifications, and server requests.
|
||||
- Capture stderr for debug logs.
|
||||
- Support abort and timeout.
|
||||
|
||||
2. Provider lifecycle
|
||||
- Register with `providerRegistry`.
|
||||
- Reuse app-server process by default.
|
||||
- Shut down child processes, pending requests, and readline handles.
|
||||
- Disable reuse when `deep_tracing` is enabled.
|
||||
|
||||
3. Thread and turn execution
|
||||
- Validate working directory.
|
||||
- Start/resume threads.
|
||||
- Start turns with prompt input, model, cwd, sandbox, approvals, effort, personality,
|
||||
service tier, and output schema.
|
||||
- Serialize turns per reused thread.
|
||||
- Unsubscribe/archive non-persistent threads during cleanup.
|
||||
|
||||
4. Streaming aggregation
|
||||
- Track `turnId`.
|
||||
- Assemble assistant deltas.
|
||||
- Store completed items.
|
||||
- Build item counts and trajectory metadata.
|
||||
- Capture command output, file changes, MCP calls, dynamic tool calls, web search,
|
||||
plans, reasoning summaries, and review output.
|
||||
|
||||
5. Server request handling
|
||||
- Deterministically answer command approvals.
|
||||
- Deterministically answer file-change approvals.
|
||||
- Return empty permission grants by default.
|
||||
- Support configured answers for `tool/requestUserInput`.
|
||||
- Decline/cancel MCP elicitation by default.
|
||||
- Support static dynamic-tool responses.
|
||||
- Record all requests and decisions in metadata.
|
||||
|
||||
6. Tracing
|
||||
- Wrap `callApi` in `withGenAISpan`.
|
||||
- Add item-level spans when streaming notifications arrive.
|
||||
- Sanitize command output, tool arguments, and message text before trace attributes.
|
||||
- Inject OTEL env when `deep_tracing` is enabled.
|
||||
|
||||
7. Docs and examples
|
||||
- Add provider docs.
|
||||
- Add provider index entry.
|
||||
- Add examples for basic usage, read-only repo review, structured output, approval
|
||||
handling, skills, and tracing.
|
||||
|
||||
8. Verification
|
||||
- Unit tests with mocked child process and mocked protocol frames.
|
||||
- Registry tests for provider IDs and model-in-path parsing.
|
||||
- Docs/examples lint where applicable.
|
||||
- Local smoke config using a harmless prompt and `sandbox_mode: read-only` if
|
||||
credentials/login are available.
|
||||
- Final dogfood: run the new provider against the git diff and iterate on comments.
|
||||
|
||||
## Progress Log
|
||||
|
||||
### 2026-04-09
|
||||
|
||||
- Added initial provider implementation at `src/providers/openai/codex-app-server.ts`.
|
||||
- Added provider IDs under the OpenAI registry:
|
||||
- `openai:codex-app-server`
|
||||
- `openai:codex-app-server:<model>`
|
||||
- `openai:codex-desktop`
|
||||
- `openai:codex-desktop:<model>`
|
||||
- Implemented stdio JSON-RPC lifecycle:
|
||||
- spawn `codex app-server --listen stdio://`
|
||||
- `initialize`
|
||||
- `initialized`
|
||||
- `thread/start`
|
||||
- `thread/resume`
|
||||
- `turn/start`
|
||||
- notification handling through `turn/completed`
|
||||
- `thread/unsubscribe`/`thread/archive` cleanup modes
|
||||
- Implemented safe config defaults:
|
||||
- `approval_policy: never`
|
||||
- `sandbox_mode: read-only`
|
||||
- `ephemeral: true`
|
||||
- `thread_cleanup: unsubscribe`
|
||||
- process env isolation unless `inherit_process_env: true`
|
||||
- Implemented deterministic server request responses:
|
||||
- command execution approvals default to `decline`
|
||||
- file changes default to `decline`
|
||||
- permission requests default to empty grants
|
||||
- user input requests default to empty answers
|
||||
- MCP elicitations default to `decline`
|
||||
- dynamic tools can use static configured responses
|
||||
- Implemented output normalization:
|
||||
- assistant delta aggregation
|
||||
- completed `agentMessage` fallback/preference
|
||||
- token usage from `thread/tokenUsage/updated`
|
||||
- cost estimate for known Codex models
|
||||
- metadata with item counts, items, server request decisions, thread/turn ids
|
||||
- Implemented provider-level GenAI tracing and item spans.
|
||||
- Added mocked protocol tests in `test/providers/openai-codex-app-server.test.ts`.
|
||||
- Added registry tests in `test/providers/index.test.ts`.
|
||||
- Verification so far:
|
||||
- `npx vitest run test/providers/openai-codex-app-server.test.ts --sequence.shuffle=false`
|
||||
- `npx vitest run test/providers/index.test.ts -t "Codex app-server|Codex desktop" --sequence.shuffle=false`
|
||||
- `npm run tsc -- --pretty false`
|
||||
- Expanded mocked protocol tests to cover:
|
||||
- `thread/resume`
|
||||
- structured prompt input normalization
|
||||
- default `thread/unsubscribe` cleanup
|
||||
- user input request policy
|
||||
- dynamic tool static response policy
|
||||
- metadata sanitization
|
||||
- Ran a real local smoke eval through `npm run local -- eval -c examples/openai-codex-app-server/promptfooconfig.yaml --no-cache -o /tmp/promptfoo-codex-app-server-example.json`.
|
||||
- Result: pass.
|
||||
- Provider returned Codex app-server `sessionId`, token usage, item counts, thread id,
|
||||
turn id, and structured JSON output.
|
||||
- Ran docs build:
|
||||
- `cd site && SKIP_OG_GENERATION=true npm run build`
|
||||
- Result: pass.
|
||||
- First dogfood review through `examples/openai-codex-app-server/review-diff/promptfooconfig.yaml`
|
||||
found four actionable provider issues:
|
||||
- startup timeout could leak a spawned app-server and leave a rejected reusable
|
||||
connection promise cached
|
||||
- reused connections closed over the first turn's server request policy
|
||||
- app-server exit during a turn could leave the eval waiting forever when no turn
|
||||
timeout was configured
|
||||
- `raw` response payload serialized unsanitized protocol items
|
||||
- Fixed the dogfood findings and added regression coverage:
|
||||
- failed startup closes the process and a later call spawns a fresh process
|
||||
- active turns store their effective config so prompt-level server request policies are
|
||||
honored on reused servers
|
||||
- connection exit resolves active turns with a provider error and removes the dead
|
||||
connection from reuse maps
|
||||
- `raw` now contains sanitized thread, turn, token usage, notifications, and item
|
||||
metadata
|
||||
- final output now uses the last completed `agentMessage`, which avoids concatenating
|
||||
progress messages with final structured review output
|
||||
- Verification after fixes:
|
||||
- `npm run tsc -- --pretty false`
|
||||
- `npx vitest run test/providers/openai-codex-app-server.test.ts --sequence.shuffle=false`
|
||||
- Result: pass, 13 provider tests.
|
||||
- Second dogfood review passed the Promptfoo eval and returned valid JSON, but still
|
||||
reported two provider comments:
|
||||
- legacy `execCommandApproval` / `applyPatchApproval` requests identify the active
|
||||
thread with `conversationId` and expect legacy review decisions
|
||||
- persistent thread-pool eviction deleted local handles without unsubscribing the
|
||||
evicted loaded thread
|
||||
- Additional hardening from the second dogfood pass:
|
||||
- stdio parser now buffers partial JSON-RPC lines and rejoins literal newlines inside
|
||||
command-output strings as escaped newlines before parsing
|
||||
- legacy approval requests now map prompt-level policy to `approved`,
|
||||
`approved_for_session`, `denied`, and `abort`
|
||||
- legacy server requests can find active turn state by `conversationId`
|
||||
- evicted persistent cached threads now send `thread/unsubscribe` before being removed
|
||||
- added regression coverage for literal-newline JSON-RPC notifications, legacy
|
||||
approval requests, and persistent thread-pool eviction
|
||||
- Verification after second dogfood fixes:
|
||||
- `npm run tsc -- --pretty false`
|
||||
- `npx vitest run test/providers/openai-codex-app-server.test.ts --sequence.shuffle=false`
|
||||
- Result: pass, 16 provider tests.
|
||||
- Third dogfood review passed transport/eval and reported two lifecycle comments:
|
||||
- stale persistent thread handles remained after a reused app-server process exited
|
||||
- JSON-RPC request timeout cleanup removed the pending request but left an abort
|
||||
listener attached
|
||||
- Additional hardening from the third dogfood pass:
|
||||
- connection close now removes cached thread handles owned by that connection key
|
||||
- per-request timeout cleanup now removes abort listeners before rejecting
|
||||
- added regression coverage for cached-thread invalidation after process exit and
|
||||
abort-listener cleanup on JSON-RPC timeout
|
||||
- Verification after third dogfood fixes:
|
||||
- `npm run tsc -- --pretty false`
|
||||
- `npx vitest run test/providers/openai-codex-app-server.test.ts --sequence.shuffle=false`
|
||||
- Result: pass, 18 provider tests.
|
||||
- Fourth dogfood review passed transport/eval and reported two thread-cache comments:
|
||||
- persistent thread caching was still enabled for fresh-per-call app-server processes
|
||||
(`reuse_server: false` or `deep_tracing`)
|
||||
- pool eviction could unsubscribe an active cached thread before its turn completed
|
||||
- Additional hardening from the fourth dogfood pass:
|
||||
- thread caching is now allowed only when the app-server connection itself is reusable
|
||||
- active/reserved thread ids are protected with a small refcount while a call is using
|
||||
them
|
||||
- thread-pool eviction skips protected threads and temporarily allows the pool to
|
||||
exceed its soft cap rather than evicting an in-flight turn
|
||||
- added regression coverage for non-reusable persistent-thread configs and active-turn
|
||||
eviction avoidance
|
||||
- Verification after fourth dogfood fixes:
|
||||
- `npm run tsc -- --pretty false`
|
||||
- `npx vitest run test/providers/openai-codex-app-server.test.ts --sequence.shuffle=false`
|
||||
- Result: pass, 20 provider tests.
|
||||
- Fifth dogfood review passed transport/eval and reported one persistent-thread race:
|
||||
- concurrent calls with the same persistent-thread cache key could both miss the cache
|
||||
while the first `thread/start` was still pending, creating duplicate persistent
|
||||
threads and leaking the earlier one
|
||||
- Additional hardening from the fifth dogfood pass:
|
||||
- added an in-flight thread promise map keyed by thread cache key
|
||||
- concurrent same-cache `thread/start` / `thread/resume` callers now share the same
|
||||
pending thread handle
|
||||
- added regression coverage for concurrent same-cache persistent calls, ensuring only
|
||||
one `thread/start` is sent and both turns use the shared thread
|
||||
- Verification after fifth dogfood fix:
|
||||
- `npm run tsc -- --pretty false`
|
||||
- `npx vitest run test/providers/openai-codex-app-server.test.ts --sequence.shuffle=false`
|
||||
- Result: pass, 21 provider tests.
|
||||
- Sixth dogfood review passed transport/eval and reported two cache/default comments:
|
||||
- reusable connections could keep the first request timeout for requests that did not
|
||||
pass a per-call timeout
|
||||
- persistent thread cache keys omitted thread-start options such as `ephemeral`,
|
||||
`experimental_raw_events`, and `persist_extended_history`
|
||||
- Additional hardening from the sixth dogfood pass:
|
||||
- all provider-owned app-server requests now pass the effective per-call request
|
||||
timeout explicitly
|
||||
- persistent thread cache keys now include thread-start options that can change thread
|
||||
semantics
|
||||
- added regression coverage for prompt-level request timeouts on reused connections and
|
||||
thread-start option changes in persistent cache keys
|
||||
- Verification after sixth dogfood fixes:
|
||||
- `npm run tsc -- --pretty false`
|
||||
- `npx vitest run test/providers/openai-codex-app-server.test.ts --sequence.shuffle=false`
|
||||
- Result: pass, 23 provider tests.
|
||||
- Seventh dogfood retry:
|
||||
- first attempt hit an external Codex connectivity failure (`Network is unreachable`,
|
||||
`Reconnecting... 2/5`), which the provider surfaced as a clean provider error
|
||||
- retry completed transport/eval and reported one metadata issue: skill-root detection
|
||||
used the parent process env instead of the resolved app-server child env
|
||||
- Additional hardening from the seventh dogfood pass:
|
||||
- turn state now carries the resolved app-server environment produced by
|
||||
`prepareEnvironment`
|
||||
- skill root detection now uses the child env for `CODEX_HOME`, `HOME`, and
|
||||
`USERPROFILE`
|
||||
- added regression coverage for `cli_env.HOME` skill-call metadata detection
|
||||
- Verification after seventh dogfood fix:
|
||||
- `npm run tsc -- --pretty false`
|
||||
- `npx vitest run test/providers/openai-codex-app-server.test.ts --sequence.shuffle=false`
|
||||
- Result: pass, 24 provider tests.
|
||||
- Eighth dogfood review:
|
||||
- `npm run local -- eval -c examples/openai-codex-app-server/review-diff/promptfooconfig.yaml --no-cache --no-share -o /tmp/promptfoo-codex-app-server-review.json`
|
||||
- Result: pass.
|
||||
- Provider output: `{"comments":[],"summary":"No actionable findings; TypeScript and focused provider tests passed."}`
|
||||
- This confirms the provider can be used to review the current git diff and return
|
||||
schema-valid JSON with no remaining actionable comments from the dogfood reviewer.
|
||||
- Final verification sweep:
|
||||
- `npm run f`: pass with existing complexity warnings only; no formatting changes needed
|
||||
- `npm run tsc -- --pretty false`: pass
|
||||
- `npx vitest run test/providers/openai-codex-app-server.test.ts --sequence.shuffle=false`:
|
||||
pass, 24 provider tests
|
||||
- `npx vitest run test/providers/index.test.ts -t "Codex app-server|Codex desktop" --sequence.shuffle=false`:
|
||||
pass, 2 registry tests
|
||||
- `npm run l`: pass with existing complexity warnings only
|
||||
- `cd site && SKIP_OG_GENERATION=true npm run build`: pass
|
||||
- `npm run local -- eval -c examples/openai-codex-app-server/promptfooconfig.yaml --no-cache --no-share -o /tmp/promptfoo-codex-app-server-example.json`:
|
||||
pass
|
||||
|
||||
## QA Matrix
|
||||
|
||||
Required mocked unit tests:
|
||||
|
||||
- Constructor defaults and strict config validation.
|
||||
- Prompt-level unknown config stripping.
|
||||
- Missing/inaccessible/non-directory working directory.
|
||||
- Git check and `skip_git_repo_check`.
|
||||
- API key/env isolation and explicit `cli_env`.
|
||||
- Handshake order: `initialize`, `initialized`, `thread/start`, `turn/start`.
|
||||
- Model from provider path overrides/defaults correctly.
|
||||
- `thread_id` uses `thread/resume`.
|
||||
- `persist_threads` reuses cached thread and serializes turns.
|
||||
- Non-persistent calls unsubscribe/archive as configured.
|
||||
- Assistant deltas aggregate into final output.
|
||||
- Completed `agentMessage` fallback works when deltas are missing.
|
||||
- Token usage from `thread/tokenUsage/updated`.
|
||||
- Error notification produces provider error.
|
||||
- Failed turn produces provider error.
|
||||
- Abort before start.
|
||||
- Abort during turn sends `turn/interrupt` and returns aborted error.
|
||||
- Command approval request default decline.
|
||||
- File change request default decline.
|
||||
- Permission request default empty grant.
|
||||
- User input request configured answers.
|
||||
- Dynamic tool call configured static response.
|
||||
- MCP elicitation default decline/cancel.
|
||||
- Metadata contains item counts, trajectories, approvals, raw notifications, and server
|
||||
request decisions without leaking API keys.
|
||||
- `cleanup` kills child process and unregisters provider.
|
||||
- `deep_tracing` injects OTEL env and disables reuse/thread persistence.
|
||||
- Provider-level GenAI tracing records response body, token usage, session id, and item
|
||||
count attributes.
|
||||
|
||||
Required docs/examples checks:
|
||||
|
||||
- Provider docs render in Docusaurus.
|
||||
- Examples are listed and runnable from repo root with `npm run local -- eval ... --no-cache`.
|
||||
- Config docs call out experimental status, safety defaults, and difference from Codex SDK.
|
||||
|
||||
## Open Questions
|
||||
|
||||
- Whether to expose WebSocket transport in the first public version. Stdio is enough for
|
||||
Promptfoo-managed app-server processes; WebSocket is useful for external clients but
|
||||
adds auth and lifecycle complexity.
|
||||
- Whether to support top-level `codex:*` aliases immediately or keep all new IDs under
|
||||
`openai:*` for consistency with the existing Codex SDK provider.
|
||||
- Whether to persist generated app-server protocol types in source. The current plan is
|
||||
to implement a narrow local type surface and document how to regenerate schemas instead
|
||||
of committing a large generated bundle.
|
||||
|
||||
## Critical Audit Follow-up
|
||||
|
||||
Review feedback and red-team audit items addressed after the initial dogfood pass:
|
||||
|
||||
- Fixed registry env propagation for object-shaped provider configs. `loadApiProvider`
|
||||
already merges suite-level and provider-level env into `providerOptions.env`; the
|
||||
registry now passes that merged env to `OpenAICodexAppServerProvider`.
|
||||
- Fixed `service_tier` to match the generated app-server schema from `codex-cli 0.118.0`:
|
||||
`fast` and `flex` only.
|
||||
- Reset the hoisted `spawn` mock implementation in `beforeEach` to satisfy `test/AGENTS.md`
|
||||
mock isolation rules.
|
||||
- Regenerated app-server TypeScript and JSON Schema into
|
||||
`/tmp/codex-app-server-schema-current.XVTCwL` during the audit and compared rare
|
||||
app-server fields against the implementation.
|
||||
- Added coverage for schema-supported rare fields:
|
||||
- `model_reasoning_effort: none`
|
||||
- exact `personality` values: `none`, `friendly`, `pragmatic`
|
||||
- granular approval policy objects
|
||||
- app-server command approval amendment objects
|
||||
- session-scoped permission grants
|
||||
- accepted MCP elicitation responses with content and metadata
|
||||
- `base_instructions`, `developer_instructions`, and `collaboration_mode`
|
||||
- Dogfood review then found additional issues:
|
||||
- reusable app-server connections stayed alive after JSON-RPC request timeouts, which
|
||||
could leave late side-effecting responses unmanaged
|
||||
- docs listed `thread_pool_size` as unlimited even though the implementation defaults
|
||||
to `1`
|
||||
- provider cleanup cleared active turns before resolving them, which could hang
|
||||
shutdown while a turn was in flight
|
||||
- raw JSON-RPC notifications were retained even when `include_raw_events` was false
|
||||
- spawned app-server processes could be missed if cleanup ran while `initialize` was
|
||||
still pending
|
||||
- concurrent persistent thread starts could temporarily exceed `thread_pool_size` and
|
||||
remain over capacity after active turns finished
|
||||
- deep-tracing calls that shared a `thread_id` could overlap turns because the queue key
|
||||
returned early
|
||||
- the OpenAI provider docs heading change would have broken the existing `#codex-sdk`
|
||||
anchor
|
||||
- default `thread_id` resumes skipped unsubscribe cleanup
|
||||
- sent JSON-RPC request aborts kept the reusable app-server alive
|
||||
- retryable app-server `error` notifications with `willRetry: true` were treated as
|
||||
terminal
|
||||
- concurrent default-cleanup `thread_id` rows could unsubscribe while another row was
|
||||
queued for the same thread
|
||||
- Fixed these by closing/evicting connections on timeout and abort, resolving active
|
||||
turns during cleanup, tracking pending initialization processes, making raw event
|
||||
retention opt-in, rebalancing the persistent thread pool after turns finish, serializing
|
||||
explicit `thread_id` turns even under deep tracing, preserving the OpenAI docs
|
||||
`#codex-sdk` heading, default-unsubscribing non-persistent resumed threads, honoring
|
||||
retryable app-server errors, and deferring resumed-thread unsubscribe until no other
|
||||
protected queued caller remains.
|
||||
- Updated docs to explain why the app-server provider should stay separate from the
|
||||
Codex SDK provider: the SDK is the right default for CI and automation, while app-server
|
||||
is for rich-client protocol event surfaces and does not attach to a running Codex
|
||||
Desktop app.
|
||||
|
||||
Latest focused verification after these fixes:
|
||||
|
||||
- `npx vitest run test/providers/openai-codex-app-server.test.ts --sequence.shuffle=false`:
|
||||
pass, 35 provider tests.
|
||||
- `npm run local -- eval -c examples/openai-codex-app-server/review-diff/promptfooconfig.yaml --no-cache --no-share`:
|
||||
pass with `{"comments":[],"summary":"No actionable issues found in the current diff."}`.
|
||||
@@ -0,0 +1,565 @@
|
||||
# Coding Agent Provider Taxonomy
|
||||
|
||||
This document summarizes how promptfoo should think about coding-agent providers,
|
||||
what has been implemented so far, and what should come next. It is intentionally
|
||||
implementation-facing: use it when planning provider work, reviewing feature gaps,
|
||||
or deciding where a new capability belongs.
|
||||
|
||||
## Scope
|
||||
|
||||
This taxonomy covers providers that run an agentic coding runtime, not ordinary
|
||||
single-turn model APIs. A coding-agent provider usually has some combination of:
|
||||
|
||||
- A workspace or project directory.
|
||||
- Tool use for files, shell commands, MCP, search, or app connectors.
|
||||
- A session, thread, or server lifecycle.
|
||||
- Permission, sandbox, or approval controls.
|
||||
- Rich metadata beyond final assistant text.
|
||||
|
||||
The main providers in this family today are:
|
||||
|
||||
| Provider family | Provider IDs | Runtime boundary |
|
||||
| ----------------------- | ----------------------------------------------------- | ------------------------------------------ |
|
||||
| OpenAI Codex SDK | `openai:codex-sdk`, `openai:codex` | `@openai/codex-sdk` library |
|
||||
| OpenAI Codex app-server | `openai:codex-app-server`, `openai:codex-desktop` | Local `codex app-server` JSON-RPC process |
|
||||
| Claude Agent SDK | `anthropic:claude-agent-sdk`, `anthropic:claude-code` | `@anthropic-ai/claude-agent-sdk` library |
|
||||
| OpenCode SDK | `opencode:sdk`, `opencode` | OpenCode SDK plus local or existing server |
|
||||
|
||||
Standard OpenAI, Anthropic, Bedrock, Azure, and other model providers still matter
|
||||
for grading and comparison, but they are outside this taxonomy unless they expose a
|
||||
stateful coding-agent runtime.
|
||||
|
||||
## Taxonomy Axes
|
||||
|
||||
### 1. Runtime Boundary
|
||||
|
||||
The first question is where promptfoo stops and the agent runtime starts.
|
||||
|
||||
| Boundary | Meaning | Current examples |
|
||||
| ------------------------ | ------------------------------------------------------------- | ---------------------------------------- |
|
||||
| In-process SDK | promptfoo calls a package API directly. | Codex SDK, Claude Agent SDK |
|
||||
| Managed local server | promptfoo starts a server, then talks to it through a client. | OpenCode when `baseUrl` is unset |
|
||||
| Existing server | promptfoo connects to a runtime it does not configure. | OpenCode with `baseUrl` |
|
||||
| Local app-server process | promptfoo starts a rich-client protocol server over stdio. | Codex app-server |
|
||||
| Desktop UI process | Human-facing native app process. | Codex Desktop app, not directly attached |
|
||||
|
||||
This distinction matters because it controls what promptfoo can guarantee. If
|
||||
promptfoo starts the runtime, it can set env vars, working directories, sandbox
|
||||
options, tracing, and cleanup behavior. If promptfoo attaches to an existing server,
|
||||
that server owns authentication, installed tools, app connectors, and runtime state.
|
||||
|
||||
### 2. Session and Thread State
|
||||
|
||||
Coding agents are rarely stateless. Each provider needs explicit semantics for:
|
||||
|
||||
- Ephemeral sessions: safe default for independent eval rows.
|
||||
- Persistent sessions: useful for memory, multi-turn tasks, or regression suites.
|
||||
- User-supplied sessions: resume an existing thread/session by id.
|
||||
- Pooling: preserve concurrency without cross-contaminating rows.
|
||||
- Cleanup: unsubscribe, archive, delete temporary directories, or leave state alone.
|
||||
|
||||
The important design rule is that session reuse must be opt-in or very clearly
|
||||
scoped. Reusing state silently makes eval results order-dependent.
|
||||
|
||||
### 3. Workspace and Side Effects
|
||||
|
||||
Agent evals should separate filesystem access, network access, and shell access.
|
||||
Those are different risks.
|
||||
|
||||
| Surface | Safe default | Higher-risk mode |
|
||||
| ----------- | ------------------------------------------ | ------------------------------------------------------- |
|
||||
| Filesystem | Temporary directory or read-only workspace | Workspace write or full filesystem access |
|
||||
| Shell | Disabled or approval-gated | Allowed command execution |
|
||||
| Network | Disabled unless explicitly requested | Host allow-lists, live web/search, package installs |
|
||||
| App/plugin | Not installed or not invoked by default | App connectors, plugin installs, config writes |
|
||||
| Environment | Minimal env | Inherited process env with secrets and local auth state |
|
||||
|
||||
Provider docs should make clear that `danger-full-access` is not the same as
|
||||
network access, and read-only filesystem mode does not automatically sanitize env
|
||||
vars. Each surface should have its own option and its own tests.
|
||||
|
||||
### 4. Permission and Interaction Model
|
||||
|
||||
Promptfoo evals are non-interactive by default. Agent runtimes often expect a human
|
||||
to answer approval prompts, permission requests, or clarification questions.
|
||||
Providers should convert those into deterministic policies.
|
||||
|
||||
Common policy categories:
|
||||
|
||||
- Command approval.
|
||||
- File-change approval.
|
||||
- Permission grants.
|
||||
- User-input or ask-user-question tools.
|
||||
- MCP elicitation.
|
||||
- Dynamic tool calls.
|
||||
- Plugin or app connector requests.
|
||||
|
||||
Default policy should decline, cancel, or return empty answers unless a config opts
|
||||
into side effects. Every accepted side effect should be visible in metadata.
|
||||
|
||||
### 5. Inputs
|
||||
|
||||
The baseline input is a prompt string. Coding-agent providers increasingly need
|
||||
structured inputs:
|
||||
|
||||
- Text items.
|
||||
- Local images or image URLs.
|
||||
- Skills or plugin references.
|
||||
- Mentions/app connector references.
|
||||
- File, diff, or workspace context.
|
||||
|
||||
Provider-specific JSON input arrays are acceptable when the underlying runtime has
|
||||
typed input items. Unknown JSON shapes should usually degrade to plain text rather
|
||||
than crashing an eval row, unless the provider docs promise strict input parsing.
|
||||
|
||||
### 6. Outputs and Metadata
|
||||
|
||||
All coding-agent providers should return a normal promptfoo provider result:
|
||||
|
||||
- `output`: final assistant-facing text.
|
||||
- `sessionId`: session/thread id when available.
|
||||
- `tokenUsage`: runtime usage when available.
|
||||
- `cost`: estimate when usage and model pricing are known.
|
||||
- `metadata`: normalized agent metadata.
|
||||
- `raw`: raw or summarized protocol data when useful and safe.
|
||||
|
||||
Provider-specific metadata is still valuable, but consumers need a shared shape for
|
||||
cross-provider assertions and dashboards. A future shared schema should include:
|
||||
|
||||
- Runtime family and version.
|
||||
- Workspace and sandbox settings.
|
||||
- Session/thread/turn identifiers.
|
||||
- Tool trajectories.
|
||||
- Approval decisions.
|
||||
- File changes and command executions.
|
||||
- MCP and dynamic tool calls.
|
||||
- Skill/plugin/app connector usage.
|
||||
- Trace ids and span links.
|
||||
|
||||
### 7. Observability
|
||||
|
||||
Tracing should answer two questions:
|
||||
|
||||
- What did the model decide?
|
||||
- What did the agent runtime do?
|
||||
|
||||
Provider tracing should include the top-level `callApi` span, item/tool-level spans
|
||||
where possible, and sanitized attributes for prompts, commands, tool inputs, file
|
||||
paths, and outputs. Deep tracing should be opt-in when it requires injecting
|
||||
OpenTelemetry env vars into a child process.
|
||||
|
||||
## What Is Implemented So Far
|
||||
|
||||
### Shared Building Blocks
|
||||
|
||||
The coding-agent providers already share several practical patterns:
|
||||
|
||||
- Optional dependencies are loaded lazily so normal promptfoo installs do not need
|
||||
every agent SDK.
|
||||
- Working directories are validated or created before the agent call.
|
||||
- Temporary workspaces are cleaned up after evals.
|
||||
- Session or thread caches are keyed by provider config.
|
||||
- Provider-level config is stricter than prompt-level merged config.
|
||||
- Tool and skill usage are surfaced through metadata where possible.
|
||||
- Tracing is supported for Codex and is partially shared through OpenAI agent
|
||||
tracing helpers.
|
||||
|
||||
Useful files:
|
||||
|
||||
- `src/providers/agentic-utils.ts`
|
||||
- `src/providers/claude-agent-sdk.ts`
|
||||
- `src/providers/opencode-sdk.ts`
|
||||
- `src/providers/openai/codex-sdk.ts`
|
||||
- `src/providers/openai/codex-app-server.ts`
|
||||
- `src/providers/registry.ts`
|
||||
|
||||
### OpenAI Codex SDK
|
||||
|
||||
Status: implemented and documented.
|
||||
|
||||
Provider IDs:
|
||||
|
||||
- `openai:codex-sdk`
|
||||
- `openai:codex-sdk:<model>`
|
||||
- `openai:codex`
|
||||
- `openai:codex:<model>`
|
||||
|
||||
Implemented capabilities:
|
||||
|
||||
- Lazy loading for `@openai/codex-sdk`.
|
||||
- API-key and local Codex login authentication paths.
|
||||
- Working directory and Git repository safety checks.
|
||||
- Sandbox, network, web search, approval, and reasoning controls.
|
||||
- Thread resume and persistent thread pooling.
|
||||
- JSON schema output.
|
||||
- Text and local-image input items.
|
||||
- Skill usage heuristics from `SKILL.md` reads.
|
||||
- Token usage and cost estimation for known Codex models.
|
||||
- Streaming aggregation for metadata and tracing.
|
||||
- Deep tracing into the Codex runtime.
|
||||
- Default-provider support for grading when Codex credentials are available.
|
||||
|
||||
Important limits:
|
||||
|
||||
- It is the right default for CI and automation, but it does not expose every rich
|
||||
app-server protocol event.
|
||||
- Skill detection is heuristic.
|
||||
- Promptfoo still receives a final provider response, not live partial output in
|
||||
assertions.
|
||||
|
||||
Docs and examples:
|
||||
|
||||
- `site/docs/providers/openai-codex-sdk.md`
|
||||
- `examples/openai-codex-sdk/`
|
||||
|
||||
### OpenAI Codex App Server
|
||||
|
||||
Status: implemented, documented, and validated with mocked protocol tests plus a
|
||||
real local eval.
|
||||
|
||||
Provider IDs:
|
||||
|
||||
- `openai:codex-app-server`
|
||||
- `openai:codex-app-server:<model>`
|
||||
- `openai:codex-desktop`
|
||||
- `openai:codex-desktop:<model>`
|
||||
|
||||
Implemented capabilities:
|
||||
|
||||
- Spawns `codex app-server --listen stdio://`.
|
||||
- Drives the app-server JSON-RPC handshake.
|
||||
- Starts, resumes, unsubscribes, and archives threads according to config.
|
||||
- Starts turns with model, cwd, sandbox, approval, reasoning, personality,
|
||||
collaboration mode, service tier, output schema, and instructions.
|
||||
- Accepts plain text plus JSON arrays of app-server input items.
|
||||
- Aggregates streamed item notifications into final text and metadata.
|
||||
- Captures item counts, command/file/MCP/tool/web-search/reasoning trajectories.
|
||||
- Handles server requests deterministically through `server_request_policy`.
|
||||
- Uses safe defaults: read-only sandbox, no approvals, ephemeral threads, minimal env.
|
||||
- Supports thread pooling and persistent thread cache invalidation.
|
||||
- Supports raw events, token usage, cost estimation, request timeouts, turn timeouts,
|
||||
cleanup, aborts, and process failure handling.
|
||||
- Supports deep tracing by creating a fresh app-server process per row and injecting
|
||||
OpenTelemetry env vars.
|
||||
- Differentiates the app-server protocol from the Codex Desktop app: promptfoo
|
||||
starts its own app-server child process and does not attach to a running Desktop
|
||||
app UI process.
|
||||
|
||||
Important limits:
|
||||
|
||||
- WebSocket transport is not implemented; stdio is the supported transport.
|
||||
- Live partial output is not exposed to assertions.
|
||||
- Direct attachment to a running Codex Desktop app is not implemented.
|
||||
- High-risk protocol requests such as config writes, plugin installs, and arbitrary
|
||||
filesystem writes should remain unavailable or explicitly policy-gated.
|
||||
|
||||
Docs and examples:
|
||||
|
||||
- `site/docs/providers/openai-codex-app-server.md`
|
||||
- `docs/agents/codex-app-server-provider-notes.md`
|
||||
- `examples/openai-codex-app-server/`
|
||||
|
||||
### Claude Agent SDK
|
||||
|
||||
Status: implemented and documented.
|
||||
|
||||
Provider IDs:
|
||||
|
||||
- `anthropic:claude-agent-sdk`
|
||||
- `anthropic:claude-code`
|
||||
|
||||
Implemented capabilities:
|
||||
|
||||
- Lazy loading for `@anthropic-ai/claude-agent-sdk`.
|
||||
- Anthropic API key, Bedrock, Vertex, and local Claude Code auth flows.
|
||||
- Temporary or configured working directories.
|
||||
- Built-in tool controls, allowed/disallowed tools, and permission modes.
|
||||
- Explicit unsafe permission skip flag.
|
||||
- MCP server configuration and optional MCP caching.
|
||||
- AskUserQuestion handling.
|
||||
- Plugin and local skill support.
|
||||
- Custom agents, hooks, system prompt overrides, betas, thinking, and effort options.
|
||||
- Session resume, fork, continue, persistence, and file checkpointing options.
|
||||
- Sandbox and executable configuration.
|
||||
- Usage/cost controls such as max budget.
|
||||
|
||||
Important limits:
|
||||
|
||||
- The SDK owns many semantics, so promptfoo must keep docs aligned with SDK changes.
|
||||
- Side-effectful modes require external workspace reset discipline.
|
||||
|
||||
Docs and examples:
|
||||
|
||||
- `site/docs/providers/claude-agent-sdk.md`
|
||||
- `examples/claude-agent-sdk/`
|
||||
|
||||
### OpenCode SDK
|
||||
|
||||
Status: implemented and documented.
|
||||
|
||||
Provider IDs:
|
||||
|
||||
- `opencode:sdk`
|
||||
- `opencode`
|
||||
|
||||
Implemented capabilities:
|
||||
|
||||
- Lazy loading for `@opencode-ai/sdk` v1 or v2.
|
||||
- Starts an OpenCode server when `baseUrl` is unset.
|
||||
- Connects to an existing OpenCode server when `baseUrl` is provided.
|
||||
- Supports provider/model selection, variants, workspaces, and custom agents.
|
||||
- Supports temporary or configured working directories.
|
||||
- Uses read-only default tools for configured workspaces.
|
||||
- Supports write/edit/bash tools with explicit permission config.
|
||||
- Supports JSON Schema structured output through OpenCode `format`.
|
||||
- Supports sessions and persistent session caching.
|
||||
- Supports MCP configuration and optional MCP caching when promptfoo starts the
|
||||
server.
|
||||
|
||||
Important limits:
|
||||
|
||||
- When using `baseUrl`, the existing server owns auth, MCP setup, installed agents,
|
||||
and server-side configuration.
|
||||
- OpenCode model support is delegated to OpenCode/models.dev rather than promptfoo's
|
||||
normal provider model tables.
|
||||
|
||||
Docs and examples:
|
||||
|
||||
- `site/docs/providers/opencode-sdk.md`
|
||||
- `examples/provider-opencode-sdk/`
|
||||
|
||||
## Current Naming Guidance
|
||||
|
||||
Use provider IDs that encode the runtime boundary, not just the model vendor.
|
||||
|
||||
- `openai:codex` should continue to mean the Codex SDK alias because that is the
|
||||
best default for automation.
|
||||
- `openai:codex-app-server` should mean the app-server JSON-RPC protocol.
|
||||
- `openai:codex-desktop` should remain an alias for app-server behavior unless or
|
||||
until promptfoo can actually attach to the Desktop app process.
|
||||
- `anthropic:claude-code` should remain an alias for Claude Agent SDK because the
|
||||
SDK is still built on Claude Code.
|
||||
- `opencode` can remain a convenience alias for `opencode:sdk`.
|
||||
|
||||
Avoid adding unscoped top-level aliases such as `codex:desktop` until the OpenAI
|
||||
scoped names are stable and the docs can clearly explain the difference between
|
||||
SDK, app-server, and Desktop UI attachment.
|
||||
|
||||
## What To Implement Next
|
||||
|
||||
### 1. Shared Agent Metadata Schema
|
||||
|
||||
Create a cross-provider `metadata.agent` shape while preserving provider-specific
|
||||
metadata namespaces such as `metadata.codexAppServer`.
|
||||
|
||||
Proposed fields:
|
||||
|
||||
- `runtime`: `codex-sdk`, `codex-app-server`, `claude-agent-sdk`, `opencode`.
|
||||
- `runtimeVersion`: runtime-reported version when available.
|
||||
- `sessionId`, `threadId`, `turnId`.
|
||||
- `workingDir`, `sandbox`, `network`, `approvalPolicy`.
|
||||
- `tools`: normalized tool calls and outcomes.
|
||||
- `commands`: normalized shell command executions.
|
||||
- `fileChanges`: normalized file write/edit/delete attempts.
|
||||
- `approvals`: normalized approval prompts and decisions.
|
||||
- `mcp`: normalized MCP calls and elicitations.
|
||||
- `skills`: confirmed and attempted skill usage.
|
||||
- `trace`: trace ids and span ids when available.
|
||||
|
||||
Acceptance criteria:
|
||||
|
||||
- Existing provider-specific metadata remains backward compatible.
|
||||
- Cross-provider assertions can target the same metadata path.
|
||||
- Tests cover at least Codex SDK, Codex app-server, Claude Agent SDK, and OpenCode.
|
||||
|
||||
### 2. Shared Coding-Agent Provider Test Contract
|
||||
|
||||
Add a reusable test contract for coding-agent providers. Each provider can implement
|
||||
the same scenarios with its own mocked runtime.
|
||||
|
||||
Core scenarios:
|
||||
|
||||
- Missing optional dependency.
|
||||
- API key/env precedence.
|
||||
- Working directory validation.
|
||||
- Prompt-level config merge.
|
||||
- Safe default sandbox and permissions.
|
||||
- Structured output.
|
||||
- Session persistence and cleanup.
|
||||
- Timeout and abort.
|
||||
- Runtime process/server failure.
|
||||
- Tool approval decline by default.
|
||||
- Metadata normalization.
|
||||
- Trace sanitization.
|
||||
|
||||
Acceptance criteria:
|
||||
|
||||
- New providers cannot skip lifecycle and safety cases.
|
||||
- Mock implementations reset hoisted mocks in `beforeEach`.
|
||||
- Concurrency tests prove one session/process failure cannot fail unrelated rows.
|
||||
|
||||
### 3. Provider Capability Matrix
|
||||
|
||||
Add a docs page or generated table that compares coding-agent provider capabilities.
|
||||
|
||||
Suggested columns:
|
||||
|
||||
- Provider IDs.
|
||||
- Runtime boundary.
|
||||
- Optional dependency or CLI requirement.
|
||||
- Auth modes.
|
||||
- Workspace model.
|
||||
- Sandbox controls.
|
||||
- Shell/file/network controls.
|
||||
- MCP support.
|
||||
- Skills/plugins/custom agents.
|
||||
- Structured output.
|
||||
- Session persistence.
|
||||
- Tracing.
|
||||
- Raw protocol metadata.
|
||||
- Existing-server attachment.
|
||||
|
||||
Acceptance criteria:
|
||||
|
||||
- The matrix links to each provider doc.
|
||||
- It clearly says that Codex Desktop attachment is not currently supported.
|
||||
- It is tested in the docs build.
|
||||
|
||||
### 4. Real Eval QA Matrix
|
||||
|
||||
Create a small set of real eval examples that can be run selectively by maintainers.
|
||||
|
||||
Minimum scenarios:
|
||||
|
||||
- Read-only repo review.
|
||||
- Structured JSON output.
|
||||
- Sandbox denial for attempted writes.
|
||||
- Workspace-write side effect in a disposable fixture.
|
||||
- Session persistence across two rows.
|
||||
- Skill or plugin invocation.
|
||||
- MCP tool call with a deterministic local MCP server.
|
||||
- Deep tracing smoke test.
|
||||
|
||||
Acceptance criteria:
|
||||
|
||||
- Each scenario records expected pass/fail/error behavior.
|
||||
- Each scenario can run with `--no-cache`.
|
||||
- Side-effectful scenarios use disposable workspaces.
|
||||
- CI can run a lightweight subset without requiring every optional agent runtime.
|
||||
|
||||
### 5. Security and Redteam Coverage
|
||||
|
||||
Add adversarial tests for the risky parts of agent runtimes.
|
||||
|
||||
High-value cases:
|
||||
|
||||
- Prompt injection that asks the agent to reveal env vars.
|
||||
- Path traversal through input items, tool args, or MCP responses.
|
||||
- Approval bypass attempts.
|
||||
- Plugin install/config-write attempts.
|
||||
- Network enablement attempts when network should be off.
|
||||
- Symlink writes out of the workspace.
|
||||
- Malicious `SKILL.md` or plugin instructions.
|
||||
- Tool output containing secrets that must be redacted from traces.
|
||||
|
||||
Acceptance criteria:
|
||||
|
||||
- Providers decline or isolate side effects by default.
|
||||
- Redaction is tested for command output, MCP arguments, tool arguments, prompts,
|
||||
and final metadata.
|
||||
- Any accepted dangerous operation requires explicit config and is visible in
|
||||
metadata.
|
||||
|
||||
### 6. Codex App Server Follow-Up Features
|
||||
|
||||
The app-server provider now covers the core eval path. The next app-server-specific
|
||||
work should focus on rare protocol features and product integration boundaries.
|
||||
|
||||
Candidates:
|
||||
|
||||
- `model/list`, `skills/list`, `plugin/list`, `plugin/read`, and `app/list`
|
||||
metadata discovery.
|
||||
- `review/start` support for native review flows.
|
||||
- `turn/steer` and `turn/interrupt` tests for cancellation and mid-turn control.
|
||||
- Optional WebSocket transport only if upstream stabilizes it.
|
||||
- Better raw event snapshots for protocol regression tests.
|
||||
- Stronger docs around Desktop alias semantics and why promptfoo starts a separate
|
||||
app-server process.
|
||||
|
||||
Acceptance criteria:
|
||||
|
||||
- Rare features are opt-in and deterministic.
|
||||
- High-risk operations are policy-gated.
|
||||
- Protocol additions have mocked tests and at least one documented example when
|
||||
user-facing.
|
||||
|
||||
### 7. Side-Effect Harness
|
||||
|
||||
Build a reusable harness for tests and examples that need write access.
|
||||
|
||||
It should provide:
|
||||
|
||||
- Disposable workspace creation.
|
||||
- Git repository initialization when needed.
|
||||
- Snapshot before and after agent runs.
|
||||
- Symlink and path traversal fixtures.
|
||||
- Cleanup guarantees.
|
||||
- Helpers for expected file diffs.
|
||||
|
||||
Acceptance criteria:
|
||||
|
||||
- No side-effectful provider test mutates the source checkout.
|
||||
- Tests can assert exact changed files.
|
||||
- Harness works on macOS, Linux, and Windows CI.
|
||||
|
||||
### 8. Documentation Cleanup
|
||||
|
||||
Improve the information architecture around coding-agent providers.
|
||||
|
||||
Recommended docs:
|
||||
|
||||
- A public provider comparison matrix.
|
||||
- A "Choosing a coding-agent provider" guide.
|
||||
- A "Managing side effects in agent evals" guide.
|
||||
- Provider-specific "SDK vs app-server vs Desktop app" sections for Codex.
|
||||
- Example READMEs that all share the same run, auth, and safety structure.
|
||||
|
||||
Acceptance criteria:
|
||||
|
||||
- The public docs answer which provider to use for CI, Desktop-like protocol evals,
|
||||
Claude Code compatibility, and OpenCode multi-provider setups.
|
||||
- Example configs validate locally.
|
||||
- Docs state safe defaults and side-effect responsibilities near every write-capable
|
||||
example.
|
||||
|
||||
## Review Checklist For New Coding-Agent Provider Work
|
||||
|
||||
Before merging a provider in this family, verify:
|
||||
|
||||
- Provider IDs are explicit about runtime boundary.
|
||||
- Optional dependencies fail with actionable install guidance.
|
||||
- API keys and local-auth modes are documented.
|
||||
- Env precedence is tested.
|
||||
- Working directory, sandbox, network, and shell controls are documented and tested.
|
||||
- Prompt-level config merges do not drop nested provider defaults.
|
||||
- Hoisted mocks with implementations reset in `beforeEach`.
|
||||
- Session/thread reuse is opt-in or explicitly scoped.
|
||||
- Runtime failures do not poison unrelated sessions or rows.
|
||||
- Approval/user-input/MCP requests have deterministic non-interactive defaults.
|
||||
- Metadata includes session ids, tool/command/file activity, and approval decisions.
|
||||
- Trace attributes are sanitized.
|
||||
- Examples validate and at least one real eval has been run when the runtime is
|
||||
locally available.
|
||||
|
||||
## Open Questions
|
||||
|
||||
- Should promptfoo expose one public `metadata.agent` schema now, or keep it internal
|
||||
until at least two providers use it in docs examples?
|
||||
- Should Codex app-server discovery operations be exposed as provider metadata on
|
||||
every call, or only behind an explicit config flag?
|
||||
- Should any provider support a hard "no side effects" verifier that snapshots the
|
||||
workspace and fails if files changed?
|
||||
- Should remote/existing-server modes be marked as less reproducible in promptfoo
|
||||
output metadata?
|
||||
- Should top-level aliases like `codex:app-server` wait for a broader provider naming
|
||||
cleanup?
|
||||
@@ -0,0 +1,113 @@
|
||||
# Database Security (SQL Injection Prevention)
|
||||
|
||||
This codebase uses Drizzle ORM with SQLite. All database queries must use parameterized SQL so user-controlled input never changes query structure.
|
||||
|
||||
## Quick Rules
|
||||
|
||||
- Use Drizzle's `sql` tagged template literals for dynamic values.
|
||||
- Use `sql.join()` for dynamic lists such as `IN (...)` clauses.
|
||||
- Pass `SQL<unknown>` fragments between functions, not strings.
|
||||
- Do not build queries with `sql.raw()` or string interpolation.
|
||||
- Prefer `json_each()` for user-selected JSON keys. When `json_extract()` is appropriate, bind a path built with the vetted `buildSafeJsonPath` helper in `src/models/eval.ts`.
|
||||
|
||||
## Required Pattern: Use `sql` Template Strings
|
||||
|
||||
Use parameterized queries for single values:
|
||||
|
||||
```typescript
|
||||
import { sql } from 'drizzle-orm';
|
||||
|
||||
const query = sql`SELECT * FROM eval_results WHERE eval_id = ${evalId}`;
|
||||
```
|
||||
|
||||
Use parameterized queries for multiple values:
|
||||
|
||||
```typescript
|
||||
import { sql } from 'drizzle-orm';
|
||||
|
||||
const query = sql`
|
||||
SELECT * FROM eval_results
|
||||
WHERE eval_id = ${evalId} AND success = ${1}
|
||||
`;
|
||||
```
|
||||
|
||||
Use `sql.join()` for dynamic `IN (...)` lists:
|
||||
|
||||
```typescript
|
||||
import { sql } from 'drizzle-orm';
|
||||
|
||||
const ids = ['id1', 'id2', 'id3'];
|
||||
const query = sql`SELECT * FROM evals WHERE id IN (${sql.join(ids, sql`, `)})`;
|
||||
```
|
||||
|
||||
## Forbidden Pattern: Raw SQL with Dynamic Content
|
||||
|
||||
Do not interpolate user-controlled values into raw SQL:
|
||||
|
||||
```typescript
|
||||
const query = sql.raw(`SELECT * FROM eval_results WHERE eval_id = '${evalId}'`);
|
||||
```
|
||||
|
||||
Do not build SQL conditions with string concatenation:
|
||||
|
||||
```typescript
|
||||
import { sql } from 'drizzle-orm';
|
||||
|
||||
const whereClause = `eval_id = '${evalId}'`;
|
||||
const query = sql.raw(`SELECT * FROM eval_results WHERE ${whereClause}`);
|
||||
```
|
||||
|
||||
## JSON Paths in SQLite
|
||||
|
||||
SQLite's `json_extract()` accepts its JSON path as a bound value. Do not use `sql.raw()` to splice user-controlled JSON paths into SQL. Prefer `json_each()` when filtering by dynamic keys, because the key can be compared as a normal parameter:
|
||||
|
||||
```typescript
|
||||
import { sql } from 'drizzle-orm';
|
||||
|
||||
const query = sql`
|
||||
SELECT *
|
||||
FROM eval_results
|
||||
WHERE EXISTS (
|
||||
SELECT 1
|
||||
FROM json_each(metadata)
|
||||
WHERE json_each.key = ${field} AND json_each.value = ${value}
|
||||
)
|
||||
`;
|
||||
```
|
||||
|
||||
If you construct a JSON path for `json_extract()`, use `buildSafeJsonPath` from `src/models/eval.ts`, which escapes backslashes and double quotes for JSON path syntax and returns a value to bind:
|
||||
|
||||
```typescript
|
||||
import { sql } from 'drizzle-orm';
|
||||
import { buildSafeJsonPath } from '../../src/models/eval';
|
||||
|
||||
const jsonPath = buildSafeJsonPath(userField);
|
||||
const query = sql`
|
||||
SELECT * FROM eval_results
|
||||
WHERE json_extract(metadata, ${jsonPath}) = ${value}
|
||||
`;
|
||||
```
|
||||
|
||||
If you need a new JSON-path helper, match the guarantees in `buildSafeJsonPath` and keep the implementation audited in one shared utility.
|
||||
|
||||
## Passing SQL Fragments Between Functions
|
||||
|
||||
When building complex queries, pass `SQL<unknown>` fragments instead of strings:
|
||||
|
||||
```typescript
|
||||
import { type SQL, sql } from 'drizzle-orm';
|
||||
|
||||
function queryWithFilter(whereSql: SQL<unknown>): Promise<Result[]> {
|
||||
const query = sql`SELECT * FROM eval_results WHERE ${whereSql}`;
|
||||
return db.all(query);
|
||||
}
|
||||
|
||||
const filter = sql`eval_id = ${evalId} AND success = ${1}`;
|
||||
const results = await queryWithFilter(filter);
|
||||
```
|
||||
|
||||
## Key Files with Database Queries
|
||||
|
||||
- `src/models/eval.ts` - Main eval queries and JSON-path helper
|
||||
- `src/util/calculateFilteredMetrics.ts` - Metrics aggregation queries
|
||||
- `src/database/index.ts` - Database connection
|
||||
@@ -0,0 +1,89 @@
|
||||
# Dependency Management
|
||||
|
||||
## Safe Update Workflow
|
||||
|
||||
Use `--target minor` for safe minor/patch updates only:
|
||||
|
||||
```bash
|
||||
# Check all three locations for available updates
|
||||
npx npm-check-updates --target minor # Root
|
||||
npx npm-check-updates --target minor --cwd site # Site workspace
|
||||
npx npm-check-updates --target minor --cwd src/app # App workspace
|
||||
|
||||
# Apply updates with -u flag
|
||||
npx npm-check-updates --target minor -u
|
||||
npx npm-check-updates --target minor -u --cwd site
|
||||
npx npm-check-updates --target minor -u --cwd src/app
|
||||
|
||||
# Install and verify
|
||||
npm install
|
||||
npm run build && npm test && npm run lint && npm run format:check
|
||||
|
||||
# Check version consistency (required by CI)
|
||||
npx check-dependency-version-consistency
|
||||
```
|
||||
|
||||
## Critical Rules
|
||||
|
||||
1. **Version consistency across workspaces** - All workspaces must use the same version of shared dependencies. CI enforces this via `check-dependency-version-consistency`.
|
||||
|
||||
2. **Update examples/** - 20+ package.json files in examples/ are user-facing; keep them current when updating dependencies.
|
||||
|
||||
3. **Run `npm audit`** - Use `npm audit` or `npm run audit:fix` to check for security vulnerabilities across all workspaces. Do not let `npm audit fix` lockfile drift ride along with an unrelated change; ship audit-driven updates as their own PR.
|
||||
|
||||
4. **If updates fail** - Revert the problematic package and keep the current version. Don't force incompatible updates.
|
||||
|
||||
5. **Test before committing** - Always run `npm run build && npm test` after updating dependencies.
|
||||
|
||||
## Major Updates
|
||||
|
||||
```bash
|
||||
# See available major updates (don't apply automatically)
|
||||
npx npm-check-updates --target latest
|
||||
|
||||
# Major updates often require code changes - evaluate each carefully
|
||||
```
|
||||
|
||||
Major updates require careful evaluation:
|
||||
|
||||
- Check the changelog for breaking changes
|
||||
- Look for migration guides
|
||||
- Test thoroughly before merging
|
||||
|
||||
## Workspaces
|
||||
|
||||
The project uses npm workspaces. Updates must be checked in all three locations:
|
||||
|
||||
- Root (`/`) - Core library dependencies
|
||||
- Site (`/site`) - Documentation site (Docusaurus)
|
||||
- App (`/src/app`) - Web UI (React/Vite)
|
||||
|
||||
## Working on Renovate Branches
|
||||
|
||||
Renovate force-pushes its branches whenever `main` changes or someone comments
|
||||
`@renovate rebase`. Any manual commit you add may be overwritten without warning.
|
||||
|
||||
- Push fixes quickly and expect them to survive only until the next Renovate rebase.
|
||||
- For non-trivial manual work on a Renovate-managed dependency, create a sibling
|
||||
branch off the Renovate branch and open a separate PR that Renovate will not touch.
|
||||
- On a major-version Renovate PR, read the upstream changelog, run gap analysis on
|
||||
our matching provider/integration and its docs, then test end-to-end with real evals
|
||||
(`npm run local -- eval -c <example>.yaml --no-cache -o output.json`, adding
|
||||
`--env-file .env` when credentials are needed and the file exists)
|
||||
before deciding whether the upgrade needs code changes.
|
||||
|
||||
## Useful Commands
|
||||
|
||||
```bash
|
||||
# Fix security vulnerabilities in all workspaces
|
||||
npm run audit:fix
|
||||
|
||||
# Check for outdated packages
|
||||
npm outdated
|
||||
|
||||
# See why a package is installed
|
||||
npm explain <package-name>
|
||||
|
||||
# Check for unused dependencies
|
||||
npm run depcheck
|
||||
```
|
||||
@@ -0,0 +1,77 @@
|
||||
# Git Workflow
|
||||
|
||||
## Critical Rules
|
||||
|
||||
1. **NEVER** commit directly to main
|
||||
2. **NEVER** merge branches into main directly
|
||||
3. **NEVER** push to main - EVER
|
||||
4. **NEVER** use `--force` without explicit approval
|
||||
5. **ALWAYS** create new commits - never amend or rebase unless explicitly asked
|
||||
|
||||
**Forbidden commands:**
|
||||
|
||||
- `git push origin main`
|
||||
- `git merge feature-branch` while on main
|
||||
- Any direct commits to main
|
||||
|
||||
All changes to main MUST go through pull requests.
|
||||
|
||||
## Commit Policy
|
||||
|
||||
**"Explicitly asked"** = user says "amend", "squash", "rebase", or "fix up the commit".
|
||||
|
||||
"Looks good" or "go ahead" is NOT permission to rewrite history.
|
||||
|
||||
## Standard Workflow
|
||||
|
||||
### 1. Create Feature Branch
|
||||
|
||||
```bash
|
||||
git checkout main
|
||||
git pull origin main
|
||||
git checkout -b feature/your-branch-name
|
||||
```
|
||||
|
||||
### 2. Make Changes and Commit
|
||||
|
||||
```bash
|
||||
git add <specific-files> # NEVER blindly add everything
|
||||
git commit -m "type(scope): description"
|
||||
```
|
||||
|
||||
See `docs/agents/pr-conventions.md` for commit message format.
|
||||
|
||||
### 3. Lint and Format
|
||||
|
||||
```bash
|
||||
npm run l # Lint changed files
|
||||
npm run f # Format changed files
|
||||
```
|
||||
|
||||
Fix any errors before proceeding.
|
||||
|
||||
### 4. Sync with Main
|
||||
|
||||
```bash
|
||||
git fetch origin main
|
||||
git merge origin/main
|
||||
```
|
||||
|
||||
Resolve any conflicts before pushing.
|
||||
|
||||
### 5. Push and Create PR
|
||||
|
||||
```bash
|
||||
git push -u origin feature/your-branch-name
|
||||
gh pr create --title "type(scope): description" --body "PR description"
|
||||
```
|
||||
|
||||
### 6. Wait for Review
|
||||
|
||||
Wait for CI checks to pass and code review approval before merging.
|
||||
|
||||
## Key Points
|
||||
|
||||
- **Never blindly `git add .`** - there may be unrelated files
|
||||
- **Always sync with main** before creating PR to avoid conflicts
|
||||
- **Don't edit CHANGELOG.md** - it's auto-generated
|
||||
@@ -0,0 +1,72 @@
|
||||
# Logging Guidelines
|
||||
|
||||
## The Rule
|
||||
|
||||
Always use the logger with an object as the second parameter:
|
||||
|
||||
```typescript
|
||||
logger.debug('[Component] Message', { headers, body, config });
|
||||
```
|
||||
|
||||
The object is **auto-sanitized** - sensitive fields are automatically redacted.
|
||||
|
||||
## Why This Matters
|
||||
|
||||
- **Security**: Prevents accidental exposure of secrets in logs
|
||||
- **Consistency**: Structured logs are easier to search and analyze
|
||||
- **Safety**: Red team test content may contain harmful/sensitive data
|
||||
|
||||
## Correct Usage
|
||||
|
||||
```typescript
|
||||
import logger from './logger';
|
||||
|
||||
// Good - context object is auto-sanitized
|
||||
logger.debug('[Provider]: Making API request', {
|
||||
url: 'https://api.example.com',
|
||||
method: 'POST',
|
||||
headers: { Authorization: 'Bearer secret-token' },
|
||||
body: { apiKey: 'secret-key', data: 'value' },
|
||||
});
|
||||
// Output: Authorization and apiKey are [REDACTED]
|
||||
|
||||
logger.error('Request failed', {
|
||||
headers: response.headers,
|
||||
body: errorResponse,
|
||||
});
|
||||
```
|
||||
|
||||
## Anti-Pattern
|
||||
|
||||
```typescript
|
||||
// WRONG - exposes secrets, bypasses sanitization
|
||||
logger.debug(`Config: ${JSON.stringify(config)}`);
|
||||
logger.debug(`Calling ${url} with headers: ${JSON.stringify(headers)}`);
|
||||
```
|
||||
|
||||
## Manual Sanitization
|
||||
|
||||
For non-logging contexts:
|
||||
|
||||
```typescript
|
||||
import { sanitizeObject } from './util/sanitizer';
|
||||
|
||||
const sanitizedConfig = sanitizeObject(providerConfig, {
|
||||
context: 'provider config',
|
||||
});
|
||||
```
|
||||
|
||||
## What Gets Sanitized
|
||||
|
||||
Field names containing (case-insensitive, works with `-`, `_`, camelCase):
|
||||
|
||||
| Category | Fields |
|
||||
| ------------ | -------------------------------------------------------------------------- |
|
||||
| Passwords | `password`, `passwd`, `pwd`, `passphrase` |
|
||||
| API Keys | `apiKey`, `api_key`, `token`, `accessToken`, `refreshToken`, `bearerToken` |
|
||||
| Secrets | `secret`, `clientSecret`, `webhookSecret` |
|
||||
| Headers | `authorization`, `cookie`, `x-api-key`, `x-auth-token` |
|
||||
| Certificates | `privateKey`, `certificatePassword`, `keystorePassword` |
|
||||
| Signatures | `signature`, `sig`, `signingKey` |
|
||||
|
||||
See `src/util/sanitizer.ts` for the complete list.
|
||||
@@ -0,0 +1,219 @@
|
||||
# Pull Request Conventions
|
||||
|
||||
PR titles follow Conventional Commits format. They become squash-merge commit messages and changelog entries.
|
||||
|
||||
## Format
|
||||
|
||||
```plaintext
|
||||
<type>(<scope>): <description>
|
||||
<type>(<scope>)!: <description> # Breaking changes
|
||||
```
|
||||
|
||||
### Description Guidelines
|
||||
|
||||
- **Imperative mood**: "add feature" not "added" or "adds"
|
||||
- **Lowercase**: except proper nouns and acronyms (FERPA, OAuth, MUI)
|
||||
- **No trailing period**
|
||||
- **Be specific**: describe what changed, not that something changed
|
||||
- **~50 characters**: GitHub truncates long titles
|
||||
|
||||
## Types
|
||||
|
||||
| Type | Use For |
|
||||
| ---------- | ----------------------------------------------------- |
|
||||
| `feat` | New CLI feature or major webui feature |
|
||||
| `fix` | Bug fix in CLI or major webui bug fix |
|
||||
| `chore` | Maintenance, upgrades, minor fixes, non-user-facing |
|
||||
| `refactor` | Code restructuring without behavior change |
|
||||
| `docs` | Documentation only (use with `site` scope for site/) |
|
||||
| `test` | Test-only changes (new tests, test fixes, test infra) |
|
||||
| `ci` | CI/CD changes |
|
||||
| `revert` | Revert previous change |
|
||||
| `perf` | Performance improvement |
|
||||
|
||||
**Changelog visibility:** Only `feat`, `fix`, and breaking changes (`!`) appear in release notes. Use `ci`, `chore`, `test`, `docs`, or `refactor` for changes that shouldn't be user-facing.
|
||||
|
||||
**Breaking changes:** Add `!` after scope: `feat(api)!:`, `chore(deps)!:`
|
||||
|
||||
### Test vs Fix
|
||||
|
||||
Use `test:` when the PR **only** contains test changes:
|
||||
|
||||
- Adding new tests
|
||||
- Fixing broken/flaky tests
|
||||
- Fixing lint errors in test files
|
||||
- Test infrastructure changes
|
||||
|
||||
Use `fix:` when fixing bugs in **application code** (even if tests are included):
|
||||
|
||||
- Bug fix in `src/` with accompanying test changes → `fix:`
|
||||
- Lint error in test file only → `test:`
|
||||
|
||||
### Type Selection for Mixed Changes
|
||||
|
||||
Use the **primary change** to determine type:
|
||||
|
||||
| PR Contains | Type | Why |
|
||||
| --------------------------------- | ---------- | -------------------------------- |
|
||||
| Bug fix + new tests | `fix` | Fix is primary, tests support it |
|
||||
| Feature + documentation | `feat` | Feature is primary |
|
||||
| Only test changes | `test` | No application code changed |
|
||||
| Only doc changes | `docs` | No application code changed |
|
||||
| Minor webui fix (styling, typos) | `chore` | Not a major user-facing fix |
|
||||
| Refactor + minor fixes discovered | `refactor` | Refactor was the intent |
|
||||
|
||||
**Major webui changes** = new pages, significant UX changes, core functionality bugs
|
||||
|
||||
**Minor webui changes** = styling tweaks, copy changes, internal refactors → use `chore`
|
||||
|
||||
## Scope Selection (Priority Order)
|
||||
|
||||
### 1. Feature Domains (HIGHEST PRIORITY)
|
||||
|
||||
**`redteam` - MANDATORY when redteam is the PR's primary change or product surface:**
|
||||
|
||||
- Plugins, strategies, grading
|
||||
- UI components (setup, report, config dialogs)
|
||||
- CLI commands, server endpoints
|
||||
- Documentation, examples
|
||||
- Redteam-specific tests, fixtures, utilities, and behavior changes
|
||||
|
||||
**Other feature domains:** `providers`, `assertions`, `eval`, `api`, `db`
|
||||
|
||||
### 2. Product Areas
|
||||
|
||||
- `webui` - React app in `src/app/`
|
||||
- `cli` - CLI in `src/`
|
||||
- `server` - Web server in `src/server/`
|
||||
|
||||
**Note:** Documentation site changes use `docs(site):`, not a standalone `site` scope.
|
||||
|
||||
### 3. Technical/Infrastructure
|
||||
|
||||
- `deps` - Dependency updates
|
||||
- `ci` - CI/CD pipelines, GitHub Actions
|
||||
- `tests` - Test infrastructure
|
||||
- `build` - Build tooling
|
||||
- `examples` - Non-redteam examples
|
||||
|
||||
### 4. Specialized
|
||||
|
||||
`auth`, `cache`, `config`, `python`, `mcp`, `code-scan`
|
||||
|
||||
### 5. No Scope
|
||||
|
||||
For generic/cross-cutting changes: `chore: bump version 0.119.11`
|
||||
|
||||
## THE REDTEAM RULE
|
||||
|
||||
**If a PR is primarily redteam-related, use `(redteam)` scope.**
|
||||
|
||||
This applies even if the redteam change is only in UI, CLI, docs, examples, utilities, tests, or server endpoints.
|
||||
|
||||
For broad, cross-cutting maintenance PRs, do **not** choose `(redteam)` solely because one touched file lives under `src/redteam/` or because one generic helper is also used by redteam. Use the PR's primary purpose/scope and call out the redteam-adjacent touch in the PR description when it is review-relevant.
|
||||
|
||||
❌ **Wrong:**
|
||||
|
||||
```plaintext
|
||||
fix(webui): fix Basic strategy checkbox in red team setup
|
||||
feat(cli): add redteam validate command
|
||||
chore(redteam): resolve repo-wide lint findings
|
||||
```
|
||||
|
||||
✅ **Correct:**
|
||||
|
||||
```plaintext
|
||||
fix(redteam): fix Basic strategy checkbox in setup UI
|
||||
feat(redteam): add validate target CLI command
|
||||
chore: resolve repo-wide lint findings
|
||||
```
|
||||
|
||||
**Why?** Redteam spans CLI, webui, server, docs, and examples. Consistent scoping makes it easy to find all redteam work.
|
||||
|
||||
## Decision Tree
|
||||
|
||||
```plaintext
|
||||
1. Is the PR primarily redteam-related? → Use (redteam)
|
||||
2. Is it another feature domain? → Use that scope
|
||||
3. Is it localized to one product area? → Use that scope
|
||||
4. Is it infrastructure? → Use that scope
|
||||
5. Otherwise → No scope
|
||||
```
|
||||
|
||||
## Dependency Updates
|
||||
|
||||
- **`fix(deps)`** - Patch versions (security/bug fixes)
|
||||
- **`chore(deps)`** - Minor/major upgrades, bulk updates, dev dependencies
|
||||
|
||||
## Examples
|
||||
|
||||
✅ **Good:**
|
||||
|
||||
```plaintext
|
||||
feat(redteam): add FERPA compliance plugin
|
||||
feat(cli): add --json output flag to eval command
|
||||
fix(cli): handle empty config file gracefully
|
||||
fix(webui): fix pagination crash on empty results
|
||||
chore(webui): update button styling on settings page
|
||||
docs(site): add guide for custom providers
|
||||
chore(deps): update Material-UI monorepo to v8 (major)
|
||||
fix(deps): update dependency zod to v4.2.0
|
||||
feat(api)!: simplify provider interface
|
||||
chore: bump version 0.119.11
|
||||
test: add smoke tests for CLI commands
|
||||
test(redteam): fix flaky plugin integration tests
|
||||
```
|
||||
|
||||
❌ **Bad:**
|
||||
|
||||
```plaintext
|
||||
feat: add new redteam thing # Missing (redteam) scope
|
||||
fix(webui): red team checkbox # Should be fix(redteam)
|
||||
chore(webui): update dependency # Should be chore(deps)
|
||||
feat: stuff # Too vague
|
||||
fix: bug fix # What bug? Be specific
|
||||
Fix(cli): Add feature # Wrong case, not imperative
|
||||
fix(test): resolve lint errors # Should be test: (test-only)
|
||||
docs: update site # Should be docs(site):
|
||||
site: update guides # Should be docs(site):
|
||||
feat(webui): minor styling update # Minor = chore, not feat
|
||||
```
|
||||
|
||||
## Draft vs Ready
|
||||
|
||||
Open PRs **ready for review** by default:
|
||||
|
||||
```bash
|
||||
gh pr create --title "feat(scope): description" --body "..."
|
||||
```
|
||||
|
||||
Use `--draft` only when:
|
||||
|
||||
- The user explicitly asks for a draft
|
||||
- The work is an intentional WIP parked for a hand-off
|
||||
- The PR blocks on an external dependency that must land first
|
||||
- The PR addresses an unpublished security advisory (see root `AGENTS.md`
|
||||
"Security-Sensitive PRs")
|
||||
|
||||
## Commit & PR Attribution
|
||||
|
||||
- **Never attribute commits or PR bodies to Claude / Claude Code.** Do not add
|
||||
`Co-Authored-By: Claude…` trailers, "Generated with Claude Code" footers, or
|
||||
similar markers. Use your configured git identity only.
|
||||
- Do not add marketing-style suffixes to commit subjects.
|
||||
|
||||
## GitHub Interaction Rules
|
||||
|
||||
- **NEVER comment on GitHub issues** - Only create PRs to address issues
|
||||
- **NEVER close issues** - Let maintainers close issues after PR merge
|
||||
- Focus on creating high-quality PRs that fully address the issue
|
||||
|
||||
## Checklist Before Creating PR
|
||||
|
||||
1. Is this PR primarily redteam-related? → Use `(redteam)` scope
|
||||
2. Choose correct type
|
||||
3. Choose correct scope using priority order
|
||||
4. Breaking change? Add `!` after scope
|
||||
5. Run `npm run l && npm run f`
|
||||
6. Open ready-for-review (omit `--draft`) unless one of the exceptions above applies
|
||||
7. Do **not** add Claude attribution trailers or footers
|
||||
@@ -0,0 +1,62 @@
|
||||
# Python Guidelines
|
||||
|
||||
For Python providers, prompts, assertions, and scripts.
|
||||
|
||||
## Requirements
|
||||
|
||||
- Python 3.8 or later
|
||||
- Follow [Google Python Style Guide](https://google.github.io/styleguide/pyguide.html)
|
||||
- Use type hints for readability and error catching
|
||||
|
||||
## Linting & Formatting
|
||||
|
||||
Use `ruff` for both:
|
||||
|
||||
```bash
|
||||
ruff check --fix # Lint with auto-fix
|
||||
ruff check --select I --fix # Sort imports
|
||||
ruff format # Format code
|
||||
```
|
||||
|
||||
## Testing
|
||||
|
||||
Use the built-in `unittest` module for new Python tests.
|
||||
|
||||
## Best Practices
|
||||
|
||||
- Keep dependencies minimal - avoid unnecessary external packages
|
||||
- When adding examples, update relevant `requirements.txt` files
|
||||
- Follow promptfoo API patterns for custom providers/prompts/assertions
|
||||
- Write unit tests for new Python functions
|
||||
|
||||
## Provider Pattern
|
||||
|
||||
```python
|
||||
def call_api(prompt: str, options: dict, context: dict) -> dict:
|
||||
"""
|
||||
Args:
|
||||
prompt: The prompt string
|
||||
options: Provider configuration
|
||||
context: Variables from the test case
|
||||
|
||||
Returns:
|
||||
dict with 'output' key (and optionally 'error')
|
||||
"""
|
||||
# Implementation
|
||||
return {"output": response_text}
|
||||
```
|
||||
|
||||
## Assertion Pattern
|
||||
|
||||
```python
|
||||
def get_assert(output: str, context: dict) -> dict:
|
||||
"""
|
||||
Args:
|
||||
output: The provider's output
|
||||
context: Test context including vars
|
||||
|
||||
Returns:
|
||||
dict with 'pass' (bool) and optionally 'reason'
|
||||
"""
|
||||
return {"pass": True, "reason": "Validation passed"}
|
||||
```
|
||||
Reference in New Issue
Block a user