chore: import upstream snapshot with attribution
CI / Shell Format Check (push) Has been cancelled
CI / Check Ruby (3.4) (push) Has been cancelled
CI / CI Config (push) Has been cancelled
CI / Test on Node ${{ matrix.node }} and ${{ matrix.os }}${{ matrix.shard && format(' (shard {0}/3)', matrix.shard) || '' }} (push) Has been cancelled
CI / Build on Node ${{ matrix.node }} (push) Has been cancelled
CI / Style Check (push) Has been cancelled
CI / Generate Assets (push) Has been cancelled
CI / Check Python (3.14) (push) Has been cancelled
CI / Check Python (3.9) (push) Has been cancelled
CI / Build Docs (push) Has been cancelled
CI / Code Scan Action (push) Has been cancelled
CI / Site tests (push) Has been cancelled
CI / webui tests (push) Has been cancelled
CI / Run Integration Tests (push) Has been cancelled
CI / Run Smoke Tests (push) Has been cancelled
CI / Go Tests (push) Has been cancelled
CI / Share Test (push) Has been cancelled
CI / Redteam (Production API) (push) Has been cancelled
CI / Redteam (Staging API) (push) Has been cancelled
CI / GitHub Actions Lint (push) Has been cancelled
CI / Check Ruby (3.0) (push) Has been cancelled
release-please / release-please (push) Has been cancelled
release-please / build (push) Has been cancelled
release-please / publish-npm (push) Has been cancelled
release-please / publish-npm-backfill (push) Has been cancelled
release-please / docker (push) Has been cancelled
release-please / publish-code-scan-action (push) Has been cancelled
release-please / attest-code-scan-action (push) Has been cancelled
Deploy local.promptfoo.app / Deploy to Cloudflare Pages (push) Has been cancelled
Test and Publish Multi-arch Docker Image / test (push) Has been cancelled
Test and Publish Multi-arch Docker Image / build-docker-and-push-digests (map[digest-suffix:linux-amd64 platform:linux/amd64 runner:ubuntu-latest]) (push) Has been cancelled
Test and Publish Multi-arch Docker Image / build-docker-and-push-digests (map[digest-suffix:linux-arm64 platform:linux/arm64 runner:ubuntu-24.04-arm]) (push) Has been cancelled
Test and Publish Multi-arch Docker Image / merge-docker-digests (push) Has been cancelled
Test and Publish Multi-arch Docker Image / Attest Multi-arch Image (push) Has been cancelled
Validate Renovate Config / Validate Renovate Configuration (push) Has been cancelled
CI / Shell Format Check (push) Has been cancelled
CI / Check Ruby (3.4) (push) Has been cancelled
CI / CI Config (push) Has been cancelled
CI / Test on Node ${{ matrix.node }} and ${{ matrix.os }}${{ matrix.shard && format(' (shard {0}/3)', matrix.shard) || '' }} (push) Has been cancelled
CI / Build on Node ${{ matrix.node }} (push) Has been cancelled
CI / Style Check (push) Has been cancelled
CI / Generate Assets (push) Has been cancelled
CI / Check Python (3.14) (push) Has been cancelled
CI / Check Python (3.9) (push) Has been cancelled
CI / Build Docs (push) Has been cancelled
CI / Code Scan Action (push) Has been cancelled
CI / Site tests (push) Has been cancelled
CI / webui tests (push) Has been cancelled
CI / Run Integration Tests (push) Has been cancelled
CI / Run Smoke Tests (push) Has been cancelled
CI / Go Tests (push) Has been cancelled
CI / Share Test (push) Has been cancelled
CI / Redteam (Production API) (push) Has been cancelled
CI / Redteam (Staging API) (push) Has been cancelled
CI / GitHub Actions Lint (push) Has been cancelled
CI / Check Ruby (3.0) (push) Has been cancelled
release-please / release-please (push) Has been cancelled
release-please / build (push) Has been cancelled
release-please / publish-npm (push) Has been cancelled
release-please / publish-npm-backfill (push) Has been cancelled
release-please / docker (push) Has been cancelled
release-please / publish-code-scan-action (push) Has been cancelled
release-please / attest-code-scan-action (push) Has been cancelled
Deploy local.promptfoo.app / Deploy to Cloudflare Pages (push) Has been cancelled
Test and Publish Multi-arch Docker Image / test (push) Has been cancelled
Test and Publish Multi-arch Docker Image / build-docker-and-push-digests (map[digest-suffix:linux-amd64 platform:linux/amd64 runner:ubuntu-latest]) (push) Has been cancelled
Test and Publish Multi-arch Docker Image / build-docker-and-push-digests (map[digest-suffix:linux-arm64 platform:linux/arm64 runner:ubuntu-24.04-arm]) (push) Has been cancelled
Test and Publish Multi-arch Docker Image / merge-docker-digests (push) Has been cancelled
Test and Publish Multi-arch Docker Image / Attest Multi-arch Image (push) Has been cancelled
Validate Renovate Config / Validate Renovate Configuration (push) Has been cancelled
This commit is contained in:
wehub-resource-sync
@@ -0,0 +1,27 @@
|
||||
# Agent Documentation
|
||||
|
||||
Guidance for AI agents editing the reusable docs in `docs/agents/`.
|
||||
|
||||
## Rules
|
||||
|
||||
- When changing PR workflow guidance, update both `docs/agents/pr-conventions.md` and
|
||||
the root `AGENTS.md` "Pull Request Creation" section in the same PR.
|
||||
- When changing dependency workflow guidance, keep
|
||||
`docs/agents/dependency-management.md` aligned with the root npm/audit guidance.
|
||||
- Do not introduce hard-coded local env-file requirements into generic eval commands.
|
||||
Use `--env-file .env` only when credentials are needed and the file exists.
|
||||
- Do not claim that eval exit code 0 means all test cases passed unless the text
|
||||
accounts for `PROMPTFOO_PASS_RATE_THRESHOLD`.
|
||||
- Verify implementation claims against source before documenting paths, defaults, or
|
||||
environment variables, and avoid machine-specific paths.
|
||||
- Prefer concrete command examples over broad advice, and avoid duplicating long
|
||||
guidance from root docs.
|
||||
|
||||
## Validation
|
||||
|
||||
For docs-only edits here, run:
|
||||
|
||||
```bash
|
||||
npm run f
|
||||
git diff --check
|
||||
```
|
||||
@@ -0,0 +1,576 @@
|
||||
# Codex App Server Provider Notes
|
||||
|
||||
These notes track the planned Promptfoo integration for the Codex app-server protocol.
|
||||
They are intentionally implementation-facing: keep them current as the provider, docs,
|
||||
examples, and verification expand.
|
||||
|
||||
For the broader coding-agent provider taxonomy, see
|
||||
[`coding-agent-provider-taxonomy.md`](./coding-agent-provider-taxonomy.md).
|
||||
|
||||
## Objective
|
||||
|
||||
Add an experimental Promptfoo provider that drives `codex app-server` directly. The
|
||||
provider should complement, not replace, the existing OpenAI Codex SDK provider:
|
||||
|
||||
- Codex SDK provider: best default for CI and automation.
|
||||
- Codex app-server provider: best for evaluating rich-client behavior exposed by the
|
||||
Codex app-server protocol, including streamed item events, approvals, skills,
|
||||
plugins, apps, filesystem requests, and thread lifecycle primitives.
|
||||
|
||||
Primary provider IDs:
|
||||
|
||||
- `openai:codex-app-server`
|
||||
- `openai:codex-app-server:<model>`
|
||||
- `openai:codex-desktop`
|
||||
- `openai:codex-desktop:<model>`
|
||||
|
||||
Optional top-level aliases may be added after the OpenAI-scoped provider is stable:
|
||||
|
||||
- `codex:app-server`
|
||||
- `codex:desktop`
|
||||
|
||||
## Source Material
|
||||
|
||||
- Official docs: https://developers.openai.com/codex/app-server
|
||||
- Local CLI: `/Applications/Codex.app/Contents/Resources/codex app-server --help`
|
||||
- Local generated schema command:
|
||||
|
||||
```bash
|
||||
codex app-server generate-ts --out /tmp/codex-app-server-schema/ts
|
||||
codex app-server generate-json-schema --out /tmp/codex-app-server-schema/json
|
||||
```
|
||||
|
||||
Current local schema inspection was generated from `codex-cli 0.118.0`.
|
||||
|
||||
## Protocol Shape
|
||||
|
||||
Transport:
|
||||
|
||||
- `stdio://` default, JSONL messages.
|
||||
- `ws://IP:PORT` experimental, one JSON-RPC message per WebSocket text frame.
|
||||
|
||||
Handshake:
|
||||
|
||||
1. Send `initialize` with Promptfoo client metadata.
|
||||
2. Send `initialized` notification.
|
||||
3. Start or resume a thread.
|
||||
4. Start a turn.
|
||||
5. Read notifications until `turn/completed`.
|
||||
|
||||
Core client requests:
|
||||
|
||||
- `initialize`
|
||||
- `thread/start`
|
||||
- `thread/resume`
|
||||
- `thread/archive`
|
||||
- `thread/unsubscribe`
|
||||
- `thread/read`
|
||||
- `turn/start`
|
||||
- `turn/steer`
|
||||
- `turn/interrupt`
|
||||
- `review/start`
|
||||
- `model/list`
|
||||
- `skills/list`
|
||||
- `plugin/list`
|
||||
- `plugin/read`
|
||||
- `app/list`
|
||||
|
||||
High-risk client requests that should not be exposed casually:
|
||||
|
||||
- `fs/writeFile`
|
||||
- `fs/remove`
|
||||
- `fs/copy`
|
||||
- `config/value/write`
|
||||
- `config/batchWrite`
|
||||
- `plugin/install`
|
||||
- `plugin/uninstall`
|
||||
- `command/exec`
|
||||
|
||||
Core server notifications:
|
||||
|
||||
- `thread/started`
|
||||
- `thread/status/changed`
|
||||
- `turn/started`
|
||||
- `turn/completed`
|
||||
- `item/started`
|
||||
- `item/completed`
|
||||
- `item/agentMessage/delta`
|
||||
- `item/commandExecution/outputDelta`
|
||||
- `item/fileChange/outputDelta`
|
||||
- `item/mcpToolCall/progress`
|
||||
- `serverRequest/resolved`
|
||||
- `thread/tokenUsage/updated`
|
||||
- `error`
|
||||
|
||||
Core server requests requiring deterministic Promptfoo responses:
|
||||
|
||||
- `item/commandExecution/requestApproval`
|
||||
- `item/fileChange/requestApproval`
|
||||
- `item/permissions/requestApproval`
|
||||
- `item/tool/requestUserInput`
|
||||
- `mcpServer/elicitation/request`
|
||||
- `item/tool/call`
|
||||
|
||||
## Provider Contract
|
||||
|
||||
### Inputs
|
||||
|
||||
Promptfoo prompt strings remain the default. The provider should also accept a JSON
|
||||
array of Codex input items:
|
||||
|
||||
```json
|
||||
[
|
||||
{ "type": "text", "text": "Review this project" },
|
||||
{ "type": "local_image", "path": "/absolute/path/to/screenshot.png" },
|
||||
{ "type": "skill", "name": "skill-creator", "path": "/absolute/path/SKILL.md" }
|
||||
]
|
||||
```
|
||||
|
||||
Supported input item types for the first implementation:
|
||||
|
||||
- `text`
|
||||
- `local_image`, mapped to app-server `inputImage`
|
||||
- `skill`
|
||||
|
||||
Unknown prompt JSON should be treated as plain text instead of throwing.
|
||||
|
||||
### Output
|
||||
|
||||
The provider response should include:
|
||||
|
||||
- `output`: final assistant text, assembled from `item/agentMessage/delta` and
|
||||
completed `agentMessage` items.
|
||||
- `sessionId`: thread id.
|
||||
- `raw`: serialized protocol-level turn summary and selected notifications.
|
||||
- `metadata.codexAppServer`: thread id, turn id, model, cwd, sandbox, approvals,
|
||||
server requests, item counts, command/file/tool trajectories, and app-server
|
||||
protocol data useful for debugging.
|
||||
- `metadata.skillCalls` / `metadata.attemptedSkillCalls`: heuristic skill usage
|
||||
where available.
|
||||
- `tokenUsage`: from `thread/tokenUsage/updated` if emitted.
|
||||
- `cost`: estimated from Promptfoo's Codex pricing table when model is known.
|
||||
|
||||
### Config
|
||||
|
||||
Provider-level config should be strict. Prompt-level merged config should strip unknown
|
||||
keys so generic Promptfoo prompt config does not break rows.
|
||||
|
||||
Core config:
|
||||
|
||||
- `apiKey`
|
||||
- `base_url`
|
||||
- `working_dir`
|
||||
- `additional_directories`
|
||||
- `skip_git_repo_check`
|
||||
- `codex_path_override`
|
||||
- `model`
|
||||
- `model_provider`
|
||||
- `service_tier`
|
||||
- `sandbox_mode`
|
||||
- `sandbox_policy`
|
||||
- `approval_policy`
|
||||
- `approvals_reviewer`
|
||||
- `model_reasoning_effort`
|
||||
- `reasoning_summary`
|
||||
- `personality`
|
||||
- `output_schema`
|
||||
- `thread_id`
|
||||
- `persist_threads`
|
||||
- `thread_pool_size`
|
||||
- `ephemeral`
|
||||
- `persist_extended_history`
|
||||
- `experimental_raw_events`
|
||||
- `experimental_api`
|
||||
- `cli_config`
|
||||
- `cli_env`
|
||||
- `inherit_process_env`
|
||||
- `reuse_server`
|
||||
- `deep_tracing`
|
||||
- `request_timeout_ms`
|
||||
- `startup_timeout_ms`
|
||||
- `server_request_policy`
|
||||
|
||||
### Safety Defaults
|
||||
|
||||
Default stance should favor repeatable evals over convenience:
|
||||
|
||||
- `approval_policy`: `never`
|
||||
- `sandbox_mode`: `read-only`
|
||||
- `network_access_enabled`: `false`
|
||||
- `ephemeral`: `true`
|
||||
- `reuse_server`: `true` unless `deep_tracing` is enabled
|
||||
- `inherit_process_env`: `false`
|
||||
- Server-side approval requests: decline/cancel or empty grants unless explicitly
|
||||
configured.
|
||||
|
||||
Rationale:
|
||||
|
||||
- The app-server exposes shell, filesystem, app connector, plugin, and config surfaces.
|
||||
- Promptfoo evals should be deterministic and should not block on human approval.
|
||||
- Eval prompts and target behavior can be adversarial.
|
||||
|
||||
## Implementation Phases
|
||||
|
||||
1. Stdio JSON-RPC client
|
||||
- Spawn `codex app-server`.
|
||||
- Parse JSONL stdout.
|
||||
- Route responses, notifications, and server requests.
|
||||
- Capture stderr for debug logs.
|
||||
- Support abort and timeout.
|
||||
|
||||
2. Provider lifecycle
|
||||
- Register with `providerRegistry`.
|
||||
- Reuse app-server process by default.
|
||||
- Shut down child processes, pending requests, and readline handles.
|
||||
- Disable reuse when `deep_tracing` is enabled.
|
||||
|
||||
3. Thread and turn execution
|
||||
- Validate working directory.
|
||||
- Start/resume threads.
|
||||
- Start turns with prompt input, model, cwd, sandbox, approvals, effort, personality,
|
||||
service tier, and output schema.
|
||||
- Serialize turns per reused thread.
|
||||
- Unsubscribe/archive non-persistent threads during cleanup.
|
||||
|
||||
4. Streaming aggregation
|
||||
- Track `turnId`.
|
||||
- Assemble assistant deltas.
|
||||
- Store completed items.
|
||||
- Build item counts and trajectory metadata.
|
||||
- Capture command output, file changes, MCP calls, dynamic tool calls, web search,
|
||||
plans, reasoning summaries, and review output.
|
||||
|
||||
5. Server request handling
|
||||
- Deterministically answer command approvals.
|
||||
- Deterministically answer file-change approvals.
|
||||
- Return empty permission grants by default.
|
||||
- Support configured answers for `tool/requestUserInput`.
|
||||
- Decline/cancel MCP elicitation by default.
|
||||
- Support static dynamic-tool responses.
|
||||
- Record all requests and decisions in metadata.
|
||||
|
||||
6. Tracing
|
||||
- Wrap `callApi` in `withGenAISpan`.
|
||||
- Add item-level spans when streaming notifications arrive.
|
||||
- Sanitize command output, tool arguments, and message text before trace attributes.
|
||||
- Inject OTEL env when `deep_tracing` is enabled.
|
||||
|
||||
7. Docs and examples
|
||||
- Add provider docs.
|
||||
- Add provider index entry.
|
||||
- Add examples for basic usage, read-only repo review, structured output, approval
|
||||
handling, skills, and tracing.
|
||||
|
||||
8. Verification
|
||||
- Unit tests with mocked child process and mocked protocol frames.
|
||||
- Registry tests for provider IDs and model-in-path parsing.
|
||||
- Docs/examples lint where applicable.
|
||||
- Local smoke config using a harmless prompt and `sandbox_mode: read-only` if
|
||||
credentials/login are available.
|
||||
- Final dogfood: run the new provider against the git diff and iterate on comments.
|
||||
|
||||
## Progress Log
|
||||
|
||||
### 2026-04-09
|
||||
|
||||
- Added initial provider implementation at `src/providers/openai/codex-app-server.ts`.
|
||||
- Added provider IDs under the OpenAI registry:
|
||||
- `openai:codex-app-server`
|
||||
- `openai:codex-app-server:<model>`
|
||||
- `openai:codex-desktop`
|
||||
- `openai:codex-desktop:<model>`
|
||||
- Implemented stdio JSON-RPC lifecycle:
|
||||
- spawn `codex app-server --listen stdio://`
|
||||
- `initialize`
|
||||
- `initialized`
|
||||
- `thread/start`
|
||||
- `thread/resume`
|
||||
- `turn/start`
|
||||
- notification handling through `turn/completed`
|
||||
- `thread/unsubscribe`/`thread/archive` cleanup modes
|
||||
- Implemented safe config defaults:
|
||||
- `approval_policy: never`
|
||||
- `sandbox_mode: read-only`
|
||||
- `ephemeral: true`
|
||||
- `thread_cleanup: unsubscribe`
|
||||
- process env isolation unless `inherit_process_env: true`
|
||||
- Implemented deterministic server request responses:
|
||||
- command execution approvals default to `decline`
|
||||
- file changes default to `decline`
|
||||
- permission requests default to empty grants
|
||||
- user input requests default to empty answers
|
||||
- MCP elicitations default to `decline`
|
||||
- dynamic tools can use static configured responses
|
||||
- Implemented output normalization:
|
||||
- assistant delta aggregation
|
||||
- completed `agentMessage` fallback/preference
|
||||
- token usage from `thread/tokenUsage/updated`
|
||||
- cost estimate for known Codex models
|
||||
- metadata with item counts, items, server request decisions, thread/turn ids
|
||||
- Implemented provider-level GenAI tracing and item spans.
|
||||
- Added mocked protocol tests in `test/providers/openai-codex-app-server.test.ts`.
|
||||
- Added registry tests in `test/providers/index.test.ts`.
|
||||
- Verification so far:
|
||||
- `npx vitest run test/providers/openai-codex-app-server.test.ts --sequence.shuffle=false`
|
||||
- `npx vitest run test/providers/index.test.ts -t "Codex app-server|Codex desktop" --sequence.shuffle=false`
|
||||
- `npm run tsc -- --pretty false`
|
||||
- Expanded mocked protocol tests to cover:
|
||||
- `thread/resume`
|
||||
- structured prompt input normalization
|
||||
- default `thread/unsubscribe` cleanup
|
||||
- user input request policy
|
||||
- dynamic tool static response policy
|
||||
- metadata sanitization
|
||||
- Ran a real local smoke eval through `npm run local -- eval -c examples/openai-codex-app-server/promptfooconfig.yaml --no-cache -o /tmp/promptfoo-codex-app-server-example.json`.
|
||||
- Result: pass.
|
||||
- Provider returned Codex app-server `sessionId`, token usage, item counts, thread id,
|
||||
turn id, and structured JSON output.
|
||||
- Ran docs build:
|
||||
- `cd site && SKIP_OG_GENERATION=true npm run build`
|
||||
- Result: pass.
|
||||
- First dogfood review through `examples/openai-codex-app-server/review-diff/promptfooconfig.yaml`
|
||||
found four actionable provider issues:
|
||||
- startup timeout could leak a spawned app-server and leave a rejected reusable
|
||||
connection promise cached
|
||||
- reused connections closed over the first turn's server request policy
|
||||
- app-server exit during a turn could leave the eval waiting forever when no turn
|
||||
timeout was configured
|
||||
- `raw` response payload serialized unsanitized protocol items
|
||||
- Fixed the dogfood findings and added regression coverage:
|
||||
- failed startup closes the process and a later call spawns a fresh process
|
||||
- active turns store their effective config so prompt-level server request policies are
|
||||
honored on reused servers
|
||||
- connection exit resolves active turns with a provider error and removes the dead
|
||||
connection from reuse maps
|
||||
- `raw` now contains sanitized thread, turn, token usage, notifications, and item
|
||||
metadata
|
||||
- final output now uses the last completed `agentMessage`, which avoids concatenating
|
||||
progress messages with final structured review output
|
||||
- Verification after fixes:
|
||||
- `npm run tsc -- --pretty false`
|
||||
- `npx vitest run test/providers/openai-codex-app-server.test.ts --sequence.shuffle=false`
|
||||
- Result: pass, 13 provider tests.
|
||||
- Second dogfood review passed the Promptfoo eval and returned valid JSON, but still
|
||||
reported two provider comments:
|
||||
- legacy `execCommandApproval` / `applyPatchApproval` requests identify the active
|
||||
thread with `conversationId` and expect legacy review decisions
|
||||
- persistent thread-pool eviction deleted local handles without unsubscribing the
|
||||
evicted loaded thread
|
||||
- Additional hardening from the second dogfood pass:
|
||||
- stdio parser now buffers partial JSON-RPC lines and rejoins literal newlines inside
|
||||
command-output strings as escaped newlines before parsing
|
||||
- legacy approval requests now map prompt-level policy to `approved`,
|
||||
`approved_for_session`, `denied`, and `abort`
|
||||
- legacy server requests can find active turn state by `conversationId`
|
||||
- evicted persistent cached threads now send `thread/unsubscribe` before being removed
|
||||
- added regression coverage for literal-newline JSON-RPC notifications, legacy
|
||||
approval requests, and persistent thread-pool eviction
|
||||
- Verification after second dogfood fixes:
|
||||
- `npm run tsc -- --pretty false`
|
||||
- `npx vitest run test/providers/openai-codex-app-server.test.ts --sequence.shuffle=false`
|
||||
- Result: pass, 16 provider tests.
|
||||
- Third dogfood review passed transport/eval and reported two lifecycle comments:
|
||||
- stale persistent thread handles remained after a reused app-server process exited
|
||||
- JSON-RPC request timeout cleanup removed the pending request but left an abort
|
||||
listener attached
|
||||
- Additional hardening from the third dogfood pass:
|
||||
- connection close now removes cached thread handles owned by that connection key
|
||||
- per-request timeout cleanup now removes abort listeners before rejecting
|
||||
- added regression coverage for cached-thread invalidation after process exit and
|
||||
abort-listener cleanup on JSON-RPC timeout
|
||||
- Verification after third dogfood fixes:
|
||||
- `npm run tsc -- --pretty false`
|
||||
- `npx vitest run test/providers/openai-codex-app-server.test.ts --sequence.shuffle=false`
|
||||
- Result: pass, 18 provider tests.
|
||||
- Fourth dogfood review passed transport/eval and reported two thread-cache comments:
|
||||
- persistent thread caching was still enabled for fresh-per-call app-server processes
|
||||
(`reuse_server: false` or `deep_tracing`)
|
||||
- pool eviction could unsubscribe an active cached thread before its turn completed
|
||||
- Additional hardening from the fourth dogfood pass:
|
||||
- thread caching is now allowed only when the app-server connection itself is reusable
|
||||
- active/reserved thread ids are protected with a small refcount while a call is using
|
||||
them
|
||||
- thread-pool eviction skips protected threads and temporarily allows the pool to
|
||||
exceed its soft cap rather than evicting an in-flight turn
|
||||
- added regression coverage for non-reusable persistent-thread configs and active-turn
|
||||
eviction avoidance
|
||||
- Verification after fourth dogfood fixes:
|
||||
- `npm run tsc -- --pretty false`
|
||||
- `npx vitest run test/providers/openai-codex-app-server.test.ts --sequence.shuffle=false`
|
||||
- Result: pass, 20 provider tests.
|
||||
- Fifth dogfood review passed transport/eval and reported one persistent-thread race:
|
||||
- concurrent calls with the same persistent-thread cache key could both miss the cache
|
||||
while the first `thread/start` was still pending, creating duplicate persistent
|
||||
threads and leaking the earlier one
|
||||
- Additional hardening from the fifth dogfood pass:
|
||||
- added an in-flight thread promise map keyed by thread cache key
|
||||
- concurrent same-cache `thread/start` / `thread/resume` callers now share the same
|
||||
pending thread handle
|
||||
- added regression coverage for concurrent same-cache persistent calls, ensuring only
|
||||
one `thread/start` is sent and both turns use the shared thread
|
||||
- Verification after fifth dogfood fix:
|
||||
- `npm run tsc -- --pretty false`
|
||||
- `npx vitest run test/providers/openai-codex-app-server.test.ts --sequence.shuffle=false`
|
||||
- Result: pass, 21 provider tests.
|
||||
- Sixth dogfood review passed transport/eval and reported two cache/default comments:
|
||||
- reusable connections could keep the first request timeout for requests that did not
|
||||
pass a per-call timeout
|
||||
- persistent thread cache keys omitted thread-start options such as `ephemeral`,
|
||||
`experimental_raw_events`, and `persist_extended_history`
|
||||
- Additional hardening from the sixth dogfood pass:
|
||||
- all provider-owned app-server requests now pass the effective per-call request
|
||||
timeout explicitly
|
||||
- persistent thread cache keys now include thread-start options that can change thread
|
||||
semantics
|
||||
- added regression coverage for prompt-level request timeouts on reused connections and
|
||||
thread-start option changes in persistent cache keys
|
||||
- Verification after sixth dogfood fixes:
|
||||
- `npm run tsc -- --pretty false`
|
||||
- `npx vitest run test/providers/openai-codex-app-server.test.ts --sequence.shuffle=false`
|
||||
- Result: pass, 23 provider tests.
|
||||
- Seventh dogfood retry:
|
||||
- first attempt hit an external Codex connectivity failure (`Network is unreachable`,
|
||||
`Reconnecting... 2/5`), which the provider surfaced as a clean provider error
|
||||
- retry completed transport/eval and reported one metadata issue: skill-root detection
|
||||
used the parent process env instead of the resolved app-server child env
|
||||
- Additional hardening from the seventh dogfood pass:
|
||||
- turn state now carries the resolved app-server environment produced by
|
||||
`prepareEnvironment`
|
||||
- skill root detection now uses the child env for `CODEX_HOME`, `HOME`, and
|
||||
`USERPROFILE`
|
||||
- added regression coverage for `cli_env.HOME` skill-call metadata detection
|
||||
- Verification after seventh dogfood fix:
|
||||
- `npm run tsc -- --pretty false`
|
||||
- `npx vitest run test/providers/openai-codex-app-server.test.ts --sequence.shuffle=false`
|
||||
- Result: pass, 24 provider tests.
|
||||
- Eighth dogfood review:
|
||||
- `npm run local -- eval -c examples/openai-codex-app-server/review-diff/promptfooconfig.yaml --no-cache --no-share -o /tmp/promptfoo-codex-app-server-review.json`
|
||||
- Result: pass.
|
||||
- Provider output: `{"comments":[],"summary":"No actionable findings; TypeScript and focused provider tests passed."}`
|
||||
- This confirms the provider can be used to review the current git diff and return
|
||||
schema-valid JSON with no remaining actionable comments from the dogfood reviewer.
|
||||
- Final verification sweep:
|
||||
- `npm run f`: pass with existing complexity warnings only; no formatting changes needed
|
||||
- `npm run tsc -- --pretty false`: pass
|
||||
- `npx vitest run test/providers/openai-codex-app-server.test.ts --sequence.shuffle=false`:
|
||||
pass, 24 provider tests
|
||||
- `npx vitest run test/providers/index.test.ts -t "Codex app-server|Codex desktop" --sequence.shuffle=false`:
|
||||
pass, 2 registry tests
|
||||
- `npm run l`: pass with existing complexity warnings only
|
||||
- `cd site && SKIP_OG_GENERATION=true npm run build`: pass
|
||||
- `npm run local -- eval -c examples/openai-codex-app-server/promptfooconfig.yaml --no-cache --no-share -o /tmp/promptfoo-codex-app-server-example.json`:
|
||||
pass
|
||||
|
||||
## QA Matrix
|
||||
|
||||
Required mocked unit tests:
|
||||
|
||||
- Constructor defaults and strict config validation.
|
||||
- Prompt-level unknown config stripping.
|
||||
- Missing/inaccessible/non-directory working directory.
|
||||
- Git check and `skip_git_repo_check`.
|
||||
- API key/env isolation and explicit `cli_env`.
|
||||
- Handshake order: `initialize`, `initialized`, `thread/start`, `turn/start`.
|
||||
- Model from provider path overrides/defaults correctly.
|
||||
- `thread_id` uses `thread/resume`.
|
||||
- `persist_threads` reuses cached thread and serializes turns.
|
||||
- Non-persistent calls unsubscribe/archive as configured.
|
||||
- Assistant deltas aggregate into final output.
|
||||
- Completed `agentMessage` fallback works when deltas are missing.
|
||||
- Token usage from `thread/tokenUsage/updated`.
|
||||
- Error notification produces provider error.
|
||||
- Failed turn produces provider error.
|
||||
- Abort before start.
|
||||
- Abort during turn sends `turn/interrupt` and returns aborted error.
|
||||
- Command approval request default decline.
|
||||
- File change request default decline.
|
||||
- Permission request default empty grant.
|
||||
- User input request configured answers.
|
||||
- Dynamic tool call configured static response.
|
||||
- MCP elicitation default decline/cancel.
|
||||
- Metadata contains item counts, trajectories, approvals, raw notifications, and server
|
||||
request decisions without leaking API keys.
|
||||
- `cleanup` kills child process and unregisters provider.
|
||||
- `deep_tracing` injects OTEL env and disables reuse/thread persistence.
|
||||
- Provider-level GenAI tracing records response body, token usage, session id, and item
|
||||
count attributes.
|
||||
|
||||
Required docs/examples checks:
|
||||
|
||||
- Provider docs render in Docusaurus.
|
||||
- Examples are listed and runnable from repo root with `npm run local -- eval ... --no-cache`.
|
||||
- Config docs call out experimental status, safety defaults, and difference from Codex SDK.
|
||||
|
||||
## Open Questions
|
||||
|
||||
- Whether to expose WebSocket transport in the first public version. Stdio is enough for
|
||||
Promptfoo-managed app-server processes; WebSocket is useful for external clients but
|
||||
adds auth and lifecycle complexity.
|
||||
- Whether to support top-level `codex:*` aliases immediately or keep all new IDs under
|
||||
`openai:*` for consistency with the existing Codex SDK provider.
|
||||
- Whether to persist generated app-server protocol types in source. The current plan is
|
||||
to implement a narrow local type surface and document how to regenerate schemas instead
|
||||
of committing a large generated bundle.
|
||||
|
||||
## Critical Audit Follow-up
|
||||
|
||||
Review feedback and red-team audit items addressed after the initial dogfood pass:
|
||||
|
||||
- Fixed registry env propagation for object-shaped provider configs. `loadApiProvider`
|
||||
already merges suite-level and provider-level env into `providerOptions.env`; the
|
||||
registry now passes that merged env to `OpenAICodexAppServerProvider`.
|
||||
- Fixed `service_tier` to match the generated app-server schema from `codex-cli 0.118.0`:
|
||||
`fast` and `flex` only.
|
||||
- Reset the hoisted `spawn` mock implementation in `beforeEach` to satisfy `test/AGENTS.md`
|
||||
mock isolation rules.
|
||||
- Regenerated app-server TypeScript and JSON Schema into
|
||||
`/tmp/codex-app-server-schema-current.XVTCwL` during the audit and compared rare
|
||||
app-server fields against the implementation.
|
||||
- Added coverage for schema-supported rare fields:
|
||||
- `model_reasoning_effort: none`
|
||||
- exact `personality` values: `none`, `friendly`, `pragmatic`
|
||||
- granular approval policy objects
|
||||
- app-server command approval amendment objects
|
||||
- session-scoped permission grants
|
||||
- accepted MCP elicitation responses with content and metadata
|
||||
- `base_instructions`, `developer_instructions`, and `collaboration_mode`
|
||||
- Dogfood review then found additional issues:
|
||||
- reusable app-server connections stayed alive after JSON-RPC request timeouts, which
|
||||
could leave late side-effecting responses unmanaged
|
||||
- docs listed `thread_pool_size` as unlimited even though the implementation defaults
|
||||
to `1`
|
||||
- provider cleanup cleared active turns before resolving them, which could hang
|
||||
shutdown while a turn was in flight
|
||||
- raw JSON-RPC notifications were retained even when `include_raw_events` was false
|
||||
- spawned app-server processes could be missed if cleanup ran while `initialize` was
|
||||
still pending
|
||||
- concurrent persistent thread starts could temporarily exceed `thread_pool_size` and
|
||||
remain over capacity after active turns finished
|
||||
- deep-tracing calls that shared a `thread_id` could overlap turns because the queue key
|
||||
returned early
|
||||
- the OpenAI provider docs heading change would have broken the existing `#codex-sdk`
|
||||
anchor
|
||||
- default `thread_id` resumes skipped unsubscribe cleanup
|
||||
- sent JSON-RPC request aborts kept the reusable app-server alive
|
||||
- retryable app-server `error` notifications with `willRetry: true` were treated as
|
||||
terminal
|
||||
- concurrent default-cleanup `thread_id` rows could unsubscribe while another row was
|
||||
queued for the same thread
|
||||
- Fixed these by closing/evicting connections on timeout and abort, resolving active
|
||||
turns during cleanup, tracking pending initialization processes, making raw event
|
||||
retention opt-in, rebalancing the persistent thread pool after turns finish, serializing
|
||||
explicit `thread_id` turns even under deep tracing, preserving the OpenAI docs
|
||||
`#codex-sdk` heading, default-unsubscribing non-persistent resumed threads, honoring
|
||||
retryable app-server errors, and deferring resumed-thread unsubscribe until no other
|
||||
protected queued caller remains.
|
||||
- Updated docs to explain why the app-server provider should stay separate from the
|
||||
Codex SDK provider: the SDK is the right default for CI and automation, while app-server
|
||||
is for rich-client protocol event surfaces and does not attach to a running Codex
|
||||
Desktop app.
|
||||
|
||||
Latest focused verification after these fixes:
|
||||
|
||||
- `npx vitest run test/providers/openai-codex-app-server.test.ts --sequence.shuffle=false`:
|
||||
pass, 35 provider tests.
|
||||
- `npm run local -- eval -c examples/openai-codex-app-server/review-diff/promptfooconfig.yaml --no-cache --no-share`:
|
||||
pass with `{"comments":[],"summary":"No actionable issues found in the current diff."}`.
|
||||
@@ -0,0 +1,565 @@
|
||||
# Coding Agent Provider Taxonomy
|
||||
|
||||
This document summarizes how promptfoo should think about coding-agent providers,
|
||||
what has been implemented so far, and what should come next. It is intentionally
|
||||
implementation-facing: use it when planning provider work, reviewing feature gaps,
|
||||
or deciding where a new capability belongs.
|
||||
|
||||
## Scope
|
||||
|
||||
This taxonomy covers providers that run an agentic coding runtime, not ordinary
|
||||
single-turn model APIs. A coding-agent provider usually has some combination of:
|
||||
|
||||
- A workspace or project directory.
|
||||
- Tool use for files, shell commands, MCP, search, or app connectors.
|
||||
- A session, thread, or server lifecycle.
|
||||
- Permission, sandbox, or approval controls.
|
||||
- Rich metadata beyond final assistant text.
|
||||
|
||||
The main providers in this family today are:
|
||||
|
||||
| Provider family | Provider IDs | Runtime boundary |
|
||||
| ----------------------- | ----------------------------------------------------- | ------------------------------------------ |
|
||||
| OpenAI Codex SDK | `openai:codex-sdk`, `openai:codex` | `@openai/codex-sdk` library |
|
||||
| OpenAI Codex app-server | `openai:codex-app-server`, `openai:codex-desktop` | Local `codex app-server` JSON-RPC process |
|
||||
| Claude Agent SDK | `anthropic:claude-agent-sdk`, `anthropic:claude-code` | `@anthropic-ai/claude-agent-sdk` library |
|
||||
| OpenCode SDK | `opencode:sdk`, `opencode` | OpenCode SDK plus local or existing server |
|
||||
|
||||
Standard OpenAI, Anthropic, Bedrock, Azure, and other model providers still matter
|
||||
for grading and comparison, but they are outside this taxonomy unless they expose a
|
||||
stateful coding-agent runtime.
|
||||
|
||||
## Taxonomy Axes
|
||||
|
||||
### 1. Runtime Boundary
|
||||
|
||||
The first question is where promptfoo stops and the agent runtime starts.
|
||||
|
||||
| Boundary | Meaning | Current examples |
|
||||
| ------------------------ | ------------------------------------------------------------- | ---------------------------------------- |
|
||||
| In-process SDK | promptfoo calls a package API directly. | Codex SDK, Claude Agent SDK |
|
||||
| Managed local server | promptfoo starts a server, then talks to it through a client. | OpenCode when `baseUrl` is unset |
|
||||
| Existing server | promptfoo connects to a runtime it does not configure. | OpenCode with `baseUrl` |
|
||||
| Local app-server process | promptfoo starts a rich-client protocol server over stdio. | Codex app-server |
|
||||
| Desktop UI process | Human-facing native app process. | Codex Desktop app, not directly attached |
|
||||
|
||||
This distinction matters because it controls what promptfoo can guarantee. If
|
||||
promptfoo starts the runtime, it can set env vars, working directories, sandbox
|
||||
options, tracing, and cleanup behavior. If promptfoo attaches to an existing server,
|
||||
that server owns authentication, installed tools, app connectors, and runtime state.
|
||||
|
||||
### 2. Session and Thread State
|
||||
|
||||
Coding agents are rarely stateless. Each provider needs explicit semantics for:
|
||||
|
||||
- Ephemeral sessions: safe default for independent eval rows.
|
||||
- Persistent sessions: useful for memory, multi-turn tasks, or regression suites.
|
||||
- User-supplied sessions: resume an existing thread/session by id.
|
||||
- Pooling: preserve concurrency without cross-contaminating rows.
|
||||
- Cleanup: unsubscribe, archive, delete temporary directories, or leave state alone.
|
||||
|
||||
The important design rule is that session reuse must be opt-in or very clearly
|
||||
scoped. Reusing state silently makes eval results order-dependent.
|
||||
|
||||
### 3. Workspace and Side Effects
|
||||
|
||||
Agent evals should separate filesystem access, network access, and shell access.
|
||||
Those are different risks.
|
||||
|
||||
| Surface | Safe default | Higher-risk mode |
|
||||
| ----------- | ------------------------------------------ | ------------------------------------------------------- |
|
||||
| Filesystem | Temporary directory or read-only workspace | Workspace write or full filesystem access |
|
||||
| Shell | Disabled or approval-gated | Allowed command execution |
|
||||
| Network | Disabled unless explicitly requested | Host allow-lists, live web/search, package installs |
|
||||
| App/plugin | Not installed or not invoked by default | App connectors, plugin installs, config writes |
|
||||
| Environment | Minimal env | Inherited process env with secrets and local auth state |
|
||||
|
||||
Provider docs should make clear that `danger-full-access` is not the same as
|
||||
network access, and read-only filesystem mode does not automatically sanitize env
|
||||
vars. Each surface should have its own option and its own tests.
|
||||
|
||||
### 4. Permission and Interaction Model
|
||||
|
||||
Promptfoo evals are non-interactive by default. Agent runtimes often expect a human
|
||||
to answer approval prompts, permission requests, or clarification questions.
|
||||
Providers should convert those into deterministic policies.
|
||||
|
||||
Common policy categories:
|
||||
|
||||
- Command approval.
|
||||
- File-change approval.
|
||||
- Permission grants.
|
||||
- User-input or ask-user-question tools.
|
||||
- MCP elicitation.
|
||||
- Dynamic tool calls.
|
||||
- Plugin or app connector requests.
|
||||
|
||||
Default policy should decline, cancel, or return empty answers unless a config opts
|
||||
into side effects. Every accepted side effect should be visible in metadata.
|
||||
|
||||
### 5. Inputs
|
||||
|
||||
The baseline input is a prompt string. Coding-agent providers increasingly need
|
||||
structured inputs:
|
||||
|
||||
- Text items.
|
||||
- Local images or image URLs.
|
||||
- Skills or plugin references.
|
||||
- Mentions/app connector references.
|
||||
- File, diff, or workspace context.
|
||||
|
||||
Provider-specific JSON input arrays are acceptable when the underlying runtime has
|
||||
typed input items. Unknown JSON shapes should usually degrade to plain text rather
|
||||
than crashing an eval row, unless the provider docs promise strict input parsing.
|
||||
|
||||
### 6. Outputs and Metadata
|
||||
|
||||
All coding-agent providers should return a normal promptfoo provider result:
|
||||
|
||||
- `output`: final assistant-facing text.
|
||||
- `sessionId`: session/thread id when available.
|
||||
- `tokenUsage`: runtime usage when available.
|
||||
- `cost`: estimate when usage and model pricing are known.
|
||||
- `metadata`: normalized agent metadata.
|
||||
- `raw`: raw or summarized protocol data when useful and safe.
|
||||
|
||||
Provider-specific metadata is still valuable, but consumers need a shared shape for
|
||||
cross-provider assertions and dashboards. A future shared schema should include:
|
||||
|
||||
- Runtime family and version.
|
||||
- Workspace and sandbox settings.
|
||||
- Session/thread/turn identifiers.
|
||||
- Tool trajectories.
|
||||
- Approval decisions.
|
||||
- File changes and command executions.
|
||||
- MCP and dynamic tool calls.
|
||||
- Skill/plugin/app connector usage.
|
||||
- Trace ids and span links.
|
||||
|
||||
### 7. Observability
|
||||
|
||||
Tracing should answer two questions:
|
||||
|
||||
- What did the model decide?
|
||||
- What did the agent runtime do?
|
||||
|
||||
Provider tracing should include the top-level `callApi` span, item/tool-level spans
|
||||
where possible, and sanitized attributes for prompts, commands, tool inputs, file
|
||||
paths, and outputs. Deep tracing should be opt-in when it requires injecting
|
||||
OpenTelemetry env vars into a child process.
|
||||
|
||||
## What Is Implemented So Far
|
||||
|
||||
### Shared Building Blocks
|
||||
|
||||
The coding-agent providers already share several practical patterns:
|
||||
|
||||
- Optional dependencies are loaded lazily so normal promptfoo installs do not need
|
||||
every agent SDK.
|
||||
- Working directories are validated or created before the agent call.
|
||||
- Temporary workspaces are cleaned up after evals.
|
||||
- Session or thread caches are keyed by provider config.
|
||||
- Provider-level config is stricter than prompt-level merged config.
|
||||
- Tool and skill usage are surfaced through metadata where possible.
|
||||
- Tracing is supported for Codex and is partially shared through OpenAI agent
|
||||
tracing helpers.
|
||||
|
||||
Useful files:
|
||||
|
||||
- `src/providers/agentic-utils.ts`
|
||||
- `src/providers/claude-agent-sdk.ts`
|
||||
- `src/providers/opencode-sdk.ts`
|
||||
- `src/providers/openai/codex-sdk.ts`
|
||||
- `src/providers/openai/codex-app-server.ts`
|
||||
- `src/providers/registry.ts`
|
||||
|
||||
### OpenAI Codex SDK
|
||||
|
||||
Status: implemented and documented.
|
||||
|
||||
Provider IDs:
|
||||
|
||||
- `openai:codex-sdk`
|
||||
- `openai:codex-sdk:<model>`
|
||||
- `openai:codex`
|
||||
- `openai:codex:<model>`
|
||||
|
||||
Implemented capabilities:
|
||||
|
||||
- Lazy loading for `@openai/codex-sdk`.
|
||||
- API-key and local Codex login authentication paths.
|
||||
- Working directory and Git repository safety checks.
|
||||
- Sandbox, network, web search, approval, and reasoning controls.
|
||||
- Thread resume and persistent thread pooling.
|
||||
- JSON schema output.
|
||||
- Text and local-image input items.
|
||||
- Skill usage heuristics from `SKILL.md` reads.
|
||||
- Token usage and cost estimation for known Codex models.
|
||||
- Streaming aggregation for metadata and tracing.
|
||||
- Deep tracing into the Codex runtime.
|
||||
- Default-provider support for grading when Codex credentials are available.
|
||||
|
||||
Important limits:
|
||||
|
||||
- It is the right default for CI and automation, but it does not expose every rich
|
||||
app-server protocol event.
|
||||
- Skill detection is heuristic.
|
||||
- Promptfoo still receives a final provider response, not live partial output in
|
||||
assertions.
|
||||
|
||||
Docs and examples:
|
||||
|
||||
- `site/docs/providers/openai-codex-sdk.md`
|
||||
- `examples/openai-codex-sdk/`
|
||||
|
||||
### OpenAI Codex App Server
|
||||
|
||||
Status: implemented, documented, and validated with mocked protocol tests plus a
|
||||
real local eval.
|
||||
|
||||
Provider IDs:
|
||||
|
||||
- `openai:codex-app-server`
|
||||
- `openai:codex-app-server:<model>`
|
||||
- `openai:codex-desktop`
|
||||
- `openai:codex-desktop:<model>`
|
||||
|
||||
Implemented capabilities:
|
||||
|
||||
- Spawns `codex app-server --listen stdio://`.
|
||||
- Drives the app-server JSON-RPC handshake.
|
||||
- Starts, resumes, unsubscribes, and archives threads according to config.
|
||||
- Starts turns with model, cwd, sandbox, approval, reasoning, personality,
|
||||
collaboration mode, service tier, output schema, and instructions.
|
||||
- Accepts plain text plus JSON arrays of app-server input items.
|
||||
- Aggregates streamed item notifications into final text and metadata.
|
||||
- Captures item counts, command/file/MCP/tool/web-search/reasoning trajectories.
|
||||
- Handles server requests deterministically through `server_request_policy`.
|
||||
- Uses safe defaults: read-only sandbox, no approvals, ephemeral threads, minimal env.
|
||||
- Supports thread pooling and persistent thread cache invalidation.
|
||||
- Supports raw events, token usage, cost estimation, request timeouts, turn timeouts,
|
||||
cleanup, aborts, and process failure handling.
|
||||
- Supports deep tracing by creating a fresh app-server process per row and injecting
|
||||
OpenTelemetry env vars.
|
||||
- Differentiates the app-server protocol from the Codex Desktop app: promptfoo
|
||||
starts its own app-server child process and does not attach to a running Desktop
|
||||
app UI process.
|
||||
|
||||
Important limits:
|
||||
|
||||
- WebSocket transport is not implemented; stdio is the supported transport.
|
||||
- Live partial output is not exposed to assertions.
|
||||
- Direct attachment to a running Codex Desktop app is not implemented.
|
||||
- High-risk protocol requests such as config writes, plugin installs, and arbitrary
|
||||
filesystem writes should remain unavailable or explicitly policy-gated.
|
||||
|
||||
Docs and examples:
|
||||
|
||||
- `site/docs/providers/openai-codex-app-server.md`
|
||||
- `docs/agents/codex-app-server-provider-notes.md`
|
||||
- `examples/openai-codex-app-server/`
|
||||
|
||||
### Claude Agent SDK
|
||||
|
||||
Status: implemented and documented.
|
||||
|
||||
Provider IDs:
|
||||
|
||||
- `anthropic:claude-agent-sdk`
|
||||
- `anthropic:claude-code`
|
||||
|
||||
Implemented capabilities:
|
||||
|
||||
- Lazy loading for `@anthropic-ai/claude-agent-sdk`.
|
||||
- Anthropic API key, Bedrock, Vertex, and local Claude Code auth flows.
|
||||
- Temporary or configured working directories.
|
||||
- Built-in tool controls, allowed/disallowed tools, and permission modes.
|
||||
- Explicit unsafe permission skip flag.
|
||||
- MCP server configuration and optional MCP caching.
|
||||
- AskUserQuestion handling.
|
||||
- Plugin and local skill support.
|
||||
- Custom agents, hooks, system prompt overrides, betas, thinking, and effort options.
|
||||
- Session resume, fork, continue, persistence, and file checkpointing options.
|
||||
- Sandbox and executable configuration.
|
||||
- Usage/cost controls such as max budget.
|
||||
|
||||
Important limits:
|
||||
|
||||
- The SDK owns many semantics, so promptfoo must keep docs aligned with SDK changes.
|
||||
- Side-effectful modes require external workspace reset discipline.
|
||||
|
||||
Docs and examples:
|
||||
|
||||
- `site/docs/providers/claude-agent-sdk.md`
|
||||
- `examples/claude-agent-sdk/`
|
||||
|
||||
### OpenCode SDK
|
||||
|
||||
Status: implemented and documented.
|
||||
|
||||
Provider IDs:
|
||||
|
||||
- `opencode:sdk`
|
||||
- `opencode`
|
||||
|
||||
Implemented capabilities:
|
||||
|
||||
- Lazy loading for `@opencode-ai/sdk` v1 or v2.
|
||||
- Starts an OpenCode server when `baseUrl` is unset.
|
||||
- Connects to an existing OpenCode server when `baseUrl` is provided.
|
||||
- Supports provider/model selection, variants, workspaces, and custom agents.
|
||||
- Supports temporary or configured working directories.
|
||||
- Uses read-only default tools for configured workspaces.
|
||||
- Supports write/edit/bash tools with explicit permission config.
|
||||
- Supports JSON Schema structured output through OpenCode `format`.
|
||||
- Supports sessions and persistent session caching.
|
||||
- Supports MCP configuration and optional MCP caching when promptfoo starts the
|
||||
server.
|
||||
|
||||
Important limits:
|
||||
|
||||
- When using `baseUrl`, the existing server owns auth, MCP setup, installed agents,
|
||||
and server-side configuration.
|
||||
- OpenCode model support is delegated to OpenCode/models.dev rather than promptfoo's
|
||||
normal provider model tables.
|
||||
|
||||
Docs and examples:
|
||||
|
||||
- `site/docs/providers/opencode-sdk.md`
|
||||
- `examples/provider-opencode-sdk/`
|
||||
|
||||
## Current Naming Guidance
|
||||
|
||||
Use provider IDs that encode the runtime boundary, not just the model vendor.
|
||||
|
||||
- `openai:codex` should continue to mean the Codex SDK alias because that is the
|
||||
best default for automation.
|
||||
- `openai:codex-app-server` should mean the app-server JSON-RPC protocol.
|
||||
- `openai:codex-desktop` should remain an alias for app-server behavior unless or
|
||||
until promptfoo can actually attach to the Desktop app process.
|
||||
- `anthropic:claude-code` should remain an alias for Claude Agent SDK because the
|
||||
SDK is still built on Claude Code.
|
||||
- `opencode` can remain a convenience alias for `opencode:sdk`.
|
||||
|
||||
Avoid adding unscoped top-level aliases such as `codex:desktop` until the OpenAI
|
||||
scoped names are stable and the docs can clearly explain the difference between
|
||||
SDK, app-server, and Desktop UI attachment.
|
||||
|
||||
## What To Implement Next
|
||||
|
||||
### 1. Shared Agent Metadata Schema
|
||||
|
||||
Create a cross-provider `metadata.agent` shape while preserving provider-specific
|
||||
metadata namespaces such as `metadata.codexAppServer`.
|
||||
|
||||
Proposed fields:
|
||||
|
||||
- `runtime`: `codex-sdk`, `codex-app-server`, `claude-agent-sdk`, `opencode`.
|
||||
- `runtimeVersion`: runtime-reported version when available.
|
||||
- `sessionId`, `threadId`, `turnId`.
|
||||
- `workingDir`, `sandbox`, `network`, `approvalPolicy`.
|
||||
- `tools`: normalized tool calls and outcomes.
|
||||
- `commands`: normalized shell command executions.
|
||||
- `fileChanges`: normalized file write/edit/delete attempts.
|
||||
- `approvals`: normalized approval prompts and decisions.
|
||||
- `mcp`: normalized MCP calls and elicitations.
|
||||
- `skills`: confirmed and attempted skill usage.
|
||||
- `trace`: trace ids and span ids when available.
|
||||
|
||||
Acceptance criteria:
|
||||
|
||||
- Existing provider-specific metadata remains backward compatible.
|
||||
- Cross-provider assertions can target the same metadata path.
|
||||
- Tests cover at least Codex SDK, Codex app-server, Claude Agent SDK, and OpenCode.
|
||||
|
||||
### 2. Shared Coding-Agent Provider Test Contract
|
||||
|
||||
Add a reusable test contract for coding-agent providers. Each provider can implement
|
||||
the same scenarios with its own mocked runtime.
|
||||
|
||||
Core scenarios:
|
||||
|
||||
- Missing optional dependency.
|
||||
- API key/env precedence.
|
||||
- Working directory validation.
|
||||
- Prompt-level config merge.
|
||||
- Safe default sandbox and permissions.
|
||||
- Structured output.
|
||||
- Session persistence and cleanup.
|
||||
- Timeout and abort.
|
||||
- Runtime process/server failure.
|
||||
- Tool approval decline by default.
|
||||
- Metadata normalization.
|
||||
- Trace sanitization.
|
||||
|
||||
Acceptance criteria:
|
||||
|
||||
- New providers cannot skip lifecycle and safety cases.
|
||||
- Mock implementations reset hoisted mocks in `beforeEach`.
|
||||
- Concurrency tests prove one session/process failure cannot fail unrelated rows.
|
||||
|
||||
### 3. Provider Capability Matrix
|
||||
|
||||
Add a docs page or generated table that compares coding-agent provider capabilities.
|
||||
|
||||
Suggested columns:
|
||||
|
||||
- Provider IDs.
|
||||
- Runtime boundary.
|
||||
- Optional dependency or CLI requirement.
|
||||
- Auth modes.
|
||||
- Workspace model.
|
||||
- Sandbox controls.
|
||||
- Shell/file/network controls.
|
||||
- MCP support.
|
||||
- Skills/plugins/custom agents.
|
||||
- Structured output.
|
||||
- Session persistence.
|
||||
- Tracing.
|
||||
- Raw protocol metadata.
|
||||
- Existing-server attachment.
|
||||
|
||||
Acceptance criteria:
|
||||
|
||||
- The matrix links to each provider doc.
|
||||
- It clearly says that Codex Desktop attachment is not currently supported.
|
||||
- It is tested in the docs build.
|
||||
|
||||
### 4. Real Eval QA Matrix
|
||||
|
||||
Create a small set of real eval examples that can be run selectively by maintainers.
|
||||
|
||||
Minimum scenarios:
|
||||
|
||||
- Read-only repo review.
|
||||
- Structured JSON output.
|
||||
- Sandbox denial for attempted writes.
|
||||
- Workspace-write side effect in a disposable fixture.
|
||||
- Session persistence across two rows.
|
||||
- Skill or plugin invocation.
|
||||
- MCP tool call with a deterministic local MCP server.
|
||||
- Deep tracing smoke test.
|
||||
|
||||
Acceptance criteria:
|
||||
|
||||
- Each scenario records expected pass/fail/error behavior.
|
||||
- Each scenario can run with `--no-cache`.
|
||||
- Side-effectful scenarios use disposable workspaces.
|
||||
- CI can run a lightweight subset without requiring every optional agent runtime.
|
||||
|
||||
### 5. Security and Redteam Coverage
|
||||
|
||||
Add adversarial tests for the risky parts of agent runtimes.
|
||||
|
||||
High-value cases:
|
||||
|
||||
- Prompt injection that asks the agent to reveal env vars.
|
||||
- Path traversal through input items, tool args, or MCP responses.
|
||||
- Approval bypass attempts.
|
||||
- Plugin install/config-write attempts.
|
||||
- Network enablement attempts when network should be off.
|
||||
- Symlink writes out of the workspace.
|
||||
- Malicious `SKILL.md` or plugin instructions.
|
||||
- Tool output containing secrets that must be redacted from traces.
|
||||
|
||||
Acceptance criteria:
|
||||
|
||||
- Providers decline or isolate side effects by default.
|
||||
- Redaction is tested for command output, MCP arguments, tool arguments, prompts,
|
||||
and final metadata.
|
||||
- Any accepted dangerous operation requires explicit config and is visible in
|
||||
metadata.
|
||||
|
||||
### 6. Codex App Server Follow-Up Features
|
||||
|
||||
The app-server provider now covers the core eval path. The next app-server-specific
|
||||
work should focus on rare protocol features and product integration boundaries.
|
||||
|
||||
Candidates:
|
||||
|
||||
- `model/list`, `skills/list`, `plugin/list`, `plugin/read`, and `app/list`
|
||||
metadata discovery.
|
||||
- `review/start` support for native review flows.
|
||||
- `turn/steer` and `turn/interrupt` tests for cancellation and mid-turn control.
|
||||
- Optional WebSocket transport only if upstream stabilizes it.
|
||||
- Better raw event snapshots for protocol regression tests.
|
||||
- Stronger docs around Desktop alias semantics and why promptfoo starts a separate
|
||||
app-server process.
|
||||
|
||||
Acceptance criteria:
|
||||
|
||||
- Rare features are opt-in and deterministic.
|
||||
- High-risk operations are policy-gated.
|
||||
- Protocol additions have mocked tests and at least one documented example when
|
||||
user-facing.
|
||||
|
||||
### 7. Side-Effect Harness
|
||||
|
||||
Build a reusable harness for tests and examples that need write access.
|
||||
|
||||
It should provide:
|
||||
|
||||
- Disposable workspace creation.
|
||||
- Git repository initialization when needed.
|
||||
- Snapshot before and after agent runs.
|
||||
- Symlink and path traversal fixtures.
|
||||
- Cleanup guarantees.
|
||||
- Helpers for expected file diffs.
|
||||
|
||||
Acceptance criteria:
|
||||
|
||||
- No side-effectful provider test mutates the source checkout.
|
||||
- Tests can assert exact changed files.
|
||||
- Harness works on macOS, Linux, and Windows CI.
|
||||
|
||||
### 8. Documentation Cleanup
|
||||
|
||||
Improve the information architecture around coding-agent providers.
|
||||
|
||||
Recommended docs:
|
||||
|
||||
- A public provider comparison matrix.
|
||||
- A "Choosing a coding-agent provider" guide.
|
||||
- A "Managing side effects in agent evals" guide.
|
||||
- Provider-specific "SDK vs app-server vs Desktop app" sections for Codex.
|
||||
- Example READMEs that all share the same run, auth, and safety structure.
|
||||
|
||||
Acceptance criteria:
|
||||
|
||||
- The public docs answer which provider to use for CI, Desktop-like protocol evals,
|
||||
Claude Code compatibility, and OpenCode multi-provider setups.
|
||||
- Example configs validate locally.
|
||||
- Docs state safe defaults and side-effect responsibilities near every write-capable
|
||||
example.
|
||||
|
||||
## Review Checklist For New Coding-Agent Provider Work
|
||||
|
||||
Before merging a provider in this family, verify:
|
||||
|
||||
- Provider IDs are explicit about runtime boundary.
|
||||
- Optional dependencies fail with actionable install guidance.
|
||||
- API keys and local-auth modes are documented.
|
||||
- Env precedence is tested.
|
||||
- Working directory, sandbox, network, and shell controls are documented and tested.
|
||||
- Prompt-level config merges do not drop nested provider defaults.
|
||||
- Hoisted mocks with implementations reset in `beforeEach`.
|
||||
- Session/thread reuse is opt-in or explicitly scoped.
|
||||
- Runtime failures do not poison unrelated sessions or rows.
|
||||
- Approval/user-input/MCP requests have deterministic non-interactive defaults.
|
||||
- Metadata includes session ids, tool/command/file activity, and approval decisions.
|
||||
- Trace attributes are sanitized.
|
||||
- Examples validate and at least one real eval has been run when the runtime is
|
||||
locally available.
|
||||
|
||||
## Open Questions
|
||||
|
||||
- Should promptfoo expose one public `metadata.agent` schema now, or keep it internal
|
||||
until at least two providers use it in docs examples?
|
||||
- Should Codex app-server discovery operations be exposed as provider metadata on
|
||||
every call, or only behind an explicit config flag?
|
||||
- Should any provider support a hard "no side effects" verifier that snapshots the
|
||||
workspace and fails if files changed?
|
||||
- Should remote/existing-server modes be marked as less reproducible in promptfoo
|
||||
output metadata?
|
||||
- Should top-level aliases like `codex:app-server` wait for a broader provider naming
|
||||
cleanup?
|
||||
@@ -0,0 +1,113 @@
|
||||
# Database Security (SQL Injection Prevention)
|
||||
|
||||
This codebase uses Drizzle ORM with SQLite. All database queries must use parameterized SQL so user-controlled input never changes query structure.
|
||||
|
||||
## Quick Rules
|
||||
|
||||
- Use Drizzle's `sql` tagged template literals for dynamic values.
|
||||
- Use `sql.join()` for dynamic lists such as `IN (...)` clauses.
|
||||
- Pass `SQL<unknown>` fragments between functions, not strings.
|
||||
- Do not build queries with `sql.raw()` or string interpolation.
|
||||
- Prefer `json_each()` for user-selected JSON keys. When `json_extract()` is appropriate, bind a path built with the vetted `buildSafeJsonPath` helper in `src/models/eval.ts`.
|
||||
|
||||
## Required Pattern: Use `sql` Template Strings
|
||||
|
||||
Use parameterized queries for single values:
|
||||
|
||||
```typescript
|
||||
import { sql } from 'drizzle-orm';
|
||||
|
||||
const query = sql`SELECT * FROM eval_results WHERE eval_id = ${evalId}`;
|
||||
```
|
||||
|
||||
Use parameterized queries for multiple values:
|
||||
|
||||
```typescript
|
||||
import { sql } from 'drizzle-orm';
|
||||
|
||||
const query = sql`
|
||||
SELECT * FROM eval_results
|
||||
WHERE eval_id = ${evalId} AND success = ${1}
|
||||
`;
|
||||
```
|
||||
|
||||
Use `sql.join()` for dynamic `IN (...)` lists:
|
||||
|
||||
```typescript
|
||||
import { sql } from 'drizzle-orm';
|
||||
|
||||
const ids = ['id1', 'id2', 'id3'];
|
||||
const query = sql`SELECT * FROM evals WHERE id IN (${sql.join(ids, sql`, `)})`;
|
||||
```
|
||||
|
||||
## Forbidden Pattern: Raw SQL with Dynamic Content
|
||||
|
||||
Do not interpolate user-controlled values into raw SQL:
|
||||
|
||||
```typescript
|
||||
const query = sql.raw(`SELECT * FROM eval_results WHERE eval_id = '${evalId}'`);
|
||||
```
|
||||
|
||||
Do not build SQL conditions with string concatenation:
|
||||
|
||||
```typescript
|
||||
import { sql } from 'drizzle-orm';
|
||||
|
||||
const whereClause = `eval_id = '${evalId}'`;
|
||||
const query = sql.raw(`SELECT * FROM eval_results WHERE ${whereClause}`);
|
||||
```
|
||||
|
||||
## JSON Paths in SQLite
|
||||
|
||||
SQLite's `json_extract()` accepts its JSON path as a bound value. Do not use `sql.raw()` to splice user-controlled JSON paths into SQL. Prefer `json_each()` when filtering by dynamic keys, because the key can be compared as a normal parameter:
|
||||
|
||||
```typescript
|
||||
import { sql } from 'drizzle-orm';
|
||||
|
||||
const query = sql`
|
||||
SELECT *
|
||||
FROM eval_results
|
||||
WHERE EXISTS (
|
||||
SELECT 1
|
||||
FROM json_each(metadata)
|
||||
WHERE json_each.key = ${field} AND json_each.value = ${value}
|
||||
)
|
||||
`;
|
||||
```
|
||||
|
||||
If you construct a JSON path for `json_extract()`, use `buildSafeJsonPath` from `src/models/eval.ts`, which escapes backslashes and double quotes for JSON path syntax and returns a value to bind:
|
||||
|
||||
```typescript
|
||||
import { sql } from 'drizzle-orm';
|
||||
import { buildSafeJsonPath } from '../../src/models/eval';
|
||||
|
||||
const jsonPath = buildSafeJsonPath(userField);
|
||||
const query = sql`
|
||||
SELECT * FROM eval_results
|
||||
WHERE json_extract(metadata, ${jsonPath}) = ${value}
|
||||
`;
|
||||
```
|
||||
|
||||
If you need a new JSON-path helper, match the guarantees in `buildSafeJsonPath` and keep the implementation audited in one shared utility.
|
||||
|
||||
## Passing SQL Fragments Between Functions
|
||||
|
||||
When building complex queries, pass `SQL<unknown>` fragments instead of strings:
|
||||
|
||||
```typescript
|
||||
import { type SQL, sql } from 'drizzle-orm';
|
||||
|
||||
function queryWithFilter(whereSql: SQL<unknown>): Promise<Result[]> {
|
||||
const query = sql`SELECT * FROM eval_results WHERE ${whereSql}`;
|
||||
return db.all(query);
|
||||
}
|
||||
|
||||
const filter = sql`eval_id = ${evalId} AND success = ${1}`;
|
||||
const results = await queryWithFilter(filter);
|
||||
```
|
||||
|
||||
## Key Files with Database Queries
|
||||
|
||||
- `src/models/eval.ts` - Main eval queries and JSON-path helper
|
||||
- `src/util/calculateFilteredMetrics.ts` - Metrics aggregation queries
|
||||
- `src/database/index.ts` - Database connection
|
||||
@@ -0,0 +1,89 @@
|
||||
# Dependency Management
|
||||
|
||||
## Safe Update Workflow
|
||||
|
||||
Use `--target minor` for safe minor/patch updates only:
|
||||
|
||||
```bash
|
||||
# Check all three locations for available updates
|
||||
npx npm-check-updates --target minor # Root
|
||||
npx npm-check-updates --target minor --cwd site # Site workspace
|
||||
npx npm-check-updates --target minor --cwd src/app # App workspace
|
||||
|
||||
# Apply updates with -u flag
|
||||
npx npm-check-updates --target minor -u
|
||||
npx npm-check-updates --target minor -u --cwd site
|
||||
npx npm-check-updates --target minor -u --cwd src/app
|
||||
|
||||
# Install and verify
|
||||
npm install
|
||||
npm run build && npm test && npm run lint && npm run format:check
|
||||
|
||||
# Check version consistency (required by CI)
|
||||
npx check-dependency-version-consistency
|
||||
```
|
||||
|
||||
## Critical Rules
|
||||
|
||||
1. **Version consistency across workspaces** - All workspaces must use the same version of shared dependencies. CI enforces this via `check-dependency-version-consistency`.
|
||||
|
||||
2. **Update examples/** - 20+ package.json files in examples/ are user-facing; keep them current when updating dependencies.
|
||||
|
||||
3. **Run `npm audit`** - Use `npm audit` or `npm run audit:fix` to check for security vulnerabilities across all workspaces. Do not let `npm audit fix` lockfile drift ride along with an unrelated change; ship audit-driven updates as their own PR.
|
||||
|
||||
4. **If updates fail** - Revert the problematic package and keep the current version. Don't force incompatible updates.
|
||||
|
||||
5. **Test before committing** - Always run `npm run build && npm test` after updating dependencies.
|
||||
|
||||
## Major Updates
|
||||
|
||||
```bash
|
||||
# See available major updates (don't apply automatically)
|
||||
npx npm-check-updates --target latest
|
||||
|
||||
# Major updates often require code changes - evaluate each carefully
|
||||
```
|
||||
|
||||
Major updates require careful evaluation:
|
||||
|
||||
- Check the changelog for breaking changes
|
||||
- Look for migration guides
|
||||
- Test thoroughly before merging
|
||||
|
||||
## Workspaces
|
||||
|
||||
The project uses npm workspaces. Updates must be checked in all three locations:
|
||||
|
||||
- Root (`/`) - Core library dependencies
|
||||
- Site (`/site`) - Documentation site (Docusaurus)
|
||||
- App (`/src/app`) - Web UI (React/Vite)
|
||||
|
||||
## Working on Renovate Branches
|
||||
|
||||
Renovate force-pushes its branches whenever `main` changes or someone comments
|
||||
`@renovate rebase`. Any manual commit you add may be overwritten without warning.
|
||||
|
||||
- Push fixes quickly and expect them to survive only until the next Renovate rebase.
|
||||
- For non-trivial manual work on a Renovate-managed dependency, create a sibling
|
||||
branch off the Renovate branch and open a separate PR that Renovate will not touch.
|
||||
- On a major-version Renovate PR, read the upstream changelog, run gap analysis on
|
||||
our matching provider/integration and its docs, then test end-to-end with real evals
|
||||
(`npm run local -- eval -c <example>.yaml --no-cache -o output.json`, adding
|
||||
`--env-file .env` when credentials are needed and the file exists)
|
||||
before deciding whether the upgrade needs code changes.
|
||||
|
||||
## Useful Commands
|
||||
|
||||
```bash
|
||||
# Fix security vulnerabilities in all workspaces
|
||||
npm run audit:fix
|
||||
|
||||
# Check for outdated packages
|
||||
npm outdated
|
||||
|
||||
# See why a package is installed
|
||||
npm explain <package-name>
|
||||
|
||||
# Check for unused dependencies
|
||||
npm run depcheck
|
||||
```
|
||||
@@ -0,0 +1,77 @@
|
||||
# Git Workflow
|
||||
|
||||
## Critical Rules
|
||||
|
||||
1. **NEVER** commit directly to main
|
||||
2. **NEVER** merge branches into main directly
|
||||
3. **NEVER** push to main - EVER
|
||||
4. **NEVER** use `--force` without explicit approval
|
||||
5. **ALWAYS** create new commits - never amend or rebase unless explicitly asked
|
||||
|
||||
**Forbidden commands:**
|
||||
|
||||
- `git push origin main`
|
||||
- `git merge feature-branch` while on main
|
||||
- Any direct commits to main
|
||||
|
||||
All changes to main MUST go through pull requests.
|
||||
|
||||
## Commit Policy
|
||||
|
||||
**"Explicitly asked"** = user says "amend", "squash", "rebase", or "fix up the commit".
|
||||
|
||||
"Looks good" or "go ahead" is NOT permission to rewrite history.
|
||||
|
||||
## Standard Workflow
|
||||
|
||||
### 1. Create Feature Branch
|
||||
|
||||
```bash
|
||||
git checkout main
|
||||
git pull origin main
|
||||
git checkout -b feature/your-branch-name
|
||||
```
|
||||
|
||||
### 2. Make Changes and Commit
|
||||
|
||||
```bash
|
||||
git add <specific-files> # NEVER blindly add everything
|
||||
git commit -m "type(scope): description"
|
||||
```
|
||||
|
||||
See `docs/agents/pr-conventions.md` for commit message format.
|
||||
|
||||
### 3. Lint and Format
|
||||
|
||||
```bash
|
||||
npm run l # Lint changed files
|
||||
npm run f # Format changed files
|
||||
```
|
||||
|
||||
Fix any errors before proceeding.
|
||||
|
||||
### 4. Sync with Main
|
||||
|
||||
```bash
|
||||
git fetch origin main
|
||||
git merge origin/main
|
||||
```
|
||||
|
||||
Resolve any conflicts before pushing.
|
||||
|
||||
### 5. Push and Create PR
|
||||
|
||||
```bash
|
||||
git push -u origin feature/your-branch-name
|
||||
gh pr create --title "type(scope): description" --body "PR description"
|
||||
```
|
||||
|
||||
### 6. Wait for Review
|
||||
|
||||
Wait for CI checks to pass and code review approval before merging.
|
||||
|
||||
## Key Points
|
||||
|
||||
- **Never blindly `git add .`** - there may be unrelated files
|
||||
- **Always sync with main** before creating PR to avoid conflicts
|
||||
- **Don't edit CHANGELOG.md** - it's auto-generated
|
||||
@@ -0,0 +1,72 @@
|
||||
# Logging Guidelines
|
||||
|
||||
## The Rule
|
||||
|
||||
Always use the logger with an object as the second parameter:
|
||||
|
||||
```typescript
|
||||
logger.debug('[Component] Message', { headers, body, config });
|
||||
```
|
||||
|
||||
The object is **auto-sanitized** - sensitive fields are automatically redacted.
|
||||
|
||||
## Why This Matters
|
||||
|
||||
- **Security**: Prevents accidental exposure of secrets in logs
|
||||
- **Consistency**: Structured logs are easier to search and analyze
|
||||
- **Safety**: Red team test content may contain harmful/sensitive data
|
||||
|
||||
## Correct Usage
|
||||
|
||||
```typescript
|
||||
import logger from './logger';
|
||||
|
||||
// Good - context object is auto-sanitized
|
||||
logger.debug('[Provider]: Making API request', {
|
||||
url: 'https://api.example.com',
|
||||
method: 'POST',
|
||||
headers: { Authorization: 'Bearer secret-token' },
|
||||
body: { apiKey: 'secret-key', data: 'value' },
|
||||
});
|
||||
// Output: Authorization and apiKey are [REDACTED]
|
||||
|
||||
logger.error('Request failed', {
|
||||
headers: response.headers,
|
||||
body: errorResponse,
|
||||
});
|
||||
```
|
||||
|
||||
## Anti-Pattern
|
||||
|
||||
```typescript
|
||||
// WRONG - exposes secrets, bypasses sanitization
|
||||
logger.debug(`Config: ${JSON.stringify(config)}`);
|
||||
logger.debug(`Calling ${url} with headers: ${JSON.stringify(headers)}`);
|
||||
```
|
||||
|
||||
## Manual Sanitization
|
||||
|
||||
For non-logging contexts:
|
||||
|
||||
```typescript
|
||||
import { sanitizeObject } from './util/sanitizer';
|
||||
|
||||
const sanitizedConfig = sanitizeObject(providerConfig, {
|
||||
context: 'provider config',
|
||||
});
|
||||
```
|
||||
|
||||
## What Gets Sanitized
|
||||
|
||||
Field names containing (case-insensitive, works with `-`, `_`, camelCase):
|
||||
|
||||
| Category | Fields |
|
||||
| ------------ | -------------------------------------------------------------------------- |
|
||||
| Passwords | `password`, `passwd`, `pwd`, `passphrase` |
|
||||
| API Keys | `apiKey`, `api_key`, `token`, `accessToken`, `refreshToken`, `bearerToken` |
|
||||
| Secrets | `secret`, `clientSecret`, `webhookSecret` |
|
||||
| Headers | `authorization`, `cookie`, `x-api-key`, `x-auth-token` |
|
||||
| Certificates | `privateKey`, `certificatePassword`, `keystorePassword` |
|
||||
| Signatures | `signature`, `sig`, `signingKey` |
|
||||
|
||||
See `src/util/sanitizer.ts` for the complete list.
|
||||
@@ -0,0 +1,219 @@
|
||||
# Pull Request Conventions
|
||||
|
||||
PR titles follow Conventional Commits format. They become squash-merge commit messages and changelog entries.
|
||||
|
||||
## Format
|
||||
|
||||
```plaintext
|
||||
<type>(<scope>): <description>
|
||||
<type>(<scope>)!: <description> # Breaking changes
|
||||
```
|
||||
|
||||
### Description Guidelines
|
||||
|
||||
- **Imperative mood**: "add feature" not "added" or "adds"
|
||||
- **Lowercase**: except proper nouns and acronyms (FERPA, OAuth, MUI)
|
||||
- **No trailing period**
|
||||
- **Be specific**: describe what changed, not that something changed
|
||||
- **~50 characters**: GitHub truncates long titles
|
||||
|
||||
## Types
|
||||
|
||||
| Type | Use For |
|
||||
| ---------- | ----------------------------------------------------- |
|
||||
| `feat` | New CLI feature or major webui feature |
|
||||
| `fix` | Bug fix in CLI or major webui bug fix |
|
||||
| `chore` | Maintenance, upgrades, minor fixes, non-user-facing |
|
||||
| `refactor` | Code restructuring without behavior change |
|
||||
| `docs` | Documentation only (use with `site` scope for site/) |
|
||||
| `test` | Test-only changes (new tests, test fixes, test infra) |
|
||||
| `ci` | CI/CD changes |
|
||||
| `revert` | Revert previous change |
|
||||
| `perf` | Performance improvement |
|
||||
|
||||
**Changelog visibility:** Only `feat`, `fix`, and breaking changes (`!`) appear in release notes. Use `ci`, `chore`, `test`, `docs`, or `refactor` for changes that shouldn't be user-facing.
|
||||
|
||||
**Breaking changes:** Add `!` after scope: `feat(api)!:`, `chore(deps)!:`
|
||||
|
||||
### Test vs Fix
|
||||
|
||||
Use `test:` when the PR **only** contains test changes:
|
||||
|
||||
- Adding new tests
|
||||
- Fixing broken/flaky tests
|
||||
- Fixing lint errors in test files
|
||||
- Test infrastructure changes
|
||||
|
||||
Use `fix:` when fixing bugs in **application code** (even if tests are included):
|
||||
|
||||
- Bug fix in `src/` with accompanying test changes → `fix:`
|
||||
- Lint error in test file only → `test:`
|
||||
|
||||
### Type Selection for Mixed Changes
|
||||
|
||||
Use the **primary change** to determine type:
|
||||
|
||||
| PR Contains | Type | Why |
|
||||
| --------------------------------- | ---------- | -------------------------------- |
|
||||
| Bug fix + new tests | `fix` | Fix is primary, tests support it |
|
||||
| Feature + documentation | `feat` | Feature is primary |
|
||||
| Only test changes | `test` | No application code changed |
|
||||
| Only doc changes | `docs` | No application code changed |
|
||||
| Minor webui fix (styling, typos) | `chore` | Not a major user-facing fix |
|
||||
| Refactor + minor fixes discovered | `refactor` | Refactor was the intent |
|
||||
|
||||
**Major webui changes** = new pages, significant UX changes, core functionality bugs
|
||||
|
||||
**Minor webui changes** = styling tweaks, copy changes, internal refactors → use `chore`
|
||||
|
||||
## Scope Selection (Priority Order)
|
||||
|
||||
### 1. Feature Domains (HIGHEST PRIORITY)
|
||||
|
||||
**`redteam` - MANDATORY when redteam is the PR's primary change or product surface:**
|
||||
|
||||
- Plugins, strategies, grading
|
||||
- UI components (setup, report, config dialogs)
|
||||
- CLI commands, server endpoints
|
||||
- Documentation, examples
|
||||
- Redteam-specific tests, fixtures, utilities, and behavior changes
|
||||
|
||||
**Other feature domains:** `providers`, `assertions`, `eval`, `api`, `db`
|
||||
|
||||
### 2. Product Areas
|
||||
|
||||
- `webui` - React app in `src/app/`
|
||||
- `cli` - CLI in `src/`
|
||||
- `server` - Web server in `src/server/`
|
||||
|
||||
**Note:** Documentation site changes use `docs(site):`, not a standalone `site` scope.
|
||||
|
||||
### 3. Technical/Infrastructure
|
||||
|
||||
- `deps` - Dependency updates
|
||||
- `ci` - CI/CD pipelines, GitHub Actions
|
||||
- `tests` - Test infrastructure
|
||||
- `build` - Build tooling
|
||||
- `examples` - Non-redteam examples
|
||||
|
||||
### 4. Specialized
|
||||
|
||||
`auth`, `cache`, `config`, `python`, `mcp`, `code-scan`
|
||||
|
||||
### 5. No Scope
|
||||
|
||||
For generic/cross-cutting changes: `chore: bump version 0.119.11`
|
||||
|
||||
## THE REDTEAM RULE
|
||||
|
||||
**If a PR is primarily redteam-related, use `(redteam)` scope.**
|
||||
|
||||
This applies even if the redteam change is only in UI, CLI, docs, examples, utilities, tests, or server endpoints.
|
||||
|
||||
For broad, cross-cutting maintenance PRs, do **not** choose `(redteam)` solely because one touched file lives under `src/redteam/` or because one generic helper is also used by redteam. Use the PR's primary purpose/scope and call out the redteam-adjacent touch in the PR description when it is review-relevant.
|
||||
|
||||
❌ **Wrong:**
|
||||
|
||||
```plaintext
|
||||
fix(webui): fix Basic strategy checkbox in red team setup
|
||||
feat(cli): add redteam validate command
|
||||
chore(redteam): resolve repo-wide lint findings
|
||||
```
|
||||
|
||||
✅ **Correct:**
|
||||
|
||||
```plaintext
|
||||
fix(redteam): fix Basic strategy checkbox in setup UI
|
||||
feat(redteam): add validate target CLI command
|
||||
chore: resolve repo-wide lint findings
|
||||
```
|
||||
|
||||
**Why?** Redteam spans CLI, webui, server, docs, and examples. Consistent scoping makes it easy to find all redteam work.
|
||||
|
||||
## Decision Tree
|
||||
|
||||
```plaintext
|
||||
1. Is the PR primarily redteam-related? → Use (redteam)
|
||||
2. Is it another feature domain? → Use that scope
|
||||
3. Is it localized to one product area? → Use that scope
|
||||
4. Is it infrastructure? → Use that scope
|
||||
5. Otherwise → No scope
|
||||
```
|
||||
|
||||
## Dependency Updates
|
||||
|
||||
- **`fix(deps)`** - Patch versions (security/bug fixes)
|
||||
- **`chore(deps)`** - Minor/major upgrades, bulk updates, dev dependencies
|
||||
|
||||
## Examples
|
||||
|
||||
✅ **Good:**
|
||||
|
||||
```plaintext
|
||||
feat(redteam): add FERPA compliance plugin
|
||||
feat(cli): add --json output flag to eval command
|
||||
fix(cli): handle empty config file gracefully
|
||||
fix(webui): fix pagination crash on empty results
|
||||
chore(webui): update button styling on settings page
|
||||
docs(site): add guide for custom providers
|
||||
chore(deps): update Material-UI monorepo to v8 (major)
|
||||
fix(deps): update dependency zod to v4.2.0
|
||||
feat(api)!: simplify provider interface
|
||||
chore: bump version 0.119.11
|
||||
test: add smoke tests for CLI commands
|
||||
test(redteam): fix flaky plugin integration tests
|
||||
```
|
||||
|
||||
❌ **Bad:**
|
||||
|
||||
```plaintext
|
||||
feat: add new redteam thing # Missing (redteam) scope
|
||||
fix(webui): red team checkbox # Should be fix(redteam)
|
||||
chore(webui): update dependency # Should be chore(deps)
|
||||
feat: stuff # Too vague
|
||||
fix: bug fix # What bug? Be specific
|
||||
Fix(cli): Add feature # Wrong case, not imperative
|
||||
fix(test): resolve lint errors # Should be test: (test-only)
|
||||
docs: update site # Should be docs(site):
|
||||
site: update guides # Should be docs(site):
|
||||
feat(webui): minor styling update # Minor = chore, not feat
|
||||
```
|
||||
|
||||
## Draft vs Ready
|
||||
|
||||
Open PRs **ready for review** by default:
|
||||
|
||||
```bash
|
||||
gh pr create --title "feat(scope): description" --body "..."
|
||||
```
|
||||
|
||||
Use `--draft` only when:
|
||||
|
||||
- The user explicitly asks for a draft
|
||||
- The work is an intentional WIP parked for a hand-off
|
||||
- The PR blocks on an external dependency that must land first
|
||||
- The PR addresses an unpublished security advisory (see root `AGENTS.md`
|
||||
"Security-Sensitive PRs")
|
||||
|
||||
## Commit & PR Attribution
|
||||
|
||||
- **Never attribute commits or PR bodies to Claude / Claude Code.** Do not add
|
||||
`Co-Authored-By: Claude…` trailers, "Generated with Claude Code" footers, or
|
||||
similar markers. Use your configured git identity only.
|
||||
- Do not add marketing-style suffixes to commit subjects.
|
||||
|
||||
## GitHub Interaction Rules
|
||||
|
||||
- **NEVER comment on GitHub issues** - Only create PRs to address issues
|
||||
- **NEVER close issues** - Let maintainers close issues after PR merge
|
||||
- Focus on creating high-quality PRs that fully address the issue
|
||||
|
||||
## Checklist Before Creating PR
|
||||
|
||||
1. Is this PR primarily redteam-related? → Use `(redteam)` scope
|
||||
2. Choose correct type
|
||||
3. Choose correct scope using priority order
|
||||
4. Breaking change? Add `!` after scope
|
||||
5. Run `npm run l && npm run f`
|
||||
6. Open ready-for-review (omit `--draft`) unless one of the exceptions above applies
|
||||
7. Do **not** add Claude attribution trailers or footers
|
||||
@@ -0,0 +1,62 @@
|
||||
# Python Guidelines
|
||||
|
||||
For Python providers, prompts, assertions, and scripts.
|
||||
|
||||
## Requirements
|
||||
|
||||
- Python 3.8 or later
|
||||
- Follow [Google Python Style Guide](https://google.github.io/styleguide/pyguide.html)
|
||||
- Use type hints for readability and error catching
|
||||
|
||||
## Linting & Formatting
|
||||
|
||||
Use `ruff` for both:
|
||||
|
||||
```bash
|
||||
ruff check --fix # Lint with auto-fix
|
||||
ruff check --select I --fix # Sort imports
|
||||
ruff format # Format code
|
||||
```
|
||||
|
||||
## Testing
|
||||
|
||||
Use the built-in `unittest` module for new Python tests.
|
||||
|
||||
## Best Practices
|
||||
|
||||
- Keep dependencies minimal - avoid unnecessary external packages
|
||||
- When adding examples, update relevant `requirements.txt` files
|
||||
- Follow promptfoo API patterns for custom providers/prompts/assertions
|
||||
- Write unit tests for new Python functions
|
||||
|
||||
## Provider Pattern
|
||||
|
||||
```python
|
||||
def call_api(prompt: str, options: dict, context: dict) -> dict:
|
||||
"""
|
||||
Args:
|
||||
prompt: The prompt string
|
||||
options: Provider configuration
|
||||
context: Variables from the test case
|
||||
|
||||
Returns:
|
||||
dict with 'output' key (and optionally 'error')
|
||||
"""
|
||||
# Implementation
|
||||
return {"output": response_text}
|
||||
```
|
||||
|
||||
## Assertion Pattern
|
||||
|
||||
```python
|
||||
def get_assert(output: str, context: dict) -> dict:
|
||||
"""
|
||||
Args:
|
||||
output: The provider's output
|
||||
context: Test context including vars
|
||||
|
||||
Returns:
|
||||
dict with 'pass' (bool) and optionally 'reason'
|
||||
"""
|
||||
return {"pass": True, "reason": "Validation passed"}
|
||||
```
|
||||
@@ -0,0 +1,130 @@
|
||||
# Internal Package Boundaries
|
||||
|
||||
Promptfoo still publishes one package today, but the repository is beginning to
|
||||
model the internal boundaries that would support a future multi-package split.
|
||||
|
||||
## Current Private Layers
|
||||
|
||||
| Layer | Current roots | Intended role |
|
||||
| ------------------ | ---------------------------------------------------------------- | ----------------------------------------------- |
|
||||
| `facade` | `src/index.ts` | Public compatibility surface |
|
||||
| `contracts` | `src/contracts`, `src/contracts.ts` | Leaf-safe shared contracts and schemas |
|
||||
| `legacy-contracts` | `src/types`, `src/validators` | Transitional mixed runtime types and validators |
|
||||
| `core` | assertions, matchers, prompts, scheduler, test-case logic | Evaluation domain logic |
|
||||
| `node` | database, models, config, storage, `src/evaluate.ts`, `src/node` | Node runtime adapters |
|
||||
| `providers` | `src/providers` | Concrete provider implementations |
|
||||
| `redteam` | `src/redteam` | Red-team workflows |
|
||||
| `view-server` | `src/server` | Local server and API routes |
|
||||
| `cli` | `src/main.ts`, `src/commands` | Command-line orchestration |
|
||||
| `app` | `src/app` | Browser UI and build configuration |
|
||||
| `legacy-runtime` | Mixed top-level runtime modules | Transitional modules awaiting narrower owners |
|
||||
|
||||
The source of truth for these temporary private layers is
|
||||
`architecture/layers.json`.
|
||||
|
||||
## First Enforced Rule
|
||||
|
||||
Internal modules must not import `src/index.ts`.
|
||||
|
||||
`src/index.ts` is the public facade. Importing it from inside the product makes
|
||||
the dependency graph point inward through the public API, which makes later
|
||||
package extraction harder and can hide cycles.
|
||||
|
||||
Run the check with:
|
||||
|
||||
```bash
|
||||
npm run architecture:check
|
||||
```
|
||||
|
||||
## First Leaf Layer
|
||||
|
||||
`src/contracts` and its `src/contracts.ts` public entrypoint are the first intentionally
|
||||
leaf-safe surface. They currently own the dependency-free-or-`zod` subset that can plausibly become a future
|
||||
`@promptfoo/schema` package:
|
||||
|
||||
- shared token/input contracts
|
||||
- browser-safe common and user API DTOs
|
||||
- portable blob references and provider-neutral capability/result contracts
|
||||
- provider environment override schema
|
||||
- prompt contracts and prompt validation
|
||||
- transform contracts and shared transform validation
|
||||
|
||||
The older `src/types` and `src/validators` paths remain as compatibility shims or
|
||||
mixed transitional modules. They are useful public/internal surfaces today, but
|
||||
they are not yet clean enough to call a package boundary.
|
||||
|
||||
Leaf layers may import only themselves plus the external packages on their
|
||||
`allowedExternal` allowlist in `architecture/layers.json` (`contracts` allows only
|
||||
`zod`). The same architecture check enforces both halves of the rule, so this first
|
||||
extracted surface can neither grow back upward into Node, provider, or redteam code,
|
||||
nor quietly pick up a new npm dependency or Node builtin such as `node:fs`. A
|
||||
`node:` prefix is ignored when matching, so `"fs"` and `"node:fs"` are equivalent.
|
||||
|
||||
## Layer Dependency Ratchet
|
||||
|
||||
Each private layer declares its currently allowed dependencies in
|
||||
`architecture/layers.json`. The current graph still has transitional edges, so
|
||||
the allowlist records today's honest baseline rather than pretending the final
|
||||
package topology already exists. New cross-layer relationships fail
|
||||
`npm run architecture:check` until they are reviewed explicitly.
|
||||
|
||||
Mixed modules that do not yet have a stable package owner belong to
|
||||
`legacy-runtime`. This keeps migration debt visible. New checked source files
|
||||
must be assigned to a layer instead of silently becoming unclassified.
|
||||
|
||||
`src/evaluator/runtime.ts` defines the evaluator's narrow runtime port. The
|
||||
`EvaluationStore` contract owns result append/read, prompt updates, resume lookup,
|
||||
comparison-result saves, and final evaluation persistence. The default
|
||||
`src/node/evaluationStore.ts` adapter maps that port to the existing `Eval` and
|
||||
`EvalResult` models, while `src/evaluator/inMemoryStore.ts` provides a
|
||||
dependency-light state implementation for embedded evaluators and focused tests.
|
||||
`src/node/evaluatorRuntime.ts` continues to own JSONL writer construction and
|
||||
resume append behavior. The evaluator orchestrates evaluation behavior without
|
||||
importing the concrete `Eval` model.
|
||||
|
||||
The checker also resolves cross-layer source aliases such as `@promptfoo/*`.
|
||||
The browser-only `@app/*` alias stays inside the `app` layer. Alias spelling
|
||||
does not exempt a browser import from the same layer and path checks as a
|
||||
relative import.
|
||||
|
||||
## DAG Progress Ratchets
|
||||
|
||||
The architecture check also measures the layer dependency graph so it can move
|
||||
toward a directed acyclic graph without regressing:
|
||||
|
||||
- `maxStronglyConnectedComponentSize` limits the size of the largest remaining
|
||||
layer cycle.
|
||||
- `architecture/edge-baseline.json` limits every existing cross-layer edge to
|
||||
its reviewed import count and rejects new edges.
|
||||
- `forbiddenDependencies` permanently locks layer pairs whose direct or
|
||||
transitive dependency has been removed.
|
||||
- `tierOrder` lists every layer in the intended bottom-to-top topology so the
|
||||
checker can report the remaining back-edges.
|
||||
|
||||
After intentionally reducing or otherwise reviewing cross-layer coupling, run
|
||||
`npm run architecture:baseline` and include the baseline change in review. Do
|
||||
not refresh the baseline merely to make a newly introduced dependency pass.
|
||||
|
||||
## Browser Import Ratchet
|
||||
|
||||
The `app` layer has an additional internal-path allowlist. It pins the existing
|
||||
browser-to-runtime imports while DTOs and presentation helpers move into a
|
||||
browser-safe package surface. A new app import from root runtime code fails the
|
||||
architecture check even when the broader layer relationship already exists.
|
||||
|
||||
When moving an existing browser import to a narrower surface, remove its old
|
||||
path from the allowlist. Avoid adding paths unless the dependency is
|
||||
intentionally browser-safe. Allowlist entries are exact files, not directory
|
||||
roots.
|
||||
|
||||
## Dependency Ownership Report
|
||||
|
||||
The dependency report groups direct runtime imports by the private layer that
|
||||
currently uses them:
|
||||
|
||||
```bash
|
||||
npm run deps:ownership
|
||||
```
|
||||
|
||||
The report is intentionally descriptive for now. It gives us the evidence needed
|
||||
to move dependencies into future packages without guessing at ownership.
|
||||
@@ -0,0 +1,544 @@
|
||||
# Plugins State Management Refactor Implementation Plan
|
||||
|
||||
## Overview
|
||||
|
||||
Refactor `Plugins.tsx` to remove interstitial state management between `PluginsTab.tsx` and the Zustand store. The current implementation maintains duplicate local state (`selectedPlugins`, `pluginConfig`, `hasUserInteracted`) with bi-directional sync effects. This refactor will derive state directly from the store and update the store directly on user interactions.
|
||||
|
||||
## Current State Analysis
|
||||
|
||||
### Three-Layer Architecture (to be simplified)
|
||||
|
||||
1. **Global State**: Zustand store (`useRedTeamConfig.ts`) - source of truth
|
||||
2. **Local State**: `Plugins.tsx` maintains `selectedPlugins`, `pluginConfig`, `hasUserInteracted`
|
||||
3. **Props**: `PluginsTab.tsx` receives state via props
|
||||
|
||||
### Key Files
|
||||
|
||||
- `src/app/src/pages/redteam/setup/components/Plugins.tsx` - Parent component with local state
|
||||
- `src/app/src/pages/redteam/setup/components/PluginsTab.tsx` - Child component receiving props
|
||||
- `src/app/src/pages/redteam/setup/hooks/useRedTeamConfig.ts` - Zustand store
|
||||
|
||||
### Current State Variables to Remove (Plugins.tsx)
|
||||
|
||||
- `const [selectedPlugins, setSelectedPlugins]` - lines 112-118
|
||||
- `const [hasUserInteracted, setHasUserInteracted]` - line 135
|
||||
- `const [pluginConfig, setPluginConfig]` - lines 136-144
|
||||
|
||||
### Current Sync Effects to Remove (Plugins.tsx)
|
||||
|
||||
- Effect 1 (lines 152-168): Syncs store → local state when `!hasUserInteracted`
|
||||
- Effect 2 (lines 171-199): Syncs local state → store when `hasUserInteracted`
|
||||
|
||||
### Store Protection Already Exists
|
||||
|
||||
The `updatePlugins` method (useRedTeamConfig.ts:272-311) already:
|
||||
|
||||
- Merges new plugin configs with existing configs
|
||||
- Compares output vs current state to prevent infinite loops
|
||||
- Only triggers updates when state actually changed
|
||||
|
||||
## Desired End State
|
||||
|
||||
After this refactor:
|
||||
|
||||
1. `Plugins.tsx` derives `selectedPlugins` and `pluginConfig` from `config.plugins` using `useMemo`
|
||||
2. All plugin mutations go directly to the Zustand store
|
||||
3. No local state synchronization effects
|
||||
4. `PluginsTab` has a new `setSelectedPlugins` prop for efficient bulk operations
|
||||
5. `onUserInteraction` prop is removed from `PluginsTab`
|
||||
|
||||
### How to Verify
|
||||
|
||||
1. All tests in `PluginsTab.test.tsx` pass
|
||||
2. Preset selection works (plugins are set correctly in store)
|
||||
3. Individual plugin toggle works (checkbox toggles update store)
|
||||
4. Select All/None buttons work correctly
|
||||
5. Clear All button works correctly
|
||||
6. Plugin configs (e.g., for `indirect-prompt-injection`) are preserved when toggling
|
||||
|
||||
## What We're NOT Doing
|
||||
|
||||
- Changing the Zustand store implementation
|
||||
- Modifying `CustomIntentsTab` or `CustomPoliciesTab`
|
||||
- Changing how policy/intent plugins are handled
|
||||
- Refactoring the `PluginConfigDialog` component
|
||||
- Changing test structure or test helpers
|
||||
|
||||
## Implementation Approach
|
||||
|
||||
The refactoring follows these principles:
|
||||
|
||||
1. **Remove duplicate state** - No local state that mirrors the store
|
||||
2. **Derive, don't store** - Use `useMemo` to compute `selectedPlugins` and `pluginConfig` from `config.plugins`
|
||||
3. **Direct store updates** - All mutations go straight to `updatePlugins`
|
||||
4. **Efficient bulk operations** - Add `setSelectedPlugins` for presets and bulk selection
|
||||
|
||||
---
|
||||
|
||||
## Phase 1: Refactor `Plugins.tsx`
|
||||
|
||||
### Overview
|
||||
|
||||
Remove local state and sync effects, replace with derived values and direct store updates.
|
||||
|
||||
### Changes Required:
|
||||
|
||||
#### 1. Remove Local State Variables
|
||||
|
||||
**File**: `src/app/src/pages/redteam/setup/components/Plugins.tsx`
|
||||
|
||||
**Remove lines 112-118** (selectedPlugins state):
|
||||
|
||||
```tsx
|
||||
// REMOVE THIS:
|
||||
const [selectedPlugins, setSelectedPlugins] = useState<Set<Plugin>>(() => {
|
||||
return new Set(
|
||||
config.plugins
|
||||
.map((plugin) => (typeof plugin === 'string' ? plugin : plugin.id))
|
||||
.filter((id) => id !== 'policy' && id !== 'intent') as Plugin[],
|
||||
);
|
||||
});
|
||||
```
|
||||
|
||||
**Remove line 135** (hasUserInteracted state):
|
||||
|
||||
```tsx
|
||||
// REMOVE THIS:
|
||||
const [hasUserInteracted, setHasUserInteracted] = useState(false);
|
||||
```
|
||||
|
||||
**Remove lines 136-144** (pluginConfig state):
|
||||
|
||||
```tsx
|
||||
// REMOVE THIS:
|
||||
const [pluginConfig, setPluginConfig] = useState<LocalPluginConfig>(() => {
|
||||
const initialConfig: LocalPluginConfig = {};
|
||||
config.plugins.forEach((plugin) => {
|
||||
if (typeof plugin === 'object' && plugin.config) {
|
||||
initialConfig[plugin.id] = plugin.config;
|
||||
}
|
||||
});
|
||||
return initialConfig;
|
||||
});
|
||||
```
|
||||
|
||||
#### 2. Add Derived Values
|
||||
|
||||
**Add after the store hook calls (after line 108):**
|
||||
|
||||
```tsx
|
||||
// Derive selectedPlugins from config.plugins
|
||||
const selectedPlugins = useMemo(() => {
|
||||
return new Set(
|
||||
config.plugins
|
||||
.map((plugin) => (typeof plugin === 'string' ? plugin : plugin.id))
|
||||
.filter((id) => id !== 'policy' && id !== 'intent') as Plugin[],
|
||||
);
|
||||
}, [config.plugins]);
|
||||
|
||||
// Derive pluginConfig from config.plugins
|
||||
const pluginConfig = useMemo(() => {
|
||||
const configs: LocalPluginConfig = {};
|
||||
config.plugins.forEach((plugin) => {
|
||||
if (typeof plugin === 'object' && plugin.config) {
|
||||
configs[plugin.id] = plugin.config;
|
||||
}
|
||||
});
|
||||
return configs;
|
||||
}, [config.plugins]);
|
||||
```
|
||||
|
||||
#### 3. Remove Sync Effects
|
||||
|
||||
**Remove lines 152-168** (Effect 1 - config → local sync):
|
||||
|
||||
```tsx
|
||||
// REMOVE THIS ENTIRE EFFECT:
|
||||
useEffect(() => {
|
||||
if (!hasUserInteracted) {
|
||||
const configPlugins = new Set(
|
||||
config.plugins
|
||||
.map((plugin) => (typeof plugin === 'string' ? plugin : plugin.id))
|
||||
.filter((id) => id !== 'policy' && id !== 'intent') as Plugin[],
|
||||
);
|
||||
if (
|
||||
configPlugins.size !== selectedPlugins.size ||
|
||||
!Array.from(configPlugins).every((plugin) => selectedPlugins.has(plugin))
|
||||
) {
|
||||
setSelectedPlugins(configPlugins);
|
||||
}
|
||||
}
|
||||
}, [config.plugins, hasUserInteracted, selectedPlugins]);
|
||||
```
|
||||
|
||||
**Remove lines 171-199** (Effect 2 - local → config sync):
|
||||
|
||||
```tsx
|
||||
// REMOVE THIS ENTIRE EFFECT:
|
||||
useEffect(() => {
|
||||
if (hasUserInteracted) {
|
||||
const policyPlugins = config.plugins.filter((p) => typeof p === 'object' && p.id === 'policy');
|
||||
const intentPlugins = config.plugins.filter((p) => typeof p === 'object' && p.id === 'intent');
|
||||
const regularPlugins = Array.from(selectedPlugins).map((plugin) => {
|
||||
const existingConfig = pluginConfig[plugin];
|
||||
if (existingConfig && Object.keys(existingConfig).length > 0) {
|
||||
return {
|
||||
id: plugin,
|
||||
config: existingConfig,
|
||||
};
|
||||
}
|
||||
return plugin;
|
||||
});
|
||||
const allPlugins = [...regularPlugins, ...policyPlugins, ...intentPlugins];
|
||||
updatePlugins(allPlugins as Array<string | { id: string; config: any }>);
|
||||
}
|
||||
}, [selectedPlugins, pluginConfig, hasUserInteracted, config.plugins, updatePlugins]);
|
||||
```
|
||||
|
||||
#### 4. Refactor `handlePluginToggle`
|
||||
|
||||
**Replace the current implementation (lines 201-236) with:**
|
||||
|
||||
```tsx
|
||||
const handlePluginToggle = useCallback(
|
||||
(plugin: Plugin) => {
|
||||
// Preserve policy and intent plugins
|
||||
const policyPlugins = config.plugins.filter((p) => typeof p === 'object' && p.id === 'policy');
|
||||
const intentPlugins = config.plugins.filter((p) => typeof p === 'object' && p.id === 'intent');
|
||||
|
||||
// Get current regular plugins (excluding policy/intent)
|
||||
const currentRegularPlugins = config.plugins.filter((p) => {
|
||||
const id = typeof p === 'string' ? p : p.id;
|
||||
return id !== 'policy' && id !== 'intent';
|
||||
});
|
||||
|
||||
const isCurrentlySelected = selectedPlugins.has(plugin);
|
||||
|
||||
let newRegularPlugins: Config['plugins'];
|
||||
|
||||
if (isCurrentlySelected) {
|
||||
// Remove the plugin
|
||||
newRegularPlugins = currentRegularPlugins.filter((p) => {
|
||||
const id = typeof p === 'string' ? p : p.id;
|
||||
return id !== plugin;
|
||||
});
|
||||
} else {
|
||||
// Add the plugin
|
||||
addPlugin(plugin); // Add to recently used
|
||||
newRegularPlugins = [...currentRegularPlugins, plugin];
|
||||
}
|
||||
|
||||
// Combine all plugins and update store
|
||||
const allPlugins = [...newRegularPlugins, ...policyPlugins, ...intentPlugins];
|
||||
updatePlugins(allPlugins);
|
||||
},
|
||||
[config.plugins, selectedPlugins, updatePlugins, addPlugin],
|
||||
);
|
||||
```
|
||||
|
||||
#### 5. Add `setSelectedPlugins` Handler for Bulk Operations
|
||||
|
||||
**Add after `handlePluginToggle`:**
|
||||
|
||||
```tsx
|
||||
const setSelectedPlugins = useCallback(
|
||||
(newSelectedPlugins: Set<Plugin>) => {
|
||||
// Preserve policy and intent plugins
|
||||
const policyPlugins = config.plugins.filter((p) => typeof p === 'object' && p.id === 'policy');
|
||||
const intentPlugins = config.plugins.filter((p) => typeof p === 'object' && p.id === 'intent');
|
||||
|
||||
// Create new plugins array, preserving configs from existing plugins
|
||||
const newPluginsArray: Config['plugins'] = Array.from(newSelectedPlugins).map((plugin) => {
|
||||
const existing = config.plugins.find((p) => (typeof p === 'string' ? p : p.id) === plugin);
|
||||
if (existing && typeof existing === 'object' && existing.config) {
|
||||
return existing; // Preserve existing config
|
||||
}
|
||||
return plugin;
|
||||
});
|
||||
|
||||
// Combine all plugins and update store
|
||||
const allPlugins = [...newPluginsArray, ...policyPlugins, ...intentPlugins];
|
||||
updatePlugins(allPlugins);
|
||||
},
|
||||
[config.plugins, updatePlugins],
|
||||
);
|
||||
```
|
||||
|
||||
#### 6. Refactor `updatePluginConfig`
|
||||
|
||||
**Replace the current implementation (lines 238-257) with:**
|
||||
|
||||
```tsx
|
||||
const updatePluginConfig = useCallback(
|
||||
(plugin: string, newConfig: Partial<LocalPluginConfig[string]>) => {
|
||||
// Build new plugins array with updated config
|
||||
const newPlugins = config.plugins.map((p) => {
|
||||
const id = typeof p === 'string' ? p : p.id;
|
||||
if (id === plugin) {
|
||||
const existingConfig = typeof p === 'object' ? p.config || {} : {};
|
||||
return {
|
||||
id: plugin,
|
||||
config: { ...existingConfig, ...newConfig },
|
||||
};
|
||||
}
|
||||
return p;
|
||||
});
|
||||
|
||||
updatePlugins(newPlugins);
|
||||
},
|
||||
[config.plugins, updatePlugins],
|
||||
);
|
||||
```
|
||||
|
||||
#### 7. Update PluginsTab Props
|
||||
|
||||
**Update the PluginsTab component call (around line 431):**
|
||||
|
||||
```tsx
|
||||
<PluginsTab
|
||||
selectedPlugins={selectedPlugins}
|
||||
handlePluginToggle={handlePluginToggle}
|
||||
setSelectedPlugins={setSelectedPlugins} // NEW PROP
|
||||
pluginConfig={pluginConfig}
|
||||
updatePluginConfig={updatePluginConfig}
|
||||
recentlyUsedPlugins={recentlyUsedSnapshot}
|
||||
isRemoteGenerationDisabled={isRemoteGenerationDisabled}
|
||||
/>
|
||||
```
|
||||
|
||||
**Remove** the `onUserInteraction` prop.
|
||||
|
||||
### Success Criteria:
|
||||
|
||||
#### Automated Verification:
|
||||
|
||||
- [x] TypeScript compilation passes: `npm run tsc` from `src/app`
|
||||
- [x] Linting passes: `npm run lint`
|
||||
- [x] All PluginsTab tests pass: `npm run test:app -- src/pages/redteam/setup/components/PluginsTab.test.tsx`
|
||||
|
||||
**Implementation Note**: After completing this phase and all automated verification passes, proceed to Phase 2.
|
||||
|
||||
---
|
||||
|
||||
## Phase 2: Update `PluginsTab.tsx`
|
||||
|
||||
### Overview
|
||||
|
||||
Update the PluginsTab component to use the new `setSelectedPlugins` prop and remove `onUserInteraction`.
|
||||
|
||||
### Changes Required:
|
||||
|
||||
#### 1. Update Props Interface
|
||||
|
||||
**File**: `src/app/src/pages/redteam/setup/components/PluginsTab.tsx`
|
||||
|
||||
**Replace lines 64-72:**
|
||||
|
||||
```tsx
|
||||
export interface PluginsTabProps {
|
||||
selectedPlugins: Set<Plugin>;
|
||||
handlePluginToggle: (plugin: Plugin) => void;
|
||||
setSelectedPlugins: (plugins: Set<Plugin>) => void; // NEW
|
||||
pluginConfig: LocalPluginConfig;
|
||||
updatePluginConfig: (plugin: string, newConfig: Partial<LocalPluginConfig[string]>) => void;
|
||||
recentlyUsedPlugins: Plugin[];
|
||||
isRemoteGenerationDisabled: boolean;
|
||||
// REMOVED: onUserInteraction
|
||||
}
|
||||
```
|
||||
|
||||
#### 2. Update Component Parameters
|
||||
|
||||
**Update lines 74-82:**
|
||||
|
||||
```tsx
|
||||
export default function PluginsTab({
|
||||
selectedPlugins,
|
||||
handlePluginToggle,
|
||||
setSelectedPlugins, // NEW
|
||||
pluginConfig,
|
||||
updatePluginConfig,
|
||||
recentlyUsedPlugins,
|
||||
isRemoteGenerationDisabled,
|
||||
}: PluginsTabProps): React.ReactElement {
|
||||
```
|
||||
|
||||
#### 3. Refactor `handlePresetSelect`
|
||||
|
||||
**Replace the current implementation (around lines 367-392):**
|
||||
|
||||
```tsx
|
||||
const handlePresetSelect = useCallback(
|
||||
(preset: { name: string; plugins: Set<Plugin> | ReadonlySet<Plugin> }) => {
|
||||
recordEvent('feature_used', {
|
||||
feature: 'redteam_config_plugins_preset_selected',
|
||||
preset: preset.name,
|
||||
});
|
||||
if (preset.name === 'Custom') {
|
||||
setIsCustomMode(true);
|
||||
} else {
|
||||
// Use setSelectedPlugins for efficient bulk update
|
||||
setSelectedPlugins(new Set(preset.plugins as Set<Plugin>));
|
||||
setIsCustomMode(false);
|
||||
}
|
||||
},
|
||||
[recordEvent, setSelectedPlugins],
|
||||
);
|
||||
```
|
||||
|
||||
#### 4. Refactor "Select All" Button
|
||||
|
||||
**Replace lines 485-494:**
|
||||
|
||||
```tsx
|
||||
onClick={() => {
|
||||
// Collect all filtered plugins and merge with existing selection
|
||||
const newSelected = new Set(selectedPlugins);
|
||||
filteredPlugins.forEach(({ plugin }) => {
|
||||
newSelected.add(plugin);
|
||||
});
|
||||
setSelectedPlugins(newSelected);
|
||||
}}
|
||||
```
|
||||
|
||||
#### 5. Refactor "Select None" Button
|
||||
|
||||
**Replace lines 499-508:**
|
||||
|
||||
```tsx
|
||||
onClick={() => {
|
||||
// Remove only the filtered plugins from selection
|
||||
const filteredPluginIds = new Set(filteredPlugins.map((p) => p.plugin));
|
||||
const newSelected = new Set(
|
||||
[...selectedPlugins].filter((p) => !filteredPluginIds.has(p)),
|
||||
);
|
||||
setSelectedPlugins(newSelected);
|
||||
}}
|
||||
```
|
||||
|
||||
#### 6. Refactor "Clear All" Button
|
||||
|
||||
**Replace lines 816-820:**
|
||||
|
||||
```tsx
|
||||
onClick={() => {
|
||||
setSelectedPlugins(new Set());
|
||||
}}
|
||||
```
|
||||
|
||||
### Success Criteria:
|
||||
|
||||
#### Automated Verification:
|
||||
|
||||
- [x] TypeScript compilation passes: `npm run tsc` from `src/app`
|
||||
- [x] Linting passes: `npm run lint`
|
||||
- [x] All PluginsTab tests pass: `npm run test:app -- src/pages/redteam/setup/components/PluginsTab.test.tsx`
|
||||
|
||||
**Implementation Note**: After completing this phase and all automated verification passes, proceed to Phase 3.
|
||||
|
||||
---
|
||||
|
||||
## Phase 3: Verify and Fix Tests
|
||||
|
||||
### Overview
|
||||
|
||||
Run the full test suite and fix any test failures. Tests should mostly pass since they test end-to-end behavior (user interaction → store state), not implementation details.
|
||||
|
||||
### Changes Required:
|
||||
|
||||
#### 1. Run Full Test Suite
|
||||
|
||||
```bash
|
||||
cd src/app
|
||||
npm run test -- src/pages/redteam/setup/components/PluginsTab.test.tsx
|
||||
```
|
||||
|
||||
#### 2. Potential Test Adjustments
|
||||
|
||||
The tests should largely pass as-is because they:
|
||||
|
||||
- Verify store state after interactions (still works)
|
||||
- Use `userEvent` to simulate clicks (still works)
|
||||
- Don't mock the intermediate state management
|
||||
|
||||
However, if any tests reference `onUserInteraction` in expectations or setup, they will need updates.
|
||||
|
||||
**If needed, remove references to `onUserInteraction` in test mocks or assertions.**
|
||||
|
||||
### Success Criteria:
|
||||
|
||||
#### Automated Verification:
|
||||
|
||||
- [x] All 30+ tests in `PluginsTab.test.tsx` pass (44 tests passed)
|
||||
- [x] No TypeScript errors
|
||||
- [x] No linting errors
|
||||
|
||||
#### Manual Verification:
|
||||
|
||||
- [x] Start the dev server: `npm run dev:app`
|
||||
- [x] Navigate to Red Team Setup → Plugins page
|
||||
- [x] Select a preset (e.g., "Recommended") → verify plugins appear selected
|
||||
- [x] Toggle individual plugins → verify selection updates
|
||||
- [x] Use "Select all" → verify all visible plugins are selected
|
||||
- [x] Use "Select none" → verify filtered plugins are deselected
|
||||
- [x] Use "Clear All" in sidebar → verify all plugins are cleared
|
||||
- [x] Select `indirect-prompt-injection`, configure it, then toggle other plugins → verify config is preserved
|
||||
- [x] Refresh page → verify plugin selection persists (Zustand persistence)
|
||||
|
||||
**Implementation Note**: ✅ All automated and manual verification complete. Refactor is complete.
|
||||
|
||||
---
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
### Unit Tests (Existing)
|
||||
|
||||
The existing test suite in `PluginsTab.test.tsx` covers:
|
||||
|
||||
- Component rendering
|
||||
- Plugin search filtering
|
||||
- Category filtering
|
||||
- Selected plugins list display
|
||||
- Preset selection → store update
|
||||
- Plugin list item toggle → store update
|
||||
- Select All/None → store update
|
||||
- Clear All → store update
|
||||
|
||||
### Key Test Scenarios to Verify
|
||||
|
||||
1. **Preset Selection**: Clicking a preset card results in correct plugins in store
|
||||
2. **Single Plugin Toggle**: Clicking checkbox adds/removes plugin from store
|
||||
3. **Select All**: All filtered plugins added to store
|
||||
4. **Select None**: All filtered plugins removed from store (preserving others)
|
||||
5. **Clear All**: All plugins removed from store
|
||||
6. **Config Preservation**: Plugin configs survive toggle operations
|
||||
|
||||
### Edge Cases
|
||||
|
||||
- Toggling a plugin that requires configuration
|
||||
- Preserving policy/intent plugins during regular plugin operations
|
||||
- Rapid toggling (React batching)
|
||||
|
||||
## Performance Considerations
|
||||
|
||||
### Improvements
|
||||
|
||||
- **Bulk operations** now update the store once instead of N times (N = number of plugins in operation)
|
||||
- **No duplicate renders** from local state → store → local state sync cycle
|
||||
- **Memoized derivation** prevents unnecessary recomputation
|
||||
|
||||
### Potential Concerns
|
||||
|
||||
- None expected; `useMemo` derivation is O(n) where n = number of plugins
|
||||
- Store's `updatePlugins` already has JSON comparison optimization
|
||||
|
||||
## Migration Notes
|
||||
|
||||
- No data migration needed; store format unchanged
|
||||
- Existing persisted configs will work without changes
|
||||
|
||||
## References
|
||||
|
||||
- Research document: `docs/research/2026-01-08-redteam-plugins-state-management.md`
|
||||
- Test file: `src/app/src/pages/redteam/setup/components/PluginsTab.test.tsx`
|
||||
- Zustand store: `src/app/src/pages/redteam/setup/hooks/useRedTeamConfig.ts`
|
||||
@@ -0,0 +1,754 @@
|
||||
# Proposal: A Layered Package System for Promptfoo
|
||||
|
||||
## Executive Summary
|
||||
|
||||
Promptfoo should move from "one published package that happens to contain many
|
||||
systems" to "one familiar full package backed by a small set of explicit package
|
||||
layers."
|
||||
|
||||
The recommendation is:
|
||||
|
||||
1. Keep `promptfoo` as the default full install and compatibility facade.
|
||||
2. Create private workspace packages first, then publish only the packages whose
|
||||
boundaries prove useful.
|
||||
3. Separate the low-dependency evaluation kernel from Node adapters, CLI,
|
||||
server/UI hosting, redteam, and provider families.
|
||||
4. Make dependency ownership visible and test the packed artifacts users install,
|
||||
not only the source tree.
|
||||
5. Preserve dual ESM/CommonJS support at public package boundaries during the
|
||||
transition.
|
||||
|
||||
This gives lightweight consumers a smaller dependency graph without making the
|
||||
normal `npm install promptfoo` experience worse. It also gives the team a safer
|
||||
path to provider packs and future products without forcing a flag day.
|
||||
|
||||
## Why Change
|
||||
|
||||
Today, the published root package owns several quite different responsibilities:
|
||||
|
||||
- Node library entrypoint
|
||||
- CLI
|
||||
- evaluation engine
|
||||
- database and migrations
|
||||
- sharing and persistence
|
||||
- view server
|
||||
- redteam workflows
|
||||
- many provider SDKs
|
||||
- the built web UI
|
||||
|
||||
That makes the root package convenient, but also broad:
|
||||
|
||||
- The root package currently has `83` direct runtime dependencies and `36`
|
||||
optional dependencies.
|
||||
- Before this prototype, the public library entrypoint in `src/index.ts`
|
||||
imported migrations, models, sharing, provider loading, and redteam APIs.
|
||||
The prototype starts separating that shape by moving the public Node API
|
||||
behind `src/node/evaluate.ts`, keeping the internal orchestration in the Node
|
||||
layer at `src/evaluate.ts`, and carving a first leaf-safe contract subset into
|
||||
`src/contracts/**` while `promptfoo` remains the facade.
|
||||
- `src/main.ts` and `src/commands/view.ts` are already outer-shell concerns,
|
||||
not core evaluation concerns.
|
||||
- Provider loading is centralized enough that optional/provider dependencies are
|
||||
still effectively part of the product shape.
|
||||
|
||||
The system has grown past the point where one package boundary expresses the
|
||||
architecture well.
|
||||
|
||||
## Non-Goals
|
||||
|
||||
- Do not make existing users choose packages on day one.
|
||||
- Do not publish every internal boundary just because it exists.
|
||||
- Do not require a package-manager migration before the architecture improves.
|
||||
- Do not turn providers into runtime-installed plugins as a prerequisite for the
|
||||
split.
|
||||
- Do not combine the package split with an ESM-only migration.
|
||||
|
||||
## Design Principles
|
||||
|
||||
1. **One obvious default.**
|
||||
`promptfoo` remains the package most users install.
|
||||
|
||||
2. **Narrow leaves, convenient facade.**
|
||||
Lightweight consumers should not pay for servers, databases, CLIs, or provider
|
||||
SDKs they do not use.
|
||||
|
||||
3. **No runtime dependency magic.**
|
||||
Dependency installation happens at install/build/publish time, not when an eval
|
||||
starts or a server boots.
|
||||
|
||||
4. **Private boundaries before public promises.**
|
||||
We should first make internal ownership real inside the monorepo, then publish
|
||||
only the packages that survive real use.
|
||||
|
||||
5. **Dual-format correctness over format ideology.**
|
||||
Promptfoo already supports both ESM and CommonJS. Public packages should keep
|
||||
doing that until we intentionally decide otherwise.
|
||||
|
||||
6. **Package artifacts are the contract.**
|
||||
A source-tree green build is not enough; every published package must be packed,
|
||||
installed, imported, required, and exercised as a user would consume it.
|
||||
|
||||
## Recommended Package Topology
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
schema["@promptfoo/schema"]
|
||||
core["@promptfoo/core"]
|
||||
node["@promptfoo/node"]
|
||||
redteam["@promptfoo/redteam"]
|
||||
providers["@promptfoo/provider-*"]
|
||||
view["@promptfoo/view-server"]
|
||||
cli["@promptfoo/cli"]
|
||||
facade["promptfoo"]
|
||||
|
||||
schema --> core
|
||||
core --> node
|
||||
core --> redteam
|
||||
core --> providers
|
||||
node --> view
|
||||
node --> cli
|
||||
redteam --> cli
|
||||
providers --> facade
|
||||
node --> facade
|
||||
redteam --> facade
|
||||
view --> facade
|
||||
cli --> facade
|
||||
```
|
||||
|
||||
### `@promptfoo/schema`
|
||||
|
||||
**Purpose**
|
||||
|
||||
- Config types and runtime schemas
|
||||
- JSON schema generation
|
||||
- Browser-safe shared contracts
|
||||
- Stable serialized result shapes where appropriate
|
||||
|
||||
**May depend on**
|
||||
|
||||
- schema libraries such as `zod`
|
||||
|
||||
**Must not depend on**
|
||||
|
||||
- `fs`, `@libsql/client`, Express, provider SDKs, CLI libraries, server code
|
||||
|
||||
**Why it exists**
|
||||
|
||||
- It is the cleanest shared layer between library, CLI, server, UI, and docs.
|
||||
- It is also the safest first extraction.
|
||||
|
||||
### `@promptfoo/core`
|
||||
|
||||
**Purpose**
|
||||
|
||||
- Pure evaluation domain model
|
||||
- Test planning, prompt expansion, assertions, scoring, result aggregation
|
||||
- Provider/assertion interfaces
|
||||
- No direct filesystem, DB, HTTP-server, or provider-SDK assumptions
|
||||
|
||||
**May depend on**
|
||||
|
||||
- `@promptfoo/schema`
|
||||
- small domain utilities
|
||||
|
||||
**Must not depend on**
|
||||
|
||||
- persistence
|
||||
- web server
|
||||
- CLI
|
||||
- concrete provider SDK packages
|
||||
|
||||
**Why it exists**
|
||||
|
||||
- This is the actual reusable engine people mean when they say "the Node package."
|
||||
- It should be possible to run it with fake providers and in-memory adapters.
|
||||
|
||||
### `@promptfoo/node`
|
||||
|
||||
**Purpose**
|
||||
|
||||
- Node adapters around core
|
||||
- config/file loading
|
||||
- local persistence
|
||||
- migrations
|
||||
- cache
|
||||
- share/report persistence helpers
|
||||
- current `evaluate()`-style Node API
|
||||
|
||||
**May depend on**
|
||||
|
||||
- `@promptfoo/core`
|
||||
- `@promptfoo/schema`
|
||||
- Node-only dependencies such as `@libsql/client`, `glob`, `chokidar`, `dotenv`
|
||||
|
||||
**Why it exists**
|
||||
|
||||
- Most library users still want the batteries-included Node API.
|
||||
- This gives them that without forcing the same dependencies onto future
|
||||
browser-safe or embedded consumers.
|
||||
|
||||
### `@promptfoo/redteam`
|
||||
|
||||
**Purpose**
|
||||
|
||||
- redteam generation, strategies, graders, plugins, reporting
|
||||
|
||||
**May depend on**
|
||||
|
||||
- `@promptfoo/core`
|
||||
- `@promptfoo/node` only where it truly needs Node adapters
|
||||
- redteam-specific dependencies
|
||||
|
||||
**Why it exists**
|
||||
|
||||
- Redteam is now a substantial product surface with its own cadence,
|
||||
dependencies, docs, and CLI flows.
|
||||
|
||||
### `@promptfoo/view-server`
|
||||
|
||||
**Purpose**
|
||||
|
||||
- Express/socket server
|
||||
- API routes
|
||||
- static app serving
|
||||
- local UI hosting
|
||||
|
||||
**May depend on**
|
||||
|
||||
- `@promptfoo/node`
|
||||
- server-specific dependencies
|
||||
|
||||
**Why it exists**
|
||||
|
||||
- The server is useful, but it is not intrinsic to every library consumer.
|
||||
|
||||
### `@promptfoo/cli`
|
||||
|
||||
**Purpose**
|
||||
|
||||
- command registration
|
||||
- process lifecycle
|
||||
- update checks
|
||||
- terminal UX
|
||||
- orchestration across `node`, `redteam`, and `view-server`
|
||||
|
||||
**May depend on**
|
||||
|
||||
- `@promptfoo/node`
|
||||
- `@promptfoo/redteam`
|
||||
- `@promptfoo/view-server`
|
||||
|
||||
**Why it exists**
|
||||
|
||||
- The CLI is a shell around the system, not the system itself.
|
||||
|
||||
### `@promptfoo/provider-*`
|
||||
|
||||
**Purpose**
|
||||
|
||||
- Provider-specific implementations and SDK dependencies
|
||||
- Examples:
|
||||
- `@promptfoo/provider-openai`
|
||||
- `@promptfoo/provider-anthropic`
|
||||
- `@promptfoo/provider-aws`
|
||||
- `@promptfoo/provider-google`
|
||||
- eventually a small `@promptfoo/providers-core` for zero-extra-dependency or
|
||||
very common providers
|
||||
|
||||
**May depend on**
|
||||
|
||||
- `@promptfoo/core`
|
||||
- provider SDKs owned by that package
|
||||
|
||||
**Why it exists**
|
||||
|
||||
- This is where the largest optional dependency savings eventually come from.
|
||||
- It also makes provider ownership and release notes much clearer.
|
||||
|
||||
**Important**
|
||||
|
||||
- Do not start by publishing dozens of provider packages.
|
||||
- First make provider registration explicit and let built-in providers move behind
|
||||
the same registry shape internally.
|
||||
|
||||
### `promptfoo`
|
||||
|
||||
**Purpose**
|
||||
|
||||
- Familiar full package
|
||||
- current CLI binaries
|
||||
- compatibility imports
|
||||
- "everything included" experience
|
||||
|
||||
**Behavior**
|
||||
|
||||
- Depends on the packages that define the full Promptfoo distribution.
|
||||
- Re-exports the stable Node API from `@promptfoo/node`.
|
||||
- Continues to ship the CLI users know.
|
||||
- Becomes the migration shield while the rest of the topology matures.
|
||||
|
||||
## Dependency Policy
|
||||
|
||||
### Ownership Rule
|
||||
|
||||
Every runtime dependency must have one declared owner:
|
||||
|
||||
- core runtime
|
||||
- node runtime
|
||||
- cli runtime
|
||||
- view-server runtime
|
||||
- redteam runtime
|
||||
- provider runtime
|
||||
- docs/app/dev-only
|
||||
|
||||
If a dependency is used by more than one package, we should prefer:
|
||||
|
||||
1. a smaller shared package only when the shared code is real, or
|
||||
2. duplicate declarations when the runtime ownership is legitimately separate.
|
||||
|
||||
Do not keep dependencies at the root merely because multiple packages happen to
|
||||
use them during the transition.
|
||||
|
||||
### Boundary Rule
|
||||
|
||||
- A package may import only from packages below it in the topology.
|
||||
- No package may deep-import another package's `src/**`.
|
||||
- Public imports go through explicit package exports.
|
||||
- UI code depends on schema/browser-safe contracts, not Node internals.
|
||||
|
||||
### Provider Rule
|
||||
|
||||
- Provider SDK dependencies belong to provider packages.
|
||||
- `core` knows only provider interfaces and registration metadata.
|
||||
- `node` may ship a default provider registry, but should not be the owner of every
|
||||
provider SDK forever.
|
||||
|
||||
### Optional Dependency Rule
|
||||
|
||||
- Use `optionalDependencies` only for genuinely optional platform/native
|
||||
capability, not as a substitute for package ownership.
|
||||
- Once a provider family has its own package, its SDK should leave the root
|
||||
optional-dependency bucket.
|
||||
|
||||
## Build and Repository Model
|
||||
|
||||
### Keep npm Workspaces for the First Stage
|
||||
|
||||
The repository already uses npm workspaces and has established CI around them.
|
||||
The split does not require an immediate package-manager migration.
|
||||
|
||||
Recommended first-stage workspace layout:
|
||||
|
||||
```text
|
||||
packages/
|
||||
schema/
|
||||
core/
|
||||
node/
|
||||
redteam/
|
||||
view-server/
|
||||
cli/
|
||||
provider-openai/
|
||||
provider-anthropic/
|
||||
src/app/
|
||||
site/
|
||||
```
|
||||
|
||||
`src/` can migrate inward over time. During the first phase, thin package entry
|
||||
files may point at existing source while we move code by ownership.
|
||||
|
||||
### Build Outputs
|
||||
|
||||
Each publishable package should produce:
|
||||
|
||||
```text
|
||||
dist/
|
||||
esm/
|
||||
cjs/
|
||||
types/
|
||||
```
|
||||
|
||||
and expose only supported entrypoints through `exports`.
|
||||
|
||||
Recommended package export shape:
|
||||
|
||||
```json
|
||||
{
|
||||
"type": "module",
|
||||
"exports": {
|
||||
".": {
|
||||
"types": "./dist/types/index.d.ts",
|
||||
"import": "./dist/esm/index.js",
|
||||
"require": "./dist/cjs/index.cjs"
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
For Node-targeted packages, TypeScript should be checked in `nodenext` mode so
|
||||
the compiler models Node's dual-format resolver. Browser/bundler packages can
|
||||
keep bundler-oriented settings where appropriate.
|
||||
|
||||
### Package Boundary Checks
|
||||
|
||||
Add CI checks for:
|
||||
|
||||
- illegal cross-package imports
|
||||
- public-export drift
|
||||
- missing dependency declarations
|
||||
- duplicate accidental dependencies
|
||||
- root dependency ownership
|
||||
- packed-tarball contents
|
||||
- ESM import smoke
|
||||
- CommonJS require smoke
|
||||
- type-resolution smoke
|
||||
|
||||
The strongest useful addition from the OpenClaw study is a generated dependency
|
||||
ownership report:
|
||||
|
||||
```text
|
||||
dependency owner direct users transitive size
|
||||
@libsql/client @promptfoo/node node ...
|
||||
express @promptfoo/view-server view-server ...
|
||||
@anthropic-ai/sdk provider-anthropic provider package ...
|
||||
```
|
||||
|
||||
That report should fail CI when:
|
||||
|
||||
- a dependency has no owner
|
||||
- a package imports a dependency it does not declare
|
||||
- a dependency owned by a leaf is still imported from a higher layer
|
||||
|
||||
### What We Should Borrow From OpenClaw
|
||||
|
||||
The useful lesson from OpenClaw is not "copy its exact repo layout." It is that
|
||||
dependency management gets simpler when ownership is explicit and package
|
||||
artifacts are tested like user-facing products.
|
||||
|
||||
Adopt:
|
||||
|
||||
- private workspaces as the first architectural boundary
|
||||
- leaf-package ownership of runtime dependencies
|
||||
- dependency ownership reports in CI
|
||||
- tarball checks and clean-install acceptance tests
|
||||
- install/update flows that are explicit, never hidden inside startup paths
|
||||
|
||||
Do not adopt blindly:
|
||||
|
||||
- an immediate package-manager migration
|
||||
- runtime-installable provider plugins before Promptfoo needs that product model
|
||||
- ESM-only publishing while Promptfoo still supports both `import` and `require`
|
||||
|
||||
## Publishing Model
|
||||
|
||||
### Recommendation: One Version Line at First
|
||||
|
||||
Use a fixed-version monorepo release for the first public split:
|
||||
|
||||
- all public `@promptfoo/*` packages share the same version
|
||||
- `promptfoo` depends on exact matching versions of internal public packages
|
||||
- release notes can still call out package-specific changes
|
||||
|
||||
This is easier for users, support, and rollback while boundaries are still
|
||||
forming. Independent versions become worth revisiting only after package
|
||||
consumption patterns are stable.
|
||||
|
||||
### Release Automation
|
||||
|
||||
Keep release automation centralized and extend the current flow rather than
|
||||
inventing a second release system immediately:
|
||||
|
||||
1. build all publishable packages
|
||||
2. run package-graph checks
|
||||
3. pack each package
|
||||
4. install packed artifacts into clean temp projects
|
||||
5. run ESM, CJS, CLI, server, and upgrade smokes
|
||||
6. publish with npm trusted publishing / provenance
|
||||
7. publish the `promptfoo` facade last
|
||||
|
||||
Promptfoo already publishes with provenance in the current release workflow, so
|
||||
this should be an extension of the existing release path rather than a parallel
|
||||
system.
|
||||
|
||||
### Artifact Acceptance
|
||||
|
||||
Before publish, every public package should prove:
|
||||
|
||||
- `npm pack` contents are complete
|
||||
- no source-only files are required at runtime
|
||||
- `import` works
|
||||
- `require` works
|
||||
- TypeScript resolves exported types
|
||||
- `promptfoo` facade still exposes current behavior
|
||||
- upgrade from the last published version works for the top-level package
|
||||
|
||||
The current smoke-test philosophy already says "test the built package, not
|
||||
source code." This proposal extends that idea from the root package to every
|
||||
public package.
|
||||
|
||||
## ESM and CommonJS Strategy
|
||||
|
||||
Promptfoo should remain dual-mode during the split.
|
||||
|
||||
### Rules
|
||||
|
||||
- Public packages ship both `import` and `require` conditions until we make a
|
||||
separate deprecation decision.
|
||||
- Source-of-truth implementation may remain ESM-first.
|
||||
- CJS builds are compatibility artifacts, not a second architecture.
|
||||
- No package may rely on unexported internal paths from another package.
|
||||
- Every public package gets both `import` and `require` smoke tests.
|
||||
|
||||
### Why
|
||||
|
||||
Modern Node can load more ESM from CommonJS than older Node versions could, but
|
||||
Promptfoo already promises a `require` entrypoint today. Keeping that promise
|
||||
while the package graph changes avoids combining two migrations into one.
|
||||
|
||||
## Developer Experience
|
||||
|
||||
The system should feel no worse locally than the repo does today.
|
||||
|
||||
### Required Properties
|
||||
|
||||
- one install at the repo root
|
||||
- one normal full build command
|
||||
- package-local test commands when working narrowly
|
||||
- no manual linking
|
||||
- no publishing knowledge required for ordinary feature work
|
||||
- docs/examples keep using `promptfoo` unless a smaller package is the point of
|
||||
the example
|
||||
|
||||
### Useful Commands
|
||||
|
||||
```bash
|
||||
npm run build
|
||||
npm test
|
||||
npm run test:package -- --package @promptfoo/node
|
||||
npm run deps:ownership
|
||||
npm run pack:check
|
||||
```
|
||||
|
||||
### Golden Path
|
||||
|
||||
Most contributors should continue to:
|
||||
|
||||
1. install once
|
||||
2. edit one package or normal source file
|
||||
3. run focused tests
|
||||
4. run the normal repo checks before merging
|
||||
|
||||
Package boundaries should make reasoning easier, not force every engineer to
|
||||
become a release engineer.
|
||||
|
||||
## Documentation System
|
||||
|
||||
### New Docs to Add
|
||||
|
||||
1. `docs/architecture/packages.md`
|
||||
- package graph
|
||||
- dependency rules
|
||||
- when to add a package
|
||||
|
||||
2. `docs/agents/package-development.md`
|
||||
- how to choose the owning package
|
||||
- package-boundary checks
|
||||
- ESM/CJS export rules
|
||||
|
||||
3. `site/docs/usage/packages.md`
|
||||
- "which package should I install?"
|
||||
- full package vs lightweight library vs provider packs
|
||||
|
||||
4. Per-package README template
|
||||
- purpose
|
||||
- install
|
||||
- supported imports
|
||||
- dependency notes
|
||||
- compatibility guarantees
|
||||
|
||||
### Public Message
|
||||
|
||||
For users, the story should be simple:
|
||||
|
||||
- Install `promptfoo` when you want the normal product.
|
||||
- Install `@promptfoo/node` when you want the Node library without the CLI/server.
|
||||
- Install provider packages only when you want explicit fine-grained control.
|
||||
|
||||
## Migration Plan
|
||||
|
||||
### Phase 0: Add Guardrails Without Moving Code
|
||||
|
||||
- add dependency ownership inventory
|
||||
- add package-artifact smoke harness
|
||||
- add package-boundary linting
|
||||
- add architecture docs
|
||||
|
||||
**Exit criterion**
|
||||
|
||||
- We can explain which dependency belongs to which future package.
|
||||
|
||||
### Phase 1: Extract the Safest Layers Privately
|
||||
|
||||
- create private workspaces for `schema`, `core`, and `node`
|
||||
- move exports and types behind those boundaries
|
||||
- keep `promptfoo` behavior unchanged
|
||||
|
||||
**Exit criterion**
|
||||
|
||||
- Existing CLI and Node consumers pass unchanged through the facade.
|
||||
|
||||
### Phase 2: Split Outer Shells Privately
|
||||
|
||||
- create private `cli` and `view-server`
|
||||
- move server dependencies out of node/core
|
||||
- move command orchestration out of the library layer
|
||||
|
||||
**Exit criterion**
|
||||
|
||||
- `@promptfoo/node` can build and test without CLI/server dependencies.
|
||||
|
||||
### Phase 3: Make Provider Registration Explicit
|
||||
|
||||
- define provider registration API
|
||||
- move a small pilot set behind provider packages internally
|
||||
- start with packages that have obvious heavy dependencies or ownership
|
||||
|
||||
**Exit criterion**
|
||||
|
||||
- Providers can be loaded through one registry path whether bundled or packaged.
|
||||
|
||||
### Phase 4: Publish the First Useful Subpackages
|
||||
|
||||
Recommended first public packages:
|
||||
|
||||
1. `@promptfoo/schema`
|
||||
2. `@promptfoo/node`
|
||||
3. `@promptfoo/view-server`
|
||||
|
||||
Keep `promptfoo` as the full facade.
|
||||
|
||||
**Exit criterion**
|
||||
|
||||
- Real users can consume the smaller packages without undocumented imports or
|
||||
hidden dependencies.
|
||||
|
||||
### Phase 5: Publish Provider Packs Selectively
|
||||
|
||||
Only publish provider packages where there is a clear benefit:
|
||||
|
||||
- large SDK footprint
|
||||
- unusual native/platform dependency
|
||||
- clear ownership
|
||||
- meaningful install-size reduction
|
||||
- independent release pressure
|
||||
|
||||
**Exit criterion**
|
||||
|
||||
- Provider packages reduce the root graph without making provider selection
|
||||
confusing.
|
||||
|
||||
## Alternatives Considered
|
||||
|
||||
### A. Keep One Package Forever
|
||||
|
||||
**Pros**
|
||||
|
||||
- simplest publishing story
|
||||
- zero package churn
|
||||
|
||||
**Cons**
|
||||
|
||||
- dependency graph keeps growing
|
||||
- library consumers keep paying for unrelated surfaces
|
||||
- harder ownership and release reasoning
|
||||
|
||||
### B. Split Everything Immediately
|
||||
|
||||
**Pros**
|
||||
|
||||
- cleanest theoretical end state
|
||||
|
||||
**Cons**
|
||||
|
||||
- too many public promises at once
|
||||
- high migration risk
|
||||
- poor signal about which boundaries users actually need
|
||||
|
||||
### C. Recommended: Layered Split With a Facade
|
||||
|
||||
**Pros**
|
||||
|
||||
- gives smaller packages to users who need them
|
||||
- keeps today's default UX
|
||||
- makes ownership real before multiplying public APIs
|
||||
- lets us stop after any phase if the benefits flatten out
|
||||
|
||||
**Cons**
|
||||
|
||||
- a transitional period with some duplicated packaging work
|
||||
- requires disciplined export and dependency checks
|
||||
|
||||
## Risks and Mitigations
|
||||
|
||||
| Risk | Mitigation |
|
||||
| --------------------------------------------- | ------------------------------------------------------------------------------- |
|
||||
| Internal package churn leaks into public API | Publish only after private boundary use stabilizes |
|
||||
| Dual ESM/CJS builds become inconsistent | Artifact smokes for both import modes on every public package |
|
||||
| Developers slow down in a new monorepo layout | Keep root install/build/test commands as the golden path |
|
||||
| Provider package count becomes confusing | Publish provider packs selectively; keep `promptfoo` full install |
|
||||
| Version skew across packages | Start with fixed versions and exact internal deps |
|
||||
| Release workflow becomes fragile | Reuse current release pipeline, add package-graph and tarball acceptance checks |
|
||||
|
||||
## Decisions Requested
|
||||
|
||||
1. Approve the layered topology as the target direction.
|
||||
2. Approve a private-workspace-first migration rather than immediate public
|
||||
package proliferation.
|
||||
3. Approve `promptfoo` as the long-lived full facade.
|
||||
4. Approve fixed-version public packages for the first release cycle.
|
||||
5. Approve dual ESM/CommonJS support for all first-wave public packages.
|
||||
6. Approve investment in dependency ownership and package-acceptance CI before
|
||||
publishing subpackages.
|
||||
|
||||
## First Concrete Milestone
|
||||
|
||||
The first milestone should be deliberately modest:
|
||||
|
||||
1. Add dependency ownership reporting.
|
||||
2. Create private `packages/schema`, `packages/core`, and `packages/node`.
|
||||
3. Move the public Node API behind `@promptfoo/node` while keeping
|
||||
`promptfoo` exports unchanged.
|
||||
4. Prove that `@promptfoo/node` can build and test without CLI/server imports.
|
||||
5. Add packed-artifact smoke tests for the root package and the private node
|
||||
package.
|
||||
|
||||
If that milestone is not useful, we learn cheaply. If it is useful, the rest of
|
||||
the package system has a clear path.
|
||||
|
||||
## Appendix: Current-Code Signals
|
||||
|
||||
- Before this prototype, `src/index.ts` mixed the Node API with migrations,
|
||||
sharing, provider loading, and redteam exports. The prototype moves the Node
|
||||
orchestration into `src/node/evaluate.ts` as the first concrete seam.
|
||||
- `src/main.ts` is already a CLI shell around lower-level functionality.
|
||||
- `src/commands/view.ts` and `src/server/**` are natural `view-server` owners.
|
||||
- `src/providers/**` is already a natural future provider-package boundary.
|
||||
- `src/types/index.ts` is already being deconstructed, which points toward
|
||||
`schema` as a first extraction.
|
||||
|
||||
## Appendix: 2026 Packaging Notes
|
||||
|
||||
- Use explicit `exports` maps for public packages.
|
||||
- For Node-targeted packages, type-check in `nodenext` mode so TypeScript models
|
||||
the same `import`/`require` behavior Node uses.
|
||||
- Keep npm trusted publishing / provenance in the release path for every public
|
||||
package.
|
||||
- Treat `npm pack` plus clean-install smoke tests as the release contract, not an
|
||||
optional extra.
|
||||
|
||||
## References
|
||||
|
||||
- [Node.js package exports documentation](https://nodejs.org/api/packages.html)
|
||||
- [TypeScript module-system reference](https://www.typescriptlang.org/docs/handbook/modules/reference)
|
||||
- [npm trusted publishing documentation](https://docs.npmjs.com/trusted-publishers)
|
||||
- [npm provenance documentation](https://docs.npmjs.com/generating-provenance-statements)
|
||||
- [npm workspaces documentation](https://docs.npmjs.com/cli/v8/using-npm/workspaces/)
|
||||
@@ -0,0 +1,125 @@
|
||||
# Plan: `promptfoo eval -c <uuid>` Cloud Config Support
|
||||
|
||||
## Context
|
||||
|
||||
Currently, `promptfoo redteam run -c <uuid>` supports loading configs from Promptfoo Cloud by UUID, but `promptfoo eval -c` only accepts local file paths. This feature extends the same cloud config loading pattern to the eval command, allowing users to run `promptfoo eval -c <cloud-uuid>` to fetch and execute a config stored in Promptfoo Cloud.
|
||||
|
||||
The ticket (ENG-1770) has two parts:
|
||||
|
||||
1. **Open Source (this PR)**: Make the CLI accept a cloud UUID for `eval -c`
|
||||
2. **Cloud**: Show the `promptfoo eval -c <uuid>` command in the Cloud run modal (separate repo)
|
||||
|
||||
## Changes
|
||||
|
||||
### 1. Add `getEvalConfigFromCloud()` to `src/util/cloud.ts`
|
||||
|
||||
Create a new function modeled after the existing `getConfigFromCloud()` (line 88-114) but hitting a different endpoint for eval configs:
|
||||
|
||||
```typescript
|
||||
export async function getEvalConfigFromCloud(id: string): Promise<UnifiedConfig> {
|
||||
// Same pattern as getConfigFromCloud but using `configs/${id}` endpoint
|
||||
}
|
||||
```
|
||||
|
||||
- Endpoint: `GET /api/v1/configs/${id}` (eval configs, not redteam-specific)
|
||||
- Reuse existing `makeRequest()` helper (line 25) and `cloudConfig.isEnabled()` check pattern
|
||||
- Same error handling pattern as `getConfigFromCloud`
|
||||
|
||||
### 2. Add UUID detection to `src/commands/eval.ts`
|
||||
|
||||
In `doEval()` (line 107), add UUID detection **before** the existing config path processing at line 142. This mirrors the pattern in `src/redteam/commands/run.ts:62-79`.
|
||||
|
||||
```typescript
|
||||
// Before the existing config path processing (line 142)
|
||||
const UUID_REGEX = /^[A-Fa-f0-9]{8}-[A-Fa-f0-9]{4}-[A-Fa-f0-9]{4}-[A-Fa-f0-9]{4}-[A-Fa-f0-9]{12}$/;
|
||||
|
||||
if (cmdObj.config?.length === 1 && UUID_REGEX.test(cmdObj.config[0])) {
|
||||
const cloudConfigObj = await getEvalConfigFromCloud(cmdObj.config[0]);
|
||||
defaultConfig = cloudConfigObj;
|
||||
cmdObj.config = undefined;
|
||||
}
|
||||
```
|
||||
|
||||
Key behaviors:
|
||||
|
||||
- Only trigger when exactly one config path is provided and it's a UUID
|
||||
- Fetch the config from cloud and use it as `defaultConfig`
|
||||
- Clear `cmdObj.config` so `resolveConfigs` uses `defaultConfig` instead of trying to read a file
|
||||
- Import `getEvalConfigFromCloud` from `../../util/cloud`
|
||||
|
||||
### 3. Update eval command description
|
||||
|
||||
Update the `-c` option description at line 903-906 to mention cloud UUID support:
|
||||
|
||||
```typescript
|
||||
.option(
|
||||
'-c, --config <paths...>',
|
||||
'Path to configuration file or cloud config UUID. Automatically loads promptfooconfig.yaml',
|
||||
)
|
||||
```
|
||||
|
||||
### 4. Add tests
|
||||
|
||||
Create test cases in a new test file or add to existing eval command tests, following the pattern from `test/redteam/commands/run.test.ts`:
|
||||
|
||||
- Test UUID detection triggers cloud fetch
|
||||
- Test local file paths bypass UUID detection
|
||||
- Test error when cloud is not enabled
|
||||
- Test multiple config paths with UUID (should not trigger UUID detection)
|
||||
|
||||
## Files to Modify
|
||||
|
||||
| File | Change |
|
||||
| --------------------------------------------- | ----------------------------------------------------------------------- |
|
||||
| `src/util/cloud.ts` | Add `getEvalConfigFromCloud()` function |
|
||||
| `src/commands/eval.ts` | Add UUID detection + cloud fetch in `doEval()`, update `-c` description |
|
||||
| `test/commands/eval.test.ts` or new test file | Add tests for UUID cloud config |
|
||||
|
||||
## Existing Code to Reuse
|
||||
|
||||
- `makeRequest()` from `src/util/cloud.ts:25` - authenticated HTTP helper
|
||||
- `cloudConfig.isEnabled()` from `src/globalConfig/cloud.ts` - auth check
|
||||
- UUID regex pattern from `src/redteam/commands/run.ts:17`
|
||||
- `resolveConfigs()` from `src/util/config/load.ts:481` - existing config resolution (no changes needed)
|
||||
|
||||
## Verification
|
||||
|
||||
1. **Build**: `npm run build` should succeed
|
||||
2. **Lint**: `npm run l && npm run f`
|
||||
3. **Unit tests**: Run existing + new tests with `npx vitest src/commands/eval` and `npx vitest src/util/cloud`
|
||||
4. **Manual test** (if cloud access available):
|
||||
- `npm run local -- eval -c <valid-uuid> --env-file .env` should fetch config from cloud
|
||||
- `npm run local -- eval -c path/to/config.yaml` should still work as before
|
||||
- `npm run local -- eval -c <invalid-uuid-format>` should fall through to file resolution
|
||||
|
||||
## Decisions (Reconciled)
|
||||
|
||||
1. Use the OSS endpoint contract: `GET /api/v1/configs/:id` returning an envelope (`{ config: ... }`), not a raw payload.
|
||||
2. Support persisted config shape with `providers` and `tests`; loading must not require additional requests.
|
||||
3. Provider references in the config should use `promptfoo://provider/<uuid>`.
|
||||
4. Prompts should be emitted as plain strings.
|
||||
5. `tests` defaults to `[]` when missing.
|
||||
6. `description` falls back to `config.name` when missing.
|
||||
7. `--watch` is not supported when `-c <uuid>` is used. CLI should fail fast with a clear error.
|
||||
8. If `-c <value>` matches UUID format but cloud fetch fails (404/auth disabled), hard-fail.
|
||||
9. If multiple `-c` values are supplied and any value is a UUID, fail with an explicit error stating only one `-c` value is allowed for cloud UUID mode.
|
||||
10. Add tests in both places:
|
||||
- `test/commands/eval.test.ts` for UUID detection/CLI behavior
|
||||
- `test/util/cloud.test.ts` for `getEvalConfigFromCloud()` contract/error handling
|
||||
11. Scope is `eval` only (not `redteam eval`).
|
||||
12. No temporary UI note or minimum-version gating is required.
|
||||
13. In UUID mode, clear/ignore `defaultConfigPath` for the run to prevent accidental local reload/fallback behavior.
|
||||
14. Read-time schema normalization should normalize legacy fields (`providerIds`/`testCases`) into canonical fields (`providers`/`tests`).
|
||||
|
||||
### CLI Error Messages (exact draft text)
|
||||
|
||||
1. Multiple `-c` values with UUID mode:
|
||||
- `Cloud config UUID mode supports exactly one -c value. Use: promptfoo eval -c <cloud-config-uuid>`
|
||||
2. UUID mode with `--watch`:
|
||||
- `--watch is not supported when using a cloud config UUID with -c. Use a local config file path for watch mode.`
|
||||
3. UUID-shaped value with failed cloud fetch:
|
||||
- `Failed to load cloud eval config "<uuid>". <reason>. Cloud UUID inputs do not fall back to local file paths. Check authentication and that the UUID exists.`
|
||||
|
||||
## Remaining Follow-ups
|
||||
|
||||
None.
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,228 @@
|
||||
# Adaptive Rate Limit Scheduler - Architecture
|
||||
|
||||
## Overview
|
||||
|
||||
The adaptive rate limit scheduler automatically handles provider rate limits during evaluations. It's **zero-configuration** - users don't need to change anything. The scheduler transparently wraps all provider calls with intelligent rate limit detection, retry logic, and adaptive concurrency management.
|
||||
|
||||
## Design Goals
|
||||
|
||||
The scheduler addresses common challenges when running evaluations against rate-limited APIs:
|
||||
|
||||
- **No manual tuning**: Users shouldn't need to guess the right `-j` (concurrency) value
|
||||
- **Automatic recovery**: Rate limit errors (429) should be retried, not fail permanently
|
||||
- **Prevent cascading failures**: High concurrency shouldn't cause mass failures
|
||||
- **Zero configuration**: Works out of the box with sensible defaults
|
||||
|
||||
## Architecture
|
||||
|
||||
```text
|
||||
┌─────────────────────────────────────────────────────────────────────────────┐
|
||||
│ Evaluator │
|
||||
│ ┌─────────────────────────────────────────────────────────────────────┐ │
|
||||
│ │ RateLimitRegistry │ │
|
||||
│ │ (Central coordinator - one per evaluation) │ │
|
||||
│ │ │ │
|
||||
│ │ ┌──────────────────┐ ┌──────────────────┐ ┌──────────────────┐ │ │
|
||||
│ │ │ProviderRateLimit │ │ProviderRateLimit │ │ProviderRateLimit │ │ │
|
||||
│ │ │ State │ │ State │ │ State │ │ │
|
||||
│ │ │ (openai/key1) │ │ (openai/key2) │ │ (anthropic) │ │ │
|
||||
│ │ │ │ │ │ │ │ │ │
|
||||
│ │ │ ┌─────────────┐ │ │ ┌─────────────┐ │ │ ┌─────────────┐ │ │ │
|
||||
│ │ │ │ SlotQueue │ │ │ │ SlotQueue │ │ │ │ SlotQueue │ │ │ │
|
||||
│ │ │ │ (FIFO) │ │ │ │ (FIFO) │ │ │ │ (FIFO) │ │ │ │
|
||||
│ │ │ └─────────────┘ │ │ └─────────────┘ │ │ └─────────────┘ │ │ │
|
||||
│ │ │ ┌─────────────┐ │ │ ┌─────────────┐ │ │ ┌─────────────┐ │ │ │
|
||||
│ │ │ │ Adaptive │ │ │ │ Adaptive │ │ │ │ Adaptive │ │ │ │
|
||||
│ │ │ │ Concurrency │ │ │ │ Concurrency │ │ │ │ Concurrency │ │ │ │
|
||||
│ │ │ └─────────────┘ │ │ └─────────────┘ │ │ └─────────────┘ │ │ │
|
||||
│ │ └──────────────────┘ └──────────────────┘ └──────────────────┘ │ │
|
||||
│ └─────────────────────────────────────────────────────────────────────┘ │
|
||||
└─────────────────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
## Component Responsibilities
|
||||
|
||||
### RateLimitRegistry
|
||||
|
||||
**File**: `src/scheduler/rateLimitRegistry.ts`
|
||||
|
||||
Central coordinator that:
|
||||
|
||||
- Creates/retrieves per-provider state based on rate limit keys
|
||||
- Routes provider calls to the appropriate state
|
||||
- Aggregates metrics across all providers
|
||||
- Emits events for monitoring
|
||||
|
||||
```typescript
|
||||
// Usage (automatic in evaluator)
|
||||
const result = await registry.execute(provider, () => provider.callApi(...), {
|
||||
getHeaders: (result) => result.metadata?.headers,
|
||||
isRateLimited: (result, error) => error?.message?.includes('429'),
|
||||
getRetryAfter: (result, error) => parseRetryAfter(headers['retry-after']),
|
||||
});
|
||||
```
|
||||
|
||||
### ProviderRateLimitState
|
||||
|
||||
**File**: `src/scheduler/providerRateLimitState.ts`
|
||||
|
||||
Per-provider state manager that:
|
||||
|
||||
- Manages the slot queue for concurrency control
|
||||
- Tracks rate limit headers from responses
|
||||
- Implements retry logic with exponential backoff
|
||||
- Adapts concurrency based on success/failure patterns
|
||||
- Collects latency metrics
|
||||
|
||||
### SlotQueue
|
||||
|
||||
**File**: `src/scheduler/slotQueue.ts`
|
||||
|
||||
FIFO queue with concurrency limiting:
|
||||
|
||||
- Acquires/releases "slots" for concurrent requests
|
||||
- Blocks when at max concurrency or quota exhausted
|
||||
- Tracks remaining requests/tokens from headers
|
||||
- Schedules queue processing after rate limit windows
|
||||
|
||||
Key insight: **Race-condition-free** slot allocation. All requests queue, then slots are allocated in FIFO order.
|
||||
|
||||
### AdaptiveConcurrency
|
||||
|
||||
**File**: `src/scheduler/adaptiveConcurrency.ts`
|
||||
|
||||
Dynamic concurrency adjustment:
|
||||
|
||||
- **On rate limit**: Reduce concurrency by 50% (multiplicative decrease)
|
||||
- **On sustained success**: Increase by 1 (additive increase)
|
||||
- **Proactive throttling**: Reduce when approaching limits (via headers)
|
||||
|
||||
This implements AIMD (Additive Increase, Multiplicative Decrease) - the same algorithm TCP uses for congestion control.
|
||||
|
||||
### HeaderParser
|
||||
|
||||
**File**: `src/scheduler/headerParser.ts`
|
||||
|
||||
Parses rate limit headers from multiple providers:
|
||||
|
||||
- **OpenAI**: `x-ratelimit-remaining-requests`, `x-ratelimit-limit-requests`
|
||||
- **Anthropic**: `anthropic-ratelimit-requests-remaining`
|
||||
- **Generic**: `retry-after`, `retry-after-ms`, `ratelimit-reset`
|
||||
|
||||
### RetryPolicy
|
||||
|
||||
**File**: `src/scheduler/retryPolicy.ts`
|
||||
|
||||
Determines retry behavior:
|
||||
|
||||
- Exponential backoff with jitter
|
||||
- Respects server `retry-after` headers
|
||||
- Configurable max retries (default: 3)
|
||||
- Retries on: rate limits, timeouts, 502/503/504
|
||||
|
||||
## Data Flow
|
||||
|
||||
```text
|
||||
1. Evaluator calls registry.execute(provider, callFn)
|
||||
│
|
||||
▼
|
||||
2. Registry gets/creates ProviderRateLimitState for this provider
|
||||
│
|
||||
▼
|
||||
3. State.executeWithRetry() is called
|
||||
│
|
||||
▼
|
||||
4. SlotQueue.acquire() - wait for available slot
|
||||
│
|
||||
▼
|
||||
5. Execute callFn() - actual provider API call
|
||||
│
|
||||
▼
|
||||
6. Parse response headers → update rate limit state
|
||||
│
|
||||
▼
|
||||
7. Check if rate limited:
|
||||
├─ Yes → retry with backoff, reduce concurrency
|
||||
└─ No → record success, maybe increase concurrency
|
||||
│
|
||||
▼
|
||||
8. SlotQueue.release() - free slot for next request
|
||||
│
|
||||
▼
|
||||
9. Return result (or throw after max retries)
|
||||
```
|
||||
|
||||
## Rate Limit Key Generation
|
||||
|
||||
Each provider gets a unique "rate limit key" based on:
|
||||
|
||||
- Provider ID (e.g., "openai:chat:gpt-4o")
|
||||
- API key hash (different keys = different rate limits)
|
||||
- Organization ID (if applicable)
|
||||
|
||||
This ensures:
|
||||
|
||||
- Same provider + same key = shared rate limit state
|
||||
- Same provider + different keys = separate rate limits
|
||||
- Different providers = completely isolated
|
||||
|
||||
## Key Design Decisions
|
||||
|
||||
### 1. Zero Configuration
|
||||
|
||||
Users shouldn't need to tune rate limit settings. The scheduler learns from response headers and adapts automatically.
|
||||
|
||||
### 2. Fail-Safe Defaults
|
||||
|
||||
- Default max concurrency: 4 (conservative)
|
||||
- Default retry delay: 60 seconds (when no header)
|
||||
- Max retries: 3 (prevents infinite loops)
|
||||
|
||||
### 3. Proactive Throttling
|
||||
|
||||
Don't wait for 429 errors. When headers show <10% remaining quota, proactively reduce concurrency.
|
||||
|
||||
### 4. Per-Provider Isolation
|
||||
|
||||
Different providers have different rate limits. Don't let OpenAI rate limits affect Anthropic calls.
|
||||
|
||||
### 5. Transparent Integration
|
||||
|
||||
The scheduler wraps `provider.callApi()` without changing the interface. Existing code works unchanged.
|
||||
|
||||
## Metrics
|
||||
|
||||
The scheduler tracks:
|
||||
|
||||
- `totalRequests` - All requests attempted
|
||||
- `completedRequests` - Successful completions
|
||||
- `failedRequests` - Permanent failures (after retries)
|
||||
- `rateLimitHits` - Times 429 was encountered
|
||||
- `retriedRequests` - Requests that required retry
|
||||
- `avgLatencyMs`, `p50LatencyMs`, `p99LatencyMs` - Latency distribution
|
||||
|
||||
## Events
|
||||
|
||||
For monitoring/debugging, the scheduler emits:
|
||||
|
||||
- `slot:acquired` / `slot:released` - Concurrency tracking
|
||||
- `ratelimit:hit` - Rate limit encountered
|
||||
- `ratelimit:learned` - First time seeing provider's limits
|
||||
- `ratelimit:warning` - Approaching rate limit
|
||||
- `concurrency:increased` / `concurrency:decreased` - Adaptive changes
|
||||
- `request:retrying` - Retry in progress
|
||||
|
||||
## Testing
|
||||
|
||||
256 tests covering:
|
||||
|
||||
- Unit tests for each component
|
||||
- Edge cases (negative values, zero values, overflow)
|
||||
- Race condition prevention
|
||||
- Integration with evaluator
|
||||
|
||||
## Performance Characteristics
|
||||
|
||||
- **Overhead**: Minimal - just slot acquisition and header parsing
|
||||
- **Memory**: O(providers) - one state object per unique rate limit key
|
||||
- **Latency buffer**: Circular buffer, last 100 requests, O(1) insertion
|
||||
Reference in New Issue
Block a user