promptfoo--promptfoo/docs/agents/coding-agent-provider-taxonomy.md

# Coding Agent Provider Taxonomy

This document summarizes how promptfoo should think about coding-agent providers,
what has been implemented so far, and what should come next. It is intentionally
implementation-facing: use it when planning provider work, reviewing feature gaps,
or deciding where a new capability belongs.

## Scope

This taxonomy covers providers that run an agentic coding runtime, not ordinary
single-turn model APIs. A coding-agent provider usually has some combination of:

- A workspace or project directory.
- Tool use for files, shell commands, MCP, search, or app connectors.
- A session, thread, or server lifecycle.
- Permission, sandbox, or approval controls.
- Rich metadata beyond final assistant text.

The main providers in this family today are:

| Provider family         | Provider IDs                                          | Runtime boundary                           |
| ----------------------- | ----------------------------------------------------- | ------------------------------------------ |
| OpenAI Codex SDK        | `openai:codex-sdk`, `openai:codex`                    | `@openai/codex-sdk` library                |
| OpenAI Codex app-server | `openai:codex-app-server`, `openai:codex-desktop`     | Local `codex app-server` JSON-RPC process  |
| Claude Agent SDK        | `anthropic:claude-agent-sdk`, `anthropic:claude-code` | `@anthropic-ai/claude-agent-sdk` library   |
| OpenCode SDK            | `opencode:sdk`, `opencode`                            | OpenCode SDK plus local or existing server |

Standard OpenAI, Anthropic, Bedrock, Azure, and other model providers still matter
for grading and comparison, but they are outside this taxonomy unless they expose a
stateful coding-agent runtime.

## Taxonomy Axes

### 1. Runtime Boundary

The first question is where promptfoo stops and the agent runtime starts.

| Boundary                 | Meaning                                                       | Current examples                         |
| ------------------------ | ------------------------------------------------------------- | ---------------------------------------- |
| In-process SDK           | promptfoo calls a package API directly.                       | Codex SDK, Claude Agent SDK              |
| Managed local server     | promptfoo starts a server, then talks to it through a client. | OpenCode when `baseUrl` is unset         |
| Existing server          | promptfoo connects to a runtime it does not configure.        | OpenCode with `baseUrl`                  |
| Local app-server process | promptfoo starts a rich-client protocol server over stdio.    | Codex app-server                         |
| Desktop UI process       | Human-facing native app process.                              | Codex Desktop app, not directly attached |

This distinction matters because it controls what promptfoo can guarantee. If
promptfoo starts the runtime, it can set env vars, working directories, sandbox
options, tracing, and cleanup behavior. If promptfoo attaches to an existing server,
that server owns authentication, installed tools, app connectors, and runtime state.

### 2. Session and Thread State

Coding agents are rarely stateless. Each provider needs explicit semantics for:

- Ephemeral sessions: safe default for independent eval rows.
- Persistent sessions: useful for memory, multi-turn tasks, or regression suites.
- User-supplied sessions: resume an existing thread/session by id.
- Pooling: preserve concurrency without cross-contaminating rows.
- Cleanup: unsubscribe, archive, delete temporary directories, or leave state alone.

The important design rule is that session reuse must be opt-in or very clearly
scoped. Reusing state silently makes eval results order-dependent.

### 3. Workspace and Side Effects

Agent evals should separate filesystem access, network access, and shell access.
Those are different risks.

| Surface     | Safe default                               | Higher-risk mode                                        |
| ----------- | ------------------------------------------ | ------------------------------------------------------- |
| Filesystem  | Temporary directory or read-only workspace | Workspace write or full filesystem access               |
| Shell       | Disabled or approval-gated                 | Allowed command execution                               |
| Network     | Disabled unless explicitly requested       | Host allow-lists, live web/search, package installs     |
| App/plugin  | Not installed or not invoked by default    | App connectors, plugin installs, config writes          |
| Environment | Minimal env                                | Inherited process env with secrets and local auth state |

Provider docs should make clear that `danger-full-access` is not the same as
network access, and read-only filesystem mode does not automatically sanitize env
vars. Each surface should have its own option and its own tests.

### 4. Permission and Interaction Model

Promptfoo evals are non-interactive by default. Agent runtimes often expect a human
to answer approval prompts, permission requests, or clarification questions.
Providers should convert those into deterministic policies.

Common policy categories:

- Command approval.
- File-change approval.
- Permission grants.
- User-input or ask-user-question tools.
- MCP elicitation.
- Dynamic tool calls.
- Plugin or app connector requests.

Default policy should decline, cancel, or return empty answers unless a config opts
into side effects. Every accepted side effect should be visible in metadata.

### 5. Inputs

The baseline input is a prompt string. Coding-agent providers increasingly need
structured inputs:

- Text items.
- Local images or image URLs.
- Skills or plugin references.
- Mentions/app connector references.
- File, diff, or workspace context.

Provider-specific JSON input arrays are acceptable when the underlying runtime has
typed input items. Unknown JSON shapes should usually degrade to plain text rather
than crashing an eval row, unless the provider docs promise strict input parsing.

### 6. Outputs and Metadata

All coding-agent providers should return a normal promptfoo provider result:

- `output`: final assistant-facing text.
- `sessionId`: session/thread id when available.
- `tokenUsage`: runtime usage when available.
- `cost`: estimate when usage and model pricing are known.
- `metadata`: normalized agent metadata.
- `raw`: raw or summarized protocol data when useful and safe.

Provider-specific metadata is still valuable, but consumers need a shared shape for
cross-provider assertions and dashboards. A future shared schema should include:

- Runtime family and version.
- Workspace and sandbox settings.
- Session/thread/turn identifiers.
- Tool trajectories.
- Approval decisions.
- File changes and command executions.
- MCP and dynamic tool calls.
- Skill/plugin/app connector usage.
- Trace ids and span links.

### 7. Observability

Tracing should answer two questions:

- What did the model decide?
- What did the agent runtime do?

Provider tracing should include the top-level `callApi` span, item/tool-level spans
where possible, and sanitized attributes for prompts, commands, tool inputs, file
paths, and outputs. Deep tracing should be opt-in when it requires injecting
OpenTelemetry env vars into a child process.

## What Is Implemented So Far

### Shared Building Blocks

The coding-agent providers already share several practical patterns:

- Optional dependencies are loaded lazily so normal promptfoo installs do not need
  every agent SDK.
- Working directories are validated or created before the agent call.
- Temporary workspaces are cleaned up after evals.
- Session or thread caches are keyed by provider config.
- Provider-level config is stricter than prompt-level merged config.
- Tool and skill usage are surfaced through metadata where possible.
- Tracing is supported for Codex and is partially shared through OpenAI agent
  tracing helpers.

Useful files:

- `src/providers/agentic-utils.ts`
- `src/providers/claude-agent-sdk.ts`
- `src/providers/opencode-sdk.ts`
- `src/providers/openai/codex-sdk.ts`
- `src/providers/openai/codex-app-server.ts`
- `src/providers/registry.ts`

### OpenAI Codex SDK

Status: implemented and documented.

Provider IDs:

- `openai:codex-sdk`
- `openai:codex-sdk:<model>`
- `openai:codex`
- `openai:codex:<model>`

Implemented capabilities:

- Lazy loading for `@openai/codex-sdk`.
- API-key and local Codex login authentication paths.
- Working directory and Git repository safety checks.
- Sandbox, network, web search, approval, and reasoning controls.
- Thread resume and persistent thread pooling.
- JSON schema output.
- Text and local-image input items.
- Skill usage heuristics from `SKILL.md` reads.
- Token usage and cost estimation for known Codex models.
- Streaming aggregation for metadata and tracing.
- Deep tracing into the Codex runtime.
- Default-provider support for grading when Codex credentials are available.

Important limits:

- It is the right default for CI and automation, but it does not expose every rich
  app-server protocol event.
- Skill detection is heuristic.
- Promptfoo still receives a final provider response, not live partial output in
  assertions.

Docs and examples:

- `site/docs/providers/openai-codex-sdk.md`
- `examples/openai-codex-sdk/`

### OpenAI Codex App Server

Status: implemented, documented, and validated with mocked protocol tests plus a
real local eval.

Provider IDs:

- `openai:codex-app-server`
- `openai:codex-app-server:<model>`
- `openai:codex-desktop`
- `openai:codex-desktop:<model>`

Implemented capabilities:

- Spawns `codex app-server --listen stdio://`.
- Drives the app-server JSON-RPC handshake.
- Starts, resumes, unsubscribes, and archives threads according to config.
- Starts turns with model, cwd, sandbox, approval, reasoning, personality,
  collaboration mode, service tier, output schema, and instructions.
- Accepts plain text plus JSON arrays of app-server input items.
- Aggregates streamed item notifications into final text and metadata.
- Captures item counts, command/file/MCP/tool/web-search/reasoning trajectories.
- Handles server requests deterministically through `server_request_policy`.
- Uses safe defaults: read-only sandbox, no approvals, ephemeral threads, minimal env.
- Supports thread pooling and persistent thread cache invalidation.
- Supports raw events, token usage, cost estimation, request timeouts, turn timeouts,
  cleanup, aborts, and process failure handling.
- Supports deep tracing by creating a fresh app-server process per row and injecting
  OpenTelemetry env vars.
- Differentiates the app-server protocol from the Codex Desktop app: promptfoo
  starts its own app-server child process and does not attach to a running Desktop
  app UI process.

Important limits:

- WebSocket transport is not implemented; stdio is the supported transport.
- Live partial output is not exposed to assertions.
- Direct attachment to a running Codex Desktop app is not implemented.
- High-risk protocol requests such as config writes, plugin installs, and arbitrary
  filesystem writes should remain unavailable or explicitly policy-gated.

Docs and examples:

- `site/docs/providers/openai-codex-app-server.md`
- `docs/agents/codex-app-server-provider-notes.md`
- `examples/openai-codex-app-server/`

### Claude Agent SDK

Status: implemented and documented.

Provider IDs:

- `anthropic:claude-agent-sdk`
- `anthropic:claude-code`

Implemented capabilities:

- Lazy loading for `@anthropic-ai/claude-agent-sdk`.
- Anthropic API key, Bedrock, Vertex, and local Claude Code auth flows.
- Temporary or configured working directories.
- Built-in tool controls, allowed/disallowed tools, and permission modes.
- Explicit unsafe permission skip flag.
- MCP server configuration and optional MCP caching.
- AskUserQuestion handling.
- Plugin and local skill support.
- Custom agents, hooks, system prompt overrides, betas, thinking, and effort options.
- Session resume, fork, continue, persistence, and file checkpointing options.
- Sandbox and executable configuration.
- Usage/cost controls such as max budget.

Important limits:

- The SDK owns many semantics, so promptfoo must keep docs aligned with SDK changes.
- Side-effectful modes require external workspace reset discipline.

Docs and examples:

- `site/docs/providers/claude-agent-sdk.md`
- `examples/claude-agent-sdk/`

### OpenCode SDK

Status: implemented and documented.

Provider IDs:

- `opencode:sdk`
- `opencode`

Implemented capabilities:

- Lazy loading for `@opencode-ai/sdk` v1 or v2.
- Starts an OpenCode server when `baseUrl` is unset.
- Connects to an existing OpenCode server when `baseUrl` is provided.
- Supports provider/model selection, variants, workspaces, and custom agents.
- Supports temporary or configured working directories.
- Uses read-only default tools for configured workspaces.
- Supports write/edit/bash tools with explicit permission config.
- Supports JSON Schema structured output through OpenCode `format`.
- Supports sessions and persistent session caching.
- Supports MCP configuration and optional MCP caching when promptfoo starts the
  server.

Important limits:

- When using `baseUrl`, the existing server owns auth, MCP setup, installed agents,
  and server-side configuration.
- OpenCode model support is delegated to OpenCode/models.dev rather than promptfoo's
  normal provider model tables.

Docs and examples:

- `site/docs/providers/opencode-sdk.md`
- `examples/provider-opencode-sdk/`

## Current Naming Guidance

Use provider IDs that encode the runtime boundary, not just the model vendor.

- `openai:codex` should continue to mean the Codex SDK alias because that is the
  best default for automation.
- `openai:codex-app-server` should mean the app-server JSON-RPC protocol.
- `openai:codex-desktop` should remain an alias for app-server behavior unless or
  until promptfoo can actually attach to the Desktop app process.
- `anthropic:claude-code` should remain an alias for Claude Agent SDK because the
  SDK is still built on Claude Code.
- `opencode` can remain a convenience alias for `opencode:sdk`.

Avoid adding unscoped top-level aliases such as `codex:desktop` until the OpenAI
scoped names are stable and the docs can clearly explain the difference between
SDK, app-server, and Desktop UI attachment.

## What To Implement Next

### 1. Shared Agent Metadata Schema

Create a cross-provider `metadata.agent` shape while preserving provider-specific
metadata namespaces such as `metadata.codexAppServer`.

Proposed fields:

- `runtime`: `codex-sdk`, `codex-app-server`, `claude-agent-sdk`, `opencode`.
- `runtimeVersion`: runtime-reported version when available.
- `sessionId`, `threadId`, `turnId`.
- `workingDir`, `sandbox`, `network`, `approvalPolicy`.
- `tools`: normalized tool calls and outcomes.
- `commands`: normalized shell command executions.
- `fileChanges`: normalized file write/edit/delete attempts.
- `approvals`: normalized approval prompts and decisions.
- `mcp`: normalized MCP calls and elicitations.
- `skills`: confirmed and attempted skill usage.
- `trace`: trace ids and span ids when available.

Acceptance criteria:

- Existing provider-specific metadata remains backward compatible.
- Cross-provider assertions can target the same metadata path.
- Tests cover at least Codex SDK, Codex app-server, Claude Agent SDK, and OpenCode.

### 2. Shared Coding-Agent Provider Test Contract

Add a reusable test contract for coding-agent providers. Each provider can implement
the same scenarios with its own mocked runtime.

Core scenarios:

- Missing optional dependency.
- API key/env precedence.
- Working directory validation.
- Prompt-level config merge.
- Safe default sandbox and permissions.
- Structured output.
- Session persistence and cleanup.
- Timeout and abort.
- Runtime process/server failure.
- Tool approval decline by default.
- Metadata normalization.
- Trace sanitization.

Acceptance criteria:

- New providers cannot skip lifecycle and safety cases.
- Mock implementations reset hoisted mocks in `beforeEach`.
- Concurrency tests prove one session/process failure cannot fail unrelated rows.

### 3. Provider Capability Matrix

Add a docs page or generated table that compares coding-agent provider capabilities.

Suggested columns:

- Provider IDs.
- Runtime boundary.
- Optional dependency or CLI requirement.
- Auth modes.
- Workspace model.
- Sandbox controls.
- Shell/file/network controls.
- MCP support.
- Skills/plugins/custom agents.
- Structured output.
- Session persistence.
- Tracing.
- Raw protocol metadata.
- Existing-server attachment.

Acceptance criteria:

- The matrix links to each provider doc.
- It clearly says that Codex Desktop attachment is not currently supported.
- It is tested in the docs build.

### 4. Real Eval QA Matrix

Create a small set of real eval examples that can be run selectively by maintainers.

Minimum scenarios:

- Read-only repo review.
- Structured JSON output.
- Sandbox denial for attempted writes.
- Workspace-write side effect in a disposable fixture.
- Session persistence across two rows.
- Skill or plugin invocation.
- MCP tool call with a deterministic local MCP server.
- Deep tracing smoke test.

Acceptance criteria:

- Each scenario records expected pass/fail/error behavior.
- Each scenario can run with `--no-cache`.
- Side-effectful scenarios use disposable workspaces.
- CI can run a lightweight subset without requiring every optional agent runtime.

### 5. Security and Redteam Coverage

Add adversarial tests for the risky parts of agent runtimes.

High-value cases:

- Prompt injection that asks the agent to reveal env vars.
- Path traversal through input items, tool args, or MCP responses.
- Approval bypass attempts.
- Plugin install/config-write attempts.
- Network enablement attempts when network should be off.
- Symlink writes out of the workspace.
- Malicious `SKILL.md` or plugin instructions.
- Tool output containing secrets that must be redacted from traces.

Acceptance criteria:

- Providers decline or isolate side effects by default.
- Redaction is tested for command output, MCP arguments, tool arguments, prompts,
  and final metadata.
- Any accepted dangerous operation requires explicit config and is visible in
  metadata.

### 6. Codex App Server Follow-Up Features

The app-server provider now covers the core eval path. The next app-server-specific
work should focus on rare protocol features and product integration boundaries.

Candidates:

- `model/list`, `skills/list`, `plugin/list`, `plugin/read`, and `app/list`
  metadata discovery.
- `review/start` support for native review flows.
- `turn/steer` and `turn/interrupt` tests for cancellation and mid-turn control.
- Optional WebSocket transport only if upstream stabilizes it.
- Better raw event snapshots for protocol regression tests.
- Stronger docs around Desktop alias semantics and why promptfoo starts a separate
  app-server process.

Acceptance criteria:

- Rare features are opt-in and deterministic.
- High-risk operations are policy-gated.
- Protocol additions have mocked tests and at least one documented example when
  user-facing.

### 7. Side-Effect Harness

Build a reusable harness for tests and examples that need write access.

It should provide:

- Disposable workspace creation.
- Git repository initialization when needed.
- Snapshot before and after agent runs.
- Symlink and path traversal fixtures.
- Cleanup guarantees.
- Helpers for expected file diffs.

Acceptance criteria:

- No side-effectful provider test mutates the source checkout.
- Tests can assert exact changed files.
- Harness works on macOS, Linux, and Windows CI.

### 8. Documentation Cleanup

Improve the information architecture around coding-agent providers.

Recommended docs:

- A public provider comparison matrix.
- A "Choosing a coding-agent provider" guide.
- A "Managing side effects in agent evals" guide.
- Provider-specific "SDK vs app-server vs Desktop app" sections for Codex.
- Example READMEs that all share the same run, auth, and safety structure.

Acceptance criteria:

- The public docs answer which provider to use for CI, Desktop-like protocol evals,
  Claude Code compatibility, and OpenCode multi-provider setups.
- Example configs validate locally.
- Docs state safe defaults and side-effect responsibilities near every write-capable
  example.

## Review Checklist For New Coding-Agent Provider Work

Before merging a provider in this family, verify:

- Provider IDs are explicit about runtime boundary.
- Optional dependencies fail with actionable install guidance.
- API keys and local-auth modes are documented.
- Env precedence is tested.
- Working directory, sandbox, network, and shell controls are documented and tested.
- Prompt-level config merges do not drop nested provider defaults.
- Hoisted mocks with implementations reset in `beforeEach`.
- Session/thread reuse is opt-in or explicitly scoped.
- Runtime failures do not poison unrelated sessions or rows.
- Approval/user-input/MCP requests have deterministic non-interactive defaults.
- Metadata includes session ids, tool/command/file activity, and approval decisions.
- Trace attributes are sanitized.
- Examples validate and at least one real eval has been run when the runtime is
  locally available.

## Open Questions

- Should promptfoo expose one public `metadata.agent` schema now, or keep it internal
  until at least two providers use it in docs examples?
- Should Codex app-server discovery operations be exposed as provider metadata on
  every call, or only behind an explicit config flag?
- Should any provider support a hard "no side effects" verifier that snapshots the
  workspace and fails if files changed?
- Should remote/existing-server modes be marked as less reproducible in promptfoo
  output metadata?
- Should top-level aliases like `codex:app-server` wait for a broader provider naming
  cleanup?