wehub-resource-sync
0d3cb498a3
CI / Shell Format Check (push) Has been cancelled
CI / Check Ruby (3.4) (push) Has been cancelled
CI / CI Config (push) Has been cancelled
CI / Test on Node ${{ matrix.node }} and ${{ matrix.os }}${{ matrix.shard && format(' (shard {0}/3)', matrix.shard) || '' }} (push) Has been cancelled
CI / Build on Node ${{ matrix.node }} (push) Has been cancelled
CI / Style Check (push) Has been cancelled
CI / Generate Assets (push) Has been cancelled
CI / Check Python (3.14) (push) Has been cancelled
CI / Check Python (3.9) (push) Has been cancelled
CI / Build Docs (push) Has been cancelled
CI / Code Scan Action (push) Has been cancelled
CI / Site tests (push) Has been cancelled
CI / webui tests (push) Has been cancelled
CI / Run Integration Tests (push) Has been cancelled
CI / Run Smoke Tests (push) Has been cancelled
CI / Go Tests (push) Has been cancelled
CI / Share Test (push) Has been cancelled
CI / Redteam (Production API) (push) Has been cancelled
CI / Redteam (Staging API) (push) Has been cancelled
CI / GitHub Actions Lint (push) Has been cancelled
CI / Check Ruby (3.0) (push) Has been cancelled
release-please / release-please (push) Has been cancelled
release-please / build (push) Has been cancelled
release-please / publish-npm (push) Has been cancelled
release-please / publish-npm-backfill (push) Has been cancelled
release-please / docker (push) Has been cancelled
release-please / publish-code-scan-action (push) Has been cancelled
release-please / attest-code-scan-action (push) Has been cancelled
Deploy local.promptfoo.app / Deploy to Cloudflare Pages (push) Has been cancelled
Test and Publish Multi-arch Docker Image / test (push) Has been cancelled
Test and Publish Multi-arch Docker Image / build-docker-and-push-digests (map[digest-suffix:linux-amd64 platform:linux/amd64 runner:ubuntu-latest]) (push) Has been cancelled
Test and Publish Multi-arch Docker Image / build-docker-and-push-digests (map[digest-suffix:linux-arm64 platform:linux/arm64 runner:ubuntu-24.04-arm]) (push) Has been cancelled
Test and Publish Multi-arch Docker Image / merge-docker-digests (push) Has been cancelled
Test and Publish Multi-arch Docker Image / Attest Multi-arch Image (push) Has been cancelled
Validate Renovate Config / Validate Renovate Configuration (push) Has been cancelled
56 lines
2.0 KiB
Markdown
56 lines
2.0 KiB
Markdown
# compare-agentic-sdks (Agentic SDK Comparison)
|
|
|
|
Compare OpenAI Codex SDK, Claude Agent SDK, and OpenCode SDK on a security audit task.
|
|
|
|
## Quick Start
|
|
|
|
```bash
|
|
npx promptfoo@latest init --example compare-agentic-sdks
|
|
npx promptfoo eval
|
|
npx promptfoo view
|
|
```
|
|
|
|
## What This Compares
|
|
|
|
Four providers analyze an intentionally vulnerable Python codebase:
|
|
|
|
| Provider | How It Works | Output |
|
|
| -------------------- | -------------------------------------------- | --------------------- |
|
|
| **Codex SDK** | Reads files implicitly, uses `output_schema` | Structured JSON |
|
|
| **Claude Agent SDK** | Uses Read/Grep/Glob tools explicitly | Natural language |
|
|
| **OpenCode SDK** | Uses read/grep/glob tools, provider-agnostic | Natural language |
|
|
| **Plain LLM** | No file access (baseline) | Explains how to audit |
|
|
|
|
## Vulnerabilities Planted
|
|
|
|
The vulnerable code lives in the [test-codebase](./test-codebase/) directory.
|
|
|
|
**`user_service.py`:**
|
|
|
|
- MD5 password hashing
|
|
- Timing attack in authentication
|
|
- Predictable session tokens
|
|
|
|
**`payment_processor.py`:**
|
|
|
|
- Float for currency (precision loss)
|
|
- PCI-DSS violations (storing CVV)
|
|
- Sensitive data in logs
|
|
|
|
## Key Differences
|
|
|
|
**Codex SDK** returns structured JSON matching the schema. Fast, predictable, good for automation. OpenAI only.
|
|
|
|
**Claude Agent SDK** uses file system tools to explore, returns natural language. More flexible, shows reasoning. Anthropic only.
|
|
|
|
**OpenCode SDK** uses file system tools similar to Claude Agent SDK, but supports 75+ LLM providers including Anthropic, OpenAI, Google, Ollama (local), and more.
|
|
|
|
**Plain LLM** can't read files, so it explains how to do a security audit instead of doing one.
|
|
|
|
## Learn More
|
|
|
|
- [Evaluate Coding Agents Guide](/docs/guides/evaluate-coding-agents)
|
|
- [OpenAI Codex SDK](/docs/providers/openai-codex-sdk)
|
|
- [Claude Agent SDK](/docs/providers/claude-agent-sdk)
|
|
- [OpenCode SDK](/docs/providers/opencode-sdk)
|