wehub-resource-sync
0d3cb498a3
Deploy local.promptfoo.app / Deploy to Cloudflare Pages (push) Waiting to run
Test and Publish Multi-arch Docker Image / test (push) Waiting to run
Test and Publish Multi-arch Docker Image / build-docker-and-push-digests (map[digest-suffix:linux-amd64 platform:linux/amd64 runner:ubuntu-latest]) (push) Blocked by required conditions
Test and Publish Multi-arch Docker Image / build-docker-and-push-digests (map[digest-suffix:linux-arm64 platform:linux/arm64 runner:ubuntu-24.04-arm]) (push) Blocked by required conditions
Test and Publish Multi-arch Docker Image / merge-docker-digests (push) Blocked by required conditions
Test and Publish Multi-arch Docker Image / Attest Multi-arch Image (push) Blocked by required conditions
CI / Shell Format Check (push) Waiting to run
CI / Check Ruby (3.4) (push) Waiting to run
CI / CI Config (push) Waiting to run
CI / Test on Node ${{ matrix.node }} and ${{ matrix.os }}${{ matrix.shard && format(' (shard {0}/3)', matrix.shard) || '' }} (push) Blocked by required conditions
CI / Build on Node ${{ matrix.node }} (push) Blocked by required conditions
CI / Style Check (push) Waiting to run
CI / Generate Assets (push) Waiting to run
CI / Check Python (3.14) (push) Waiting to run
CI / Check Python (3.9) (push) Waiting to run
CI / Build Docs (push) Waiting to run
CI / Code Scan Action (push) Waiting to run
CI / Site tests (push) Waiting to run
CI / webui tests (push) Waiting to run
CI / Run Integration Tests (push) Waiting to run
CI / Run Smoke Tests (push) Waiting to run
CI / Go Tests (push) Waiting to run
CI / Share Test (push) Waiting to run
CI / Redteam (Production API) (push) Waiting to run
CI / Redteam (Staging API) (push) Waiting to run
CI / GitHub Actions Lint (push) Waiting to run
CI / Check Ruby (3.0) (push) Waiting to run
release-please / release-please (push) Waiting to run
release-please / build (push) Blocked by required conditions
release-please / publish-npm (push) Blocked by required conditions
release-please / publish-npm-backfill (push) Waiting to run
release-please / docker (push) Blocked by required conditions
release-please / publish-code-scan-action (push) Blocked by required conditions
release-please / attest-code-scan-action (push) Blocked by required conditions
Validate Renovate Config / Validate Renovate Configuration (push) Waiting to run
compare-agentic-sdks (Agentic SDK Comparison)
Compare OpenAI Codex SDK, Claude Agent SDK, and OpenCode SDK on a security audit task.
Quick Start
npx promptfoo@latest init --example compare-agentic-sdks
npx promptfoo eval
npx promptfoo view
What This Compares
Four providers analyze an intentionally vulnerable Python codebase:
| Provider | How It Works | Output |
|---|---|---|
| Codex SDK | Reads files implicitly, uses output_schema |
Structured JSON |
| Claude Agent SDK | Uses Read/Grep/Glob tools explicitly | Natural language |
| OpenCode SDK | Uses read/grep/glob tools, provider-agnostic | Natural language |
| Plain LLM | No file access (baseline) | Explains how to audit |
Vulnerabilities Planted
The vulnerable code lives in the test-codebase directory.
user_service.py:
- MD5 password hashing
- Timing attack in authentication
- Predictable session tokens
payment_processor.py:
- Float for currency (precision loss)
- PCI-DSS violations (storing CVV)
- Sensitive data in logs
Key Differences
Codex SDK returns structured JSON matching the schema. Fast, predictable, good for automation. OpenAI only.
Claude Agent SDK uses file system tools to explore, returns natural language. More flexible, shows reasoning. Anthropic only.
OpenCode SDK uses file system tools similar to Claude Agent SDK, but supports 75+ LLM providers including Anthropic, OpenAI, Google, Ollama (local), and more.
Plain LLM can't read files, so it explains how to do a security audit instead of doing one.