promptfoo--promptfoo/examples/claude-agent-sdk

claude-agent-sdk (Claude Agent SDK Examples)

The Claude Agent SDK provider (aka Claude Code provider) enables you to run agentic evals with configurable tools, permissions, and environments.

npx promptfoo@latest init --example claude-agent-sdk
cd claude-agent-sdk

Setup

Install the Claude Agent SDK:

npm install @anthropic-ai/claude-agent-sdk

Export your Anthropic API key as ANTHROPIC_API_KEY:

export ANTHROPIC_API_KEY=your_api_key_here

Examples

Basic Usage

This example shows Claude Agent SDK in its simplest form - running in a temporary directory with no file system access or tools enabled, behaving similarly to the standard Anthropic provider.

Location: ./basic/

Usage:

(cd basic && promptfoo eval)

Counting LLM turns with trace markers

./basic/promptfooconfig.tracing.yaml enables OTEL tracing and uses trace-span-count over the gen_ai.turn * marker spans the provider emits (one per assistant message / LLM round) to assert how many rounds a task took. See Turn marker spans for the per-provider table and the subagent/cache-hit caveats.

# Use --no-cache so the turn-marker spans are re-emitted on every run; a cached
# response would short-circuit before the tracing code and fail the turn-count assertions.
(cd basic && promptfoo eval -c promptfooconfig.tracing.yaml --no-cache)

Working Directory

This example provides Claude Agent SDK with read-only access to a sample project containing Python, TypeScript, and JavaScript files with intentional bugs for analysis. Because the working_dir is set, Claude Agent SDK has access to the following read-only tools:

• Read - Read file contents
• Grep - Search file contents
• Glob - Find files by pattern
• LS - List directory contents

Location: ./working-dir/

Usage:

(cd working-dir && promptfoo eval)

Advanced Editing

This example shows Claude Agent SDK's ability to modify files with:

• File editing tools: Write, Edit, and MultiEdit tools are added to the default set of read-only tools by setting append_allowed_tools
• Permission mode: permission_mode is set to acceptEdits for automatic approval of file edits
• Automatic git workspace management: The working directory (./workspace) uses beforeAll, afterEach, and afterAll extension hooks defined in hooks.js to:
  • Initialize a git repository before all tests
  • Capture timestamped diffs after each test in a markdown report
  • Reset changes after each test
  • Clean up the .git directory after all tests
• Serial execution: maxConcurrency: 1 to prevent race conditions during concurrent tests

Location: ./advanced/

Usage:

(cd advanced && promptfoo eval)

MCP Integration

This example shows Claude Agent SDK integration with:

• MCP weather server: Uses @h1deya/mcp-server-weather for weather data
• Tool permissions: Specific MCP tools (mcp__weather__get-forecast, mcp__weather__get-alerts)
• External API access: Fetches live weather data for San Francisco

Location: ./mcp/

Usage:

(cd mcp && promptfoo eval)

Structured Output

This example demonstrates Claude Agent SDK's structured output feature, which returns validated JSON that conforms to a schema. It includes:

• JSON schema validation: Define expected output structure with types, enums, and required fields
• Code analysis task: Agent analyzes a Python function for bugs
• Assertion testing: Validates that output matches expected schema and contains correct analysis

Location: ./structured-output/

Usage:

(cd structured-output && promptfoo eval)

Advanced Options

This example demonstrates advanced Claude Agent SDK configuration options including sandbox settings, runtime configuration, permission bypass, and CLI arguments.

Location: ./advanced-options/

Usage:

(cd advanced-options && promptfoo eval)

Features demonstrated:

• Sandbox configuration: Run commands in isolated environments with network restrictions
• Runtime configuration: Specify JavaScript runtime (node, bun, deno)
• Extra CLI arguments: Pass additional flags to Claude Code
• Setting sources: Control where SDK loads settings from
• Permission bypass: Safely bypass permissions for automated testing

AskUserQuestion Handling

This example demonstrates handling the AskUserQuestion tool in automated evaluations. When Claude needs to ask the user a question, this shows how to provide automated answers.

Location: ./ask-user-question/

Usage:

(cd ask-user-question && promptfoo eval)

Features demonstrated:

• Convenience option: Use ask_user_question.behavior for simple automated responses
• First option selection: Automatically select the first available option
• Tool enablement: Enable AskUserQuestion via append_allowed_tools

Skills Testing

This example demonstrates testing Agent Skills with the Claude Agent SDK. Skills are reusable capabilities defined as SKILL.md files that Claude automatically invokes when relevant.

• Skill discovery: Uses setting_sources: ['project'] to load skills from .claude/skills/
• Skill filtering: Uses skills: ['code-review'] (SDK 0.2.120+) to scope the test to a single skill and auto-allow the Skill tool
• Skill assertions: Verifies normalized metadata.skillCalls with the skill-used assertion
• Sample skill: A code review skill that identifies bugs and security issues

Location: ./skills/

Usage:

(cd skills && promptfoo eval)

Skill Comparison

This example compares two versions of the same Claude Agent SDK skill against identical review tasks. It is the Claude companion to examples/openai-codex-sdk/skill-comparison and the runnable form of the agent-skill testing guide.

• Versioned fixtures: Each provider points at a different working_dir with its own .claude/skills/review-standards/SKILL.md
• Skill filter: Uses skills: ['review-standards'] (SDK 0.2.120+) to auto-allow the Skill tool
• Structured output: Shares an output_format schema across both providers via a YAML anchor so JSON results are reliable without prompt gymnastics
• Outcome scoring: A JavaScript assertion scores issue recall against expectedIssues

Location: ./skill-comparison/

Usage:

(cd skill-comparison && promptfoo eval --no-cache)

Plugins

This example demonstrates loading skills from a plugin instead of from setting_sources. Plugins are self-contained directories that bundle skills, agents, hooks, and MCP servers together.

• Plugin loading: Uses plugins: [{type: local, path: ./sample-plugin}] to load a local plugin
• Skill filtering: Uses skills: all to load plugin skills and auto-allow the Skill tool
• Skill assertions: Verifies normalized metadata.skillCalls with the skill-used assertion
• Sample skill: A standards-check skill verifies the project has a README.md

Location: ./plugins/

Usage:

(cd plugins && promptfoo eval)

Cyber Espionage Red Team

This example demonstrates testing AI agents against cyber espionage attack patterns based on Anthropic's "Disrupting AI Espionage" blog post. It includes:

• Simulated target system: Workspace with configuration files, credentials, logs, and sensitive data
• Comprehensive red team plugins: harmful:cybercrime, harmful:cybercrime:malicious-code, ssrf, pii, excessive-agency, and more
• Advanced jailbreak strategies: jailbreak:meta, jailbreak:hydra, crescendo, goat for sophisticated attacks
• Reconnaissance testing: File system access tools (Read, Grep, Glob, Bash) to test security boundaries
• Authorized testing context: Demonstrates responsible security testing practices

Location: ./cyber-espionage/

Usage:

(cd cyber-espionage && promptfoo eval)

⚠️ This example is for authorized security testing only. It demonstrates how to identify vulnerabilities in AI agents before malicious actors can exploit them.