chore: import upstream snapshot with attribution

examples/anthropic/claude-code-session/README.md

@@ -0,0 +1,63 @@
+# anthropic/claude-code-session (Authenticate via Claude Code session)
+
+This example shows how to run Promptfoo evals against the Anthropic Messages
+API — including `llm-rubric` model-graded assertions — by reusing an existing
+local Claude Code session instead of creating a separate Anthropic Console API
+key.
+
+It's meant for Claude Pro / Max subscribers who already use the
+[`claude` CLI](https://docs.claude.com/en/docs/claude-code/overview) on the
+same machine.
+
+You can run this example with:
+
+```bash
+npx promptfoo@latest init --example anthropic/claude-code-session
+cd anthropic/claude-code-session
+```
+
+## Prerequisites
+
+1. Install and log in to Claude Code:
+
+   ```bash
+   claude /login
+   ```
+
+   Promptfoo reads the OAuth credential that Claude Code stores in either the
+   macOS keychain (`Claude Code-credentials`) or
+   `$HOME/.claude/.credentials.json`.
+
+2. Make sure `ANTHROPIC_API_KEY` is **unset** if you specifically want
+   Promptfoo to use your Claude Code session. If the env var is set, Promptfoo
+   will prefer it over the OAuth credential.
+
+## How it works
+
+The provider config sets `apiKeyRequired: false`, which tells Promptfoo to:
+
+- Skip its upfront API key check.
+- Load the Claude Code OAuth credential from the local session.
+- Authenticate Messages API requests with a Bearer token plus the
+  `claude-code-20250219,oauth-2025-04-20` beta headers.
+- Prepend the required Claude Code identity system block
+  (`"You are Claude Code, Anthropic's official CLI for Claude."`) before your
+  own system prompt.
+
+Requests made this way are expected to count against your Claude
+subscription the same way calls from the `claude` CLI do. Check
+[Anthropic's documentation](https://docs.claude.com/en/docs/claude-code/overview)
+for current billing behavior.
+
+## Run it
+
+```bash
+promptfoo eval
+promptfoo view
+```
+
+## Configuration
+
+See `promptfooconfig.yaml` — both the main provider and the `llm-rubric`
+grader point at `anthropic:messages:claude-sonnet-4-6` with
+`apiKeyRequired: false` so the entire eval runs on your Claude subscription.

examples/anthropic/claude-code-session/promptfooconfig.yaml

@@ -0,0 +1,47 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: |
+  Run evals against Claude via a local Claude Code session (no Anthropic
+  Console API key needed). Requires `claude /login` to have been run on the
+  same machine.
+
+prompts:
+  - |
+    You are a careful technical writer. Rewrite the following sentence for
+    clarity without changing its meaning:
+
+    "{{sentence}}"
+
+providers:
+  - id: anthropic:messages:claude-sonnet-4-6
+    config:
+      # Skip the API key preflight and authenticate via the local Claude Code
+      # OAuth credential (macOS keychain or ~/.claude/.credentials.json).
+      apiKeyRequired: false
+      max_tokens: 512
+
+# Use the same Claude Code session to run the llm-rubric grader. This lets
+# Claude Pro / Max subscribers use semantic model-graded evals without a
+# separate Anthropic Console API key.
+defaultTest:
+  options:
+    provider:
+      id: anthropic:messages:claude-sonnet-4-6
+      config:
+        apiKeyRequired: false
+
+tests:
+  - vars:
+      sentence: 'Utilizing these tools, we can facilitate the optimization of the process.'
+    assert:
+      - type: llm-rubric
+        value: |
+          The rewrite is concise, uses plain English, and preserves the
+          original meaning (improving a process using the tools). It should
+          not invent new details.
+  - vars:
+      sentence: 'The aforementioned refactor exhibits suboptimal efficiency characteristics.'
+    assert:
+      - type: llm-rubric
+        value: |
+          The rewrite is clearer and less jargon-heavy while still saying the
+          refactor is inefficient or slow. It must not add unrelated claims.

examples/anthropic/fable-5-coding/README.md

@@ -0,0 +1,54 @@
+# anthropic/fable-5-coding (Claude Fable 5 Advanced Coding)
+
+This example exercises Claude Fable 5 on hard coding tasks using the `xhigh` effort level. Fable 5 always uses adaptive thinking, so no `thinking` configuration is needed.
+
+You can run this example with:
+
+```bash
+npx promptfoo@latest init --example anthropic/fable-5-coding
+cd fable-5-coding
+```
+
+## What This Tests
+
+Claude Fable 5 is Anthropic's most powerful model — a new tier above Opus. This example evaluates:
+
+- **Distributed-systems debugging** with incomplete information
+- **Production-quality code generation** with concurrency concerns
+- **Security-focused code review** with prioritized feedback
+
+## Working with Fable 5
+
+- **Adaptive thinking is always on.** Unlike Opus 4.7/4.8 (where adaptive thinking is opt-in), Fable 5 always thinks adaptively. There is nothing to configure: promptfoo converts a legacy `thinking: { type: enabled, budget_tokens: N }` config to adaptive and omits `thinking: { type: disabled }`, because the model rejects both.
+- **Thinking summaries are opt-in.** By default the API omits thinking content (an empty thinking block, which promptfoo excludes from output). Set `thinking: { type: adaptive, display: summarized }` to include a readable summary.
+- **Sampling controls are managed for you.** Fable 5 rejects `temperature`, `top_p`, and `top_k` at the model level; promptfoo omits them automatically (don't set them in config).
+- **`effort` defaults to `high`; `xhigh` is available.** Start with `xhigh` for coding and agentic work, and pair high effort with a large `max_tokens` — thinking consumes output tokens before any visible text, so a tight budget can be spent entirely on thinking and yield an empty response.
+- **1M-token context, up to 128K output tokens.** Priced at $10/$50 per million input/output tokens.
+
+## Running the Example
+
+```bash
+# Set your API key
+export ANTHROPIC_API_KEY=your_api_key_here
+
+# Run the evaluation
+npx promptfoo@latest eval
+
+# View results
+npx promptfoo@latest view
+```
+
+## Other providers
+
+Fable 5 is also reachable through:
+
+- AWS Bedrock — `bedrock:global.anthropic.claude-fable-5` (Runtime/Converse; requires the account to opt in to provider data sharing via `aws bedrock put-account-data-retention --mode provider_data_share`), or `bedrock:messages:anthropic.claude-fable-5` for Bedrock's Anthropic-compatible Messages endpoint in `us-east-1`/`eu-north-1`
+- Google Vertex — `vertex:claude-fable-5`
+- Azure AI Foundry — point `anthropic:messages:claude-fable-5` at `https://<resource>.services.ai.azure.com/anthropic` via `apiBaseUrl`
+
+Across all providers, promptfoo automatically omits the unsupported sampling parameters and normalizes unsupported thinking configs for Fable 5. Note that Bedrock's regional and geo endpoints and Vertex's regional/multi-region endpoints (everything except `global`) carry a 10% price premium, which promptfoo includes in cost calculations.
+
+## Learn More
+
+- [Anthropic documentation](https://docs.anthropic.com)
+- [Promptfoo Anthropic provider docs](https://promptfoo.dev/docs/providers/anthropic)

examples/anthropic/fable-5-coding/promptfooconfig.yaml

@@ -0,0 +1,110 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: Claude Fable 5 advanced coding with xhigh effort and always-on adaptive thinking
+
+prompts:
+  - |
+    {{task}}
+
+providers:
+  - id: anthropic:messages:claude-fable-5
+    config:
+      # Fable 5 always uses adaptive thinking — there is no `thinking` block to
+      # set. Manual budgets (`type: enabled`) are converted to adaptive and
+      # `type: disabled` is omitted because thinking cannot be turned off.
+      #
+      # Fable 5 also rejects manual sampling controls (temperature/top_p/top_k)
+      # at the model level — promptfoo omits them automatically, so don't set
+      # them here.
+      effort: xhigh # Recommended starting point for coding/agentic work (between high and max)
+      # Always-on adaptive thinking consumes output tokens before any visible
+      # text. Budget max_tokens generously — at xhigh effort a tight budget can
+      # be spent entirely on thinking, yielding an empty response.
+      max_tokens: 16000
+
+tests:
+  # Distributed-systems debugging with incomplete information
+  - vars:
+      task: |
+        A payment service intermittently double-charges customers. Here's what you know:
+
+        1. The charge endpoint is idempotent — it checks for an existing charge by idempotency key before creating one
+        2. The idempotency check reads from a Postgres replica; writes go to the primary
+        3. Replication lag spikes to 2-3 seconds during peak traffic
+        4. Clients retry on timeout with the same idempotency key
+        5. Double charges only happen during peak hours
+        6. Each double charge shows two rows with the same idempotency key but different charge IDs
+
+        Diagnose the root cause and propose a fix. Explain why the idempotency check fails despite using the same key.
+    assert:
+      - type: contains-any
+        value: ['replica', 'replication lag', 'read-after-write', 'primary', 'race']
+        reason: Should identify the stale-replica read as the root cause
+      - type: llm-rubric
+        value: |
+          The response should:
+          1. Identify the root cause (idempotency check reads a lagging replica, so a retry during the lag window misses the in-flight charge)
+          2. Explain why it only happens at peak (replication lag exceeds the client retry interval)
+          3. Propose concrete fixes (read idempotency checks from the primary, unique constraint on idempotency key, or insert-first/conflict-based idempotency)
+          4. Note that a unique constraint is the only fix that closes the race completely
+
+  # Production-quality code generation with concurrency concerns
+  - vars:
+      task: |
+        Write a TypeScript class implementing a token-bucket rate limiter that:
+        1. Supports a configurable capacity and refill rate
+        2. Is safe to call concurrently from async code
+        3. Exposes acquire() that resolves when a token is available (with optional timeout)
+        4. Avoids busy-waiting and timer leaks
+
+        Include proper typing and comments explaining design decisions.
+    assert:
+      - type: contains
+        value: 'class'
+        reason: Should define a TypeScript class
+      - type: contains-any
+        value: ['Promise', 'async', 'await']
+        reason: Should use async primitives
+      - type: llm-rubric
+        value: |
+          The code should:
+          1. Implement correct token-bucket math (refill proportional to elapsed time, capped at capacity)
+          2. Queue waiters instead of busy-waiting
+          3. Clean up timers when acquire() times out or resolves
+          4. Use precise TypeScript types (no `any`)
+          5. Include comments explaining the concurrency-safety reasoning
+          6. Be production-ready (not a toy example)
+
+  # Code review with prioritized, nuanced feedback
+  - vars:
+      task: |
+        Review this Express handler and provide feedback:
+
+        ```js
+        app.post('/upload', async (req, res) => {
+          const file = req.files.document;
+          const dest = path.join('/uploads', req.body.filename);
+          await file.mv(dest);
+          const meta = JSON.parse(req.body.metadata);
+          db.query(`INSERT INTO uploads (path, owner) VALUES ('${dest}', '${meta.owner}')`);
+          res.json({ ok: true, path: dest });
+        });
+        ```
+
+        Identify issues, suggest improvements, and explain the reasoning behind each suggestion. Prioritize by severity.
+    assert:
+      - type: contains-any
+        value: ['injection', 'traversal', 'sanitize', 'parameterized']
+        reason: Should identify the SQL injection and path traversal vulnerabilities
+      - type: llm-rubric
+        value: |
+          The review should identify multiple issues, prioritized by severity:
+          1. SQL injection via string-interpolated query (critical)
+          2. Path traversal via user-controlled filename (critical)
+          3. Unhandled JSON.parse exception on malformed metadata
+          4. No validation that req.files.document exists
+          5. Unawaited/unchecked db.query result
+
+          For each issue, it should:
+          - Explain why it's a problem
+          - Suggest specific improvements (parameterized queries, basename/allowlist for paths)
+          - Provide example code where helpful

examples/anthropic/mcp/README.md

@@ -0,0 +1,48 @@
+# anthropic/mcp (Anthropic Messages + MCP tools)
+
+This example wires Claude up to a [Model Context Protocol](https://modelcontextprotocol.io) server through the Anthropic Messages provider. Promptfoo exposes the MCP server's tools to Claude, executes any `tool_use` blocks the model emits, and feeds the `tool_result` back into the conversation until Claude produces a final reply.
+
+```bash
+npx promptfoo@latest init --example anthropic/mcp
+cd anthropic/mcp
+```
+
+## Setup
+
+```bash
+export ANTHROPIC_API_KEY=sk-ant-...
+npx promptfoo@latest eval
+```
+
+The bundled config uses the public [deepwiki MCP server](https://mcp.deepwiki.com/) so the example works out of the box without installing anything. To swap in your own server, replace the `mcp` block:
+
+```yaml
+mcp:
+  enabled: true
+  # Local stdio server
+  server:
+    command: npx
+    args: ['-y', '@modelcontextprotocol/server-filesystem', '/tmp/workspace']
+  # …or remote SSE / streamable HTTP
+  # servers:
+  #   - name: my-server
+  #     url: https://example.com/mcp
+```
+
+## What it demonstrates
+
+- **Tool discovery**: Tools exposed by the MCP server are forwarded to Claude alongside any inline `tools` you define.
+- **Multi-round execution**: When Claude returns a `tool_use` block matching an MCP tool, promptfoo invokes the tool with the model's arguments and appends a `tool_result` on the next user turn. The loop continues until Claude returns text — bounded by `max_tool_calls` (default `8`).
+- **Error pass-through**: Tool errors come back as `tool_result` blocks with `is_error: true` so Claude can recover or report the failure.
+- **Mixed tool handling**: Non-MCP tools (regular function-calling tools, or built-ins like `web_search`) fall through to the existing tool-use output without being auto-executed.
+
+## Caching
+
+Promptfoo's disk response cache is **skipped automatically** when `mcp.enabled` is `true`, because tool results can vary between runs. Use `max_tool_calls` to bound the per-request cost.
+
+## See also
+
+- [Anthropic provider docs](https://promptfoo.dev/docs/providers/anthropic/#model-context-protocol-mcp)
+- [MCP integration guide](https://promptfoo.dev/docs/integrations/mcp/) — full server configuration (auth, timeouts, multi-server)
+- [examples/openai-mcp](../../openai-mcp/) — the same pattern via the OpenAI Responses API
+- [examples/simple-mcp](../../simple-mcp/) — testing an MCP server directly without an LLM provider

examples/anthropic/mcp/promptfooconfig.yaml

@@ -0,0 +1,46 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: 'Anthropic Messages provider with MCP tool execution'
+
+prompts:
+  - 'Answer the question using the available MCP tools, then summarize what you found in 2-3 sentences. Question: {{question}}'
+
+providers:
+  - id: anthropic:messages:claude-sonnet-4-6
+    label: 'Claude Sonnet 4.6 + deepwiki MCP'
+    config:
+      max_tokens: 1500
+      temperature: 0
+      # Cap how many MCP rounds promptfoo will run per request. Default is 8.
+      max_tool_calls: 5
+      mcp:
+        enabled: true
+        servers:
+          # Hosted MCP server with two tools: ask_question, read_wiki_structure.
+          # Streams over HTTP — no local install required to try this example.
+          - name: deepwiki
+            url: https://mcp.deepwiki.com/mcp
+
+tests:
+  - vars:
+      question: 'Which transport protocols are described in the modelcontextprotocol/modelcontextprotocol repository?'
+    assert:
+      - type: contains-any
+        value: ['stdio', 'sse', 'streamable']
+      - type: llm-rubric
+        value: 'The response identifies at least one MCP transport (stdio, SSE, or streamable HTTP) and explains its purpose.'
+
+  - vars:
+      question: 'What is the high-level architecture of the promptfoo/promptfoo repository according to its docs?'
+    assert:
+      - type: contains
+        value: 'promptfoo'
+      - type: llm-rubric
+        value: 'The response describes promptfoo as an evaluation framework for LLMs (or similar wording) and mentions at least one concrete feature such as evals, redteam, providers, or assertions.'
+
+  # Demonstrate the loop terminating gracefully when the model decides it has
+  # enough information without further tool calls.
+  - vars:
+      question: 'In one sentence, what does MCP stand for?'
+    assert:
+      - type: contains
+        value: 'Model Context Protocol'

examples/anthropic/memory-tool/README.md

@@ -0,0 +1,26 @@
+# anthropic/memory-tool (Anthropic Memory Tool)
+
+This example shows how to include Anthropic's native `memory_20250818` tool in a Promptfoo eval.
+
+```bash
+npx promptfoo@latest init --example anthropic/memory-tool
+cd anthropic/memory-tool
+```
+
+## Features Demonstrated
+
+- Passing the Anthropic memory tool through the Messages provider
+- Restricting the tool to direct model calls
+- Keeping the eval single-turn by disabling tool use with `tool_choice`
+
+## Required Environment Variables
+
+- `ANTHROPIC_API_KEY` - Your Anthropic API key from [console.anthropic.com](https://console.anthropic.com/settings/keys)
+
+## Running the Example
+
+```bash
+promptfoo eval
+```
+
+Promptfoo sends the memory tool definition to Anthropic, but it does not create memory stores or run local memory handlers. Use this pattern to validate prompt behavior around memory-tool availability, or remove `tool_choice` when you want the model to request memory operations for assertion.

examples/anthropic/memory-tool/promptfooconfig.yaml

@@ -0,0 +1,23 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: 'Anthropic memory tool'
+
+prompts:
+  - 'Return exactly: MEMORY_TOOL_AVAILABLE'
+
+providers:
+  - id: anthropic:messages:claude-sonnet-4-6
+    config:
+      temperature: 0
+      max_tokens: 64
+      tools:
+        - type: memory_20250818
+          name: memory
+          allowed_callers:
+            - direct
+      tool_choice:
+        type: none
+
+tests:
+  - assert:
+      - type: contains
+        value: MEMORY_TOOL_AVAILABLE

examples/anthropic/opus-4-6-coding/README.md

@@ -0,0 +1,55 @@
+# anthropic/opus-4-6-coding (Claude Opus 4.6 Advanced Coding)
+
+This example demonstrates Claude Opus 4.6's state-of-the-art coding and reasoning capabilities, showcasing its ability to handle complex software engineering tasks with ambiguity and tradeoff analysis.
+
+You can run this example with:
+
+```bash
+npx promptfoo@latest init --example anthropic/opus-4-6-coding
+cd anthropic/opus-4-6-coding
+```
+
+## What This Tests
+
+Claude Opus 4.6 is the best model in the world for coding, agents, and computer use. This example evaluates:
+
+- **Complex code analysis**: Understanding multi-file codebases and architectural decisions
+- **Bug diagnosis**: Identifying root causes in complex, multi-system scenarios
+- **Ambiguity handling**: Making informed decisions when requirements are unclear
+- **Tradeoff reasoning**: Evaluating different approaches and explaining pros/cons
+- **Code generation**: Writing high-quality, production-ready code
+
+## Features Demonstrated
+
+1. **State-of-the-art coding**: Opus 4.6 achieves the highest score on SWE-bench Verified among frontier models
+2. **Reasoning about tradeoffs**: The model excels at analyzing different approaches and making informed decisions
+3. **Handling ambiguity**: Unlike models that require hand-holding, Opus 4.6 figures things out
+4. **Extended thinking**: Support for thinking budgets up to 128K tokens for complex reasoning
+
+## Running the Example
+
+```bash
+# Set your API key
+export ANTHROPIC_API_KEY=your_api_key_here
+
+# Run the evaluation
+npx promptfoo@latest eval
+
+# View results
+npx promptfoo@latest view
+```
+
+## Expected Results
+
+The evaluation tests Opus 4.6's ability to:
+
+- Diagnose bugs across multiple system boundaries
+- Choose appropriate data structures with clear reasoning
+- Write production-quality code with proper error handling
+- Analyze architectural decisions and propose improvements
+
+## Learn More
+
+- [Claude Opus 4.6 announcement](https://www.anthropic.com/news/claude-opus-4-6)
+- [Anthropic documentation](https://docs.anthropic.com)
+- [Promptfoo Anthropic provider docs](https://promptfoo.dev/docs/providers/anthropic)

examples/anthropic/opus-4-6-coding/promptfooconfig.yaml

@@ -0,0 +1,164 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: Claude Opus 4.6 advanced coding capabilities
+
+prompts:
+  - |
+    {{task}}
+
+providers:
+  - id: anthropic:messages:claude-opus-4-6
+    config:
+      temperature: 0
+      max_tokens: 8000
+
+tests:
+  # Complex bug diagnosis across multiple systems
+  - vars:
+      task: |
+        You're debugging a production issue where users can't log in. Here's what you know:
+
+        1. The frontend shows "Authentication failed" after username/password submission
+        2. Backend logs show successful JWT generation
+        3. Redis cache is returning stale session data
+        4. Database shows correct user credentials
+        5. The issue only affects 10% of login attempts
+        6. It started after deploying a load balancer configuration change
+
+        Diagnose the root cause and propose a fix. Explain your reasoning about what's causing the intermittent nature of the bug.
+    assert:
+      - type: contains-any
+        value: ['load balancer', 'session', 'sticky', 'affinity', 'routing']
+        reason: Should identify load balancer session routing as the issue
+      - type: llm-rubric
+        value: |
+          The response should:
+          1. Identify the root cause (likely session affinity/sticky sessions issue with load balancer)
+          2. Explain why it's intermittent (different backend servers, inconsistent session state)
+          3. Propose concrete fixes (enable sticky sessions, shared session store, stateless tokens)
+          4. Show reasoning about the tradeoffs of different solutions
+
+  # Data structure selection with tradeoff analysis
+  - vars:
+      task: |
+        You need to implement a feature that:
+        - Stores 10 million user activity records per day
+        - Supports queries like "find all activities for user X in date range Y"
+        - Needs to return results in under 100ms
+        - Data retention is 90 days
+        - Budget allows moderate infrastructure costs
+
+        What data structure and storage approach would you use? Explain the tradeoffs you considered.
+    assert:
+      - type: llm-rubric
+        value: |
+          The response should:
+          1. Propose a specific data structure/database (e.g., time-series DB, partitioned PostgreSQL, or similar)
+          2. Explain performance characteristics and why they meet the requirements
+          3. Discuss tradeoffs (cost vs performance, complexity vs maintainability)
+          4. Consider alternatives and explain why they were not chosen
+          5. Address scalability and data retention strategies
+      - type: contains-any
+        value: ['index', 'partition', 'query', 'performance', 'scale']
+        reason: Should discuss database optimization concepts
+
+  # Production-quality code generation with error handling
+  - vars:
+      task: |
+        Write a Python function that:
+        1. Fetches user data from a REST API (may timeout or return errors)
+        2. Caches results in Redis with 5-minute TTL
+        3. Falls back to database if cache miss
+        4. Returns user object or raises appropriate exception
+
+        Include proper error handling, typing, and comments explaining design decisions.
+    assert:
+      - type: contains
+        value: 'def'
+        reason: Should include Python function definition
+      - type: contains-any
+        value: ['try', 'except', 'raise', 'error']
+        reason: Should include error handling
+      - type: contains-any
+        value: ['cache', 'redis', 'ttl']
+        reason: Should implement caching logic
+      - type: llm-rubric
+        value: |
+          The code should:
+          1. Include proper type hints (from typing import ...)
+          2. Handle network timeouts and API errors gracefully
+          3. Implement cache-aside pattern correctly
+          4. Include docstrings and comments explaining design decisions
+          5. Use appropriate exception types
+          6. Be production-ready (not a toy example)
+
+  # Architectural decision with ambiguous requirements
+  - vars:
+      task: |
+        A startup wants to build a "social media analytics dashboard." They mention:
+        - "It should be fast"
+        - "We need real-time data"
+        - "Budget is tight but we might scale quickly"
+        - "Our team knows React and Python"
+
+        The requirements are intentionally vague. Propose an initial architecture, explain what assumptions you made, what questions you'd ask to clarify requirements, and what tradeoffs you considered.
+    assert:
+      - type: llm-rubric
+        value: |
+          The response should:
+          1. Propose a concrete but appropriately simple architecture
+          2. Explicitly state assumptions made (e.g., "Assuming 'real-time' means <1 second latency")
+          3. List specific clarifying questions (user scale, data volume, analytics complexity)
+          4. Explain technology choices based on team skills and constraints
+          5. Discuss tradeoffs (e.g., managed services vs self-hosted, cost vs performance)
+          6. Acknowledge what's unknown and how that affects the design
+      - type: contains-any
+        value: ['assumption', 'clarify', 'question', 'tradeoff', 'alternative']
+        reason: Should handle ambiguity explicitly
+
+  # Code review with nuanced feedback
+  - vars:
+      task: |
+        Review this React component and provide feedback:
+
+        ```jsx
+        function UserList() {
+          const [users, setUsers] = useState([]);
+
+          useEffect(() => {
+            fetch('/api/users')
+              .then(res => res.json())
+              .then(data => setUsers(data));
+          }, []);
+
+          return (
+            <div>
+              {users.map(user => (
+                <div key={user.id}>
+                  <h3>{user.name}</h3>
+                  <p>{user.email}</p>
+                </div>
+              ))}
+            </div>
+          );
+        }
+        ```
+
+        Identify issues, suggest improvements, and explain the reasoning behind each suggestion.
+    assert:
+      - type: contains-any
+        value: ['error', 'loading', 'state', 'async']
+        reason: Should identify missing error and loading states
+      - type: llm-rubric
+        value: |
+          The review should identify multiple issues:
+          1. No error handling for failed fetch
+          2. No loading state
+          3. No cleanup for fetch in useEffect
+          4. Missing dependencies might cause issues in strict mode
+          5. No null/empty checks for users array
+
+          For each issue, it should:
+          - Explain why it's a problem
+          - Suggest specific improvements
+          - Provide example code where helpful
+          - Prioritize issues by severity

examples/anthropic/opus-4-8-coding/README.md

@@ -0,0 +1,54 @@
+# anthropic/opus-4-8-coding (Claude Opus 4.8 Advanced Coding)
+
+This example exercises Claude Opus 4.8 on hard coding tasks using the `xhigh` effort level with adaptive thinking.
+
+You can run this example with:
+
+```bash
+npx promptfoo@latest init --example anthropic/opus-4-8-coding
+cd opus-4-8-coding
+```
+
+## What This Tests
+
+Claude Opus 4.8 is Anthropic's most capable model for complex reasoning and agentic coding. This example evaluates:
+
+- **Bug diagnosis** across multiple system boundaries
+- **Production-quality code generation** with proper error handling
+- **Code review** with nuanced, prioritized feedback
+
+## Working with Opus 4.8
+
+- **Builds on Opus 4.7.** Opus 4.8 supports the same feature set as 4.7 (no breaking API changes) and improves capability on complex reasoning and long-horizon agentic coding.
+- **Adaptive thinking is opt-in.** Set `thinking: { type: adaptive }` (as this example does) to let the model decide when and how much to reason per request. Without an explicit `thinking` block the model runs **without** extended thinking, even at high effort.
+- **`effort` defaults to `high`; `xhigh` is available.** Setting `effort: high` behaves the same as omitting it. Start with `xhigh` for coding and agentic work, and pair high effort with a large `max_tokens`.
+- **Sampling controls are managed for you.** Opus 4.8 rejects `temperature`, `top_p`, and `top_k` at the model level; promptfoo omits them automatically (don't set them in config).
+
+## Running the Example
+
+```bash
+# Set your API key
+export ANTHROPIC_API_KEY=your_api_key_here
+
+# Run the evaluation
+npx promptfoo@latest eval
+
+# View results
+npx promptfoo@latest view
+```
+
+## Other providers
+
+Opus 4.8 is also reachable through:
+
+- AWS Bedrock — `bedrock:us.anthropic.claude-opus-4-8` (or `bedrock:converse:us.anthropic.claude-opus-4-8`)
+- Google Vertex — `vertex:claude-opus-4-8` with `config.region: global`
+- Azure AI Foundry — point `anthropic:messages:claude-opus-4-8` at `https://<resource>.services.ai.azure.com/anthropic` via `apiBaseUrl`
+
+Across all four providers, promptfoo automatically omits the unsupported sampling parameters (`temperature`, `top_p`, `top_k`) for Opus 4.8. The Anthropic Messages provider also logs a one-time warning if you set them explicitly; the Bedrock, Vertex, and Azure paths omit them silently.
+
+## Learn More
+
+- [Claude Opus 4.8 announcement](https://www.anthropic.com/news/claude-opus-4-8)
+- [Anthropic documentation](https://docs.anthropic.com)
+- [Promptfoo Anthropic provider docs](https://promptfoo.dev/docs/providers/anthropic)

examples/anthropic/opus-4-8-coding/promptfooconfig.yaml

@@ -0,0 +1,124 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: Claude Opus 4.8 advanced coding with xhigh effort and adaptive thinking
+
+prompts:
+  - |
+    {{task}}
+
+providers:
+  - id: anthropic:messages:claude-opus-4-8
+    config:
+      # Opus 4.8 deprecates manual sampling controls (temperature/top_p/top_k) at
+      # the model level — promptfoo omits them automatically, so don't set them here.
+      #
+      # Adaptive thinking is opt-in: without an explicit `thinking` block the model
+      # runs WITHOUT extended thinking even at high effort. Set it to let the model
+      # decide when and how much to reason per request.
+      thinking:
+        type: adaptive
+      effort: xhigh # Recommended starting point for coding/agentic work (between high and max)
+      max_tokens: 8000
+
+tests:
+  # Complex bug diagnosis across multiple systems
+  - vars:
+      task: |
+        You're debugging a production issue where users can't log in. Here's what you know:
+
+        1. The frontend shows "Authentication failed" after username/password submission
+        2. Backend logs show successful JWT generation
+        3. Redis cache is returning stale session data
+        4. Database shows correct user credentials
+        5. The issue only affects 10% of login attempts
+        6. It started after deploying a load balancer configuration change
+
+        Diagnose the root cause and propose a fix. Explain your reasoning about what's causing the intermittent nature of the bug.
+    assert:
+      - type: contains-any
+        value: ['load balancer', 'session', 'sticky', 'affinity', 'routing']
+        reason: Should identify load balancer session routing as the issue
+      - type: llm-rubric
+        value: |
+          The response should:
+          1. Identify the root cause (likely session affinity/sticky sessions issue with load balancer)
+          2. Explain why it's intermittent (different backend servers, inconsistent session state)
+          3. Propose concrete fixes (enable sticky sessions, shared session store, stateless tokens)
+          4. Show reasoning about the tradeoffs of different solutions
+
+  # Production-quality code generation with error handling
+  - vars:
+      task: |
+        Write a Python function that:
+        1. Fetches user data from a REST API (may timeout or return errors)
+        2. Caches results in Redis with 5-minute TTL
+        3. Falls back to database if cache miss
+        4. Returns user object or raises appropriate exception
+
+        Include proper error handling, typing, and comments explaining design decisions.
+    assert:
+      - type: contains
+        value: 'def'
+        reason: Should include Python function definition
+      - type: contains-any
+        value: ['try', 'except', 'raise', 'error']
+        reason: Should include error handling
+      - type: contains-any
+        value: ['cache', 'redis', 'ttl']
+        reason: Should implement caching logic
+      - type: llm-rubric
+        value: |
+          The code should:
+          1. Include proper type hints (from typing import ...)
+          2. Handle network timeouts and API errors gracefully
+          3. Implement cache-aside pattern correctly
+          4. Include docstrings and comments explaining design decisions
+          5. Use appropriate exception types
+          6. Be production-ready (not a toy example)
+
+  # Code review with nuanced feedback
+  - vars:
+      task: |
+        Review this React component and provide feedback:
+
+        ```jsx
+        function UserList() {
+          const [users, setUsers] = useState([]);
+
+          useEffect(() => {
+            fetch('/api/users')
+              .then(res => res.json())
+              .then(data => setUsers(data));
+          }, []);
+
+          return (
+            <div>
+              {users.map(user => (
+                <div key={user.id}>
+                  <h3>{user.name}</h3>
+                  <p>{user.email}</p>
+                </div>
+              ))}
+            </div>
+          );
+        }
+        ```
+
+        Identify issues, suggest improvements, and explain the reasoning behind each suggestion.
+    assert:
+      - type: contains-any
+        value: ['error', 'loading', 'state', 'async']
+        reason: Should identify missing error and loading states
+      - type: llm-rubric
+        value: |
+          The review should identify multiple issues:
+          1. No error handling for failed fetch
+          2. No loading state
+          3. No cleanup for fetch in useEffect
+          4. Missing dependencies might cause issues in strict mode
+          5. No null/empty checks for users array
+
+          For each issue, it should:
+          - Explain why it's a problem
+          - Suggest specific improvements
+          - Provide example code where helpful
+          - Prioritize issues by severity

examples/anthropic/sonnet-5/README.md

@@ -0,0 +1,53 @@
+# anthropic/sonnet-5 (Claude Sonnet 5 Agentic Reasoning)
+
+This example exercises Claude Sonnet 5 on agentic reasoning and coding tasks using the `high` effort level with adaptive thinking.
+
+You can run this example with:
+
+```bash
+npx promptfoo@latest init --example anthropic/sonnet-5
+cd sonnet-5
+```
+
+## What This Tests
+
+Claude Sonnet 5 is the Claude 5-generation Sonnet — built to be Anthropic's most agentic Sonnet, with capability approaching Opus 4.8 at Sonnet pricing and a 1M-token context window. This example evaluates:
+
+- **Multi-system bug diagnosis** of an intermittent production failure
+- **Production-quality code generation** with error handling and caching
+
+## Working with Sonnet 5
+
+- **Adaptive thinking is opt-in.** Set `thinking: { type: adaptive }` (as this example does) to let the model decide when and how much to reason per request. Without an explicit `thinking` block the model runs **without** extended thinking, even at high effort. Unlike Fable 5 / Mythos 5, Sonnet 5 also accepts `thinking: { type: disabled }`.
+- **`effort` tunes the cost/performance tradeoff.** Sonnet 5 supports `low`, `medium`, `high`, `xhigh`, and `max`. `high` is a good cost-efficient default; step up to `xhigh`/`max` for the hardest work and pair high effort with a large `max_tokens`.
+- **Sampling controls are managed for you.** Sonnet 5 rejects `temperature`, `top_p`, and `top_k` at the model level; promptfoo omits them automatically (don't set them in config).
+- **Pricing.** $3/$15 per million input/output tokens standard, with introductory pricing of $2/$10 through August 31, 2026. The full 1M-token context bills at the standard rate (no long-context surcharge).
+
+## Running the Example
+
+```bash
+# Set your API key
+export ANTHROPIC_API_KEY=your_api_key_here
+
+# Run the evaluation
+npx promptfoo@latest eval
+
+# View results
+npx promptfoo@latest view
+```
+
+## Other providers
+
+Sonnet 5 is also reachable through:
+
+- AWS Bedrock — `bedrock:us.anthropic.claude-sonnet-5` (or `bedrock:converse:us.anthropic.claude-sonnet-5`)
+- Google Vertex — `vertex:claude-sonnet-5` with `config.region: global` (availability may roll out after the Anthropic API launch)
+- Azure AI Foundry — point `anthropic:messages:claude-sonnet-5` at `https://<resource>.services.ai.azure.com/anthropic` via `apiBaseUrl`
+
+Across all four providers, promptfoo automatically omits the unsupported sampling parameters (`temperature`, `top_p`, `top_k`) for Sonnet 5. The Anthropic Messages provider — used directly and for Azure AI Foundry via `apiBaseUrl` — logs a one-time warning if you set them explicitly; the Bedrock and Vertex paths omit them silently.
+
+## Learn More
+
+- [Claude Sonnet 5 announcement](https://www.anthropic.com/news/claude-sonnet-5)
+- [Anthropic documentation](https://docs.anthropic.com)
+- [Promptfoo Anthropic provider docs](https://promptfoo.dev/docs/providers/anthropic)

examples/anthropic/sonnet-5/promptfooconfig.yaml

@@ -0,0 +1,80 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: Claude Sonnet 5 agentic reasoning with effort and adaptive thinking
+
+prompts:
+  - |
+    {{task}}
+
+providers:
+  - id: anthropic:messages:claude-sonnet-5
+    config:
+      # Sonnet 5 is the Claude 5-generation Sonnet: near-Opus agentic capability at
+      # Sonnet pricing, with a 1M-token context window.
+      #
+      # Sonnet 5 deprecates manual sampling controls (temperature/top_p/top_k) at the
+      # model level — promptfoo omits them automatically, so don't set them here.
+      #
+      # Adaptive thinking is opt-in: without an explicit `thinking` block the model
+      # runs WITHOUT extended thinking even at high effort. Set it to let the model
+      # decide when and how much to reason per request.
+      thinking:
+        type: adaptive
+      effort: high # Sonnet 5's cost-efficient sweet spot; xhigh/max are available for harder work
+      max_tokens: 8000
+
+tests:
+  # Multi-system bug diagnosis (intermittent failure)
+  - vars:
+      task: |
+        You're debugging a production issue where users can't log in. Here's what you know:
+
+        1. The frontend shows "Authentication failed" after username/password submission
+        2. Backend logs show successful JWT generation
+        3. Redis cache is returning stale session data
+        4. Database shows correct user credentials
+        5. The issue only affects 10% of login attempts
+        6. It started after deploying a load balancer configuration change
+
+        Diagnose the root cause and propose a fix. Explain your reasoning about what's
+        causing the intermittent nature of the bug.
+    assert:
+      - type: contains-any
+        value: ['load balancer', 'session', 'sticky', 'affinity', 'routing']
+        reason: Should identify load balancer session routing as the issue
+      - type: llm-rubric
+        value: |
+          The response should:
+          1. Identify the root cause (likely session affinity/sticky sessions issue with the load balancer)
+          2. Explain why it's intermittent (requests hit different backends with inconsistent session state)
+          3. Propose concrete fixes (sticky sessions, a shared session store, or stateless tokens)
+          4. Reason about the tradeoffs of the proposed solutions
+
+  # Production-quality code generation with error handling and caching
+  - vars:
+      task: |
+        Write a Python function that:
+        1. Fetches user data from a REST API (may timeout or return errors)
+        2. Caches results in Redis with a 5-minute TTL
+        3. Falls back to the database on a cache miss
+        4. Returns a user object or raises an appropriate exception
+
+        Include proper error handling, type hints, and comments explaining design decisions.
+    assert:
+      - type: contains
+        value: 'def'
+        reason: Should include a Python function definition
+      - type: contains-any
+        value: ['try', 'except', 'raise', 'error']
+        reason: Should include error handling
+      - type: contains-any
+        value: ['cache', 'redis', 'ttl']
+        reason: Should implement caching logic
+      - type: llm-rubric
+        value: |
+          The code should:
+          1. Include type hints (e.g. from typing import ...)
+          2. Handle network timeouts and API errors gracefully
+          3. Implement the cache-aside pattern correctly
+          4. Include docstrings/comments explaining design decisions
+          5. Use appropriate exception types
+          6. Be production-ready (not a toy example)

examples/anthropic/structured-outputs/README.md

@@ -0,0 +1,128 @@
+# anthropic/structured-outputs (Anthropic Structured Outputs)
+
+This example demonstrates Anthropic's structured outputs feature, which ensures Claude's responses follow a specific schema. It shows both **JSON outputs** for structured data extraction and **strict tool use** for guaranteed schema validation on tool calls.
+
+## What You'll Learn
+
+- How to use `output_format` to constrain Claude's responses to a JSON schema
+- How to use `strict: true` on tools to guarantee type-safe function parameters
+- The difference between JSON outputs and strict tool use
+- When to use each approach
+
+## Setup
+
+```bash
+export ANTHROPIC_API_KEY=your_api_key_here
+npx promptfoo@latest init --example anthropic/structured-outputs
+npx promptfoo@latest eval
+```
+
+## Features Demonstrated
+
+### JSON Outputs
+
+The first provider configuration shows how to extract structured data from unstructured text using a JSON schema:
+
+```yaml
+providers:
+  - id: anthropic:messages:claude-sonnet-4-6
+    config:
+      output_format:
+        type: json_schema
+        schema:
+          type: object
+          properties:
+            customer_name:
+              type: string
+            customer_email:
+              type: string
+            # ... more fields
+          required:
+            - customer_name
+            - customer_email
+          additionalProperties: false
+```
+
+**Use JSON outputs when:**
+
+- Extracting data from images or text
+- Generating structured reports
+- Formatting API responses
+- You need Claude's response in a specific format
+
+### Strict Tool Use
+
+The second provider configuration demonstrates how to ensure tool parameters exactly match your schema:
+
+```yaml
+providers:
+  - id: anthropic:messages:claude-sonnet-4-6
+    config:
+      tools:
+        - name: book_demo
+          strict: true # Enable strict mode
+          input_schema:
+            type: object
+            properties:
+              customer_email:
+                type: string
+              # ... more properties
+            required:
+              - customer_email
+              - customer_name
+            additionalProperties: false
+```
+
+**Use strict tool use when:**
+
+- Building agentic workflows
+- Ensuring type-safe function calls
+- Complex tools with many/nested properties
+- You need validated parameters and tool names
+
+## Key Benefits
+
+- **Always valid**: No more `JSON.parse()` errors
+- **Type safe**: Guaranteed field types and required fields
+- **Reliable**: No retries needed for schema violations
+- **Production-ready**: Build agents that work consistently at scale
+
+## Schema Requirements
+
+Both modes share these JSON Schema limitations:
+
+✅ **Supported:**
+
+- Basic types: object, array, string, integer, number, boolean, null
+- `enum` (primitives only)
+- `required` and `additionalProperties: false`
+- Array `minItems` (only 0 and 1)
+
+❌ **Not supported:**
+
+- Recursive schemas
+- Numerical constraints (`minimum`, `maximum`)
+- String constraints (`minLength`, `maxLength`)
+- Complex `{n,m}` quantifiers in regex patterns
+
+## Test Cases
+
+The example includes comprehensive tests:
+
+1. **Schema validation**: Ensures all required fields are present
+2. **Data accuracy**: Verifies extracted values match the input
+3. **No extra fields**: Confirms `additionalProperties: false` is enforced
+4. **Tool calling**: Validates tool names and parameters are correct
+5. **Type safety**: Checks that types match schema definitions
+
+## Supported Models
+
+Structured outputs are available for:
+
+- Claude Sonnet 4.6 (`claude-sonnet-4-6`)
+- Claude Opus 4.1 (`claude-opus-4-1-20250805`)
+
+## Learn More
+
+- [Anthropic Structured Outputs Documentation](https://docs.anthropic.com/en/docs/build-with-claude/structured-outputs)
+- [promptfoo Anthropic Provider Documentation](/docs/providers/anthropic)

examples/anthropic/structured-outputs/customer_schema.json

@@ -0,0 +1,25 @@
+{
+  "type": "object",
+  "properties": {
+    "customer_name": {
+      "type": "string"
+    },
+    "customer_email": {
+      "type": "string"
+    },
+    "company": {
+      "type": "string"
+    },
+    "current_plan": {
+      "type": "string"
+    },
+    "requested_plan": {
+      "type": "string"
+    },
+    "demo_requested": {
+      "type": "boolean"
+    }
+  },
+  "required": ["customer_name", "customer_email", "demo_requested"],
+  "additionalProperties": false
+}

examples/anthropic/structured-outputs/nested_format.json

@@ -0,0 +1,4 @@
+{
+  "type": "json_schema",
+  "schema": "file://customer_schema.json"
+}

examples/anthropic/structured-outputs/output_format.json

@@ -0,0 +1,55 @@
+{
+  "type": "json_schema",
+  "schema": {
+    "type": "object",
+    "properties": {
+      "customer_name": {
+        "type": "string"
+      },
+      "customer_email": {
+        "type": "string"
+      },
+      "company": {
+        "type": "string"
+      },
+      "current_plan": {
+        "type": "string"
+      },
+      "requested_plan": {
+        "type": "string"
+      },
+      "current_users": {
+        "type": "integer"
+      },
+      "requested_users": {
+        "type": "integer"
+      },
+      "demo_requested": {
+        "type": "boolean"
+      },
+      "demo_datetime": {
+        "type": "string"
+      },
+      "features_of_interest": {
+        "type": "array",
+        "items": {
+          "type": "string"
+        }
+      },
+      "questions": {
+        "type": "array",
+        "items": {
+          "type": "string"
+        }
+      }
+    },
+    "required": [
+      "customer_name",
+      "customer_email",
+      "current_plan",
+      "requested_plan",
+      "demo_requested"
+    ],
+    "additionalProperties": false
+  }
+}

examples/anthropic/structured-outputs/promptfooconfig.yaml

@@ -0,0 +1,154 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: Anthropic structured outputs for JSON responses and strict tools
+
+prompts:
+  # JSON outputs example - extract structured data
+  - |
+    Extract the key information from this customer support email:
+
+    Subject: Enterprise Plan Upgrade Request
+    From: Sarah Johnson <sarah.johnson@techcorp.com>
+
+    Hi there,
+
+    I'm reaching out to inquire about upgrading our team to the Enterprise plan. We currently have 25 users on the Professional plan, and we're interested in the advanced security features and dedicated support that come with Enterprise.
+
+    Could someone schedule a demo for next Tuesday, March 15th at 2:00 PM EST? We'd like to see the SSO integration and audit logging features in action.
+
+    Also, what's the pricing for 50 users on the Enterprise plan?
+
+    Thanks,
+    Sarah Johnson
+    Director of Engineering, TechCorp
+
+providers:
+  # JSON outputs - Inline schema
+  - id: anthropic:messages:claude-sonnet-4-6
+    label: inline-schema
+    config:
+      max_tokens: 2048
+      output_format:
+        type: json_schema
+        schema:
+          type: object
+          properties:
+            customer_name:
+              type: string
+            customer_email:
+              type: string
+            company:
+              type: string
+            current_plan:
+              type: string
+            requested_plan:
+              type: string
+            current_users:
+              type: integer
+            requested_users:
+              type: integer
+            demo_requested:
+              type: boolean
+            demo_datetime:
+              type: string
+            features_of_interest:
+              type: array
+              items:
+                type: string
+            questions:
+              type: array
+              items:
+                type: string
+          required:
+            - customer_name
+            - customer_email
+            - current_plan
+            - requested_plan
+            - demo_requested
+          additionalProperties: false
+
+  # JSON outputs - External file
+  - id: anthropic:messages:claude-sonnet-4-6
+    label: external-file
+    config:
+      max_tokens: 2048
+      output_format: file://output_format.json
+
+  # JSON outputs - Nested file reference (format file references schema file)
+  - id: anthropic:messages:claude-sonnet-4-6
+    label: nested-file-reference
+    config:
+      max_tokens: 2048
+      output_format: file://nested_format.json
+
+  # Strict tool use - Ensure tool parameters exactly match schema
+  - id: anthropic:messages:claude-sonnet-4-6
+    config:
+      max_tokens: 2048
+      tool_choice:
+        type: any
+      tools:
+        - name: book_demo
+          description: Book a demo session for a customer
+          strict: true
+          input_schema:
+            type: object
+            properties:
+              customer_email:
+                type: string
+              customer_name:
+                type: string
+              company:
+                type: string
+              demo_datetime:
+                type: string
+                description: ISO 8601 datetime string
+              features_to_demo:
+                type: array
+                items:
+                  type: string
+                  enum:
+                    - sso_integration
+                    - audit_logging
+                    - advanced_security
+                    - dedicated_support
+                    - custom_integrations
+              attendees_count:
+                type: integer
+                enum: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
+            required:
+              - customer_email
+              - customer_name
+              - demo_datetime
+            additionalProperties: false
+
+        - name: get_pricing
+          description: Get pricing information for a specific plan and user count
+          strict: true
+          input_schema:
+            type: object
+            properties:
+              plan:
+                type: string
+                enum:
+                  - professional
+                  - enterprise
+              user_count:
+                type: integer
+              billing_period:
+                type: string
+                enum:
+                  - monthly
+                  - annual
+            required:
+              - plan
+              - user_count
+            additionalProperties: false
+
+tests:
+  # Simple test that works for both providers
+  - description: Should extract customer information
+    assert:
+      - type: contains
+        value: Sarah Johnson
+      - type: contains
+        value: sarah.johnson@techcorp.com

examples/anthropic/web-tools/README.md

@@ -0,0 +1,89 @@
+# anthropic/web-tools (Anthropic Web Tools)
+
+This example demonstrates Anthropic's web search and web fetch tools, showing both basic URL fetching and comprehensive research workflows.
+
+You can run this example with:
+
+```bash
+npx promptfoo@latest init --example anthropic/web-tools
+cd anthropic/web-tools
+```
+
+## Features Demonstrated
+
+- **Web Search**: Find relevant information across the internet
+- **Web Fetch**: Retrieve full content from specific web pages and PDFs
+- **Combined Workflows**: Multi-step research processes
+- **Security Controls**: Domain filtering for trusted sources
+- **Citations**: Source attribution for transparency
+
+## Configuration Highlights
+
+```yaml
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: 'Web search and fetch tools'
+
+prompts:
+  - |
+    {{task}}
+
+providers:
+  - id: anthropic:messages:claude-sonnet-4-6
+    config:
+      temperature: 0
+      max_tokens: 2500
+      tools:
+        - type: web_search_20250305 # Search the web
+          name: web_search
+          max_uses: 2
+        - type: web_fetch_20250910 # Fetch specific URLs
+          name: web_fetch
+          max_uses: 3
+          allowed_domains: # Restrict to trusted domains
+            - docs.anthropic.com
+            - github.com
+            - openai.com
+            - arxiv.org
+          citations:
+            enabled: true # Enable source citations
+          max_content_tokens: 12000 # Content size limit
+```
+
+## Required Environment Variables
+
+This example requires the following environment variables:
+
+- `ANTHROPIC_API_KEY` - Your Anthropic API key from [console.anthropic.com](https://console.anthropic.com/settings/keys)
+
+You can set these in a `.env` file or directly in your environment.
+
+## Running the Example
+
+```bash
+cd examples/anthropic/web-tools
+promptfoo eval
+```
+
+## What It Tests
+
+1. **Direct URL Fetch**: Retrieve and summarize specific documentation pages
+2. **GitHub Repository Analysis**: Analyze code repositories and their purpose
+3. **Research Workflow**: Multi-step research with search → fetch → synthesis
+4. **Security Research**: Investigation of web tool security considerations
+
+## Use Cases
+
+Perfect for:
+
+- Documentation analysis
+- Competitive research
+- Technical investigation
+- Security assessment
+- Academic literature review
+
+## Security Notes
+
+- Only allows fetching from trusted domains (docs.anthropic.com, github.com, openai.com, arxiv.org)
+- Content limited to 12,000 tokens to prevent excessive usage
+- Citations enabled for source transparency
+- Usage limits prevent API abuse (2 searches, 3 fetches per evaluation)

examples/anthropic/web-tools/promptfooconfig.yaml

@@ -0,0 +1,82 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: 'Web search and fetch tools'
+
+prompts:
+  - |
+    {{task}}
+
+providers:
+  - id: anthropic:messages:claude-sonnet-4-6
+    config:
+      temperature: 0
+      max_tokens: 2500
+      tools:
+        - type: web_search_20250305
+          name: web_search
+          max_uses: 2
+        - type: web_fetch_20250910
+          name: web_fetch
+          max_uses: 3
+          allowed_domains:
+            - docs.anthropic.com
+            - github.com
+            - openai.com
+            - arxiv.org
+          citations:
+            enabled: true
+          max_content_tokens: 12000
+
+tests:
+  # Basic web fetch test
+  - vars:
+      task: 'Please fetch and summarize the content from this URL: https://docs.anthropic.com/en/docs/build-with-claude/tool-use'
+    assert:
+      - type: contains-any
+        value: ['tool', 'function', 'API', 'capabilities']
+        reason: 'Should mention tool-related concepts'
+      - type: llm-rubric
+        value: 'Provides a clear, structured summary of the web page content'
+
+  # Direct URL fetch test
+  - vars:
+      task: 'Fetch and analyze the GitHub repository at https://github.com/anthropics/anthropic-sdk-typescript'
+    assert:
+      - type: contains
+        value: 'TypeScript'
+      - type: llm-rubric
+        value: 'Describes the TypeScript SDK and its purpose'
+
+  # Research workflow test
+  - vars:
+      task: |
+        Research "Claude 4 AI model capabilities" and provide a comprehensive analysis.
+
+        Process:
+        1. Search for recent information about the topic
+        2. Fetch detailed content from the most authoritative sources  
+        3. Synthesize findings into a structured report
+
+        Your report should include:
+        - Overview of the topic
+        - Key capabilities or features
+        - Recent developments
+        - Sources used
+    assert:
+      - type: contains-any
+        value: ['Claude 4', 'claude-4', 'capabilities', 'model']
+        reason: 'Should discuss Claude 4 specifically'
+      - type: llm-rubric
+        value: 'Provides a comprehensive research-based analysis with multiple information sources'
+      - type: contains-any
+        value: ['search', 'found', 'according', 'research', 'sources']
+        reason: 'Should indicate research methodology was used'
+
+  # Security-focused research test
+  - vars:
+      task: 'Research web fetch tool security considerations and best practices'
+    assert:
+      - type: contains-any
+        value: ['security', 'domain', 'filtering', 'risk']
+        reason: 'Should address security aspects'
+      - type: llm-rubric
+        value: 'Discusses security considerations and best practices for web fetching'