chore: import upstream snapshot with attribution
CI / Shell Format Check (push) Has been cancelled
CI / Check Ruby (3.4) (push) Has been cancelled
CI / CI Config (push) Has been cancelled
CI / Test on Node ${{ matrix.node }} and ${{ matrix.os }}${{ matrix.shard && format(' (shard {0}/3)', matrix.shard) || '' }} (push) Has been cancelled
CI / Build on Node ${{ matrix.node }} (push) Has been cancelled
CI / Style Check (push) Has been cancelled
CI / Generate Assets (push) Has been cancelled
CI / Check Python (3.14) (push) Has been cancelled
CI / Check Python (3.9) (push) Has been cancelled
CI / Build Docs (push) Has been cancelled
CI / Code Scan Action (push) Has been cancelled
CI / Site tests (push) Has been cancelled
CI / webui tests (push) Has been cancelled
CI / Run Integration Tests (push) Has been cancelled
CI / Run Smoke Tests (push) Has been cancelled
CI / Go Tests (push) Has been cancelled
CI / Share Test (push) Has been cancelled
CI / Redteam (Production API) (push) Has been cancelled
CI / Redteam (Staging API) (push) Has been cancelled
CI / GitHub Actions Lint (push) Has been cancelled
CI / Check Ruby (3.0) (push) Has been cancelled
release-please / release-please (push) Has been cancelled
release-please / build (push) Has been cancelled
release-please / publish-npm (push) Has been cancelled
release-please / publish-npm-backfill (push) Has been cancelled
release-please / docker (push) Has been cancelled
release-please / publish-code-scan-action (push) Has been cancelled
release-please / attest-code-scan-action (push) Has been cancelled
Deploy local.promptfoo.app / Deploy to Cloudflare Pages (push) Has been cancelled
Test and Publish Multi-arch Docker Image / test (push) Has been cancelled
Test and Publish Multi-arch Docker Image / build-docker-and-push-digests (map[digest-suffix:linux-amd64 platform:linux/amd64 runner:ubuntu-latest]) (push) Has been cancelled
Test and Publish Multi-arch Docker Image / build-docker-and-push-digests (map[digest-suffix:linux-arm64 platform:linux/arm64 runner:ubuntu-24.04-arm]) (push) Has been cancelled
Test and Publish Multi-arch Docker Image / merge-docker-digests (push) Has been cancelled
Test and Publish Multi-arch Docker Image / Attest Multi-arch Image (push) Has been cancelled
Validate Renovate Config / Validate Renovate Configuration (push) Has been cancelled
CI / Shell Format Check (push) Has been cancelled
CI / Check Ruby (3.4) (push) Has been cancelled
CI / CI Config (push) Has been cancelled
CI / Test on Node ${{ matrix.node }} and ${{ matrix.os }}${{ matrix.shard && format(' (shard {0}/3)', matrix.shard) || '' }} (push) Has been cancelled
CI / Build on Node ${{ matrix.node }} (push) Has been cancelled
CI / Style Check (push) Has been cancelled
CI / Generate Assets (push) Has been cancelled
CI / Check Python (3.14) (push) Has been cancelled
CI / Check Python (3.9) (push) Has been cancelled
CI / Build Docs (push) Has been cancelled
CI / Code Scan Action (push) Has been cancelled
CI / Site tests (push) Has been cancelled
CI / webui tests (push) Has been cancelled
CI / Run Integration Tests (push) Has been cancelled
CI / Run Smoke Tests (push) Has been cancelled
CI / Go Tests (push) Has been cancelled
CI / Share Test (push) Has been cancelled
CI / Redteam (Production API) (push) Has been cancelled
CI / Redteam (Staging API) (push) Has been cancelled
CI / GitHub Actions Lint (push) Has been cancelled
CI / Check Ruby (3.0) (push) Has been cancelled
release-please / release-please (push) Has been cancelled
release-please / build (push) Has been cancelled
release-please / publish-npm (push) Has been cancelled
release-please / publish-npm-backfill (push) Has been cancelled
release-please / docker (push) Has been cancelled
release-please / publish-code-scan-action (push) Has been cancelled
release-please / attest-code-scan-action (push) Has been cancelled
Deploy local.promptfoo.app / Deploy to Cloudflare Pages (push) Has been cancelled
Test and Publish Multi-arch Docker Image / test (push) Has been cancelled
Test and Publish Multi-arch Docker Image / build-docker-and-push-digests (map[digest-suffix:linux-amd64 platform:linux/amd64 runner:ubuntu-latest]) (push) Has been cancelled
Test and Publish Multi-arch Docker Image / build-docker-and-push-digests (map[digest-suffix:linux-arm64 platform:linux/arm64 runner:ubuntu-24.04-arm]) (push) Has been cancelled
Test and Publish Multi-arch Docker Image / merge-docker-digests (push) Has been cancelled
Test and Publish Multi-arch Docker Image / Attest Multi-arch Image (push) Has been cancelled
Validate Renovate Config / Validate Renovate Configuration (push) Has been cancelled
This commit is contained in:
@@ -0,0 +1,89 @@
|
||||
# anthropic/web-tools (Anthropic Web Tools)
|
||||
|
||||
This example demonstrates Anthropic's web search and web fetch tools, showing both basic URL fetching and comprehensive research workflows.
|
||||
|
||||
You can run this example with:
|
||||
|
||||
```bash
|
||||
npx promptfoo@latest init --example anthropic/web-tools
|
||||
cd anthropic/web-tools
|
||||
```
|
||||
|
||||
## Features Demonstrated
|
||||
|
||||
- **Web Search**: Find relevant information across the internet
|
||||
- **Web Fetch**: Retrieve full content from specific web pages and PDFs
|
||||
- **Combined Workflows**: Multi-step research processes
|
||||
- **Security Controls**: Domain filtering for trusted sources
|
||||
- **Citations**: Source attribution for transparency
|
||||
|
||||
## Configuration Highlights
|
||||
|
||||
```yaml
|
||||
# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
|
||||
description: 'Web search and fetch tools'
|
||||
|
||||
prompts:
|
||||
- |
|
||||
{{task}}
|
||||
|
||||
providers:
|
||||
- id: anthropic:messages:claude-sonnet-4-6
|
||||
config:
|
||||
temperature: 0
|
||||
max_tokens: 2500
|
||||
tools:
|
||||
- type: web_search_20250305 # Search the web
|
||||
name: web_search
|
||||
max_uses: 2
|
||||
- type: web_fetch_20250910 # Fetch specific URLs
|
||||
name: web_fetch
|
||||
max_uses: 3
|
||||
allowed_domains: # Restrict to trusted domains
|
||||
- docs.anthropic.com
|
||||
- github.com
|
||||
- openai.com
|
||||
- arxiv.org
|
||||
citations:
|
||||
enabled: true # Enable source citations
|
||||
max_content_tokens: 12000 # Content size limit
|
||||
```
|
||||
|
||||
## Required Environment Variables
|
||||
|
||||
This example requires the following environment variables:
|
||||
|
||||
- `ANTHROPIC_API_KEY` - Your Anthropic API key from [console.anthropic.com](https://console.anthropic.com/settings/keys)
|
||||
|
||||
You can set these in a `.env` file or directly in your environment.
|
||||
|
||||
## Running the Example
|
||||
|
||||
```bash
|
||||
cd examples/anthropic/web-tools
|
||||
promptfoo eval
|
||||
```
|
||||
|
||||
## What It Tests
|
||||
|
||||
1. **Direct URL Fetch**: Retrieve and summarize specific documentation pages
|
||||
2. **GitHub Repository Analysis**: Analyze code repositories and their purpose
|
||||
3. **Research Workflow**: Multi-step research with search → fetch → synthesis
|
||||
4. **Security Research**: Investigation of web tool security considerations
|
||||
|
||||
## Use Cases
|
||||
|
||||
Perfect for:
|
||||
|
||||
- Documentation analysis
|
||||
- Competitive research
|
||||
- Technical investigation
|
||||
- Security assessment
|
||||
- Academic literature review
|
||||
|
||||
## Security Notes
|
||||
|
||||
- Only allows fetching from trusted domains (docs.anthropic.com, github.com, openai.com, arxiv.org)
|
||||
- Content limited to 12,000 tokens to prevent excessive usage
|
||||
- Citations enabled for source transparency
|
||||
- Usage limits prevent API abuse (2 searches, 3 fetches per evaluation)
|
||||
@@ -0,0 +1,82 @@
|
||||
# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
|
||||
description: 'Web search and fetch tools'
|
||||
|
||||
prompts:
|
||||
- |
|
||||
{{task}}
|
||||
|
||||
providers:
|
||||
- id: anthropic:messages:claude-sonnet-4-6
|
||||
config:
|
||||
temperature: 0
|
||||
max_tokens: 2500
|
||||
tools:
|
||||
- type: web_search_20250305
|
||||
name: web_search
|
||||
max_uses: 2
|
||||
- type: web_fetch_20250910
|
||||
name: web_fetch
|
||||
max_uses: 3
|
||||
allowed_domains:
|
||||
- docs.anthropic.com
|
||||
- github.com
|
||||
- openai.com
|
||||
- arxiv.org
|
||||
citations:
|
||||
enabled: true
|
||||
max_content_tokens: 12000
|
||||
|
||||
tests:
|
||||
# Basic web fetch test
|
||||
- vars:
|
||||
task: 'Please fetch and summarize the content from this URL: https://docs.anthropic.com/en/docs/build-with-claude/tool-use'
|
||||
assert:
|
||||
- type: contains-any
|
||||
value: ['tool', 'function', 'API', 'capabilities']
|
||||
reason: 'Should mention tool-related concepts'
|
||||
- type: llm-rubric
|
||||
value: 'Provides a clear, structured summary of the web page content'
|
||||
|
||||
# Direct URL fetch test
|
||||
- vars:
|
||||
task: 'Fetch and analyze the GitHub repository at https://github.com/anthropics/anthropic-sdk-typescript'
|
||||
assert:
|
||||
- type: contains
|
||||
value: 'TypeScript'
|
||||
- type: llm-rubric
|
||||
value: 'Describes the TypeScript SDK and its purpose'
|
||||
|
||||
# Research workflow test
|
||||
- vars:
|
||||
task: |
|
||||
Research "Claude 4 AI model capabilities" and provide a comprehensive analysis.
|
||||
|
||||
Process:
|
||||
1. Search for recent information about the topic
|
||||
2. Fetch detailed content from the most authoritative sources
|
||||
3. Synthesize findings into a structured report
|
||||
|
||||
Your report should include:
|
||||
- Overview of the topic
|
||||
- Key capabilities or features
|
||||
- Recent developments
|
||||
- Sources used
|
||||
assert:
|
||||
- type: contains-any
|
||||
value: ['Claude 4', 'claude-4', 'capabilities', 'model']
|
||||
reason: 'Should discuss Claude 4 specifically'
|
||||
- type: llm-rubric
|
||||
value: 'Provides a comprehensive research-based analysis with multiple information sources'
|
||||
- type: contains-any
|
||||
value: ['search', 'found', 'according', 'research', 'sources']
|
||||
reason: 'Should indicate research methodology was used'
|
||||
|
||||
# Security-focused research test
|
||||
- vars:
|
||||
task: 'Research web fetch tool security considerations and best practices'
|
||||
assert:
|
||||
- type: contains-any
|
||||
value: ['security', 'domain', 'filtering', 'risk']
|
||||
reason: 'Should address security aspects'
|
||||
- type: llm-rubric
|
||||
value: 'Discusses security considerations and best practices for web fetching'
|
||||
Reference in New Issue
Block a user