wehub-resource-sync
0d3cb498a3
Deploy local.promptfoo.app / Deploy to Cloudflare Pages (push) Waiting to run
Test and Publish Multi-arch Docker Image / test (push) Waiting to run
Test and Publish Multi-arch Docker Image / build-docker-and-push-digests (map[digest-suffix:linux-amd64 platform:linux/amd64 runner:ubuntu-latest]) (push) Blocked by required conditions
Test and Publish Multi-arch Docker Image / build-docker-and-push-digests (map[digest-suffix:linux-arm64 platform:linux/arm64 runner:ubuntu-24.04-arm]) (push) Blocked by required conditions
Test and Publish Multi-arch Docker Image / merge-docker-digests (push) Blocked by required conditions
Test and Publish Multi-arch Docker Image / Attest Multi-arch Image (push) Blocked by required conditions
CI / Shell Format Check (push) Waiting to run
CI / Check Ruby (3.4) (push) Waiting to run
CI / CI Config (push) Waiting to run
CI / Test on Node ${{ matrix.node }} and ${{ matrix.os }}${{ matrix.shard && format(' (shard {0}/3)', matrix.shard) || '' }} (push) Blocked by required conditions
CI / Build on Node ${{ matrix.node }} (push) Blocked by required conditions
CI / Style Check (push) Waiting to run
CI / Generate Assets (push) Waiting to run
CI / Check Python (3.14) (push) Waiting to run
CI / Check Python (3.9) (push) Waiting to run
CI / Build Docs (push) Waiting to run
CI / Code Scan Action (push) Waiting to run
CI / Site tests (push) Waiting to run
CI / webui tests (push) Waiting to run
CI / Run Integration Tests (push) Waiting to run
CI / Run Smoke Tests (push) Waiting to run
CI / Go Tests (push) Waiting to run
CI / Share Test (push) Waiting to run
CI / Redteam (Production API) (push) Waiting to run
CI / Redteam (Staging API) (push) Waiting to run
CI / GitHub Actions Lint (push) Waiting to run
CI / Check Ruby (3.0) (push) Waiting to run
release-please / release-please (push) Waiting to run
release-please / build (push) Blocked by required conditions
release-please / publish-npm (push) Blocked by required conditions
release-please / publish-npm-backfill (push) Waiting to run
release-please / docker (push) Blocked by required conditions
release-please / publish-code-scan-action (push) Blocked by required conditions
release-please / attest-code-scan-action (push) Blocked by required conditions
Validate Renovate Config / Validate Renovate Configuration (push) Waiting to run
claude-thinking (Claude Thinking)
This example demonstrates Claude's "thinking" capability, which allows you to see the model's step-by-step reasoning process before it provides a final answer. The example compares thinking outputs from Claude Sonnet 4 (Anthropic API) and Claude Haiku 4.5 (AWS Bedrock).
You can run this example with:
npx promptfoo@latest init --example claude-thinking
cd claude-thinking
What This Example Demonstrates
- Using Claude's thinking feature to reveal step-by-step reasoning
- Comparing thinking output quality between different Claude models
- Comparing Anthropic API vs AWS Bedrock providers
- Configuring the thinking token budget
- Using LLM-based evaluation rubrics to assess reasoning quality
Environment Variables
This example requires:
For Anthropic API
ANTHROPIC_API_KEY- Your Anthropic API key from console.anthropic.com
For AWS Bedrock
AWS_ACCESS_KEY_ID- Your AWS access keyAWS_SECRET_ACCESS_KEY- Your AWS secret key- Or configure credentials via the AWS CLI:
aws configure
Running the Example
After setting up environment variables:
# From the example directory
promptfoo eval
promptfoo view
Test Cases
This example includes several test cases of increasing complexity:
- 8 Balls Problem - A classic logic puzzle requiring careful reasoning
- Train Meeting Problem - A traditional algebra word problem
These test cases are specifically designed to showcase Claude's ability to break down complex problems and show detailed thinking steps.
How Claude Thinking Works
The thinking feature is enabled by setting special parameters in the provider configuration:
thinking:
type: 'enabled'
budget_tokens: 4096 # Controls how many tokens are allocated for thinking
max_tokens: 8192 # Must be greater than budget_tokens
When enabled, Claude's response will include a "Thinking:" section that shows its reasoning process before the final answer:
Thinking: Let me solve this step by step...
1. First, I'll divide the 8 balls into three groups...
2. In the first weighing, I'll compare groups A and B...
3. Based on the result, I can determine...
Final answer: We need exactly 2 weighings to find the heavier ball.