chore: import upstream snapshot with attribution

2026-07-13 13:24:08 +08:00
commit 0d3cb498a3
5438 changed files with 1316560 additions and 0 deletions
@@ -0,0 +1,79 @@
+# claude-thinking (Claude Thinking)
+
+This example demonstrates Claude's "thinking" capability, which allows you to see the model's step-by-step reasoning process before it provides a final answer. The example compares thinking outputs from Claude Sonnet 4 (Anthropic API) and Claude Haiku 4.5 (AWS Bedrock).
+
+You can run this example with:
+
+```bash
+npx promptfoo@latest init --example claude-thinking
+cd claude-thinking
+```
+
+## What This Example Demonstrates
+
+- Using Claude's thinking feature to reveal step-by-step reasoning
+- Comparing thinking output quality between different Claude models
+- Comparing Anthropic API vs AWS Bedrock providers
+- Configuring the thinking token budget
+- Using LLM-based evaluation rubrics to assess reasoning quality
+
+## Environment Variables
+
+This example requires:
+
+### For Anthropic API
+
+- `ANTHROPIC_API_KEY` - Your Anthropic API key from [console.anthropic.com](https://console.anthropic.com/)
+
+### For AWS Bedrock
+
+- `AWS_ACCESS_KEY_ID` - Your AWS access key
+- `AWS_SECRET_ACCESS_KEY` - Your AWS secret key
+- Or configure credentials via the AWS CLI: `aws configure`
+
+## Running the Example
+
+After setting up environment variables:
+
+```bash
+# From the example directory
+promptfoo eval
+promptfoo view
+```
+
+## Test Cases
+
+This example includes several test cases of increasing complexity:
+
+1. **8 Balls Problem** - A classic logic puzzle requiring careful reasoning
+2. **Train Meeting Problem** - A traditional algebra word problem
+
+These test cases are specifically designed to showcase Claude's ability to break down complex problems and show detailed thinking steps.
+
+## How Claude Thinking Works
+
+The thinking feature is enabled by setting special parameters in the provider configuration:
+
+```yaml
+thinking:
+  type: 'enabled'
+  budget_tokens: 4096 # Controls how many tokens are allocated for thinking
+max_tokens: 8192 # Must be greater than budget_tokens
+```
+
+When enabled, Claude's response will include a "Thinking:" section that shows its reasoning process before the final answer:
+
+```text
+Thinking: Let me solve this step by step...
+1. First, I'll divide the 8 balls into three groups...
+2. In the first weighing, I'll compare groups A and B...
+3. Based on the result, I can determine...
+
+Final answer: We need exactly 2 weighings to find the heavier ball.
+```
+
+## Additional Resources
+
+- [Claude Thinking Documentation](https://docs.anthropic.com/claude/docs/extended-thinking)
+- [AWS Bedrock Claude Guide](https://docs.aws.amazon.com/bedrock/latest/userguide/model-parameters-claude.html)
+- [Promptfoo Documentation on Claude Providers](https://promptfoo.dev/docs/providers/anthropic)
@@ -0,0 +1,49 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: 'Claude Thinking Example'
+prompts:
+  - file://prompts.txt
+providers:
+  - id: anthropic:claude-sonnet-4-6
+    config:
+      stream: true # Required for thinking operations
+      thinking:
+        type: 'enabled'
+        budget_tokens: 16000
+      max_tokens: 32000
+  - id: anthropic:messages:claude-sonnet-4-6
+    config:
+      stream: true # Required for thinking operations
+      thinking:
+        type: 'enabled'
+        budget_tokens: 16000
+      max_tokens: 32000
+  # Claude Haiku 4.5 via AWS Bedrock
+  - id: bedrock:us.anthropic.claude-haiku-4-5-20251001-v1:0
+    config:
+      temperature: 1 # Required when thinking is enabled on Bedrock
+      stream: true # Required for thinking operations
+      thinking:
+        type: 'enabled'
+        budget_tokens: 16000
+      max_tokens: 32000
+
+tests:
+  - vars:
+      question: |
+        You have 8 identical balls, but one is slightly heavier than the others.
+        Using a balance scale, what is the minimum number of weighings needed to
+        find the heavier ball?
+    assert:
+      - type: llm-rubric
+        value: |
+          The answer should explain why 2 weighings are sufficient
+
+  - vars:
+      question: |
+        A train leaves Boston at 8:00 AM traveling west at 100 mph. Another train
+        leaves Chicago at 10:00 AM traveling east at 120 mph. If the distance
+        between Boston and Chicago is 1000 miles, at what time will the trains meet?
+    assert:
+      - type: llm-rubric
+        value: |
+          The trains will meet at 1:38 PM.
@@ -0,0 +1,9 @@
+You are a helpful assistant. Reply with a concise answer to this inquiry: "{{question}}"
+---
+You are a helpful assistant. Reply with a concise answer to this inquiry: "{{question}}"
+
+- Think carefully & step-by-step.
+- Only use information available on Wikipedia.
+- You must answer the question directly, without speculation.
+- You cannot access realtime information. Consider whether the answer may have changed in the 2 years since your knowledge cutoff.
+- If you are not confident in your answer, begin your response with "Unsure".