chore: import upstream snapshot with attribution
CI / Shell Format Check (push) Has been cancelled
CI / Check Ruby (3.4) (push) Has been cancelled
CI / CI Config (push) Has been cancelled
CI / Test on Node ${{ matrix.node }} and ${{ matrix.os }}${{ matrix.shard && format(' (shard {0}/3)', matrix.shard) || '' }} (push) Has been cancelled
CI / Build on Node ${{ matrix.node }} (push) Has been cancelled
CI / Style Check (push) Has been cancelled
CI / Generate Assets (push) Has been cancelled
CI / Check Python (3.14) (push) Has been cancelled
CI / Check Python (3.9) (push) Has been cancelled
CI / Build Docs (push) Has been cancelled
CI / Code Scan Action (push) Has been cancelled
CI / Site tests (push) Has been cancelled
CI / webui tests (push) Has been cancelled
CI / Run Integration Tests (push) Has been cancelled
CI / Run Smoke Tests (push) Has been cancelled
CI / Go Tests (push) Has been cancelled
CI / Share Test (push) Has been cancelled
CI / Redteam (Production API) (push) Has been cancelled
CI / Redteam (Staging API) (push) Has been cancelled
CI / GitHub Actions Lint (push) Has been cancelled
CI / Check Ruby (3.0) (push) Has been cancelled
release-please / release-please (push) Has been cancelled
release-please / build (push) Has been cancelled
release-please / publish-npm (push) Has been cancelled
release-please / publish-npm-backfill (push) Has been cancelled
release-please / docker (push) Has been cancelled
release-please / publish-code-scan-action (push) Has been cancelled
release-please / attest-code-scan-action (push) Has been cancelled
Deploy local.promptfoo.app / Deploy to Cloudflare Pages (push) Has been cancelled
Test and Publish Multi-arch Docker Image / test (push) Has been cancelled
Test and Publish Multi-arch Docker Image / build-docker-and-push-digests (map[digest-suffix:linux-amd64 platform:linux/amd64 runner:ubuntu-latest]) (push) Has been cancelled
Test and Publish Multi-arch Docker Image / build-docker-and-push-digests (map[digest-suffix:linux-arm64 platform:linux/arm64 runner:ubuntu-24.04-arm]) (push) Has been cancelled
Test and Publish Multi-arch Docker Image / merge-docker-digests (push) Has been cancelled
Test and Publish Multi-arch Docker Image / Attest Multi-arch Image (push) Has been cancelled
Validate Renovate Config / Validate Renovate Configuration (push) Has been cancelled

This commit is contained in:
wehub-resource-sync
2026-07-13 13:24:08 +08:00
commit 0d3cb498a3
5438 changed files with 1316560 additions and 0 deletions
@@ -0,0 +1,54 @@
# anthropic/fable-5-coding (Claude Fable 5 Advanced Coding)
This example exercises Claude Fable 5 on hard coding tasks using the `xhigh` effort level. Fable 5 always uses adaptive thinking, so no `thinking` configuration is needed.
You can run this example with:
```bash
npx promptfoo@latest init --example anthropic/fable-5-coding
cd fable-5-coding
```
## What This Tests
Claude Fable 5 is Anthropic's most powerful model — a new tier above Opus. This example evaluates:
- **Distributed-systems debugging** with incomplete information
- **Production-quality code generation** with concurrency concerns
- **Security-focused code review** with prioritized feedback
## Working with Fable 5
- **Adaptive thinking is always on.** Unlike Opus 4.7/4.8 (where adaptive thinking is opt-in), Fable 5 always thinks adaptively. There is nothing to configure: promptfoo converts a legacy `thinking: { type: enabled, budget_tokens: N }` config to adaptive and omits `thinking: { type: disabled }`, because the model rejects both.
- **Thinking summaries are opt-in.** By default the API omits thinking content (an empty thinking block, which promptfoo excludes from output). Set `thinking: { type: adaptive, display: summarized }` to include a readable summary.
- **Sampling controls are managed for you.** Fable 5 rejects `temperature`, `top_p`, and `top_k` at the model level; promptfoo omits them automatically (don't set them in config).
- **`effort` defaults to `high`; `xhigh` is available.** Start with `xhigh` for coding and agentic work, and pair high effort with a large `max_tokens` — thinking consumes output tokens before any visible text, so a tight budget can be spent entirely on thinking and yield an empty response.
- **1M-token context, up to 128K output tokens.** Priced at $10/$50 per million input/output tokens.
## Running the Example
```bash
# Set your API key
export ANTHROPIC_API_KEY=your_api_key_here
# Run the evaluation
npx promptfoo@latest eval
# View results
npx promptfoo@latest view
```
## Other providers
Fable 5 is also reachable through:
- AWS Bedrock — `bedrock:global.anthropic.claude-fable-5` (Runtime/Converse; requires the account to opt in to provider data sharing via `aws bedrock put-account-data-retention --mode provider_data_share`), or `bedrock:messages:anthropic.claude-fable-5` for Bedrock's Anthropic-compatible Messages endpoint in `us-east-1`/`eu-north-1`
- Google Vertex — `vertex:claude-fable-5`
- Azure AI Foundry — point `anthropic:messages:claude-fable-5` at `https://<resource>.services.ai.azure.com/anthropic` via `apiBaseUrl`
Across all providers, promptfoo automatically omits the unsupported sampling parameters and normalizes unsupported thinking configs for Fable 5. Note that Bedrock's regional and geo endpoints and Vertex's regional/multi-region endpoints (everything except `global`) carry a 10% price premium, which promptfoo includes in cost calculations.
## Learn More
- [Anthropic documentation](https://docs.anthropic.com)
- [Promptfoo Anthropic provider docs](https://promptfoo.dev/docs/providers/anthropic)
@@ -0,0 +1,110 @@
# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
description: Claude Fable 5 advanced coding with xhigh effort and always-on adaptive thinking
prompts:
- |
{{task}}
providers:
- id: anthropic:messages:claude-fable-5
config:
# Fable 5 always uses adaptive thinking — there is no `thinking` block to
# set. Manual budgets (`type: enabled`) are converted to adaptive and
# `type: disabled` is omitted because thinking cannot be turned off.
#
# Fable 5 also rejects manual sampling controls (temperature/top_p/top_k)
# at the model level — promptfoo omits them automatically, so don't set
# them here.
effort: xhigh # Recommended starting point for coding/agentic work (between high and max)
# Always-on adaptive thinking consumes output tokens before any visible
# text. Budget max_tokens generously — at xhigh effort a tight budget can
# be spent entirely on thinking, yielding an empty response.
max_tokens: 16000
tests:
# Distributed-systems debugging with incomplete information
- vars:
task: |
A payment service intermittently double-charges customers. Here's what you know:
1. The charge endpoint is idempotent — it checks for an existing charge by idempotency key before creating one
2. The idempotency check reads from a Postgres replica; writes go to the primary
3. Replication lag spikes to 2-3 seconds during peak traffic
4. Clients retry on timeout with the same idempotency key
5. Double charges only happen during peak hours
6. Each double charge shows two rows with the same idempotency key but different charge IDs
Diagnose the root cause and propose a fix. Explain why the idempotency check fails despite using the same key.
assert:
- type: contains-any
value: ['replica', 'replication lag', 'read-after-write', 'primary', 'race']
reason: Should identify the stale-replica read as the root cause
- type: llm-rubric
value: |
The response should:
1. Identify the root cause (idempotency check reads a lagging replica, so a retry during the lag window misses the in-flight charge)
2. Explain why it only happens at peak (replication lag exceeds the client retry interval)
3. Propose concrete fixes (read idempotency checks from the primary, unique constraint on idempotency key, or insert-first/conflict-based idempotency)
4. Note that a unique constraint is the only fix that closes the race completely
# Production-quality code generation with concurrency concerns
- vars:
task: |
Write a TypeScript class implementing a token-bucket rate limiter that:
1. Supports a configurable capacity and refill rate
2. Is safe to call concurrently from async code
3. Exposes acquire() that resolves when a token is available (with optional timeout)
4. Avoids busy-waiting and timer leaks
Include proper typing and comments explaining design decisions.
assert:
- type: contains
value: 'class'
reason: Should define a TypeScript class
- type: contains-any
value: ['Promise', 'async', 'await']
reason: Should use async primitives
- type: llm-rubric
value: |
The code should:
1. Implement correct token-bucket math (refill proportional to elapsed time, capped at capacity)
2. Queue waiters instead of busy-waiting
3. Clean up timers when acquire() times out or resolves
4. Use precise TypeScript types (no `any`)
5. Include comments explaining the concurrency-safety reasoning
6. Be production-ready (not a toy example)
# Code review with prioritized, nuanced feedback
- vars:
task: |
Review this Express handler and provide feedback:
```js
app.post('/upload', async (req, res) => {
const file = req.files.document;
const dest = path.join('/uploads', req.body.filename);
await file.mv(dest);
const meta = JSON.parse(req.body.metadata);
db.query(`INSERT INTO uploads (path, owner) VALUES ('${dest}', '${meta.owner}')`);
res.json({ ok: true, path: dest });
});
```
Identify issues, suggest improvements, and explain the reasoning behind each suggestion. Prioritize by severity.
assert:
- type: contains-any
value: ['injection', 'traversal', 'sanitize', 'parameterized']
reason: Should identify the SQL injection and path traversal vulnerabilities
- type: llm-rubric
value: |
The review should identify multiple issues, prioritized by severity:
1. SQL injection via string-interpolated query (critical)
2. Path traversal via user-controlled filename (critical)
3. Unhandled JSON.parse exception on malformed metadata
4. No validation that req.files.document exists
5. Unawaited/unchecked db.query result
For each issue, it should:
- Explain why it's a problem
- Suggest specific improvements (parameterized queries, basename/allowlist for paths)
- Provide example code where helpful