chore: import upstream snapshot with attribution
CI / Shell Format Check (push) Has been cancelled
CI / Check Ruby (3.4) (push) Has been cancelled
CI / CI Config (push) Has been cancelled
CI / Test on Node ${{ matrix.node }} and ${{ matrix.os }}${{ matrix.shard && format(' (shard {0}/3)', matrix.shard) || '' }} (push) Has been cancelled
CI / Build on Node ${{ matrix.node }} (push) Has been cancelled
CI / Style Check (push) Has been cancelled
CI / Generate Assets (push) Has been cancelled
CI / Check Python (3.14) (push) Has been cancelled
CI / Check Python (3.9) (push) Has been cancelled
CI / Build Docs (push) Has been cancelled
CI / Code Scan Action (push) Has been cancelled
CI / Site tests (push) Has been cancelled
CI / webui tests (push) Has been cancelled
CI / Run Integration Tests (push) Has been cancelled
CI / Run Smoke Tests (push) Has been cancelled
CI / Go Tests (push) Has been cancelled
CI / Share Test (push) Has been cancelled
CI / Redteam (Production API) (push) Has been cancelled
CI / Redteam (Staging API) (push) Has been cancelled
CI / GitHub Actions Lint (push) Has been cancelled
CI / Check Ruby (3.0) (push) Has been cancelled
release-please / release-please (push) Has been cancelled
release-please / build (push) Has been cancelled
release-please / publish-npm (push) Has been cancelled
release-please / publish-npm-backfill (push) Has been cancelled
release-please / docker (push) Has been cancelled
release-please / publish-code-scan-action (push) Has been cancelled
release-please / attest-code-scan-action (push) Has been cancelled
Deploy local.promptfoo.app / Deploy to Cloudflare Pages (push) Has been cancelled
Test and Publish Multi-arch Docker Image / test (push) Has been cancelled
Test and Publish Multi-arch Docker Image / build-docker-and-push-digests (map[digest-suffix:linux-amd64 platform:linux/amd64 runner:ubuntu-latest]) (push) Has been cancelled
Test and Publish Multi-arch Docker Image / build-docker-and-push-digests (map[digest-suffix:linux-arm64 platform:linux/arm64 runner:ubuntu-24.04-arm]) (push) Has been cancelled
Test and Publish Multi-arch Docker Image / merge-docker-digests (push) Has been cancelled
Test and Publish Multi-arch Docker Image / Attest Multi-arch Image (push) Has been cancelled
Validate Renovate Config / Validate Renovate Configuration (push) Has been cancelled

This commit is contained in:
wehub-resource-sync
2026-07-13 13:24:08 +08:00
commit 0d3cb498a3
5438 changed files with 1316560 additions and 0 deletions
+66
View File
@@ -0,0 +1,66 @@
# eval-search-rubric (Search Rubric)
You can run this example with:
```bash
npx promptfoo@latest init --example eval-search-rubric
cd eval-search-rubric
```
This example demonstrates how to use the `search-rubric` assertion type to verify that LLM outputs contain accurate, current information.
## Overview
The `search-rubric` assertion allows you to verify facts by searching the web in real-time. This is particularly useful for:
- Current events and news
- Stock prices and financial data
- Weather information
- Recent company information
- Any time-sensitive data
## Running the Example
```bash
npx promptfoo eval
```
## How It Works
1. The LLM generates a response to your prompt
2. The search-rubric assertion extracts the claim you want to verify
3. A provider with web search capabilities searches for current information
4. The assertion passes or fails based on whether the output matches current web data
## Provider Support
### Anthropic Claude
- Web search capabilities via tool configuration (launched in 2025)
- Requires explicit `web_search_20250305` tool configuration
- Pricing: $10 per 1,000 searches plus standard token costs
### OpenAI
- Requires `web_search_preview` tool configuration
- Works with gpt-5.1, o4-mini, and other Responses API models
### Perplexity
- Built-in web search capabilities
- No additional configuration needed
## Configuration
```yaml
assert:
- type: search-rubric
value: 'search query to verify'
threshold: 0.8 # Optional: minimum accuracy score (0-1)
```
## Notes
- Search rubric assertions add latency (2-5 seconds per assertion)
- Use caching during development: `npx promptfoo eval --cache`
- Be specific with your search queries for better results
@@ -0,0 +1,101 @@
# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
description: Search rubric assertions for fact-checking
prompts:
- '{{question}}'
providers:
# Anthropic Claude 4.6 Opus with web search tool
- id: anthropic:messages:claude-opus-4-6
config:
tools:
- type: web_search_20250305
name: web_search
max_uses: 5
# OpenAI GPT-5.4 with web search tool
- id: openai:responses:gpt-5.4
config:
tools:
- type: web_search_preview
# Google Gemini 3.1 Pro Preview with googleSearch
- id: google:gemini-3.1-pro-preview
config:
tools:
- googleSearch: {}
# xAI Grok 4.3 with Responses API web search
- id: xai:responses:grok-4.3
config:
tools:
- type: web_search
# Perplexity Sonar Pro with built-in search
- perplexity:sonar-pro
tests:
# Tech industry leadership
- vars:
question: 'Who is the current CEO of Microsoft?'
assert:
- type: search-rubric
value: 'Correctly identifies Satya Nadella as the current CEO of Microsoft'
- vars:
question: 'Who is the current CEO of OpenAI?'
assert:
- type: search-rubric
value: 'Correctly identifies the current CEO of OpenAI (Sam Altman as of late 2024)'
# Real-time financial data
- vars:
question: 'What is the current stock price of NVIDIA (NVDA)?'
assert:
- type: search-rubric
value: |
Provides a recent NVIDIA stock price that:
- Is a realistic price (should be in the hundreds of dollars range as of 2025)
- Includes the currency (USD or $)
threshold: 0.8
- vars:
question: 'What is the current market cap of Apple (AAPL)?'
assert:
- type: search-rubric
value: 'States Apple market cap in the trillions of dollars (should be around $3-4 trillion as of 2025)'
threshold: 0.8
# Current events and sports
- vars:
question: 'Who won the 2024 US Presidential Election?'
assert:
- type: search-rubric
value: 'Correctly identifies Donald Trump as the winner of the 2024 US Presidential Election'
- vars:
question: 'Who won Super Bowl LVIII in February 2024?'
assert:
- type: search-rubric
value: 'Correctly identifies the Kansas City Chiefs as the winners of Super Bowl LVIII (58) defeating the San Francisco 49ers'
# Technology facts
- vars:
question: 'What is the latest version of Node.js LTS?'
assert:
- type: search-rubric
value: |
States a currently supported Node.js LTS version:
- Names a version that is in Active or Maintenance LTS
- Does not recommend an end-of-life version
threshold: 0.7
# Weather (real-time)
- vars:
question: 'What is the current weather in Tokyo, Japan?'
assert:
- type: search-rubric
value: |
Provides current Tokyo weather including:
- Temperature with units (Celsius or Fahrenheit)
- General conditions (sunny, cloudy, rainy, etc.)
@@ -0,0 +1,21 @@
# Simple test to verify search-rubric assertion
prompts:
- 'The CEO of Microsoft is {{name}}'
providers:
# Test with OpenAI (simpler)
- id: openai:gpt-4o-mini
tests:
- vars:
name: 'Satya Nadella'
assert:
- type: search-rubric
value: 'Confirms that {{name}} is the current CEO of Microsoft'
- vars:
name: 'Bill Gates' # Intentionally wrong
assert:
- type: search-rubric
value: 'States the correct current CEO of Microsoft (should identify this is incorrect)'
threshold: 0.8