wehub-resource-sync
6e7352b6f7
Create Release / version-check (push) Has been cancelled
Create Release / release-gate (push) Has been cancelled
Create Release / ci-gate (push) Has been cancelled
Create Release / test-gate (push) Has been cancelled
Create Release / e2e-test-gate (push) Has been cancelled
Create Release / responsive-test-gate (push) Has been cancelled
Create Release / compat-test-gate (push) Has been cancelled
Create Release / compose-integration-gate (push) Has been cancelled
Create Release / vulture-gate (push) Has been cancelled
Create Release / build (push) Has been cancelled
Create Release / provenance (push) Has been cancelled
Create Release / prerelease-docker (push) Has been cancelled
Create Release / publish-docker (push) Has been cancelled
Create Release / create-release (push) Has been cancelled
Create Release / cleanup-changelog (push) Has been cancelled
Create Release / trigger-pypi (push) Has been cancelled
Create Release / monitor-pypi (push) Has been cancelled
Create Release / Clean up orphan prerelease tags and signatures (push) Has been cancelled
OSSF Scorecard / OSSF Security Scorecard Analysis (push) Failing after 0s
OSV-Scanner (Scheduled) / scan-scheduled (push) Failing after 0s
Version Auto-Bump / version-bump (push) Failing after 0s
Backwards Compatibility / Verify Encryption Constants (push) Failing after 1s
Backwards Compatibility / PyPI Version Compatibility (push) Failing after 1s
Backwards Compatibility / Database Migration Tests (push) Failing after 0s
CodeQL Advanced / Analyze (javascript-typescript) (push) Failing after 1s
CodeQL Advanced / Analyze (python) (push) Failing after 1s
Docker Tests (Consolidated) / detect-changes (push) Failing after 1s
Docker Tests (Consolidated) / Build Test Image (push) Failing after 1s
Sync repo labels / sync-labels (push) Failing after 0s
MCP Server Tests / MCP Server Tests (push) Failing after 1s
Docker Tests (Consolidated) / All Pytest Tests + Coverage (push) Has been skipped
Docker Tests (Consolidated) / UI Tests (Puppeteer) (push) Has been cancelled
Docker Tests (Consolidated) / Production Image Smoke Test (push) Has been cancelled
Docker Tests (Consolidated) / UI Tests (Puppeteer) [history-news] (push) Has been cancelled
Docker Tests (Consolidated) / UI Tests (Puppeteer) [link-analytics] (push) Has been cancelled
Docker Tests (Consolidated) / UI Tests (Puppeteer) [library] (push) Has been cancelled
Docker Tests (Consolidated) / UI Tests (Puppeteer) [settings-core] (push) Has been cancelled
Docker Tests (Consolidated) / UI Tests (Puppeteer) [settings-pages] (push) Has been cancelled
Docker Tests (Consolidated) / UI Tests (Puppeteer) [mobile] (push) Has been cancelled
Docker Tests (Consolidated) / UI Tests (Puppeteer) [auth-pages] (push) Has been cancelled
Docker Tests (Consolidated) / Accessibility Tests (push) Has been cancelled
Docker Tests (Consolidated) / UI Tests (Puppeteer) [chat-core] (push) Has been cancelled
Docker Tests (Consolidated) / UI Tests (Puppeteer) [chat-lifecycle] (push) Has been cancelled
Docker Tests (Consolidated) / UI Tests (Puppeteer) [error-benchmark] (push) Has been cancelled
Docker Tests (Consolidated) / UI Tests (Puppeteer) [research-form] (push) Has been cancelled
Docker Tests (Consolidated) / UI Tests (Puppeteer) [research-metrics] (push) Has been cancelled
Docker Tests (Consolidated) / UI Tests (Puppeteer) [research-workflow] (push) Has been cancelled
Docker Tests (Consolidated) / LLM Unit Tests (push) Has been cancelled
Docker Tests (Consolidated) / LLM Example Tests (push) Has been cancelled
Docker Tests (Consolidated) / Infrastructure Tests (push) Has been cancelled
Docker Tests (Consolidated) / UI Tests (Puppeteer) [auth-register] (push) Has been cancelled
Docker Tests (Consolidated) / UI Tests (Puppeteer) [accessibility] (push) Has been cancelled
Docker Tests (Consolidated) / UI Tests (Puppeteer) [api-crud] (push) Has been cancelled
Docker Tests (Consolidated) / UI Tests (Puppeteer) [auth-login] (push) Has been cancelled
Claude API Grading Benchmark
This benchmark integrates Claude 3 Sonnet for grading benchmark results with proper API access through the local database.
Features
- Uses Claude 3 Sonnet for grading benchmark results
- Accesses API keys from the local database
- Supports SimpleQA and BrowseComp benchmarks
- Provides composite scoring with customizable weights
- Comprehensive metrics and accuracy reports
Usage
From the project root directory:
# Run with default settings (source_based strategy, 1 iteration, 5 examples)
./examples/benchmarks/claude_grading/run_benchmark.sh
# Run with custom parameters
./examples/benchmarks/claude_grading/run_benchmark.sh --strategy source_based --iterations 2 --examples 200
How It Works
The benchmark integrates with the evaluation system by patching the grading module to use the local get_llm function, which properly retrieves API keys from the database and configures the Claude model for grading.
This approach ensures accurate grading of benchmark results and enables comparison between different strategies and configurations.
Requirements
- Valid Claude API key stored in the local database
- SearXNG search engine running locally
- Python dependencies installed
Output
Results are saved in the benchmark_results directory with comprehensive metrics:
- Accuracy scores
- Processing times
- Grading confidence
- Detailed evaluation reports