wehub-resource-sync
7a0da7932b
OSV-Scanner (Scheduled) / scan-scheduled (push) Failing after 0s
Create Release / test-gate (push) Has been cancelled
Create Release / release-gate (push) Has been cancelled
Create Release / ci-gate (push) Has been cancelled
Create Release / version-check (push) Has been cancelled
Create Release / e2e-test-gate (push) Has been cancelled
Create Release / responsive-test-gate (push) Has been cancelled
Create Release / compat-test-gate (push) Has been cancelled
Create Release / compose-integration-gate (push) Has been cancelled
Create Release / vulture-gate (push) Has been cancelled
Create Release / build (push) Has been cancelled
Create Release / provenance (push) Has been cancelled
Create Release / prerelease-docker (push) Has been cancelled
Create Release / publish-docker (push) Has been cancelled
Create Release / create-release (push) Has been cancelled
Create Release / cleanup-changelog (push) Has been cancelled
Create Release / trigger-pypi (push) Has been cancelled
Create Release / monitor-pypi (push) Has been cancelled
Create Release / Clean up orphan prerelease tags and signatures (push) Has been cancelled
Docker Tests (Consolidated) / UI Tests (Puppeteer) [research-form] (push) Has been cancelled
Docker Tests (Consolidated) / UI Tests (Puppeteer) [research-metrics] (push) Has been cancelled
Docker Tests (Consolidated) / UI Tests (Puppeteer) [research-workflow] (push) Has been cancelled
Docker Tests (Consolidated) / UI Tests (Puppeteer) [settings-core] (push) Has been cancelled
CodeQL Advanced / Analyze (javascript-typescript) (push) Has been cancelled
Docker Tests (Consolidated) / UI Tests (Puppeteer) [history-news] (push) Has been cancelled
Docker Tests (Consolidated) / UI Tests (Puppeteer) [library] (push) Has been cancelled
Docker Tests (Consolidated) / UI Tests (Puppeteer) [link-analytics] (push) Has been cancelled
Docker Tests (Consolidated) / UI Tests (Puppeteer) [chat-core] (push) Has been cancelled
Docker Tests (Consolidated) / UI Tests (Puppeteer) [chat-lifecycle] (push) Has been cancelled
Docker Tests (Consolidated) / UI Tests (Puppeteer) [error-benchmark] (push) Has been cancelled
Docker Tests (Consolidated) / UI Tests (Puppeteer) [settings-pages] (push) Has been cancelled
Docker Tests (Consolidated) / UI Tests (Puppeteer) (push) Has been cancelled
Docker Tests (Consolidated) / Accessibility Tests (push) Has been cancelled
Docker Tests (Consolidated) / LLM Unit Tests (push) Has been cancelled
Docker Tests (Consolidated) / LLM Example Tests (push) Has been cancelled
Docker Tests (Consolidated) / Production Image Smoke Test (push) Has been cancelled
Docker Tests (Consolidated) / Infrastructure Tests (push) Has been cancelled
OSSF Scorecard / OSSF Security Scorecard Analysis (push) Has been cancelled
Docker Tests (Consolidated) / UI Tests (Puppeteer) [mobile] (push) Has been cancelled
Backwards Compatibility / Verify Encryption Constants (push) Has been cancelled
Backwards Compatibility / PyPI Version Compatibility (push) Has been cancelled
Backwards Compatibility / Database Migration Tests (push) Has been cancelled
CodeQL Advanced / Analyze (python) (push) Has been cancelled
Docker Tests (Consolidated) / detect-changes (push) Has been cancelled
Docker Tests (Consolidated) / Build Test Image (push) Has been cancelled
Docker Tests (Consolidated) / All Pytest Tests + Coverage (push) Has been cancelled
Docker Tests (Consolidated) / UI Tests (Puppeteer) [accessibility] (push) Has been cancelled
Docker Tests (Consolidated) / UI Tests (Puppeteer) [api-crud] (push) Has been cancelled
Docker Tests (Consolidated) / UI Tests (Puppeteer) [auth-login] (push) Has been cancelled
Docker Tests (Consolidated) / UI Tests (Puppeteer) [auth-pages] (push) Has been cancelled
Docker Tests (Consolidated) / UI Tests (Puppeteer) [auth-register] (push) Has been cancelled
63 lines
1.9 KiB
Markdown
63 lines
1.9 KiB
Markdown
# Benchmarks for Local Deep Research
|
|
|
|
This directory contains scripts for running benchmarks to evaluate Local Deep Research's performance.
|
|
|
|
## Available Benchmarks
|
|
|
|
### SimpleQA
|
|
|
|
The SimpleQA benchmark evaluates factual question answering capabilities.
|
|
|
|
```bash
|
|
python run_simpleqa.py --examples 10 --iterations 3 --questions 3
|
|
```
|
|
|
|
Options:
|
|
- `--examples`: Number of examples to run (default: 10)
|
|
- `--iterations`: Number of search iterations (default: 3)
|
|
- `--questions`: Questions per iteration (default: 3)
|
|
- `--search-tool`: Search tool to use (default: "searxng")
|
|
- `--output-dir`: Directory to save results (default: "benchmark_results")
|
|
- `--no-eval`: Skip evaluation
|
|
- `--human-eval`: Use human evaluation
|
|
- `--eval-model`: Model to use for evaluation
|
|
- `--eval-provider`: Provider to use for evaluation
|
|
|
|
### BrowseComp
|
|
|
|
The BrowseComp benchmark evaluates web browsing comprehension and complex question answering.
|
|
|
|
```bash
|
|
python run_browsecomp.py --examples 5 --iterations 3 --questions 3
|
|
```
|
|
|
|
Options:
|
|
- `--examples`: Number of examples to run (default: 2)
|
|
- `--iterations`: Number of search iterations (default: 1)
|
|
- `--questions`: Questions per iteration (default: 1)
|
|
- `--search-tool`: Search tool to use (default: "searxng")
|
|
- `--output-dir`: Directory to save results (default: "browsecomp_results")
|
|
|
|
See `browsecomp_benchmark_readme.md` for more information on how BrowseComp works.
|
|
|
|
## Running All Benchmarks
|
|
|
|
To run both benchmarks and compare results:
|
|
|
|
```bash
|
|
# Run SimpleQA with default settings
|
|
python run_simpleqa.py
|
|
|
|
# Run BrowseComp with increased iterations and questions
|
|
python run_browsecomp.py --iterations 3 --questions 3
|
|
```
|
|
|
|
## Evaluating Results
|
|
|
|
Results are saved in the specified output directories and include:
|
|
- Raw results (JSONL format)
|
|
- Evaluation results (JSONL format)
|
|
- Summary reports (Markdown format)
|
|
|
|
The scripts will also print a summary of the results to the console, including accuracy metrics.
|