# Benchmarks for Local Deep Research

This directory contains scripts for running benchmarks to evaluate Local Deep Research's performance.

## Available Benchmarks

### SimpleQA

The SimpleQA benchmark evaluates factual question answering capabilities.

```bash
python run_simpleqa.py --examples 10 --iterations 3 --questions 3
```

Options:
- `--examples`: Number of examples to run (default: 10)
- `--iterations`: Number of search iterations (default: 3)
- `--questions`: Questions per iteration (default: 3)
- `--search-tool`: Search tool to use (default: "searxng")
- `--output-dir`: Directory to save results (default: "benchmark_results")
- `--no-eval`: Skip evaluation
- `--human-eval`: Use human evaluation
- `--eval-model`: Model to use for evaluation
- `--eval-provider`: Provider to use for evaluation

### BrowseComp

The BrowseComp benchmark evaluates web browsing comprehension and complex question answering.

```bash
python run_browsecomp.py --examples 5 --iterations 3 --questions 3
```

Options:
- `--examples`: Number of examples to run (default: 2)
- `--iterations`: Number of search iterations (default: 1)
- `--questions`: Questions per iteration (default: 1)
- `--search-tool`: Search tool to use (default: "searxng")
- `--output-dir`: Directory to save results (default: "browsecomp_results")

See `browsecomp_benchmark_readme.md` for more information on how BrowseComp works.

## Running All Benchmarks

To run both benchmarks and compare results:

```bash
# Run SimpleQA with default settings
python run_simpleqa.py

# Run BrowseComp with increased iterations and questions
python run_browsecomp.py --iterations 3 --questions 3
```

## Evaluating Results

Results are saved in the specified output directories and include:
- Raw results (JSONL format)
- Evaluation results (JSONL format)
- Summary reports (Markdown format)

The scripts will also print a summary of the results to the console, including accuracy metrics.