learningcircuit--local-deep-research/examples/benchmarks/browsecomp_benchmark_readme.md

# BrowseComp Benchmark for Local Deep Research

This document explains how to run the BrowseComp benchmark with Local Deep Research.

## Overview

BrowseComp is a benchmark created by OpenAI to evaluate models on their ability to understand complex questions that may require browsing the web for answers. The questions in BrowseComp often involve multiple criteria that must be satisfied by a single answer.

The benchmark questions are initially provided in an encrypted format, which requires decryption using a "canary" field in the dataset.

## Running the Benchmark

We've created a script called `browsecomp_fixed.py` in the root directory that properly decrypts the BrowseComp questions and runs them through Local Deep Research.

### Basic Usage

```bash
python /path/to/browsecomp_fixed.py --examples 5 --iterations 3 --questions 3
```

### Parameters

- `--examples`: Number of examples to run (default: 5)
- `--iterations`: Number of search iterations per query (default: 1)
- `--questions`: Questions per iteration (default: 1)
- `--search-tool`: Search tool to use (default: "searxng")
- `--output-dir`: Directory to save results (default: "browsecomp_results")

### Performance Tips

BrowseComp questions are challenging and may require more thorough search strategies:

1. **Use more iterations**: Increasing the number of iterations allows the system to refine its understanding of the question and explore different aspects.
   ```bash
   python browsecomp_fixed.py --iterations 5
   ```

2. **Increase questions per iteration**: More questions means more angles to explore.
   ```bash
   python browsecomp_fixed.py --questions 5
   ```

3. **Try different search engines**: Some search engines might perform better for certain types of questions.
   ```bash
   python browsecomp_fixed.py --search-tool wikipedia
   ```

4. **Combine parameters for best results**:
   ```bash
   python browsecomp_fixed.py --examples 10 --iterations 3 --questions 3 --search-tool searxng
   ```

## Understanding the Results

The script saves results in the output directory with the following files:

- `browsecomp_[timestamp]_results.jsonl`: Raw results from the benchmark
- `browsecomp_[timestamp]_evaluation.jsonl`: Evaluation of the results by a grader model

The script will also print a summary with:
- Overall accuracy
- Number of correct answers
- Average processing time

## How It Works

1. The script loads the BrowseComp dataset
2. For each example, it decrypts the problem and correct answer using the "canary" field
3. It runs the decrypted question through LDR's search system
4. It extracts the answer from LDR's response
5. A grader evaluates whether the extracted answer matches the correct answer

## Improving Benchmark Performance

To improve performance on the BrowseComp benchmark:

1. Optimize parameters for more thorough search (more iterations and questions)
2. Use search engines that provide more relevant results for complex queries
3. Consider pre-processing or reformulating the questions to better match search engine capabilities
4. Experiment with different search strategies in LDR's configuration

## Technical Details

The decryption process uses a simple XOR cipher with a key derived from the canary value using SHA-256. This matches the approach used in the original BrowseComp evaluation script.