wehub-resource-sync
7a0da7932b
3.3 KiB
BrowseComp Benchmark for Local Deep Research
This document explains how to run the BrowseComp benchmark with Local Deep Research.
Overview
BrowseComp is a benchmark created by OpenAI to evaluate models on their ability to understand complex questions that may require browsing the web for answers. The questions in BrowseComp often involve multiple criteria that must be satisfied by a single answer.
The benchmark questions are initially provided in an encrypted format, which requires decryption using a "canary" field in the dataset.
Running the Benchmark
We've created a script called browsecomp_fixed.py in the root directory that properly decrypts the BrowseComp questions and runs them through Local Deep Research.
Basic Usage
python /path/to/browsecomp_fixed.py --examples 5 --iterations 3 --questions 3
Parameters
--examples: Number of examples to run (default: 5)--iterations: Number of search iterations per query (default: 1)--questions: Questions per iteration (default: 1)--search-tool: Search tool to use (default: "searxng")--output-dir: Directory to save results (default: "browsecomp_results")
Performance Tips
BrowseComp questions are challenging and may require more thorough search strategies:
-
Use more iterations: Increasing the number of iterations allows the system to refine its understanding of the question and explore different aspects.
python browsecomp_fixed.py --iterations 5 -
Increase questions per iteration: More questions means more angles to explore.
python browsecomp_fixed.py --questions 5 -
Try different search engines: Some search engines might perform better for certain types of questions.
python browsecomp_fixed.py --search-tool wikipedia -
Combine parameters for best results:
python browsecomp_fixed.py --examples 10 --iterations 3 --questions 3 --search-tool searxng
Understanding the Results
The script saves results in the output directory with the following files:
browsecomp_[timestamp]_results.jsonl: Raw results from the benchmarkbrowsecomp_[timestamp]_evaluation.jsonl: Evaluation of the results by a grader model
The script will also print a summary with:
- Overall accuracy
- Number of correct answers
- Average processing time
How It Works
- The script loads the BrowseComp dataset
- For each example, it decrypts the problem and correct answer using the "canary" field
- It runs the decrypted question through LDR's search system
- It extracts the answer from LDR's response
- A grader evaluates whether the extracted answer matches the correct answer
Improving Benchmark Performance
To improve performance on the BrowseComp benchmark:
- Optimize parameters for more thorough search (more iterations and questions)
- Use search engines that provide more relevant results for complex queries
- Consider pre-processing or reformulating the questions to better match search engine capabilities
- Experiment with different search strategies in LDR's configuration
Technical Details
The decryption process uses a simple XOR cipher with a key derived from the canary value using SHA-256. This matches the approach used in the original BrowseComp evaluation script.