learningcircuit--local-deep-research/community_benchmark_results

Local Deep Research Benchmark Results

📦 This directory is being archived — new results go to a dedicated repo

Community benchmark results have moved to a dedicated GitHub repository (source of truth) with auto-synced leaderboard CSVs on Hugging Face:

👉 GitHub (submit PRs here): LearningCircuit/ldr-benchmarks

👉 Hugging Face (browse leaderboards): local-deep-research/ldr-benchmarks

The new setup offers:

• CI validation of every submission (schema, sharing-policy, secrets scan)
• Auto-generated leaderboard CSVs (per-benchmark and combined) synced to HF
• Dataset Viewer on Hugging Face for browsing
• One canonical place to compare runs across SimpleQA, BrowseComp, and xbench-DeepSearch

Where to submit new results: open a Pull Request against the GitHub repo. The same YAML export from the LDR web UI (/benchmark → YAML button) works unchanged — just drop it under results/{dataset}/{strategy}/{search_engine}/.

What stays here: the existing .yaml result files in this folder and benchmark_template.yaml are kept as a historical archive and for reference. They are not being deleted. New submissions, however, should go to the new repo so results stay consolidated in one place.

Why the move: keeping benchmark data in the code repo bloats git history on every clone, even though the data is static. A dedicated repo solves this cleanly and gives us a CI pipeline + viewer + leaderboards built for exactly this purpose.

---

Historical archive (pre-migration)

This directory contains community-contributed benchmark results for various LLMs tested with Local Deep Research.

Contributing Your Results

Easy Method (v0.6.0+)

1. Run benchmarks using the LDR web interface at /benchmark
2. Go to Benchmark Results page
3. Click the green "YAML" button next to your completed benchmark
4. Review the downloaded file and add any missing info (hardware specs are optional)
5. Submit a PR to add your file to this directory

Manual Method

1. Run benchmarks using the LDR web interface at /benchmark
2. Copy benchmark_template.yaml to a new file named: [model_name]_[date].yaml
   • Example: llama3.3-70b-q4_2025-01-23.yaml
   • Optional: Include your username: johnsmith_llama3.3-70b-q4_2025-01-23.yaml
3. Fill in your results manually
4. Submit a PR to add your file to this directory

Important Guidelines

• Test both strategies: focused-iteration and source-based
• Use consistent settings: Start with 20-50 SimpleQA questions
• Include all metadata: Hardware specs, configuration, and versions are crucial
• Be honest: Negative results are as valuable as positive ones
• Add notes: Your observations help others understand the results
• Review for PII: If you include individual examples in your export, review the file for any personally identifiable information before submitting a PR

Recommended Test Configuration

For Large Models (70B+)

• Context Window: 32768+ tokens
• Focused-iteration: 8 iterations, 5 questions each
• Source-based: 5 iterations, 3 questions each

For Smaller Models (<70B)

• Context Window: 16384+ tokens (adjust based on model)
• Focused-iteration: 5 iterations, 3 questions each
• Source-based: 3 iterations, 3 questions each

Current Baseline

• Model: GPT-4.1-mini
• Strategy: focused-iteration (8 iterations, 5 questions)
• Accuracy: ~95% on SimpleQA (preliminary results from 20-100 question samples)
• Search: SearXNG
• Verified by: 2 independent testers

Understanding the Results

Accuracy Ranges

• 90%+: Excellent - matches GPT-4 performance
• 80-90%: Very good - suitable for most research tasks
• 70-80%: Good - works well with human oversight
• <70%: Limited - may struggle with complex research

Common Issues

• Low accuracy: Often due to insufficient context window
• Timeouts: Model too slow for iterative research
• Memory errors: Reduce context window or batch size
• Rate limiting: SearXNG may throttle excessive requests

Viewing Results

Browse the YAML files in this directory to see how different models perform. Look for patterns like:

• Which quantization levels maintain accuracy
• Minimum viable model size for research tasks
• Best strategy for different model architectures
• Hardware requirements for acceptable performance

Questions?

Join our Discord to discuss results and get help with benchmarking.