learningcircuit--local-deep-research/docs/search-engines.md

Search Engines Guide

Local Deep Research integrates with multiple search engines to provide comprehensive research capabilities. This guide covers all available search engines, their specializations, and configuration details.

Note

: This documentation is maintained by the community and may contain inaccuracies. While we strive to keep it up-to-date, please verify critical information and report any errors via GitHub Issues.

Overview

LDR supports three categories of search engines:

• Free Search Engines - No API key required
• Premium Search Engines - Require API keys but offer enhanced features
• Custom Sources - Your own documents and databases

Search Engine Selection

Dynamic Engine Selection (Recommended)

The default langgraph-agent strategy selects the most appropriate engines dynamically per query: every enabled engine (and any registered retriever or local collection) is exposed to the research agent as a tool, and the agent decides which to call for each sub-question. You only pick a primary engine (searxng is the recommended default):

result = quick_summary(
    query="What are the latest advances in quantum computing?",
    search_tool="searxng"  # Primary engine; the langgraph-agent strategy
                           # can still pull in other enabled engines per query
)

Note

: The former auto and parallel meta engines were removed — the langgraph-agent strategy replaces them. Stored settings are migrated automatically; explicit search_tool="auto" callers should switch to a concrete engine like searxng.

Free Search Engines

Academic Search Engines

arXiv

• Specialization: Scientific papers and preprints
• Best for: Physics, mathematics, computer science, biology
• Results: Direct access to research papers
• Rate Limit: Moderate - automatic retry on limits

PubMed

• Specialization: Biomedical and life science literature
• Best for: Medical research, clinical studies, biology
• Results: Abstracts and links to full papers
• Rate Limit: Generous - rarely hits limits

Semantic Scholar

• Specialization: Academic literature across all fields
• Best for: Cross-disciplinary research, citation networks
• Results: Paper summaries with citation context
• Rate Limit: Moderate - adaptive rate limiting handles this

General Purpose

Wikipedia

• Specialization: General knowledge and encyclopedic information
• Best for: Background information, concepts, facts
• Results: Well-structured article content
• Rate Limit: Very generous

SearXNG (Highly Recommended)

• Specialization: Meta-search engine aggregating multiple sources
• Best for: Comprehensive web search with privacy
• Results: Aggregated results from Google, Bing, DuckDuckGo, etc.
• Setup:

  docker pull searxng/searxng
  docker run -d -p 8080:8080 --name searxng searxng/searxng
  

• Configuration: Set URL to http://localhost:8080 in settings

DuckDuckGo

• Specialization: Privacy-focused web search
• Best for: General web queries without tracking
• Results: Web pages, instant answers
• Rate Limit: Strict - use SearXNG for better reliability

Technical Search

GitHub

• Specialization: Code repositories and documentation
• Best for: Finding code examples, libraries, technical solutions
• Results: Repository information, code snippets, issues
• Rate Limit: Moderate when unauthenticated

Elasticsearch

• Specialization: Custom search within your Elasticsearch cluster
• Best for: Searching your own indexed data
• Configuration: See Elasticsearch Setup Guide

Historical Search

Wayback Machine

• Specialization: Historical web content
• Best for: Finding deleted content, tracking changes over time
• Results: Archived web pages with timestamps
• Rate Limit: Moderate

News Search

The Guardian

• Specialization: News articles and journalism
• Best for: Current events, news analysis
• Results: Recent news articles
• Note: Requires API key (free tier available at https://open-platform.theguardian.com/)

Wikinews

• Specialization: Open and collaboratively-written news articles on a wide range of topics
• Best for: Historical and recent news, general news coverage, quick overviews
• Results: News articles written by volunteers with verified sources

Premium Search Engines

Tavily

• Specialization: AI-optimized search for LLM applications
• Best for: High-quality, relevant results for AI research
• Pricing: Free tier available, paid plans for higher volume
• Configuration:

  # In .env file or web interface
  LDR_SEARCH_ENGINE_TAVILY_API_KEY=your-key-here
  

Google (via SerpAPI)

• Specialization: Comprehensive web search
• Best for: Most current and comprehensive results
• Pricing: Paid service with free trial
• Configuration:

  LDR_SEARCH_ENGINE_WEB_SERPAPI_API_KEY=your-key-here
  

Google Programmable Search Engine

• Specialization: Customizable Google search
• Best for: Searching specific sites or topics
• Pricing: Free tier with limits
• Configuration:

  LDR_SEARCH_ENGINE_WEB_GOOGLE_PSE_API_KEY=your-key-here
  LDR_SEARCH_ENGINE_WEB_GOOGLE_PSE_ENGINE_ID=your-engine-id
  

Brave Search

• Specialization: Independent search index with privacy focus
• Best for: Web search without big tech tracking
• Pricing: Free tier available
• Configuration:

  LDR_SEARCH_ENGINE_WEB_BRAVE_API_KEY=your-key-here
  

Custom Sources

Local Documents

• Specialization: Search your private documents
• Supported formats: PDF, TXT, MD, DOCX, CSV, and more
• Configuration: See Configuring Local Search
• Setup:
  1. Go to Settings → Search for "local"
  2. Add document collection paths
  3. Choose embedding model (CPU or Ollama)
  4. First search will index documents

LangChain Retrievers

• Specialization: Any vector store or database
• Supported: FAISS, Chroma, Pinecone, Weaviate, Elasticsearch
• Configuration: See LangChain Integration Guide

Search Performance Comparison

Engine	Speed	Quality	Privacy	Rate Limits
SearXNG	★★★★★	★★★★☆	★★★★★	★★★★★
Wikipedia	★★★★☆	★★★★☆	★★★★★	★★★★★
arXiv	★★★★☆	★★★★★	★★★★★	★★★☆☆
PubMed	★★★★☆	★★★★★	★★★★★	★★★★☆
Tavily	★★★★☆	★★★★★	★★★☆☆	★★★★☆
Google (SerpAPI)	★★★★☆	★★★★★	★★☆☆☆	★★★★★
Local Documents	★★★☆☆	★★★★★	★★★★★	★★★★★

Rate Limiting and Reliability

LDR includes intelligent adaptive rate limiting that:

• Learns optimal wait times for each engine
• Automatically retries failed requests
• Prevents your IP from being blocked
• Maintains high reliability

Managing Rate Limits

# Check rate limit status
python -m local_deep_research.web_search_engines.rate_limiting status

# Reset rate limits if needed
python -m local_deep_research.web_search_engines.rate_limiting reset

Search Strategies

LDR supports multiple search strategies that determine how queries are processed:

• langgraph-agent: Agentic research that picks engines dynamically per query (default)
• source-based: Single query, fast results
• focused_iteration: Iterative refinement for accuracy

Best Practices

1. For General Research: Use searxng with the default langgraph-agent strategy
2. For Academic Research: Combine arxiv, pubmed, and semantic_scholar
3. For Technical Questions: Use github with searxng
4. For Maximum Privacy: Use searxng with local Ollama models
5. For Best Quality: Use tavily or Google with focused_iteration strategy

Troubleshooting

SearXNG Not Working

• Verify container is running: docker ps | grep searxng
• Check URL in settings: http://localhost:8080
• Test directly: curl http://localhost:8080
• Check the logs: docker logs searxng or view them in the LDR web UI

Rate Limit Errors

• Wait a few minutes and retry
• Use the langgraph-agent strategy, which can route around rate-limited engines
• Consider adding premium engines for higher limits

No Results Found

• Try different search engines
• Broaden your query
• Check internet connectivity
• Verify API keys for premium engines

Advanced Configuration

Configuring Search Engines

You can enable/disable specific search engines and adjust their reliability parameters in the settings. This affects which engines the langgraph-agent strategy can choose from and how the system handles rate limiting.

Multi-Engine Research

The former auto and parallel meta engines (which fanned a query out over several engines) have been removed. To research across multiple engines, use the default langgraph-agent strategy: it calls any enabled engine as a tool, in parallel where useful, and picks per sub-question which engines to query.

Related Documentation

• API Quickstart
• Configuration Guide
• Full Configuration Reference
• LangChain Integration
• Elasticsearch Setup