learningcircuit--local-deep-research/docs/search-engines.md

# Search Engines Guide

Local Deep Research integrates with multiple search engines to provide comprehensive research capabilities. This guide covers all available search engines, their specializations, and configuration details.

> **Note**: This documentation is maintained by the community and may contain inaccuracies. While we strive to keep it up-to-date, please verify critical information and report any errors via [GitHub Issues](https://github.com/LearningCircuit/local-deep-research/issues).

## Overview

LDR supports three categories of search engines:
- **Free Search Engines** - No API key required
- **Premium Search Engines** - Require API keys but offer enhanced features
- **Custom Sources** - Your own documents and databases

## Search Engine Selection

### Dynamic Engine Selection (Recommended)

The default `langgraph-agent` strategy selects the most appropriate engines dynamically per query: every enabled engine (and any registered retriever or local collection) is exposed to the research agent as a tool, and the agent decides which to call for each sub-question. You only pick a primary engine (`searxng` is the recommended default):

```python
result = quick_summary(
    query="What are the latest advances in quantum computing?",
    search_tool="searxng"  # Primary engine; the langgraph-agent strategy
                           # can still pull in other enabled engines per query
)
```

> **Note**: The former `auto` and `parallel` meta engines were removed — the langgraph-agent strategy replaces them. Stored settings are migrated automatically; explicit `search_tool="auto"` callers should switch to a concrete engine like `searxng`.

## Free Search Engines

### Academic Search Engines

#### arXiv
- **Specialization**: Scientific papers and preprints
- **Best for**: Physics, mathematics, computer science, biology
- **Results**: Direct access to research papers
- **Rate Limit**: Moderate - automatic retry on limits

#### PubMed
- **Specialization**: Biomedical and life science literature
- **Best for**: Medical research, clinical studies, biology
- **Results**: Abstracts and links to full papers
- **Rate Limit**: Generous - rarely hits limits

#### Semantic Scholar
- **Specialization**: Academic literature across all fields
- **Best for**: Cross-disciplinary research, citation networks
- **Results**: Paper summaries with citation context
- **Rate Limit**: Moderate - adaptive rate limiting handles this

### General Purpose

#### Wikipedia
- **Specialization**: General knowledge and encyclopedic information
- **Best for**: Background information, concepts, facts
- **Results**: Well-structured article content
- **Rate Limit**: Very generous

#### SearXNG (Highly Recommended)
- **Specialization**: Meta-search engine aggregating multiple sources
- **Best for**: Comprehensive web search with privacy
- **Results**: Aggregated results from Google, Bing, DuckDuckGo, etc.
- **Setup**:
  ```bash
  docker pull searxng/searxng
  docker run -d -p 8080:8080 --name searxng searxng/searxng
  ```
- **Configuration**: Set URL to `http://localhost:8080` in settings

#### DuckDuckGo
- **Specialization**: Privacy-focused web search
- **Best for**: General web queries without tracking
- **Results**: Web pages, instant answers
- **Rate Limit**: Strict - use SearXNG for better reliability

### Technical Search

#### GitHub
- **Specialization**: Code repositories and documentation
- **Best for**: Finding code examples, libraries, technical solutions
- **Results**: Repository information, code snippets, issues
- **Rate Limit**: Moderate when unauthenticated

#### Elasticsearch
- **Specialization**: Custom search within your Elasticsearch cluster
- **Best for**: Searching your own indexed data
- **Configuration**: See [Elasticsearch Setup Guide](elasticsearch_search_engine.md)

### Historical Search

#### Wayback Machine
- **Specialization**: Historical web content
- **Best for**: Finding deleted content, tracking changes over time
- **Results**: Archived web pages with timestamps
- **Rate Limit**: Moderate

### News Search

#### The Guardian
- **Specialization**: News articles and journalism
- **Best for**: Current events, news analysis
- **Results**: Recent news articles
- **Note**: Requires API key (free tier available at https://open-platform.theguardian.com/)

#### Wikinews
- **Specialization**: Open and collaboratively-written news articles on a wide range of topics
- **Best for**: Historical and recent news, general news coverage, quick overviews
- **Results**: News articles written by volunteers with verified sources

## Premium Search Engines

### Tavily
- **Specialization**: AI-optimized search for LLM applications
- **Best for**: High-quality, relevant results for AI research
- **Pricing**: Free tier available, paid plans for higher volume
- **Configuration**:
  ```bash
  # In .env file or web interface
  LDR_SEARCH_ENGINE_TAVILY_API_KEY=your-key-here
  ```

### Google (via SerpAPI)
- **Specialization**: Comprehensive web search
- **Best for**: Most current and comprehensive results
- **Pricing**: Paid service with free trial
- **Configuration**:
  ```bash
  LDR_SEARCH_ENGINE_WEB_SERPAPI_API_KEY=your-key-here
  ```

### Google Programmable Search Engine
- **Specialization**: Customizable Google search
- **Best for**: Searching specific sites or topics
- **Pricing**: Free tier with limits
- **Configuration**:
  ```bash
  LDR_SEARCH_ENGINE_WEB_GOOGLE_PSE_API_KEY=your-key-here
  LDR_SEARCH_ENGINE_WEB_GOOGLE_PSE_ENGINE_ID=your-engine-id
  ```

### Brave Search
- **Specialization**: Independent search index with privacy focus
- **Best for**: Web search without big tech tracking
- **Pricing**: Free tier available
- **Configuration**:
  ```bash
  LDR_SEARCH_ENGINE_WEB_BRAVE_API_KEY=your-key-here
  ```

## Custom Sources

### Local Documents
- **Specialization**: Search your private documents
- **Supported formats**: PDF, TXT, MD, DOCX, CSV, and more
- **Configuration**: See [Configuring Local Search](https://github.com/LearningCircuit/local-deep-research/wiki/Configuring-Local-Search)
- **Setup**:
  1. Go to Settings → Search for "local"
  2. Add document collection paths
  3. Choose embedding model (CPU or Ollama)
  4. First search will index documents

### LangChain Retrievers
- **Specialization**: Any vector store or database
- **Supported**: FAISS, Chroma, Pinecone, Weaviate, Elasticsearch
- **Configuration**: See [LangChain Integration Guide](LANGCHAIN_RETRIEVER_INTEGRATION.md)


## Search Performance Comparison

| Engine | Speed | Quality | Privacy | Rate Limits |
|--------|-------|---------|---------|-------------|
| SearXNG | ★★★★★ | ★★★★☆ | ★★★★★ | ★★★★★ |
| Wikipedia | ★★★★☆ | ★★★★☆ | ★★★★★ | ★★★★★ |
| arXiv | ★★★★☆ | ★★★★★ | ★★★★★ | ★★★☆☆ |
| PubMed | ★★★★☆ | ★★★★★ | ★★★★★ | ★★★★☆ |
| Tavily | ★★★★☆ | ★★★★★ | ★★★☆☆ | ★★★★☆ |
| Google (SerpAPI) | ★★★★☆ | ★★★★★ | ★★☆☆☆ | ★★★★★ |
| Local Documents | ★★★☆☆ | ★★★★★ | ★★★★★ | ★★★★★ |

## Rate Limiting and Reliability

LDR includes intelligent adaptive rate limiting that:
- Learns optimal wait times for each engine
- Automatically retries failed requests
- Prevents your IP from being blocked
- Maintains high reliability

### Managing Rate Limits

```bash
# Check rate limit status
python -m local_deep_research.web_search_engines.rate_limiting status

# Reset rate limits if needed
python -m local_deep_research.web_search_engines.rate_limiting reset
```

## Search Strategies

LDR supports multiple search strategies that determine how queries are processed:

- **langgraph-agent**: Agentic research that picks engines dynamically per query (default)
- **source-based**: Single query, fast results
- **focused_iteration**: Iterative refinement for accuracy

## Best Practices

1. **For General Research**: Use `searxng` with the default langgraph-agent strategy
2. **For Academic Research**: Combine `arxiv`, `pubmed`, and `semantic_scholar`
3. **For Technical Questions**: Use `github` with `searxng`
4. **For Maximum Privacy**: Use `searxng` with local Ollama models
5. **For Best Quality**: Use `tavily` or Google with `focused_iteration` strategy

## Troubleshooting

### SearXNG Not Working
- Verify container is running: `docker ps | grep searxng`
- Check URL in settings: `http://localhost:8080`
- Test directly: `curl http://localhost:8080`
- Check the logs: `docker logs searxng` or view them in the LDR web UI

### Rate Limit Errors
- Wait a few minutes and retry
- Use the langgraph-agent strategy, which can route around rate-limited engines
- Consider adding premium engines for higher limits

### No Results Found
- Try different search engines
- Broaden your query
- Check internet connectivity
- Verify API keys for premium engines

## Advanced Configuration

### Configuring Search Engines

You can enable/disable specific search engines and adjust their reliability parameters in the settings. This affects which engines the langgraph-agent strategy can choose from and how the system handles rate limiting.

### Multi-Engine Research

The former `auto` and `parallel` meta engines (which fanned a query out over several engines) have been removed. To research across multiple engines, use the default langgraph-agent strategy: it calls any enabled engine as a tool, in parallel where useful, and picks per sub-question which engines to query.

## Related Documentation

- [API Quickstart](api-quickstart.md)
- [Configuration Guide](env_configuration.md)
- [Full Configuration Reference](CONFIGURATION.md)
- [LangChain Integration](LANGCHAIN_RETRIEVER_INTEGRATION.md)
- [Elasticsearch Setup](elasticsearch_search_engine.md)