learningcircuit--local-deep-research/docs/elasticsearch_search_engine.md

# Elasticsearch Search Engine

This document describes how to use and configure the Elasticsearch search engine within the Local Deep Research project.

## Overview

The Elasticsearch search engine allows you to search for documents within Elasticsearch indices. This is particularly useful for scenarios requiring searches across large volumes of structured or unstructured text data.

## Prerequisites

1.  A running Elasticsearch server (local or remote).
2.  Elasticsearch Python client library: `elasticsearch>=8.10.0`.

## Installation

Ensure that you have installed the `elasticsearch` Python package. If you are installing Local Deep Research from source, you can run:

```bash
pip install elasticsearch>=8.10.0
```

## Basic Usage

### Usage in Code

```python
from local_deep_research.web_search_engines.engines.search_engine_elasticsearch import ElasticsearchSearchEngine

# Create search engine instance
es_search = ElasticsearchSearchEngine(
    hosts=["http://localhost:9200"],  # List of Elasticsearch hosts
    index_name="my_index",            # Name of the index to search
    username="user",                  # Optional: Authentication username
    password="pass",                  # Optional: Authentication password
    max_results=10                    # Maximum number of results to return
)

# Execute search
results = es_search.run("Your search query")

# Process results
for result in results:
    print(f"Title: {result.get('title')}")
    print(f"Snippet: {result.get('snippet')}")
    print(f"Content: {result.get('content')}")
```

### Advanced Search

The ES search engine supports various advanced search methods:

```python
# Use Elasticsearch Query String syntax
results = es_search.search_by_query_string("title:keyword AND content:search_text")

# Use Elasticsearch DSL (Domain Specific Language)
results = es_search.search_by_dsl({
    "query": {
        "bool": {
            "must": {"match": {"content": "search term"}},
            "filter": {"term": {"category": "technology"}}
        }
    }
})
```

## Configuration

### Search Engine Parameters

| Parameter | Type | Default Value | Description |
| :--- | :--- | :--- | :--- |
| hosts | List[str] | ["http://localhost:9200"] | List of Elasticsearch server addresses |
| index_name | str | "documents" | Name of the index to search |
| username | Optional[str] | None | Authentication username |
| password | Optional[str] | None | Authentication password |
| api_key | Optional[str] | None | API Key authentication |
| cloud_id | Optional[str] | None | Elastic Cloud ID |
| max_results | int | 10 | Maximum number of results |
| highlight_fields | List[str] | ["content", "title"] | Fields to highlight |
| search_fields | List[str] | ["content", "title"] | Fields to search |
| filter_query | Optional[Dict] | None | Optional filter query |
| llm | Optional[BaseLLM] | None | LLM for relevance filtering |
| max_filtered_results | Optional[int] | None | Maximum number of filtered results |

## Indexing Data

For ease of use, we provide the `ElasticsearchManager` utility class to help index data:

```python
from local_deep_research.utilities.es_utils import ElasticsearchManager

# Create ES Manager
es_manager = ElasticsearchManager(
    hosts=["http://localhost:9200"]
)

# Create index
es_manager.create_index("my_index")

# Index a single document
es_manager.index_document(
    index_name="my_index",
    document={
        "title": "Document Title",
        "content": "Document content...",
    }
)

# Bulk index documents
documents = [
    {"title": "Document 1", "content": "Content 1"},
    {"title": "Document 2", "content": "Content 2"}
]
es_manager.bulk_index_documents("my_index", documents)

# Index file (automatically extract content)
es_manager.index_file("my_index", "path/to/document.pdf")

# Index all files in a directory
es_manager.index_directory(
    "my_index",
    "path/to/docs",
    file_patterns=["*.pdf", "*.docx", "*.txt"]
)
```

## Examples

Refer to `examples/elasticsearch_search_example.py` for a complete usage example.

## Running Examples

Ensure Elasticsearch is running, then execute:

```bash
python examples/elasticsearch_search_example.py
```

## Troubleshooting

### Cannot connect to Elasticsearch

-   Ensure the Elasticsearch server is running.
-   Verify the host address and port.
-   Verify authentication credentials (if required).
-   Check network connection and firewall settings.

### Empty Search Results

-   Ensure the index exists and contains data.
-   Check if the search fields are correct.
-   Try using a simpler query.
-   Check Elasticsearch logs for more information.