learningcircuit--local-deep-research/docs/library-and-rag.md

# Research Library & RAG Guide

This guide covers the Research Library for document management and the RAG (Retrieval-Augmented Generation) system for semantic search.

## Table of Contents

- [Overview](#overview)
- [Managing Documents](#managing-documents)
- [Collections](#collections)
- [RAG Indexing](#rag-indexing)
- [Semantic Search](#semantic-search)
- [Embedding Models](#embedding-models)
- [Configuration](#configuration)

---

## Overview

The Research Library allows you to:
- **Upload documents** (PDFs, text files, markdown)
- **Organize into collections** for different projects or topics
- **Index for semantic search** using RAG (vector embeddings)
- **Search your documents** using natural language queries

Access the library at: `http://localhost:5000/library`

---

## Managing Documents

### Supported File Types

| Format | Extension | Notes |
|--------|-----------|-------|
| PDF | `.pdf` | Text extracted automatically |
| Plain Text | `.txt` | Direct text storage |
| Markdown | `.md`, `.markdown` | Rendered as text |
| HTML | `.html`, `.htm` | Tags stripped, text extracted |
| Word | `.docx` (and `.doc`*) | Text extracted via `unstructured` |
| OpenDocument Text | `.odt` | Text extracted via `unstructured` |
| PowerPoint | `.pptx` (and `.ppt`*) | Slide text extracted |
| Excel | `.xlsx`, `.xls` | Cell text extracted |
| Rich Text | `.rtf` | Text extracted |
| EPUB | `.epub` | Text extracted |
| Email | `.eml` | Body text extracted |
| Data | `.csv`, `.tsv`, `.json`, `.yaml`, `.yml`, `.xml`, `.toml` | Parsed to text |
| Notebooks | `.ipynb` | Cell sources and outputs |
| Web archives | `.mhtml`, `.mht` | Saved web pages |

The upload dialog's file picker is populated from the live list of formats the
server can actually parse (`GET /library/api/config/supported-formats`), so it
only offers formats whose parser dependencies are installed.

\* The legacy binary formats `.doc` and `.ppt` are offered **only** when
LibreOffice (`soffice`) is installed, because `unstructured` converts them to
the modern format with it. Image formats (`.png`, `.jpg`, …) are offered
**only** when the optional OCR extras (`pytesseract` plus the `tesseract`
system binary) are installed. The default Docker image ships neither, so those
formats are not offered there.

### Uploading Documents

1. Navigate to **Library** in the sidebar
2. Click **Upload** or drag files into the upload area
3. Select a collection (or use the default "Library")
4. Documents are processed and text is extracted

### Storage Modes

| Mode | Description | Use Case |
|------|-------------|----------|
| **Database** | PDFs stored encrypted in SQLCipher | Default, most secure |
| **Text-only** | Only extracted text stored | Save space |

### Document Actions

- **View** - Open document details and extracted text
- **Download PDF** - Get original file (if stored)
- **Download Text** - Export extracted text
- **Delete** - Remove from library

---

## Collections

Collections organize your documents into groups.

### Creating a Collection

1. Go to **Library** → **Collections**
2. Click **Create Collection**
3. Enter a name and optional description
4. Click **Create**

### Managing Collections

- **Add documents** - Upload directly to collection or move existing docs
- **Remove documents** - Documents can exist in multiple collections
- **Delete collection** - Choose to keep or delete orphaned documents
- **Index collection** - Build RAG index for semantic search

### Default Collection

The "Library" collection is created automatically and serves as the default destination for uploads.

---

## RAG Indexing

RAG (Retrieval-Augmented Generation) enables semantic search over your documents.

### How It Works

```
Document → Split into Chunks → Generate Embeddings → Store in Vector Index
```

1. **Chunking** - Documents split into overlapping segments
2. **Embedding** - Each chunk converted to a vector using AI model
3. **Indexing** - Vectors stored in FAISS for fast similarity search

### Indexing a Collection

1. Go to **Library** → **Collections**
2. Select a collection
3. Click **Index for Search** (or **Rebuild Index**)
4. Wait for indexing to complete (progress shown)

### Index Status

| Status | Meaning |
|--------|---------|
| **Not Indexed** | Documents not searchable |
| **Indexing** | Currently processing |
| **Indexed** | Ready for semantic search |
| **Needs Reindex** | New documents added since last index |

---

## Semantic Search

Once indexed, search your documents using natural language.

### Using Collection Search

1. Select a collection with indexed documents
2. Enter a natural language query
3. Results ranked by semantic similarity

### Using in Research

When conducting research, you can:
1. Set search tool to your collection name
2. LDR will search your documents instead of the web
3. Combine with web search via the default langgraph-agent strategy, which can query your collections and web engines in the same run

Example with Python API:
```python
from local_deep_research.api import quick_summary

result = quick_summary(
    query="What does the documentation say about authentication?",
    search_tool="my_collection",  # Use your collection name
    programmatic_mode=True
)
```

---

## Embedding Models

Choose the embedding model based on your needs.

### Available Providers

#### Sentence Transformers (Local - Default)

Runs locally, no API key required.

| Model | Dimensions | Best For |
|-------|------------|----------|
| `all-MiniLM-L6-v2` | 384 | General use (fast) |
| `all-mpnet-base-v2` | 768 | Higher quality |
| `multi-qa-MiniLM-L6-cos-v1` | 384 | Q&A tasks |
| `paraphrase-multilingual-MiniLM-L12-v2` | 384 | Multi-language |

#### Ollama (Local)

Uses your local Ollama installation.

- Default model: `nomic-embed-text`
- Requires Ollama running locally
- Configure URL in Settings → LLM → Ollama

#### OpenAI (Cloud)

Uses OpenAI's embedding API.

- Default model: `text-embedding-3-small`
- Requires OpenAI API key
- Higher quality, requires internet

### Changing Embedding Model

1. Go to **Library** → **Embedding Settings**
2. Select provider and model
3. Click **Save**

> **Note:** Changing models requires reindexing existing collections.

---

## Configuration

### Chunking Settings

| Setting | Default | Description |
|---------|---------|-------------|
| Chunk Size | 1000 | Characters per chunk |
| Chunk Overlap | 200 | Overlap between chunks |
| Splitter Type | recursive | How text is split |

**Splitter Types:**
- `recursive` - Split by paragraphs, then sentences (recommended)
- `token` - Split by token count
- `sentence` - Split by sentences
- `semantic` - Split by semantic similarity

### Index Settings

| Setting | Default | Description |
|---------|---------|-------------|
| Distance Metric | cosine | Similarity calculation |
| Index Type | flat | Exact search (most accurate) |

**Distance Metrics:**
- `cosine` - Angle-based similarity (recommended)
- `l2` - Euclidean distance
- `dot_product` - Dot product similarity

### File Locations

| Data | Location |
|------|----------|
| Document database | `~/.local-deep-research/` |
| FAISS indices | `~/.cache/local_deep_research/rag_indices/` |

---

## API Reference

### Collection Endpoints

| Endpoint | Method | Description |
|----------|--------|-------------|
| `/library/api/collections` | GET | List all collections |
| `/library/api/collections` | POST | Create collection |
| `/library/api/collections/<id>` | PUT | Update collection |
| `/library/api/collections/<id>` | DELETE | Delete collection |

### Document Endpoints

| Endpoint | Method | Description |
|----------|--------|-------------|
| `/library/api/documents` | GET | List documents |
| `/library/api/document/<id>` | GET | Get document details |
| `/library/api/document/<id>` | DELETE | Delete document |
| `/library/api/document/<id>/text` | GET | Get extracted text |
| `/library/api/document/<id>/pdf` | GET | Download PDF |

### RAG Endpoints

| Endpoint | Method | Description |
|----------|--------|-------------|
| `/library/api/rag/settings` | GET | Get RAG configuration |
| `/library/api/rag/configure` | POST | Update RAG settings |
| `/library/api/rag/info` | GET | Get index statistics |
| `/library/api/collections/<id>/index` | GET | Start indexing (SSE) |

---

## Troubleshooting

### Documents Not Appearing

- Check file format is supported
- Verify upload completed successfully
- Refresh the library page

### Search Not Working

- Ensure collection is indexed (check status)
- Try rebuilding the index
- Check embedding model is configured

### Slow Indexing

- Large documents take longer
- Consider using smaller chunk sizes
- Local embedding models are slower than cloud

### Memory Issues

- Reduce chunk size
- Index fewer documents at once
- Use a lighter embedding model

---

## See Also

- [Architecture Overview](architecture/OVERVIEW.md) - System architecture
- [Extension Guide](developing/EXTENDING.md) - Adding custom retrievers
- [Full Configuration Reference](CONFIGURATION.md) - All settings and environment variables
- [API Quickstart](api-quickstart.md) - Using the API