learningcircuit--local-deep-research/docs/library-and-rag.md

Research Library & RAG Guide

This guide covers the Research Library for document management and the RAG (Retrieval-Augmented Generation) system for semantic search.

Table of Contents

• Overview
• Managing Documents
• Collections
• RAG Indexing
• Semantic Search
• Embedding Models
• Configuration

---

Overview

The Research Library allows you to:

• Upload documents (PDFs, text files, markdown)
• Organize into collections for different projects or topics
• Index for semantic search using RAG (vector embeddings)
• Search your documents using natural language queries

Access the library at: http://localhost:5000/library

---

Managing Documents

Supported File Types

Format	Extension	Notes
PDF	.pdf	Text extracted automatically
Plain Text	.txt	Direct text storage
Markdown	.md, .markdown	Rendered as text
HTML	.html, .htm	Tags stripped, text extracted
Word	.docx (and .doc*)	Text extracted via unstructured
OpenDocument Text	.odt	Text extracted via unstructured
PowerPoint	.pptx (and .ppt*)	Slide text extracted
Excel	.xlsx, .xls	Cell text extracted
Rich Text	.rtf	Text extracted
EPUB	.epub	Text extracted
Email	.eml	Body text extracted
Data	.csv, .tsv, .json, .yaml, .yml, .xml, .toml	Parsed to text
Notebooks	.ipynb	Cell sources and outputs
Web archives	.mhtml, .mht	Saved web pages

The upload dialog's file picker is populated from the live list of formats the server can actually parse (GET /library/api/config/supported-formats), so it only offers formats whose parser dependencies are installed.

* The legacy binary formats .doc and .ppt are offered only when LibreOffice (soffice) is installed, because unstructured converts them to the modern format with it. Image formats (.png, .jpg, …) are offered only when the optional OCR extras (pytesseract plus the tesseract system binary) are installed. The default Docker image ships neither, so those formats are not offered there.

Uploading Documents

1. Navigate to Library in the sidebar
2. Click Upload or drag files into the upload area
3. Select a collection (or use the default "Library")
4. Documents are processed and text is extracted

Storage Modes

Mode	Description	Use Case
Database	PDFs stored encrypted in SQLCipher	Default, most secure
Text-only	Only extracted text stored	Save space

Document Actions

• View - Open document details and extracted text
• Download PDF - Get original file (if stored)
• Download Text - Export extracted text
• Delete - Remove from library

---

Collections

Collections organize your documents into groups.

Creating a Collection

1. Go to Library → Collections
2. Click Create Collection
3. Enter a name and optional description
4. Click Create

Managing Collections

• Add documents - Upload directly to collection or move existing docs
• Remove documents - Documents can exist in multiple collections
• Delete collection - Choose to keep or delete orphaned documents
• Index collection - Build RAG index for semantic search

Default Collection

The "Library" collection is created automatically and serves as the default destination for uploads.

---

RAG Indexing

RAG (Retrieval-Augmented Generation) enables semantic search over your documents.

How It Works

Document → Split into Chunks → Generate Embeddings → Store in Vector Index

1. Chunking - Documents split into overlapping segments
2. Embedding - Each chunk converted to a vector using AI model
3. Indexing - Vectors stored in FAISS for fast similarity search

Indexing a Collection

1. Go to Library → Collections
2. Select a collection
3. Click Index for Search (or Rebuild Index)
4. Wait for indexing to complete (progress shown)

Index Status

Status	Meaning
Not Indexed	Documents not searchable
Indexing	Currently processing
Indexed	Ready for semantic search
Needs Reindex	New documents added since last index

---

Semantic Search

Once indexed, search your documents using natural language.

Using Collection Search

1. Select a collection with indexed documents
2. Enter a natural language query
3. Results ranked by semantic similarity

Using in Research

When conducting research, you can:

1. Set search tool to your collection name
2. LDR will search your documents instead of the web
3. Combine with web search via the default langgraph-agent strategy, which can query your collections and web engines in the same run

Example with Python API:

from local_deep_research.api import quick_summary

result = quick_summary(
    query="What does the documentation say about authentication?",
    search_tool="my_collection",  # Use your collection name
    programmatic_mode=True
)

---

Embedding Models

Choose the embedding model based on your needs.

Available Providers

Sentence Transformers (Local - Default)

Runs locally, no API key required.

Model	Dimensions	Best For
all-MiniLM-L6-v2	384	General use (fast)
all-mpnet-base-v2	768	Higher quality
multi-qa-MiniLM-L6-cos-v1	384	Q&A tasks
paraphrase-multilingual-MiniLM-L12-v2	384	Multi-language

Ollama (Local)

Uses your local Ollama installation.

• Default model: nomic-embed-text
• Requires Ollama running locally
• Configure URL in Settings → LLM → Ollama

OpenAI (Cloud)

Uses OpenAI's embedding API.

• Default model: text-embedding-3-small
• Requires OpenAI API key
• Higher quality, requires internet

Changing Embedding Model

1. Go to Library → Embedding Settings
2. Select provider and model
3. Click Save

Note: Changing models requires reindexing existing collections.

---

Configuration

Chunking Settings

Setting	Default	Description
Chunk Size	1000	Characters per chunk
Chunk Overlap	200	Overlap between chunks
Splitter Type	recursive	How text is split

Splitter Types:

• recursive - Split by paragraphs, then sentences (recommended)
• token - Split by token count
• sentence - Split by sentences
• semantic - Split by semantic similarity

Index Settings

Setting	Default	Description
Distance Metric	cosine	Similarity calculation
Index Type	flat	Exact search (most accurate)

Distance Metrics:

• cosine - Angle-based similarity (recommended)
• l2 - Euclidean distance
• dot_product - Dot product similarity

File Locations

Data	Location
Document database	~/.local-deep-research/
FAISS indices	~/.cache/local_deep_research/rag_indices/

---

API Reference

Collection Endpoints

Endpoint	Method	Description
/library/api/collections	GET	List all collections
/library/api/collections	POST	Create collection
/library/api/collections/<id>	PUT	Update collection
/library/api/collections/<id>	DELETE	Delete collection

Document Endpoints

Endpoint	Method	Description
/library/api/documents	GET	List documents
/library/api/document/<id>	GET	Get document details
/library/api/document/<id>	DELETE	Delete document
/library/api/document/<id>/text	GET	Get extracted text
/library/api/document/<id>/pdf	GET	Download PDF

RAG Endpoints

Endpoint	Method	Description
/library/api/rag/settings	GET	Get RAG configuration
/library/api/rag/configure	POST	Update RAG settings
/library/api/rag/info	GET	Get index statistics
/library/api/collections/<id>/index	GET	Start indexing (SSE)

---

Troubleshooting

Documents Not Appearing

• Check file format is supported
• Verify upload completed successfully
• Refresh the library page

Search Not Working

• Ensure collection is indexed (check status)
• Try rebuilding the index
• Check embedding model is configured

Slow Indexing

• Large documents take longer
• Consider using smaller chunk sizes
• Local embedding models are slower than cloud

Memory Issues

• Reduce chunk size
• Index fewer documents at once
• Use a lighter embedding model

---

See Also

• Architecture Overview - System architecture
• Extension Guide - Adding custom retrievers
• Full Configuration Reference - All settings and environment variables
• API Quickstart - Using the API