chore: import upstream snapshot with attribution

2026-07-13 13:08:55 +08:00
commit 7a0da7932b
2985 changed files with 1049377 additions and 0 deletions
@@ -0,0 +1,608 @@
+# Database Schema
+
+This document describes the database models and their relationships in Local Deep Research.
+
+## Table of Contents
+
+- [Overview](#overview)
+- [Entity Relationship Diagram](#entity-relationship-diagram)
+- [Model Groups](#model-groups)
+  - [Research Domain](#research-domain)
+  - [Authentication](#authentication)
+  - [Settings](#settings)
+  - [Library & Documents](#library--documents)
+  - [Queue Management](#queue-management)
+  - [Metrics & Analytics](#metrics--analytics)
+  - [News System](#news-system)
+  - [Benchmarking](#benchmarking)
+  - [Rate Limiting](#rate-limiting)
+  - [File Integrity](#file-integrity)
+
+---
+
+## Overview
+
+Local Deep Research uses **SQLAlchemy ORM** with **SQLCipher** for encryption.
+
+**Key Characteristics:**
+- **Per-user databases**: Each user has their own encrypted SQLite database
+- **AES-256 encryption**: User password derives the encryption key
+- **HMAC verification**: Ensures database integrity
+- **Central auth database**: Only stores usernames (no passwords)
+
+**Location:** `src/local_deep_research/database/models/`
+
+---
+
+## Entity Relationship Diagram
+
+```mermaid
+erDiagram
+    %% Research Domain
+    ResearchTask ||--o{ SearchQuery : contains
+    ResearchTask ||--o{ SearchResult : produces
+    ResearchTask ||--o{ Report : generates
+    SearchQuery ||--o{ SearchResult : returns
+
+    Research ||--o{ ResearchHistory : tracks
+    Research ||--o{ ResearchResource : uses
+
+    Report ||--o{ ReportSection : contains
+
+    %% Library
+    Document ||--o{ DocumentChunk : splits_into
+    Document }o--o{ Collection : belongs_to
+    Collection ||--o{ RAGIndex : indexes
+
+    %% News
+    NewsSubscription ||--o{ NewsCard : produces
+    NewsCard ||--o{ UserRating : receives
+
+    %% Benchmarks
+    BenchmarkRun ||--o{ BenchmarkResult : contains
+    BenchmarkRun ||--o{ BenchmarkProgress : tracks
+
+    %% Queue
+    QueuedResearch ||--o| TaskMetadata : has
+
+    %% Metrics
+    Research ||--o{ TokenUsage : tracks
+    Research ||--o{ SearchCall : logs
+```
+
+---
+
+## Model Groups
+
+### Research Domain
+
+The core models for conducting research.
+
+#### ResearchTask
+
+Top-level research container.
+
+| Column | Type | Description |
+|--------|------|-------------|
+| `id` | Integer | Primary key |
+| `title` | String(500) | Research title |
+| `description` | Text | Detailed description |
+| `status` | String(50) | pending, in_progress, completed, failed |
+| `priority` | Integer | Priority level (higher = more urgent) |
+| `tags` | JSON | List of categorization tags |
+| `research_metadata` | JSON | Flexible metadata storage |
+| `created_at` | DateTime | Creation timestamp |
+| `updated_at` | DateTime | Last update timestamp |
+| `started_at` | DateTime | When research started |
+| `completed_at` | DateTime | When research completed |
+
+**Relationships:** `searches`, `results`, `reports`
+
+#### SearchQuery
+
+Individual search queries within a research task.
+
+| Column | Type | Description |
+|--------|------|-------------|
+| `id` | Integer | Primary key |
+| `research_task_id` | Integer | FK to ResearchTask |
+| `query` | Text | The search query text |
+| `search_engine` | String(50) | Engine used (duckduckgo, arxiv, etc.) |
+| `search_type` | String(50) | Type (web, academic, news) |
+| `parameters` | JSON | Additional search parameters |
+| `status` | String(50) | pending, executing, completed, failed |
+| `error_message` | Text | Error details if failed |
+| `retry_count` | Integer | Number of retry attempts |
+| `executed_at` | DateTime | When query was executed |
+
+**Indexes:** `idx_research_task_status`, `idx_search_engine`
+
+#### SearchResult
+
+Individual results from search queries.
+
+| Column | Type | Description |
+|--------|------|-------------|
+| `id` | Integer | Primary key |
+| `research_task_id` | Integer | FK to ResearchTask |
+| `search_query_id` | Integer | FK to SearchQuery |
+| `title` | String(500) | Result title |
+| `url` | Text | Result URL (indexed) |
+| `snippet` | Text | Brief preview |
+| `content` | Text | Full fetched content |
+| `content_type` | String(50) | html, pdf, text, etc. |
+| `content_hash` | String(64) | For deduplication |
+| `relevance_score` | Float | Calculated relevance |
+| `position` | Integer | Position in results |
+| `domain` | String(255) | Source domain (indexed) |
+| `language` | String(10) | Content language |
+| `published_date` | DateTime | Publication date |
+| `fetch_status` | String(50) | pending, fetched, failed, skipped |
+
+#### Research
+
+Simplified research record (alternative to ResearchTask).
+
+| Column | Type | Description |
+|--------|------|-------------|
+| `id` | Integer | Primary key |
+| `query` | Text | Original query |
+| `mode` | Enum | ResearchMode value |
+| `strategy` | Enum | ResearchStrategy value |
+| `status` | Enum | ResearchStatus value |
+| `result` | Text | Final result/summary |
+| `iterations` | Integer | Iterations completed |
+| `created_at` | DateTime | Creation timestamp |
+
+**Enums:**
+- `ResearchMode`: quick, detailed, report
+- `ResearchStatus`: pending, queued, in_progress, completed, suspended, failed, error, cancelled
+- `ResearchStrategy`: source-based, focused-iteration, etc.
+
+#### ResearchHistory
+
+Tracks research iterations and progress.
+
+| Column | Type | Description |
+|--------|------|-------------|
+| `id` | Integer | Primary key |
+| `research_id` | Integer | FK to Research |
+| `iteration` | Integer | Iteration number |
+| `questions` | JSON | Questions asked |
+| `findings` | JSON | Findings discovered |
+| `created_at` | DateTime | When recorded |
+
+#### Report / ReportSection
+
+Generated research reports.
+
+| Column (Report) | Type | Description |
+|-----------------|------|-------------|
+| `id` | Integer | Primary key |
+| `research_task_id` | Integer | FK to ResearchTask |
+| `title` | String(500) | Report title |
+| `format` | String(50) | markdown, pdf, latex |
+| `content` | Text | Full report content |
+| `created_at` | DateTime | Generation time |
+
+| Column (ReportSection) | Type | Description |
+|------------------------|------|-------------|
+| `id` | Integer | Primary key |
+| `report_id` | Integer | FK to Report |
+| `title` | String(255) | Section title |
+| `content` | Text | Section content |
+| `order` | Integer | Display order |
+
+---
+
+### Authentication
+
+User management with per-user encrypted databases.
+
+#### User
+
+Central user registry (stored in auth database, not user database).
+
+| Column | Type | Description |
+|--------|------|-------------|
+| `id` | Integer | Primary key |
+| `username` | String(80) | Unique username (indexed) |
+| `created_at` | DateTime | Registration date |
+| `last_login` | DateTime | Last login time |
+| `database_version` | Integer | Schema version |
+
+**Note:** Passwords are NEVER stored. They derive encryption keys.
+
+#### APIKey
+
+API keys for programmatic access.
+
+| Column | Type | Description |
+|--------|------|-------------|
+| `id` | Integer | Primary key |
+| `key_hash` | String(64) | Hashed API key |
+| `name` | String(100) | Key description |
+| `created_at` | DateTime | Creation date |
+| `last_used` | DateTime | Last usage |
+| `expires_at` | DateTime | Expiration date |
+| `is_active` | Boolean | Whether key is valid |
+
+---
+
+### Settings
+
+Configuration storage.
+
+#### Setting
+
+Global application settings.
+
+| Column | Type | Description |
+|--------|------|-------------|
+| `id` | Integer | Primary key |
+| `key` | String(255) | Setting key (unique, indexed) |
+| `value` | Text | Setting value |
+| `type` | Enum | SettingType (string, int, bool, json) |
+| `category` | String(100) | Setting category |
+| `description` | Text | Human-readable description |
+| `updated_at` | DateTime | Last update |
+
+#### UserSettings
+
+Per-user setting overrides.
+
+| Column | Type | Description |
+|--------|------|-------------|
+| `id` | Integer | Primary key |
+| `key` | String(255) | Setting key |
+| `value` | Text | User's value |
+| `updated_at` | DateTime | Last update |
+
+---
+
+### Library & Documents
+
+Document management for RAG.
+
+#### Document
+
+Documents in the research library.
+
+| Column | Type | Description |
+|--------|------|-------------|
+| `id` | Integer | Primary key |
+| `title` | String(500) | Document title |
+| `source_type` | FK → SourceType | Source type (see SourceType table) |
+| `source_url` | Text | Original source URL |
+| `file_path` | Text | Local file path |
+| `file_hash` | String(64) | Content hash |
+| `mime_type` | String(100) | MIME type |
+| `file_size` | Integer | Size in bytes |
+| `text_content` | Text | Extracted text |
+| `metadata` | JSON | Additional metadata |
+| `created_at` | DateTime | When added |
+| `indexed_at` | DateTime | When indexed for RAG |
+
+**SourceType** (normalized table): research_download, user_upload, manual_entry, research_report, research_source
+
+#### Collection
+
+Document collections for organization.
+
+| Column | Type | Description |
+|--------|------|-------------|
+| `id` | Integer | Primary key |
+| `name` | String(255) | Collection name |
+| `description` | Text | Description |
+| `is_default` | Boolean | Default collection flag |
+| `created_at` | DateTime | Creation date |
+
+#### DocumentCollection
+
+Junction table for document-collection relationship.
+
+| Column | Type | Description |
+|--------|------|-------------|
+| `document_id` | Integer | FK to Document |
+| `collection_id` | Integer | FK to Collection |
+
+#### DocumentChunk
+
+Text chunks for RAG indexing.
+
+| Column | Type | Description |
+|--------|------|-------------|
+| `id` | Integer | Primary key |
+| `document_id` | Integer | FK to Document |
+| `chunk_index` | Integer | Position in document |
+| `content` | Text | Chunk text |
+| `embedding` | BLOB | Vector embedding |
+| `metadata` | JSON | Chunk metadata |
+
+#### RAGIndex
+
+Vector index metadata.
+
+| Column | Type | Description |
+|--------|------|-------------|
+| `id` | Integer | Primary key |
+| `collection_id` | Integer | FK to Collection |
+| `status` | Enum | RAGIndexStatus |
+| `embedding_model` | String(100) | Model used |
+| `chunk_count` | Integer | Number of chunks |
+| `created_at` | DateTime | Creation time |
+| `updated_at` | DateTime | Last update |
+
+---
+
+### Queue Management
+
+Background task processing.
+
+#### QueuedResearch
+
+Research waiting to be processed.
+
+| Column | Type | Description |
+|--------|------|-------------|
+| `id` | Integer | Primary key |
+| `query` | Text | Research query |
+| `mode` | String(50) | Research mode |
+| `strategy` | String(50) | Strategy name |
+| `status` | Enum | QueueStatus |
+| `priority` | Integer | Queue priority |
+| `created_at` | DateTime | When queued |
+| `started_at` | DateTime | When started |
+| `completed_at` | DateTime | When finished |
+| `error` | Text | Error message if failed |
+
+**Enum QueueStatus:** pending, running, completed, failed, cancelled
+
+#### TaskMetadata
+
+Additional task information.
+
+| Column | Type | Description |
+|--------|------|-------------|
+| `id` | Integer | Primary key |
+| `queued_research_id` | Integer | FK to QueuedResearch |
+| `key` | String(255) | Metadata key |
+| `value` | Text | Metadata value |
+
+---
+
+### Metrics & Analytics
+
+Usage tracking and analytics.
+
+#### TokenUsage
+
+LLM token consumption.
+
+| Column | Type | Description |
+|--------|------|-------------|
+| `id` | Integer | Primary key |
+| `research_id` | Integer | FK to Research |
+| `model` | String(100) | Model name |
+| `provider` | String(50) | Provider name |
+| `input_tokens` | Integer | Input token count |
+| `output_tokens` | Integer | Output token count |
+| `cost` | Float | Estimated cost |
+| `created_at` | DateTime | When recorded |
+
+#### SearchCall
+
+Search API call logging.
+
+| Column | Type | Description |
+|--------|------|-------------|
+| `id` | Integer | Primary key |
+| `research_id` | Integer | FK to Research |
+| `engine` | String(50) | Search engine |
+| `query` | Text | Query text |
+| `result_count` | Integer | Results returned |
+| `duration_ms` | Integer | Request duration |
+| `success` | Boolean | Whether succeeded |
+| `created_at` | DateTime | When called |
+
+#### ModelUsage
+
+Aggregated model usage statistics.
+
+| Column | Type | Description |
+|--------|------|-------------|
+| `id` | Integer | Primary key |
+| `model` | String(100) | Model name |
+| `provider` | String(50) | Provider name |
+| `total_input_tokens` | Integer | Cumulative input |
+| `total_output_tokens` | Integer | Cumulative output |
+| `total_cost` | Float | Cumulative cost |
+| `request_count` | Integer | Number of requests |
+| `date` | Date | Aggregation date |
+
+#### ResearchRating
+
+User ratings for research quality.
+
+| Column | Type | Description |
+|--------|------|-------------|
+| `id` | Integer | Primary key |
+| `research_id` | Integer | FK to Research |
+| `rating` | Integer | 1-5 rating |
+| `feedback` | Text | Optional feedback |
+| `created_at` | DateTime | When rated |
+
+---
+
+### News System
+
+News subscription and recommendation.
+
+#### NewsSubscription
+
+User news subscriptions.
+
+| Column | Type | Description |
+|--------|------|-------------|
+| `id` | Integer | Primary key |
+| `topic` | String(255) | Subscription topic |
+| `type` | Enum | SubscriptionType |
+| `status` | Enum | SubscriptionStatus |
+| `frequency` | String(50) | Update frequency |
+| `last_fetched` | DateTime | Last fetch time |
+| `created_at` | DateTime | Creation date |
+
+#### NewsCard
+
+Individual news items.
+
+| Column | Type | Description |
+|--------|------|-------------|
+| `id` | Integer | Primary key |
+| `subscription_id` | Integer | FK to NewsSubscription |
+| `title` | String(500) | News title |
+| `summary` | Text | News summary |
+| `url` | Text | Source URL |
+| `source` | String(100) | Source name |
+| `published_at` | DateTime | Publication date |
+| `card_type` | Enum | CardType |
+| `created_at` | DateTime | When fetched |
+
+#### UserRating / UserPreference / NewsInterest
+
+User interaction tracking for recommendations.
+
+---
+
+### Benchmarking
+
+Performance benchmarking system.
+
+#### BenchmarkRun
+
+Benchmark execution record.
+
+| Column | Type | Description |
+|--------|------|-------------|
+| `id` | Integer | Primary key |
+| `name` | String(255) | Run name |
+| `dataset_type` | Enum | DatasetType (SimpleQA, BrowseComp) |
+| `strategy` | String(100) | Strategy tested |
+| `status` | Enum | BenchmarkStatus |
+| `config` | JSON | Configuration used |
+| `started_at` | DateTime | Start time |
+| `completed_at` | DateTime | End time |
+
+#### BenchmarkResult
+
+Individual benchmark results.
+
+| Column | Type | Description |
+|--------|------|-------------|
+| `id` | Integer | Primary key |
+| `run_id` | Integer | FK to BenchmarkRun |
+| `question` | Text | Test question |
+| `expected_answer` | Text | Expected answer |
+| `actual_answer` | Text | Model's answer |
+| `is_correct` | Boolean | Whether correct |
+| `score` | Float | Quality score |
+| `latency_ms` | Integer | Response time |
+| `tokens_used` | Integer | Tokens consumed |
+
+#### BenchmarkProgress
+
+Progress tracking during runs.
+
+| Column | Type | Description |
+|--------|------|-------------|
+| `id` | Integer | Primary key |
+| `run_id` | Integer | FK to BenchmarkRun |
+| `completed` | Integer | Questions completed |
+| `total` | Integer | Total questions |
+| `current_accuracy` | Float | Running accuracy |
+| `updated_at` | DateTime | Last update |
+
+---
+
+### Rate Limiting
+
+Adaptive rate limiting data.
+
+#### RateLimitAttempt
+
+Individual rate limit events.
+
+| Column | Type | Description |
+|--------|------|-------------|
+| `id` | Integer | Primary key |
+| `engine` | String(50) | Search engine |
+| `wait_time` | Float | Wait time used |
+| `success` | Boolean | Whether request succeeded |
+| `created_at` | DateTime | When occurred |
+
+#### RateLimitEstimate
+
+Learned rate limit estimates.
+
+| Column | Type | Description |
+|--------|------|-------------|
+| `id` | Integer | Primary key |
+| `engine` | String(50) | Search engine |
+| `estimated_wait` | Float | Optimal wait time |
+| `confidence` | Float | Estimate confidence |
+| `sample_count` | Integer | Data points used |
+| `updated_at` | DateTime | Last update |
+
+---
+
+### File Integrity
+
+File verification for security.
+
+#### FileIntegrityRecord
+
+File hash records.
+
+| Column | Type | Description |
+|--------|------|-------------|
+| `id` | Integer | Primary key |
+| `file_path` | Text | File path |
+| `sha256_hash` | String(64) | SHA256 hash |
+| `blake3_hash` | String(64) | BLAKE3 hash |
+| `file_size` | Integer | Size in bytes |
+| `verified_at` | DateTime | Last verification |
+| `created_at` | DateTime | First recorded |
+
+#### FileVerificationFailure
+
+Failed verification attempts.
+
+| Column | Type | Description |
+|--------|------|-------------|
+| `id` | Integer | Primary key |
+| `file_path` | Text | File path |
+| `expected_hash` | String(64) | Expected hash |
+| `actual_hash` | String(64) | Computed hash |
+| `failure_type` | String(50) | Type of failure |
+| `created_at` | DateTime | When detected |
+
+---
+
+## Database Location
+
+```
+~/.local/share/local-deep-research/
+├── auth.db                    # Central auth database (unencrypted)
+└── users/
+    └── <username>/
+        └── research.db        # User's encrypted database
+```
+
+---
+
+## See Also
+
+- [Architecture Overview](./OVERVIEW.md) - System architecture
+- [Semantic Search](./SEMANTIC_SEARCH.md) - How Document/Collection models enable semantic search
+- [Extension Guide](../developing/EXTENDING.md) - Adding custom components
+- [Troubleshooting](../troubleshooting.md) - Common issues
@@ -0,0 +1,467 @@
+# Architecture Overview
+
+This document provides a comprehensive overview of Local Deep Research's system architecture.
+
+## Table of Contents
+
+- [System Components](#system-components)
+- [Entry Points](#entry-points)
+- [Research Execution Flow](#research-execution-flow)
+  - [Research Status Lifecycle](#research-status-lifecycle)
+- [Module Responsibilities](#module-responsibilities)
+- [Threading Model](#threading-model)
+- [Configuration System](#configuration-system)
+- [Key Interfaces](#key-interfaces)
+
+---
+
+## System Components
+
+```mermaid
+graph TB
+    subgraph "Entry Points"
+        CLI[ldr CLI]
+        WEB[ldr-web Flask App]
+        API[REST API /api/v1]
+    end
+
+    subgraph "Core Research Engine"
+        SS[SearchSystem<br/>search_system.py]
+        SSF[StrategyFactory<br/>search_system_factory.py]
+        RG[ReportGenerator<br/>report_generator.py]
+    end
+
+    subgraph "Search Layer"
+        STRAT[32 Search Strategies<br/>advanced_search_system/strategies/]
+        ENG[30+ Search Engines<br/>web_search_engines/engines/]
+        RL[Rate Limiter<br/>rate_limiting/]
+    end
+
+    subgraph "Data Layer"
+        DB[(SQLCipher DB<br/>Per-User Encrypted)]
+        MODELS[20+ ORM Models<br/>database/models/]
+        CACHE[Memory Cache]
+    end
+
+    subgraph "LLM Layer"
+        PROV[LLM Providers<br/>Ollama, OpenAI, etc.]
+        EMB[Embeddings<br/>embeddings/]
+        RERANK[Reranker<br/>reranker/]
+    end
+
+    CLI --> SS
+    WEB --> SS
+    API --> SS
+
+    SS --> SSF
+    SS --> RG
+    SSF --> STRAT
+    STRAT --> ENG
+    ENG --> RL
+
+    SS --> PROV
+    SS --> DB
+    MODELS --> DB
+
+    RG --> PROV
+    ENG --> CACHE
+```
+
+---
+
+## Entry Points
+
+### Web Application (`ldr-web`)
+
+**Location:** `src/local_deep_research/web/app.py`
+
+The primary user interface. Launches a Flask server with SocketIO for real-time updates.
+
+```mermaid
+graph LR
+    A[Browser] -->|HTTP/WS| B[Flask App]
+    B --> C[Blueprints]
+    C --> D[research_routes]
+    C --> E[api_routes]
+    C --> F[settings_routes]
+    C --> G[auth_routes]
+    B -->|Real-time| H[SocketIO]
+```
+
+**Key files:**
+- `web/app.py` - Main entry, starts server
+- `web/app_factory.py` - Flask app creation with middleware
+- `web/routes/` - Blueprint route handlers
+- `web/services/` - Business logic services
+
+### REST API (`/api/v1`)
+
+**Location:** `src/local_deep_research/web/api.py`
+
+Programmatic access for integrations.
+
+| Endpoint | Method | Purpose |
+|----------|--------|---------|
+| `/api/v1/quick_summary` | POST | Quick research summary |
+| `/api/v1/generate_report` | POST | Full research report |
+| `/api/v1/analyze_documents` | POST | Search local collections |
+| `/api/v1/health` | GET | Health check |
+
+---
+
+## Research Execution Flow
+
+```mermaid
+sequenceDiagram
+    participant User
+    participant Web as Flask App
+    participant SS as SearchSystem
+    participant SF as StrategyFactory
+    participant Strat as Strategy
+    participant Eng as SearchEngine
+    participant LLM as LLM Provider
+    participant DB as Database
+
+    User->>Web: Submit Query
+    Web->>SS: start_research()
+    SS->>SF: create_strategy(name)
+    SF-->>SS: Strategy instance
+
+    loop Research Iterations
+        SS->>Strat: analyze_topic(query)
+        Strat->>LLM: Generate questions
+        LLM-->>Strat: Questions
+
+        loop Per Question
+            Strat->>Eng: search(question)
+            Eng-->>Strat: Results
+        end
+
+        Strat->>LLM: Synthesize findings
+        LLM-->>Strat: Synthesis
+        Strat-->>SS: Findings
+    end
+
+    SS->>DB: Save results
+    SS-->>Web: Research complete
+    Web-->>User: Results via SocketIO
+```
+
+### Research Status Lifecycle
+
+```mermaid
+stateDiagram-v2
+    [*] --> QUEUED : Concurrency limit reached
+    [*] --> IN_PROGRESS : Slots available
+
+    QUEUED --> IN_PROGRESS : Worker picks up task
+    QUEUED --> SUSPENDED : User terminates
+
+    IN_PROGRESS --> COMPLETED : Research succeeds
+    IN_PROGRESS --> FAILED : Unrecoverable error
+    IN_PROGRESS --> SUSPENDED : User terminates
+
+    COMPLETED --> [*]
+    FAILED --> [*]
+    SUSPENDED --> [*]
+
+    note right of FAILED
+        Set by processor_v2 and
+        research_service on errors
+    end note
+
+    note left of SUSPENDED
+        Set when user clicks
+        terminate/stop
+    end note
+```
+
+> **Unused statuses:** `PENDING` (declared as a model default but never set by any creation
+> path), `ERROR` (never set; predates `FAILED`), and `CANCELLED` (unused by research; used
+> by benchmarks) exist in `ResearchStatus` for backward compatibility.
+
+---
+
+## Module Responsibilities
+
+### Core Modules
+
+| Module | Location | Responsibility |
+|--------|----------|----------------|
+| **SearchSystem** | `search_system.py` | Orchestrates research, coordinates strategies and engines |
+| **StrategyFactory** | `search_system_factory.py` | Creates strategy instances based on configuration |
+| **ReportGenerator** | `report_generator.py` | Generates structured reports from research findings |
+| **CitationHandler** | `citation_handler.py` | Processes and validates citations |
+
+### Search System
+
+| Module | Location | Responsibility |
+|--------|----------|----------------|
+| **BaseSearchEngine** | `web_search_engines/search_engine_base.py` | Abstract base for all search engines |
+| **SearchEngineFactory** | `web_search_engines/search_engine_factory.py` | Creates engine instances |
+| **RateLimitTracker** | `web_search_engines/rate_limiting/tracker.py` | Adaptive rate limiting |
+| **RetrieverRegistry** | `web_search_engines/retriever_registry.py` | LangChain retriever integration |
+
+### Strategy System
+
+| Module | Location | Responsibility |
+|--------|----------|----------------|
+| **BaseSearchStrategy** | `advanced_search_system/strategies/base_strategy.py` | Abstract base for strategies |
+| **FindingsRepository** | `advanced_search_system/findings/` | Accumulates research findings |
+| **QuestionGenerator** | `advanced_search_system/questions/` | Generates research questions |
+
+### Web Application
+
+| Module | Location | Responsibility |
+|--------|----------|----------------|
+| **SocketIOService** | `web/services/socket_service.py` | Real-time communication |
+| **ResearchService** | `web/services/research_service.py` | Research execution |
+| **QueueManager** | `web/queue/` | Background task queue |
+| **SessionManager** | `web/auth/session_manager.py` | User session handling |
+
+### Data Layer
+
+| Module | Location | Responsibility |
+|--------|----------|----------------|
+| **Models** | `database/models/` | SQLAlchemy ORM models |
+| **SessionContext** | `database/session_context.py` | Thread-safe DB sessions |
+| **EncryptedDB** | `database/encrypted_db.py` | SQLCipher integration |
+
+### LLM Integration
+
+| Module | Location | Responsibility |
+|--------|----------|----------------|
+| **LLM Providers** | `llm/providers/implementations/` | Provider-specific LLM wrappers |
+| **AutoDiscovery** | `llm/providers/auto_discovery.py` | Dynamic provider detection |
+| **LLMRegistry** | `llm/llm_registry.py` | Custom LLM registration |
+
+---
+
+## Threading Model
+
+```mermaid
+graph TB
+    subgraph "Main Thread"
+        FLASK[Flask Server]
+        SOCKETIO[SocketIO Handler]
+    end
+
+    subgraph "Research Threads"
+        RT1[Research Thread 1]
+        RT2[Research Thread 2]
+        RTN[Research Thread N]
+    end
+
+    subgraph "Queue Processor"
+        QP[Queue Processor Thread]
+    end
+
+    subgraph "Thread-Local Storage"
+        TC[Thread Context<br/>Settings Snapshot]
+        DBS[DB Session<br/>Per-User]
+    end
+
+    FLASK --> RT1
+    FLASK --> RT2
+    FLASK --> RTN
+
+    RT1 --> TC
+    RT2 --> TC
+    RTN --> TC
+
+    QP --> RT1
+
+    RT1 -.-> SOCKETIO
+    RT2 -.-> SOCKETIO
+```
+
+**Key Threading Concepts:**
+
+1. **Thread Context** (`config/thread_settings.py`)
+   - Each research thread has its own settings snapshot
+   - Prevents race conditions on configuration changes
+
+2. **Per-User DB Sessions** (`database/session_context.py`)
+   - Each user has an isolated SQLCipher database
+   - Sessions are thread-local via context manager
+
+3. **Queue Processing** (`web/queue/`)
+   - Background queue for long-running research
+   - Processes items from `QueuedResearch` table
+
+4. **SocketIO Updates**
+   - Research threads emit progress via SocketIO
+   - Uses threading async mode (not asyncio)
+
+---
+
+## Configuration System
+
+```mermaid
+graph TB
+    subgraph "Configuration Sources"
+        ENV[Environment Variables]
+        DB[(Database Settings<br/>per-user)]
+        JSON[Default Settings<br/>default_settings.json]
+    end
+
+    subgraph "Settings Management"
+        SM[SettingsManager<br/>manager.py]
+    end
+
+    subgraph "Runtime Access"
+        SNAP[Settings Snapshot<br/>Thread-safe copy]
+        TC[Thread Context<br/>thread_settings.py]
+    end
+
+    ENV --> SM
+    JSON --> SM
+    DB --> SM
+    SM --> SNAP
+    SNAP --> TC
+```
+
+**Configuration Flow:**
+
+1. **Defaults** - Default settings loaded from JSON
+2. **Environment** - Environment variables override defaults
+3. **Database** - User settings loaded from encrypted per-user DB
+4. **Snapshot** - Thread-safe copy created for each research
+5. **Access** - Code reads from snapshot via thread context
+
+**Key Settings Categories:**
+
+| Category | Examples |
+|----------|----------|
+| `llm.*` | Provider, model, temperature, API keys |
+| `search.*` | Engine selection, max results, rate limits |
+| `app.*` | Debug mode, logging, UI preferences |
+| `notifications.*` | Email, webhook configurations |
+
+---
+
+## Key Interfaces
+
+### Search Engine Interface
+
+All search engines implement `BaseSearchEngine`:
+
+```python
+class BaseSearchEngine(ABC):
+    # Classification flags
+    is_public: bool = True
+    is_generic: bool = True
+    is_scientific: bool = False
+    is_local: bool = False
+    is_news: bool = False
+    is_code: bool = False
+    is_lexical: bool = False
+    needs_llm_relevance_filter: bool = False
+
+    @abstractmethod
+    def run(self, query: str) -> List[Dict[str, Any]]:
+        """Execute search and return results."""
+        # Returns: [{"title": ..., "link": ..., "snippet": ...}]
+```
+
+### Strategy Interface
+
+All strategies implement `BaseSearchStrategy`:
+
+```python
+class BaseSearchStrategy(ABC):
+    def __init__(self, search, model, all_links_of_system,
+                 settings_snapshot, **kwargs):
+        ...
+
+    @abstractmethod
+    def analyze_topic(self, query: str) -> Dict:
+        """Execute research strategy."""
+        # Returns: {
+        #     "findings": [...],
+        #     "iterations": int,
+        #     "questions": {...},
+        #     "formatted_findings": str,
+        #     "current_knowledge": {...}
+        # }
+```
+
+### LLM Provider Interface
+
+All providers extend `BaseLLMProvider` (most extend `OpenAICompatibleProvider`):
+
+```python
+class OpenAICompatibleProvider(BaseLLMProvider):
+    provider_name: str
+    api_key_setting: str | None      # Settings key, or None for no-key providers
+    api_key_optional: bool = False   # If True, missing key uses placeholder
+    url_setting: str
+    default_base_url: str
+    default_model: str
+
+    @classmethod
+    def create_llm(cls, model_name, temperature, **kwargs) -> BaseChatModel:
+        """Create LangChain LLM instance."""
+```
+
+---
+
+## Directory Structure
+
+```
+src/local_deep_research/
+├── search_system.py           # Main orchestrator
+├── search_system_factory.py   # Strategy factory
+├── report_generator.py        # Report generation
+├── citation_handler.py        # Citation processing
+│
+├── web/                       # Flask application
+│   ├── app.py                # Entry point
+│   ├── app_factory.py        # App creation
+│   ├── routes/               # Blueprint handlers
+│   ├── services/             # Business logic
+│   ├── queue/                # Task queue
+│   └── auth/                 # Authentication
+│
+├── advanced_search_system/    # Search strategies
+│   ├── strategies/           # 32 strategy implementations
+│   ├── questions/            # Question generation
+│   ├── findings/             # Findings management
+│   └── ...
+│
+├── web_search_engines/        # Search engines
+│   ├── engines/              # 30+ engine implementations
+│   ├── search_engine_base.py # Abstract base
+│   ├── search_engine_factory.py
+│   └── rate_limiting/        # Adaptive rate limiting
+│
+├── database/                  # Data layer
+│   ├── models/               # 20+ ORM models
+│   ├── session_context.py    # Session management
+│   └── encrypted_db.py       # SQLCipher
+│
+├── llm/                       # LLM integration
+│   ├── providers/            # Provider implementations
+│   └── llm_registry.py       # Custom LLM registration
+│
+├── config/                    # Configuration
+│   ├── llm_config.py         # LLM setup
+│   ├── search_config.py      # Search setup
+│   └── thread_settings.py    # Thread context
+│
+├── settings/                  # Settings management
+│   └── manager.py            # SettingsManager
+│
+└── api/                       # Programmatic API
+    ├── client.py             # HTTP client
+    └── research_functions.py # Direct functions
+```
+
+---
+
+## See Also
+
+- [Database Schema](./DATABASE_SCHEMA.md) - Detailed data model documentation
+- [Semantic Search](./SEMANTIC_SEARCH.md) - Indexing pipeline, search modes, and three-tier merge algorithm
+- [Extension Guide](../developing/EXTENDING.md) - How to add custom components
+- [Troubleshooting](../troubleshooting.md) - Common issues and solutions
@@ -0,0 +1,167 @@
+# Semantic Search
+
+Semantic search lets users find research by meaning, not just title keywords. It indexes completed research reports and their sources into FAISS vector stores, then offers three search modes -- Hybrid (default), Text-Only, and AI-Only -- with a three-tier ranked merge algorithm for hybrid results.
+
+## Table of Contents
+
+- [Indexing Pipeline](#indexing-pipeline)
+- [Search Pipeline](#search-pipeline)
+- [Three-Tier Merge Algorithm](#three-tier-merge-algorithm)
+- [File Structure](#file-structure)
+- [API Routes](#api-routes)
+- [Reusing on Other Pages](#reusing-on-other-pages)
+- [See Also](#see-also)
+
+---
+
+## Indexing Pipeline
+
+```mermaid
+flowchart LR
+    RH[ResearchHistory<br/>report + sources] --> RHI[ResearchHistoryIndexer<br/>converts to Documents]
+    RHI --> DOC[Document rows<br/>+ DocumentCollection links]
+    DOC --> FACTORY[RAGServiceFactory<br/>resolves settings]
+    FACTORY --> RAG[LibraryRAGService<br/>chunk + embed]
+    RAG --> FAISS[(FAISS Index<br/>on disk)]
+```
+
+**Steps:**
+
+1. `ResearchHistoryIndexer` reads completed `ResearchHistory` rows and their `ResearchResource` sources.
+2. Each report becomes a `Document` (source\_type `research_report`). Each source with sufficient content becomes a `Document` (source\_type `research_source`).
+3. Documents are linked to the History `Collection` via `DocumentCollection`.
+4. `RAGServiceFactory` creates a `LibraryRAGService` with the collection's embedding settings.
+5. `LibraryRAGService` chunks each document, generates embeddings, and writes the FAISS index to disk.
+
+Bulk indexing uses **SSE streaming** -- the `/index` endpoint yields progress events (`start`, `progress`, `complete`, `error`) so the frontend can show a real-time progress bar.
+
+---
+
+## Search Pipeline
+
+```mermaid
+flowchart LR
+    Q[User query] --> MODE{Search Mode}
+    MODE -->|Text-Only| TF[Title filter<br/>instant, client-side]
+    MODE -->|AI-Only| API
+    MODE -->|Hybrid| TF & API
+
+    API[POST /library/api/collections/:id/search] --> CSE[CollectionSearchEngine]
+    CSE --> FAISS[(FAISS Index)]
+    FAISS --> ENRICH[Enrich metadata<br/>report/source type]
+    ENRICH --> JSON[JSON response]
+
+    TF --> RENDER
+    JSON --> MERGE[buildTieredResults<br/>three-tier merge]
+    MERGE --> RENDER[Render cards]
+```
+
+**Text filter** runs instantly on the client against cached history items. **Semantic search** is an async `POST` to the collection search endpoint, which delegates to `CollectionSearchEngine` for FAISS similarity search. In **Hybrid mode**, text results render immediately while the semantic call runs in the background; once it resolves, `buildTieredResults` merges both result sets into three tiers and re-renders.
+
+---
+
+## Three-Tier Merge Algorithm
+
+Implemented in `semantic_search.js :: buildTieredResults()`.
+
+| Tier | Contents | Sort Order |
+|------|----------|------------|
+| **Tier 1** | Matched both text filter AND semantic search | Similarity score DESC |
+| **Tier 2** | Text-only matches (no semantic hit) | Original order (recency) |
+| **Tier 3** | Semantic-only matches (below a visual divider) | Similarity score DESC |
+
+The merge groups semantic results by `research_id` (keeping the best similarity per research), then classifies each text result as Tier 1 or Tier 2 based on whether a corresponding semantic match exists. Remaining semantic-only results become Tier 3.
+
+---
+
+## File Structure
+
+### Backend
+
+```
+research_library/
+├── search/                              # Semantic search subpackage
+│   ├── __init__.py                      # Exports search_bp, ResearchHistoryIndexer
+│   ├── routes/
+│   │   └── search_routes.py             # 4 endpoints + _enrich helper
+│   └── services/
+│       └── research_history_indexer.py   # Converts ResearchHistory -> Documents -> RAG
+├── services/
+│   ├── rag_service_factory.py           # Creates LibraryRAGService with collection settings
+│   └── library_rag_service.py           # FAISS indexing, chunking, embedding management
+```
+
+### Frontend
+
+```
+js/components/
+├── semantic_search.js     # Shared module (window.SemanticSearch):
+│                          #   renderSnippet, buildTieredResults,
+│                          #   createSemanticResultCard, isSafeExternalUrl
+├── history.js             # Page controller: mode switching, hybrid merge, rendering
+└── history_search.js      # Indexing UI, semantic API calls, collection ID caching
+
+js/config/
+└── constants.js           # LDR_CONSTANTS.SEARCH_MODE: HYBRID | TEXT | SEMANTIC
+
+css/components/
+└── semantic-search.css    # All semantic search styles (badges, cards, dividers)
+```
+
+---
+
+## API Routes
+
+All routes require `@login_required`. Blueprint prefix: `/library`.
+
+| Method | Path | Purpose |
+|--------|------|---------|
+| `GET` | `/api/research-history/collection` | Get collection ID and indexing status (auto-converts unconverted entries) |
+| `POST` | `/api/research-history/convert-all` | Convert all completed research to Documents |
+| `POST` | `/api/research/<id>/add-to-collection` | Add research to a custom collection |
+| `POST` | `/api/collections/<id>/search` | Semantic search any collection (generic) |
+
+The search endpoint is **collection-agnostic** -- it works for any collection type. For `research_history` collections, results are enriched with report/source type metadata.
+
+---
+
+## Reusing on Other Pages
+
+The `semantic_search.js` shared module is designed for reuse on library or collection pages.
+
+1. **Load the shared assets** in your template (order matters):
+   ```html
+   <link rel="stylesheet" href="/static/css/components/semantic-search.css">
+   <script defer src="/static/js/components/semantic_search.js"></script>
+   ```
+
+2. **Call the search endpoint** with `POST /library/api/collections/<collection_id>/search`:
+   ```javascript
+   const response = await fetch(`/library/api/collections/${collectionId}/search`, {
+       method: 'POST',
+       headers: { 'Content-Type': 'application/json' },
+       body: JSON.stringify({ query: 'your search', limit: 10 }),
+   });
+   const { results } = await response.json();
+   ```
+
+3. **Render results** using the shared utilities:
+   ```javascript
+   for (const result of results) {
+       container.appendChild(SemanticSearch.createSemanticResultCard(result));
+   }
+
+   // Or for hybrid mode with text + semantic merge:
+   const tiered = SemanticSearch.buildTieredResults(textResults, semanticResults);
+   // tiered.tier1, tiered.tier2, tiered.tier3
+   ```
+
+No backend changes needed -- the search route and `CollectionSearchEngine` already support any collection type.
+
+---
+
+## See Also
+
+- [Architecture Overview](./OVERVIEW.md) -- System architecture
+- [Database Schema](./DATABASE_SCHEMA.md) -- Document, Collection, and DocumentCollection models
+- [Library & RAG Guide](../library-and-rag.md) -- User-facing guide to library and search features