learningcircuit--local-deep-research/docs/architecture/OVERVIEW.md

Architecture Overview

This document provides a comprehensive overview of Local Deep Research's system architecture.

Table of Contents

• System Components
• Entry Points
• Research Execution Flow
  • Research Status Lifecycle
• Module Responsibilities
• Threading Model
• Configuration System
• Key Interfaces

---

System Components

graph TB
    subgraph "Entry Points"
        CLI[ldr CLI]
        WEB[ldr-web Flask App]
        API[REST API /api/v1]
    end

    subgraph "Core Research Engine"
        SS[SearchSystem<br/>search_system.py]
        SSF[StrategyFactory<br/>search_system_factory.py]
        RG[ReportGenerator<br/>report_generator.py]
    end

    subgraph "Search Layer"
        STRAT[32 Search Strategies<br/>advanced_search_system/strategies/]
        ENG[30+ Search Engines<br/>web_search_engines/engines/]
        RL[Rate Limiter<br/>rate_limiting/]
    end

    subgraph "Data Layer"
        DB[(SQLCipher DB<br/>Per-User Encrypted)]
        MODELS[20+ ORM Models<br/>database/models/]
        CACHE[Memory Cache]
    end

    subgraph "LLM Layer"
        PROV[LLM Providers<br/>Ollama, OpenAI, etc.]
        EMB[Embeddings<br/>embeddings/]
        RERANK[Reranker<br/>reranker/]
    end

    CLI --> SS
    WEB --> SS
    API --> SS

    SS --> SSF
    SS --> RG
    SSF --> STRAT
    STRAT --> ENG
    ENG --> RL

    SS --> PROV
    SS --> DB
    MODELS --> DB

    RG --> PROV
    ENG --> CACHE

---

Entry Points

Web Application (ldr-web)

Location: src/local_deep_research/web/app.py

The primary user interface. Launches a Flask server with SocketIO for real-time updates.

graph LR
    A[Browser] -->|HTTP/WS| B[Flask App]
    B --> C[Blueprints]
    C --> D[research_routes]
    C --> E[api_routes]
    C --> F[settings_routes]
    C --> G[auth_routes]
    B -->|Real-time| H[SocketIO]

Key files:

• web/app.py - Main entry, starts server
• web/app_factory.py - Flask app creation with middleware
• web/routes/ - Blueprint route handlers
• web/services/ - Business logic services

REST API (/api/v1)

Location: src/local_deep_research/web/api.py

Programmatic access for integrations.

Endpoint	Method	Purpose
/api/v1/quick_summary	POST	Quick research summary
/api/v1/generate_report	POST	Full research report
/api/v1/analyze_documents	POST	Search local collections
/api/v1/health	GET	Health check

---

Research Execution Flow

sequenceDiagram
    participant User
    participant Web as Flask App
    participant SS as SearchSystem
    participant SF as StrategyFactory
    participant Strat as Strategy
    participant Eng as SearchEngine
    participant LLM as LLM Provider
    participant DB as Database

    User->>Web: Submit Query
    Web->>SS: start_research()
    SS->>SF: create_strategy(name)
    SF-->>SS: Strategy instance

    loop Research Iterations
        SS->>Strat: analyze_topic(query)
        Strat->>LLM: Generate questions
        LLM-->>Strat: Questions

        loop Per Question
            Strat->>Eng: search(question)
            Eng-->>Strat: Results
        end

        Strat->>LLM: Synthesize findings
        LLM-->>Strat: Synthesis
        Strat-->>SS: Findings
    end

    SS->>DB: Save results
    SS-->>Web: Research complete
    Web-->>User: Results via SocketIO

Research Status Lifecycle

stateDiagram-v2
    [*] --> QUEUED : Concurrency limit reached
    [*] --> IN_PROGRESS : Slots available

    QUEUED --> IN_PROGRESS : Worker picks up task
    QUEUED --> SUSPENDED : User terminates

    IN_PROGRESS --> COMPLETED : Research succeeds
    IN_PROGRESS --> FAILED : Unrecoverable error
    IN_PROGRESS --> SUSPENDED : User terminates

    COMPLETED --> [*]
    FAILED --> [*]
    SUSPENDED --> [*]

    note right of FAILED
        Set by processor_v2 and
        research_service on errors
    end note

    note left of SUSPENDED
        Set when user clicks
        terminate/stop
    end note

Unused statuses: PENDING (declared as a model default but never set by any creation path), ERROR (never set; predates FAILED), and CANCELLED (unused by research; used by benchmarks) exist in ResearchStatus for backward compatibility.

---

Module Responsibilities

Core Modules

Module	Location	Responsibility
SearchSystem	search_system.py	Orchestrates research, coordinates strategies and engines
StrategyFactory	search_system_factory.py	Creates strategy instances based on configuration
ReportGenerator	report_generator.py	Generates structured reports from research findings
CitationHandler	citation_handler.py	Processes and validates citations

Search System

Module	Location	Responsibility
BaseSearchEngine	web_search_engines/search_engine_base.py	Abstract base for all search engines
SearchEngineFactory	web_search_engines/search_engine_factory.py	Creates engine instances
RateLimitTracker	web_search_engines/rate_limiting/tracker.py	Adaptive rate limiting
RetrieverRegistry	web_search_engines/retriever_registry.py	LangChain retriever integration

Strategy System

Module	Location	Responsibility
BaseSearchStrategy	advanced_search_system/strategies/base_strategy.py	Abstract base for strategies
FindingsRepository	advanced_search_system/findings/	Accumulates research findings
QuestionGenerator	advanced_search_system/questions/	Generates research questions

Web Application

Module	Location	Responsibility
SocketIOService	web/services/socket_service.py	Real-time communication
ResearchService	web/services/research_service.py	Research execution
QueueManager	web/queue/	Background task queue
SessionManager	web/auth/session_manager.py	User session handling

Data Layer

Module	Location	Responsibility
Models	database/models/	SQLAlchemy ORM models
SessionContext	database/session_context.py	Thread-safe DB sessions
EncryptedDB	database/encrypted_db.py	SQLCipher integration

LLM Integration

Module	Location	Responsibility
LLM Providers	llm/providers/implementations/	Provider-specific LLM wrappers
AutoDiscovery	llm/providers/auto_discovery.py	Dynamic provider detection
LLMRegistry	llm/llm_registry.py	Custom LLM registration

---

Threading Model

graph TB
    subgraph "Main Thread"
        FLASK[Flask Server]
        SOCKETIO[SocketIO Handler]
    end

    subgraph "Research Threads"
        RT1[Research Thread 1]
        RT2[Research Thread 2]
        RTN[Research Thread N]
    end

    subgraph "Queue Processor"
        QP[Queue Processor Thread]
    end

    subgraph "Thread-Local Storage"
        TC[Thread Context<br/>Settings Snapshot]
        DBS[DB Session<br/>Per-User]
    end

    FLASK --> RT1
    FLASK --> RT2
    FLASK --> RTN

    RT1 --> TC
    RT2 --> TC
    RTN --> TC

    QP --> RT1

    RT1 -.-> SOCKETIO
    RT2 -.-> SOCKETIO

Key Threading Concepts:

1. Thread Context (config/thread_settings.py)

   • Each research thread has its own settings snapshot
   • Prevents race conditions on configuration changes

2. Per-User DB Sessions (database/session_context.py)

   • Each user has an isolated SQLCipher database
   • Sessions are thread-local via context manager

3. Queue Processing (web/queue/)

   • Background queue for long-running research
   • Processes items from QueuedResearch table

4. SocketIO Updates

   • Research threads emit progress via SocketIO
   • Uses threading async mode (not asyncio)

---

Configuration System

graph TB
    subgraph "Configuration Sources"
        ENV[Environment Variables]
        DB[(Database Settings<br/>per-user)]
        JSON[Default Settings<br/>default_settings.json]
    end

    subgraph "Settings Management"
        SM[SettingsManager<br/>manager.py]
    end

    subgraph "Runtime Access"
        SNAP[Settings Snapshot<br/>Thread-safe copy]
        TC[Thread Context<br/>thread_settings.py]
    end

    ENV --> SM
    JSON --> SM
    DB --> SM
    SM --> SNAP
    SNAP --> TC

Configuration Flow:

1. Defaults - Default settings loaded from JSON
2. Environment - Environment variables override defaults
3. Database - User settings loaded from encrypted per-user DB
4. Snapshot - Thread-safe copy created for each research
5. Access - Code reads from snapshot via thread context

Key Settings Categories:

Category	Examples
llm.*	Provider, model, temperature, API keys
search.*	Engine selection, max results, rate limits
app.*	Debug mode, logging, UI preferences
notifications.*	Email, webhook configurations

---

Key Interfaces

Search Engine Interface

All search engines implement BaseSearchEngine:

class BaseSearchEngine(ABC):
    # Classification flags
    is_public: bool = True
    is_generic: bool = True
    is_scientific: bool = False
    is_local: bool = False
    is_news: bool = False
    is_code: bool = False
    is_lexical: bool = False
    needs_llm_relevance_filter: bool = False

    @abstractmethod
    def run(self, query: str) -> List[Dict[str, Any]]:
        """Execute search and return results."""
        # Returns: [{"title": ..., "link": ..., "snippet": ...}]

Strategy Interface

All strategies implement BaseSearchStrategy:

class BaseSearchStrategy(ABC):
    def __init__(self, search, model, all_links_of_system,
                 settings_snapshot, **kwargs):
        ...

    @abstractmethod
    def analyze_topic(self, query: str) -> Dict:
        """Execute research strategy."""
        # Returns: {
        #     "findings": [...],
        #     "iterations": int,
        #     "questions": {...},
        #     "formatted_findings": str,
        #     "current_knowledge": {...}
        # }

LLM Provider Interface

All providers extend BaseLLMProvider (most extend OpenAICompatibleProvider):

class OpenAICompatibleProvider(BaseLLMProvider):
    provider_name: str
    api_key_setting: str | None      # Settings key, or None for no-key providers
    api_key_optional: bool = False   # If True, missing key uses placeholder
    url_setting: str
    default_base_url: str
    default_model: str

    @classmethod
    def create_llm(cls, model_name, temperature, **kwargs) -> BaseChatModel:
        """Create LangChain LLM instance."""

---

Directory Structure

src/local_deep_research/
├── search_system.py           # Main orchestrator
├── search_system_factory.py   # Strategy factory
├── report_generator.py        # Report generation
├── citation_handler.py        # Citation processing
│
├── web/                       # Flask application
│   ├── app.py                # Entry point
│   ├── app_factory.py        # App creation
│   ├── routes/               # Blueprint handlers
│   ├── services/             # Business logic
│   ├── queue/                # Task queue
│   └── auth/                 # Authentication
│
├── advanced_search_system/    # Search strategies
│   ├── strategies/           # 32 strategy implementations
│   ├── questions/            # Question generation
│   ├── findings/             # Findings management
│   └── ...
│
├── web_search_engines/        # Search engines
│   ├── engines/              # 30+ engine implementations
│   ├── search_engine_base.py # Abstract base
│   ├── search_engine_factory.py
│   └── rate_limiting/        # Adaptive rate limiting
│
├── database/                  # Data layer
│   ├── models/               # 20+ ORM models
│   ├── session_context.py    # Session management
│   └── encrypted_db.py       # SQLCipher
│
├── llm/                       # LLM integration
│   ├── providers/            # Provider implementations
│   └── llm_registry.py       # Custom LLM registration
│
├── config/                    # Configuration
│   ├── llm_config.py         # LLM setup
│   ├── search_config.py      # Search setup
│   └── thread_settings.py    # Thread context
│
├── settings/                  # Settings management
│   └── manager.py            # SettingsManager
│
└── api/                       # Programmatic API
    ├── client.py             # HTTP client
    └── research_functions.py # Direct functions

---

See Also

• Database Schema - Detailed data model documentation
• Semantic Search - Indexing pipeline, search modes, and three-tier merge algorithm
• Extension Guide - How to add custom components
• Troubleshooting - Common issues and solutions