chore: import upstream snapshot with attribution

docs/AI_Code_Reviewer_Setup.md

@@ -0,0 +1,141 @@
+# AI Code Reviewer Setup Guide
+
+This guide explains how to set up the automated AI PR review system using OpenRouter to analyze pull requests with your choice of AI model.
+
+## Overview
+
+The AI Code Reviewer provides automated, comprehensive code reviews covering:
+- **Security** 🔒 - Hardcoded secrets, SQL injection, XSS, authentication issues, input validation
+- **Performance** ⚡ - Inefficient algorithms, N+1 queries, memory issues, blocking operations
+- **Code Quality** 🎨 - Readability, maintainability, error handling, naming conventions
+- **Best Practices** 📋 - Coding standards, proper patterns, type safety, dead code
+
+The review is posted as a single comprehensive comment on your pull request.
+
+## Setup Instructions
+
+### 1. Get OpenRouter API Key
+
+1. Go to [OpenRouter.ai](https://openrouter.ai/)
+2. Sign up or log in
+3. Navigate to API Keys section
+4. Create a new API key
+5. Copy the key (it starts with `sk-or-v1-...`)
+
+### 2. Add API Key to GitHub Secrets
+
+1. Go to your GitHub repository
+2. Navigate to **Settings** → **Secrets and variables** → **Actions**
+3. Click **New repository secret**
+4. Name it: `OPENROUTER_API_KEY`
+5. Paste your OpenRouter API key
+6. Click **Add secret**
+
+### 3. Configure Workflow (Optional)
+
+The workflow is pre-configured with sensible defaults. You can override them **without editing the workflow** by adding GitHub **repository variables** (not secrets):
+
+1. Go to your repository → **Settings** → **Secrets and variables** → **Actions**
+2. Open the **Variables** tab → **New repository variable**
+3. Add any of:
+
+- **AI_REVIEW_MODELS**: Comma-separated list of models. Each model runs the reviewer independently, so you get one review per model in a single combined comment (see [Multiple reviewers](#multiple-reviewers)). When unset, falls back to `AI_MODEL`, then to the built-in default.
+- **AI_MODEL**: A single model (see [OpenRouter models](https://openrouter.ai/models)). Used by the reviewer when `AI_REVIEW_MODELS` is unset, and also by the release-notes workflow. Default: `moonshotai/kimi-k2-thinking`.
+- **AI_TEMPERATURE**: Adjust randomness (default: `0.1` for consistent reviews)
+- **AI_MAX_TOKENS**: Maximum response length (default: `64000`)
+- **MAX_DIFF_SIZE**: Maximum diff size in bytes (default: `800000` / 800KB)
+
+#### Multiple reviewers
+
+Set **AI_REVIEW_MODELS** to a comma-separated list to have several models review the same diff. For example:
+
+```
+minimax/minimax-m3, z-ai/glm-5.2
+```
+
+Notes:
+- Separate models with **commas** (spaces around them are fine and trimmed). A space-separated value is treated as a *single* model name and will fail.
+- Each model is shown anonymously as **Reviewer 1**, **Reviewer 2**, … (the model→reviewer mapping appears only in the workflow logs), so readers judge the feedback rather than the model.
+- All reviews are collected into one sticky PR comment. Labels are unioned across reviewers; if `FAIL_ON_REQUESTED_CHANGES` is enabled, the workflow fails when **any** reviewer requests changes. This is deliberately "any" rather than a majority vote — a single model catching a real blocker shouldn't be outvoted — and it only takes effect when you opt into `FAIL_ON_REQUESTED_CHANGES` (off by default). A reviewer that errors out (rather than returning a verdict) never counts as a "fail"; it just shows a note in its section.
+- Models run **in parallel**, so the job's wall-clock is the slowest single review rather than the sum. Each model still makes its own set of GitHub API calls for context, so a very long list means many concurrent calls; two or three models are well within limits.
+
+## Usage
+
+### Triggering AI Reviews
+
+To trigger an AI review on a PR:
+
+1. Go to the PR page
+2. Click **Labels**
+3. Add the label: `ai_code_review`
+
+The review will automatically start and post results as a comment when complete.
+
+### Re-running Reviews
+
+To re-run the AI review after making changes:
+
+1. Remove the `ai_code_review` label
+2. Add the `ai_code_review` label again
+
+This will generate a fresh review of the current PR state.
+
+## Review Results
+
+The AI posts a comprehensive comment analyzing your code across all focus areas. The review is meant to assist human reviewers, not replace them.
+
+## Cost Estimation
+
+Costs vary by model, but most code-focused models on OpenRouter are very affordable:
+- Typical small PR (< 1000 lines): $0.001 - $0.01
+- Large PR (1000-5000 lines): $0.01 - $0.05
+
+Check [OpenRouter pricing](https://openrouter.ai/models) for specific model costs.
+
+## Customization
+
+### Changing the Review Focus
+
+The review prompt lives in `ai-reviewer.sh`, which the workflow downloads at run time from the [Friendly-AI-Reviewer](https://github.com/LearningCircuit/Friendly-AI-Reviewer) repository (see the "Download AI reviewer script" step in `.github/workflows/ai-code-reviewer.yml`) — it is **not** stored in this repo. The current focus areas are:
+- Security (secrets, injection attacks, authentication)
+- Performance (algorithms, queries, memory)
+- Code Quality (readability, maintainability, error handling)
+- Best Practices (standards, patterns, type safety)
+
+To adjust them, edit the prompt in that repository (or fork it and point the workflow's download step at your fork).
+
+## Troubleshooting
+
+### Reviews Not Running
+
+- Ensure the `ai_code_review` label is added (not just present)
+- Check that `OPENROUTER_API_KEY` secret is correctly configured
+- Verify GitHub Actions permissions are properly set
+
+### API Errors
+
+- Check OpenRouter API key validity
+- Verify OpenRouter account has sufficient credits
+- Review GitHub Actions logs for specific error messages
+
+### Diff Too Large Error
+
+If you get a "Diff is too large" error:
+- Split your PR into smaller, focused changes
+- Or increase `MAX_DIFF_SIZE` in the workflow file
+- Default limit is 800KB (~200K tokens)
+
+## Security Considerations
+
+- API keys are stored securely in GitHub Secrets and passed via environment variables
+- Reviews only run when the `ai_code_review` label is manually added
+- All API calls are made through secure HTTPS connections
+- Code diffs are sent to OpenRouter/AI provider - review their data policies
+- The workflow has minimal permissions (read contents, write PR comments)
+
+## Support
+
+For issues with:
+- **OpenRouter API**: Check [OpenRouter documentation](https://openrouter.ai/docs)
+- **GitHub Actions**: Check [GitHub Actions documentation](https://docs.github.com/en/actions)
+- **Workflow issues**: Review the GitHub Actions logs for specific error details

docs/BENCHMARKING.md

@@ -0,0 +1,168 @@
+# Benchmarking System
+
+The Local Deep Research benchmarking system evaluates search configurations, models, and strategies using standardized datasets to help you optimize performance.
+
+**Important**: Benchmark results are indicators for configuration testing, not predictors of performance on your specific research topics. What works well on SimpleQA may perform differently on your actual research questions.
+
+## Quick Start
+
+### Web Interface
+1. Navigate to **Benchmark** in the web interface
+2. Configure your test:
+   - Select datasets (SimpleQA recommended)
+   - Set number of examples (start with 20-50)
+   - Uses your current Settings configuration
+3. Click **Start Benchmark** and monitor progress
+4. View results in **Benchmark Results** page
+
+## Datasets
+
+### SimpleQA (Recommended)
+- Fact-based questions with clear answers
+- Best for testing general knowledge retrieval
+- Good baseline for comparing configurations
+
+### BrowseComp (Advanced)
+- Complex browsing and comparison tasks
+- Currently limited performance - use max 20 examples for testing
+
+## Configuration Options
+
+### Search Engines
+- **Tavily**: AI-optimized commercial API
+- **SearXNG**: Meta-search aggregating multiple engines
+- **Brave**: Independent search engine
+- **Specialized engines** (ArXiv, PubMed, Wikipedia): Not suitable for general SimpleQA testing
+
+### Strategies
+- **Focused Iteration**: Best for SimpleQA fact-based questions
+- **Source-Based**: Better for comprehensive research
+
+## Interpreting Results
+
+### Key Metrics
+- **Accuracy**: Percentage of correct answers
+- **Processing Time**: Time per question (30-60s is typical)
+- **Search Results**: Number of search results retrieved per query
+
+### Performance Expectations
+- **Focused iteration with SimpleQA**: Around 95% potential with optimal setup
+- **Source-based strategy**: Around 70% accuracy, more comprehensive results
+
+## Interpretation
+
+Benchmark numbers are estimates, not exact measurements. This section helps reason about how much to trust a result and when a difference between two runs is meaningful.
+
+### Confidence Intervals for a Single Run
+
+A reported accuracy of "91%" is a point estimate with uncertainty that depends on how many examples you tested. The **Wilson score interval** gives the true range at 95% confidence (it behaves correctly near 0% and 100%, unlike the simpler normal approximation):
+
+```
+center    = (p̂ + z²/2n) / (1 + z²/n)
+half-width = z × sqrt(p̂(1−p̂)/n + z²/4n²) / (1 + z²/n)
+
+where p̂ = observed accuracy, n = examples run, z = 1.96 for 95% CI
+```
+
+**95% confidence margin of error by sample size:**
+
+| Examples (n) | ~70% accuracy | ~85% accuracy | ~91% accuracy | ~95% accuracy |
+|:---:|:---:|:---:|:---:|:---:|
+| 20  | ±21% | ±17% | ±14% | ±10% |
+| 50  | ±13% | ±10% | ±8%  | ±6%  |
+| 100 | ±9%  | ±7%  | ±6%  | ±4%  |
+| 200 | ±6%  | ±5%  | ±4%  | ±3%  |
+| 500 | ±4%  | ±3%  | ±3%  | ±2%  |
+
+> **Key takeaway:** A run of 20 examples has an uncertainty window of ±14–21%. A reported score of "91%" could plausibly be anywhere from 77% to 100%. Use at least 100 examples before drawing any conclusions, and 200+ before comparing configurations.
+
+### Comparing Two Configurations
+
+To tell whether config A is better than config B, you need enough examples that the observed difference is larger than the noise. The table below shows how many examples each configuration needs (run independently, same question set via the same seed) to reliably detect a given absolute accuracy difference at 80% statistical power (α = 0.05, two-sided):
+
+| Difference to detect | Examples needed per config |
+|:---|:---:|
+| 5 pp (e.g., 85% vs 90%)  | ~680 |
+| 10 pp (e.g., 80% vs 90%) | ~200 |
+| 15 pp (e.g., 75% vs 90%) | ~90  |
+
+**Rule of thumb:** If the observed difference between two runs is smaller than the margin of error for either run (see the table above), treat the results as a tie.
+
+### When Two Runs Cannot Be Compared
+
+Even with large sample sizes, cross-run comparison is unreliable if any of the following differ between the runs:
+
+- **LDR version** — search logic, prompt templates, and result filtering may have changed between releases
+- **Strategy** — `focused_iteration` and `source_based` answer questions differently by design; their scores measure different things
+- **Grader model** — changing the evaluation LLM changes what "correct" means; the same system response may grade differently under different graders
+- **Random seed / question sample** — some subsets of SimpleQA are inherently easier than others; always use `--seed 42` (or any fixed seed) consistently across compared runs
+- **Search engine** — Tavily, SearXNG, and Brave retrieve different content; engine latency also affects what gets retrieved within per-query time limits
+
+Treat each combination of (LDR version, strategy, search engine, grader model, seed) as a distinct experimental condition. Comparisons are only valid within the same condition.
+
+### Evaluator LLM Error
+
+The grader LLM (default: Claude 3.7 Sonnet via OpenRouter) is not perfect. On SimpleQA-style questions it mis-grades approximately **1% of responses** — consistent with calibration results reported in the original SimpleQA paper for similarly capable graders.
+
+What this means in practice:
+
+- **100 examples:** ~1 question is graded wrong. A 1 percentage-point difference (e.g., 91% vs 92%) is indistinguishable from grader noise alone.
+- **500 examples:** ~5 questions are graded wrong. A 1% gap is still inside grader noise; differences of 3–4 pp start to be interpretable.
+- The grader tends to be **conservative** — it marks ambiguous or partially-correct matches as incorrect — so reported accuracy is a slight underestimate of true accuracy.
+
+**Don't optimize for differences smaller than ~2–3 pp on runs under 500 examples.** The signal is not there.
+
+### Decision Checklist
+
+Before acting on a benchmark result:
+
+```
+[ ] n ≥ 100 examples (use ≥ 200 when comparing two configurations)
+[ ] Same random seed used across all compared runs
+[ ] Same LDR version, strategy, search engine, and grader model
+[ ] Observed difference > margin of error for each run (see table above)
+[ ] Observed difference > ~2 pp (minimum meaningful above grader noise)
+```
+
+---
+
+## Best Practices
+
+### Testing Workflow
+1. Start with 20 examples to verify configuration
+2. Check that search results are being retrieved
+3. Scale to 50-100 examples for reliable metrics
+4. Adjust settings based on results
+
+### Troubleshooting
+- **Low accuracy**: Verify API keys and search engine connectivity
+- **No search results**: Check API credentials and rate limiting
+- **Very fast processing**: Usually indicates configuration issues
+
+## Requirements
+
+### API Keys Needed
+- **Evaluation**: OpenRouter API key for automatic grading
+- **Search**: API key for your chosen search engine
+- **LLM**: API key for your language model provider
+
+## Responsible Usage
+
+- Start with small tests to verify configuration
+- Use moderate example counts for shared resources
+- Monitor API usage in the Metrics page
+- Respect rate limits and shared infrastructure
+
+## Important Limitations
+
+Benchmarks test standardized questions and may not reflect performance on:
+- Your specific domain or research topics
+- Complex, multi-step research questions
+- Real-time or recent information queries
+- Specialized knowledge areas
+
+Use benchmarks as configuration guidance, then test with your actual research topics to validate performance.
+
+---
+
+The benchmarking system helps you find starting configurations for reliable research results.

docs/CI_CD_INFRASTRUCTURE.md

@@ -0,0 +1,307 @@
+# CI/CD and Infrastructure Documentation
+
+This document describes the continuous integration, security scanning, and development infrastructure used by the Local Deep Research project.
+
+## Overview
+
+The project uses many GitHub Actions workflows and 20+ pre-commit hooks to ensure code quality, security, and reliability.
+
+> **At-a-glance health**: see [`docs/ci/workflow-status.md`](ci/workflow-status.md) — an auto-generated dashboard with live badges for every workflow, surfacing disabled, manual-only, and stale (silently-failing) ones at the top. Regenerate with `pdm run python scripts/generate_workflow_status.py`.
+
+```
+┌─────────────────────────────────────────────────────────────────┐
+│                        Developer Workflow                        │
+├─────────────────────────────────────────────────────────────────┤
+│  Local Development          │  Pull Request        │  Main/Dev  │
+│  ─────────────────          │  ────────────        │  ────────  │
+│  • Pre-commit hooks         │  • All tests         │  • Deploy  │
+│  • Unit tests               │  • Security scans    │  • Publish │
+│  • Linting                  │  • Code review       │  • Release │
+└─────────────────────────────────────────────────────────────────┘
+```
+
+## Pre-Commit Hooks
+
+Pre-commit hooks run locally before each commit. Install with:
+
+```bash
+pre-commit install
+pre-commit install-hooks
+```
+
+### Standard Hooks
+
+| Hook | Purpose |
+|------|---------|
+| `check-yaml` | Validate YAML syntax |
+| `end-of-file-fixer` | Ensure files end with newline |
+| `trailing-whitespace` | Remove trailing whitespace |
+| `check-added-large-files` | Block files >1MB |
+| `check-case-conflict` | Prevent case-sensitivity issues |
+| `forbid-new-submodules` | Prevent git submodules |
+
+### Security Hooks
+
+| Hook | Purpose |
+|------|---------|
+| `gitleaks` | Detect secrets, API keys, passwords in code |
+| `check-sensitive-logging` | Prevent logging of passwords, tokens, keys |
+| `check-safe-requests` | Enforce SSRF-safe HTTP functions (`safe_get`, `safe_post`) |
+| `check-url-security` | Validate URL handling in JavaScript (XSS prevention) |
+| `file-whitelist-check` | Only allow approved file types |
+| `check-image-pinning` | Require SHA256 digests for Docker images |
+
+### Code Quality Hooks
+
+| Hook | Purpose |
+|------|---------|
+| `ruff` | Python linter (with auto-fix) |
+| `ruff-format` | Python formatter (Black-compatible) |
+| `eslint` | JavaScript linter |
+| `shellcheck` | Shell script linter |
+| `actionlint` | GitHub Actions workflow validator |
+| `custom-code-checks` | Loguru usage, UTC datetime, raw SQL detection |
+
+### Project-Specific Hooks
+
+| Hook | Purpose |
+|------|---------|
+| `check-env-vars` | Environment variables must use `SettingsManager` |
+| `check-deprecated-db-connection` | Enforce per-user database connections |
+| `check-ldr-db-usage` | Prevent shared `ldr.db` usage |
+| `check-research-id-type` | `research_id` must be string/UUID, not int |
+| `check-datetime-timezone` | All DateTime columns (models and migrations) must use `UtcDateTime` from `sqlalchemy_utc` |
+| `check-session-context-manager` | Require context managers for DB sessions |
+| `check-pathlib-usage` | Use `pathlib.Path` instead of `os.path` |
+| `check-no-external-resources` | No external CDN/resource references |
+| `check-css-class-prefix` | CSS classes must have `ldr-` prefix |
+
+---
+
+## GitHub Actions Workflows
+
+### Test Workflows
+
+| Workflow | Trigger | Purpose |
+|----------|---------|---------|
+| `docker-tests.yml` | PR, push | Consolidated Docker tests: pytest + coverage, UI tests (51 Puppeteer tests), LLM tests, infrastructure tests (single Docker build shared across all jobs). Includes tests previously in critical-ui-tests, extended-ui-tests, metrics-analytics-tests, library-ui-tests, mobile-ui-tests, and news-tests workflows. |
+| `e2e-research-test.yml` | PR, push | End-to-end research flow |
+| `fuzz.yml` | Schedule | Fuzzing tests |
+
+### Security Scanning
+
+| Workflow | Trigger | Purpose |
+|----------|---------|---------|
+| `codeql.yml` | PR, push, schedule | GitHub CodeQL analysis |
+| `semgrep.yml` | PR, push | Semgrep static analysis |
+| `osv-scanner.yml` | PR, push, schedule | OSV vulnerability scanning (Python + npm) |
+| `gitleaks.yml` | PR, push | Secret detection |
+| `security-tests.yml` | PR, push | Security-focused test suite |
+| `devskim.yml` | PR, push | Microsoft DevSkim analysis |
+| `checkov.yml` | PR, push | Infrastructure-as-code scanning |
+| `container-security.yml` | PR, push | Container vulnerability scanning |
+| `hadolint.yml` | PR, push | Dockerfile linting |
+| `owasp-zap-scan.yml` | Schedule | OWASP ZAP dynamic scanning |
+| `retirejs.yml` | PR, push | JavaScript vulnerability scanning |
+| `zizmor-security.yml` | PR, push | Additional security checks |
+| `ossf-scorecard.yml` | Schedule | OpenSSF Scorecard |
+| `security-headers-validation.yml` | PR, push | HTTP security headers |
+| `security-file-write-check.yml` | PR, push | File write security |
+| `npm-audit.yml` | PR, push | npm audit for JS dependencies |
+
+### Dependency Management
+
+| Workflow | Trigger | Purpose |
+|----------|---------|---------|
+| `dependency-review.yml` | PR | Review dependency changes |
+| `update-dependencies.yml` | Schedule | Auto-update Python deps |
+| `update-npm-dependencies.yml` | Schedule | Auto-update npm deps |
+| `update-precommit-hooks.yml` | Schedule | Update pre-commit hooks |
+| `validate-image-pinning.yml` | PR, push | Verify Docker image pins |
+
+### UI/Accessibility
+
+| Workflow | Trigger | Purpose |
+|----------|---------|---------|
+| `responsive-ui-tests-enhanced.yml` | PR, push | Responsive design tests |
+
+### Build & Deploy
+
+| Workflow | Trigger | Purpose |
+|----------|---------|---------|
+| `prerelease-docker.yml` | `workflow_call` from release.yml | Canonical multi-arch Docker build, cosign sign, SBOM/SLSA attestations. Jobs declare `environment: release` so the first `release` env approval gates the build (env-scoped Docker Hub secrets). |
+| `docker-publish.yml` | `workflow_call` from release.yml | Retag prerelease manifest as `:1.6.9` / `:1.6` / `:latest` (gated by `release` env). No rebuild — registry-side metadata only. Inlined as a reusable workflow so its result is visible to downstream jobs in release.yml (lets create-release block on Docker success, lets cleanup-on-rejection safely scope cosign artifact deletion). |
+| `docker-multiarch-test.yml` | PR, push | Multi-architecture build test |
+| `publish.yml` | `repository_dispatch` from release.yml | Publish to PyPI. Stays on `repository_dispatch` (not `workflow_call`) because PyPI Trusted Publishing rejects OIDC claims from reusable workflows — `pypa/gh-action-pypi-publish#166`, `pypi/warehouse#11096`. |
+| `release.yml` | Push to `main`, tag `v*.*.*`, manual | Orchestrate release: gates → build → provenance → prerelease-docker → publish-docker → trigger-pypi → monitor-pypi → create-release (last) |
+
+### Code Quality
+
+| Workflow | Trigger | Purpose |
+|----------|---------|---------|
+| `pre-commit.yml` | PR, push | Run pre-commit hooks in CI |
+| `mypy-type-check.yml` | PR, push | Python type checking |
+| `ai-code-reviewer.yml` | PR | AI-assisted code review |
+| `claude-code-review.yml` | PR | Claude-based code review |
+
+### Repository Management
+
+| Workflow | Trigger | Purpose |
+|----------|---------|---------|
+| `sync-main-to-dev.yml` | Push to main | Sync main branch to dev |
+| `label-fixed-in-dev.yml` | Push to dev | Auto-label fixed issues |
+| `danger-zone-alert.yml` | PR | Alert on sensitive file changes |
+| `check-env-vars.yml` | PR, push | Environment variable validation |
+| `file-whitelist-check.yml` | PR, push | File type validation |
+| `version_check.yml` | PR, push | Version consistency check |
+
+---
+
+## Dependabot Configuration
+
+Dependabot automatically creates PRs for dependency updates:
+
+| Ecosystem | Directories | Schedule |
+|-----------|-------------|----------|
+| Python (pip) | `/` | Weekly (Monday 04:00) |
+| npm | `/`, `/tests/*` | Weekly/Daily |
+| GitHub Actions | `/` | Weekly |
+| Docker | `/` | Daily |
+
+---
+
+## Coverage Reporting
+
+Coverage reports are generated by the `docker-tests.yml` workflow (pytest-tests job):
+
+- **HTML Report**: Deployed to GitHub Pages at `https://learningcircuit.github.io/local-deep-research/coverage/`
+- **PR Comments**: Each PR receives a comment with coverage percentage
+- **Badge**: Coverage badge updated via GitHub Gist
+
+Configuration in `pyproject.toml`:
+```toml
+[tool.coverage.run]
+source = ["src"]
+omit = ["*/tests/*", "*/migrations/*"]
+
+[tool.coverage.report]
+exclude_lines = ["pragma: no cover", "if TYPE_CHECKING:"]
+```
+
+---
+
+## Security Architecture
+
+### Supply Chain Security
+
+1. **Dependency Pinning**: All GitHub Actions use SHA256 digests
+2. **Docker Image Pinning**: All base images use SHA256 digests
+3. **Lock Files**: `pdm.lock` and `package-lock.json` committed
+4. **Vulnerability Scanning**: OSV-Scanner, npm audit, RetireJS
+
+### Runtime Security
+
+1. **SSRF Protection**: `safe_get()`, `safe_post()`, `SafeSession` wrappers
+2. **XSS Prevention**: DOMPurify for HTML sanitization
+3. **SQL Injection**: SQLAlchemy ORM (no raw SQL)
+4. **Secret Management**: Environment variables via `SettingsManager`
+
+### Container Security
+
+1. **Non-root User**: Containers run as `ldruser:1000`
+2. **Minimal Base Image**: Python slim images
+3. **Health Checks**: Docker health check endpoints
+4. **Read-only Where Possible**: Minimal write permissions
+
+---
+
+## Running Tests Locally
+
+### Quick Test (Unit Tests Only)
+```bash
+pdm run pytest tests/test_settings_manager.py tests/test_utils.py -v
+```
+
+### Full Test Suite
+```bash
+pdm run pytest tests/ --ignore=tests/ui_tests --ignore=tests/fuzz -v
+```
+
+### With Coverage
+```bash
+pdm run pytest tests/ --cov=src --cov-report=html -v
+open coverage/htmlcov/index.html
+```
+
+### UI Tests (Requires Server)
+```bash
+# Terminal 1: Start server
+pdm run ldr-web
+
+# Terminal 2: Run UI tests
+cd tests/ui_tests && npm test
+```
+
+---
+
+## Docker Testing
+
+Build and run tests in Docker:
+
+```bash
+# Build test image
+docker build --target ldr-test -t ldr-test .
+
+# Run tests
+docker run --rm -v "$PWD":/app -w /app ldr-test \
+  pytest tests/ --ignore=tests/ui_tests -v
+```
+
+---
+
+## Environment Variables for CI
+
+| Variable | Purpose |
+|----------|---------|
+| `CI=true` | Indicates CI environment |
+| `LDR_TESTING_WITH_MOCKS=true` | Enable test mocks |
+| `LDR_DISABLE_RATE_LIMITING=true` | Disable HTTP rate limits in tests (canonical name). The legacy `DISABLE_RATE_LIMITING=true` is still honored but emits a deprecation warning. Distinct from `LDR_RATE_LIMITING_ENABLED`, which controls the adaptive search-engine rate limiter — different subsystem. |
+
+---
+
+## Adding New Workflows
+
+When adding a new workflow:
+
+1. Use pinned action versions with SHA256 digests
+2. Add `permissions: {}` at top level (minimal permissions)
+3. Add job-level permissions as needed
+4. Include `step-security/harden-runner` step
+5. Add workflow to this documentation
+
+Example template:
+```yaml
+name: New Workflow
+
+on:
+  pull_request:
+    branches: [main]
+
+permissions: {}
+
+jobs:
+  example:
+    runs-on: ubuntu-latest
+    permissions:
+      contents: read
+
+    steps:
+      - name: Harden the runner
+        uses: step-security/harden-runner@... # pinned
+        with:
+          egress-policy: audit
+
+      - uses: actions/checkout@... # pinned
+        with:
+          persist-credentials: false
+```

docs/CONFIGURATION.md

@@ -0,0 +1,623 @@
+# Configuration Reference
+
+This document is automatically generated from the application's default settings.
+All settings can be configured via the Web UI (Settings page), or overridden via Environment Variables.
+
+## Environment Variables
+
+To override a setting using an environment variable, convert the key to uppercase, replace dots with underscores, and prefix with `LDR_`.
+For example, `app.debug` becomes `LDR_APP_DEBUG`.
+
+Configuration Priority: Web UI Config > Environment Variables > Default Values
+> Environmental Variables are used to override default values, easing installation, while allowing for adjustments to configuration via Web UI.
+
+### System Locking
+There is a special environment variable `LDR_LOCKED_SETTINGS` that allows administrators to strictly enforce specific settings.
+
+*   **Variable**: `LDR_LOCKED_SETTINGS`
+*   **Format**: Comma-separated list of setting keys (e.g., `llm.model,app.port`)
+*   **Behavior**:
+    1.  Any setting listed here **MUST** have a corresponding value defined in the environment variables (e.g., `LDR_LLM_MODEL`). If not, the application will fail to start.
+    2.  The setting becomes **read-only** in the Web UI.
+    3.  The **Environment Variable** value takes absolute precedence, ignoring any value in the database.
+
+**Priority for Locked Settings**: Environment Variable > Database (Ignored) > Default (Ignored)
+
+
+## Pre-Database (Env-Only) Settings
+
+These settings are **required before database initialization** and can only be set via environment variables.
+They are not available in the Web UI because they are needed to start the application.
+
+| Environment Variable | Type | Default | Required | Constraints | Description | Category | Deprecated Alias |
+|----------------------|------|---------|----------|-------------|-------------|----------|------------------|
+| `LDR_BOOTSTRAP_ALLOW_UNENCRYPTED` | Boolean | `False` | No |  | Allow unencrypted database (for development) | Bootstrap | LDR_ALLOW_UNENCRYPTED |
+| `LDR_BOOTSTRAP_CONFIG_DIR` | Path | `None` | No |  | Configuration directory path | Bootstrap |  |
+| `LDR_BOOTSTRAP_DATABASE_URL` | String | `None` | No |  | Database connection URL | Bootstrap |  |
+| `LDR_BOOTSTRAP_DATA_DIR` | Path | `None` | No |  | Data directory path | Bootstrap |  |
+| `LDR_BOOTSTRAP_ENABLE_FILE_LOGGING` | Boolean | `False` | No |  | Enable logging to file | Bootstrap |  |
+| `LDR_BOOTSTRAP_ENCRYPTION_KEY` | Secret | `None` | No |  | Database encryption key | Bootstrap |  |
+| `LDR_BOOTSTRAP_LOG_DIR` | Path | `None` | No |  | Log directory path | Bootstrap |  |
+| `LDR_BOOTSTRAP_SECRET_KEY` | Secret | `None` | No |  | Application secret key for session encryption | Bootstrap |  |
+| `LDR_DB_CONFIG_CACHE_SIZE_MB` | Integer | `64` | No | 1..10000 | SQLite cache size in MB | Db Config | LDR_DB_CACHE_SIZE_MB |
+| `LDR_DB_CONFIG_CIPHER_MEMORY_SECURITY` | Enum | `'OFF'` | No | OFF, ON | SQLCipher memory security (ON=clear memory after use + mlock, OFF=faster). ON requires IPC_LOCK in Docker. | Db Config |  |
+| `LDR_DB_CONFIG_HMAC_ALGORITHM` | Enum | `'HMAC_SHA512'` | No | HMAC_SHA1, HMAC_SHA256, HMAC_SHA512 | HMAC algorithm for database integrity | Db Config | LDR_DB_HMAC_ALGORITHM |
+| `LDR_DB_CONFIG_JOURNAL_MODE` | Enum | `'WAL'` | No | DELETE, MEMORY, OFF, PERSIST, TRUNCATE, WAL | SQLite journal mode | Db Config | LDR_DB_JOURNAL_MODE |
+| `LDR_DB_CONFIG_KDF_ALGORITHM` | Enum | `'PBKDF2_HMAC_SHA512'` | No | PBKDF2_HMAC_SHA1, PBKDF2_HMAC_SHA256, PBKDF2_HMAC_SHA512 | Key derivation function algorithm | Db Config | LDR_DB_KDF_ALGORITHM |
+| `LDR_DB_CONFIG_KDF_ITERATIONS` | Integer | `256000` | No | 1000..1000000 | Number of KDF iterations for key derivation | Db Config | LDR_DB_KDF_ITERATIONS |
+| `LDR_DB_CONFIG_PAGE_SIZE` | Integer | `16384` | No | 512..65536 | SQLite page size (must be power of 2) | Db Config | LDR_DB_PAGE_SIZE |
+| `LDR_DB_CONFIG_SYNCHRONOUS` | Enum | `'NORMAL'` | No | EXTRA, FULL, NORMAL, OFF | SQLite synchronous mode | Db Config | LDR_DB_SYNCHRONOUS |
+| `LDR_DB_CONFIG_WAL_AUTOCHECKPOINT` | Integer | `250` | No | 10..10000 | WAL frames threshold for automatic PASSIVE checkpoint at commit. Lower = smaller WAL high-water-mark, faster recovery on open, slightly more frequent fsyncs. | Db Config |  |
+| `LDR_NEWS_SCHEDULER_ALLOW_API_CONTROL` | Boolean | `False` | No |  | Allow authenticated users to start/stop the global news scheduler via API. Disable in multi-user deployments to prevent users from affecting each other. | News Scheduler |  |
+| `LDR_NEWS_SCHEDULER_ENABLED` | Boolean | `True` | No |  | Enable or disable the news subscription scheduler | News Scheduler |  |
+| `LDR_NOTIFICATIONS_ALLOW_OUTBOUND` | Boolean | `False` | No |  | Master switch for outbound notification webhooks (Apprise). Disabled by default because Apprise re-resolves DNS at send time, leaving a DNS-rebinding TOCTOU window that cannot be closed in code (Apprise exposes no Session/DNS hook). See SECURITY.md 'Notification Webhook SSRF' for details. Set to true only after reviewing the residual risk. Distinct from the per-user notifications.enabled toggle in the settings UI: this is the server-level operator gate, env-only so it cannot be flipped via the user-writable settings API. | Security |  |
+| `LDR_NOTIFICATIONS_ALLOW_PRIVATE_IPS` | Boolean | `False` | No |  | Allow notification webhooks to target private/local IP addresses. Environment-only to prevent SSRF bypass via the user-writable settings API. Only enable this if your notification endpoints are on a trusted local network. | Security |  |
+| `LDR_SECURITY_ALLOW_NAT64` | Boolean | `False` | No |  | Allow outbound traffic to NAT64 prefixes (64:ff9b::/96 RFC 6052 well-known and 64:ff9b:1::/48 RFC 8215 local-use). Disabled by default to close the IPv6-wrapped SSRF bypass class — on hosts configured with NAT64 routes, attacker-supplied URLs can wrap cloud-metadata or RFC1918 destinations through these prefixes. Enable only on IPv6-only deployments (DNS64+NAT64) where outbound IPv4 traffic is synthesized through this prefix and the operator has accepted the residual SSRF risk. 6to4 (2002::/16), Teredo (2001::/32), and the discard prefix (100::/64) remain unconditionally blocked because they have no live legitimate use in 2026. The cloud-metadata block (ALWAYS_BLOCKED_METADATA_IPS) still applies via embedded-IPv4 extraction — see SECURITY.md. | Security |  |
+| `LDR_SECURITY_CORS_ALLOWED_ORIGINS` | String | `None` | No |  | Allowed CORS origins for API routes (comma-separated). Use '*' for all origins, empty for same-origin only. Example: 'https://example.com,https://app.example.com' | Security |  |
+| `LDR_SECURITY_WEBSOCKET_ALLOWED_ORIGINS` | String | `None` | No |  | Allowed origins for WebSocket/Socket.IO connections (comma-separated). Unset or empty means same-origin only (default); use '*' to allow all origins. Example: 'https://example.com,https://app.example.com' | Security |  |
+| `LDR_SERVER_MAX_CONCURRENT_RESEARCH` | Integer | `10` | No | 1..1000 | Server-wide maximum concurrent research operations. Requires restart. | Server | LDR_MAX_GLOBAL_CONCURRENT |
+| `LDR_TESTING_TEST_MODE` | Boolean | `False` | No |  | Enable test mode (adds delays for testing concurrency) | Testing |  |
+| `LDR_WEB_QUEUE_PROCESSOR_ENABLED` | Boolean | `True` | No |  | Enable or disable the background research queue processor | Queue Processor |  |
+
+
+## Settings List
+
+| Key | Environment Variable | Default Value | Description | Type |
+|-----|----------------------|---------------|-------------|------|
+| `app.allow_registrations` | `LDR_APP_ALLOW_REGISTRATIONS` | `true` | Allow new user registrations through the web interface. Configured via LDR_APP_ALLOW_REGISTRATIONS environment variable. Set to 'false' to disable on public-facing deployments. Enabled by default so initial account creation works out of the box. Requires server restart. | APP |
+| `app.debug` | `LDR_APP_DEBUG` | `false` | Raises log verbosity to DEBUG level. Configured via LDR_APP_DEBUG environment variable. Requires server restart. Does NOT enable Loguru's diagnose mode (local-variable dumps in exception tracebacks); set LDR_LOGURU_DIAGNOSE=true in addition if you need that — see docs/env_configuration.md. | APP |
+| `app.enable_file_logging` | `LDR_APP_ENABLE_FILE_LOGGING` | `false` | Enable logging to files (WARNING: Log files are unencrypted and may contain sensitive data) | APP |
+| `app.enable_notifications` | `LDR_APP_ENABLE_NOTIFICATIONS` | `true` | Enable browser push notifications to alert you when research completes or encounters errors. Requires browser permission to display notifications. | APP |
+| `app.enable_web` | `LDR_APP_ENABLE_WEB` | `true` | Enable the web interface. When disabled, the application runs in CLI-only mode. Requires server restart to take effect. | APP |
+| `app.external_url` | `LDR_APP_EXTERNAL_URL` | `` | Public URL where this application is accessible (e.g., http://localhost:5000, https://myapp.example.com). Used for generating links in notifications. Leave empty to auto-detect from request context. | APP |
+| `app.host` | `LDR_APP_HOST` | `0.0.0.0` | IP address or hostname for the web server to bind to. Use '0.0.0.0' to accept connections from any network interface, '127.0.0.1' for localhost only, or a specific IP for targeted access. Requires server restart. | APP |
+| `app.lock_settings` | `LDR_APP_LOCK_SETTINGS` | `false` | When enabled, prevents any modifications to settings through the UI. Useful for locking configuration in production environments. Must be changed directly in the database to re-enable editing. | APP |
+| `app.max_concurrent_researches` | `LDR_MAX_CONCURRENT` | `3` | Maximum number of concurrent research processes allowed per user | APP |
+| `app.max_global_concurrent_researches` | `LDR_SERVER_MAX_CONCURRENT_RESEARCH` | `10` | Maximum number of concurrent research processes allowed across all users (server-wide). Set via env var LDR_SERVER_MAX_CONCURRENT_RESEARCH. Requires server restart. | APP |
+| `app.max_user_query_length` | `LDR_APP_MAX_USER_QUERY_LENGTH` | `300` | Maximum character length for user queries before disabling direct search. Longer queries will use LLM-generated questions only. | SEARCH |
+| `app.port` | `LDR_APP_PORT` | `5000` | TCP port for the web server (1-65535). Requires server restart to take effect. | APP |
+| `app.queue_mode` | `LDR_QUEUE_MODE` | `direct` | How new research requests are handled. 'Direct' starts research immediately if a concurrent slot is available. 'Queue' always adds to a background queue for sequential processing in order. | APP |
+| `app.theme` | `LDR_APP_THEME` | `dark` | User interface color theme. Choose from 20 themes including popular developer themes. | APP |
+| `app.timezone` | `TZ` | `UTC` | Timezone for date calculations (e.g., news subscription YYYY-MM-DD placeholder). Use 'UTC' for server time or specify your local timezone. | APP |
+| `app.warnings.dismiss_adaptive_scope_info` | `LDR_APP_WARNINGS_DISMISS_ADAPTIVE_SCOPE_INFO` | `false` | Permanently hide the informational banner that states what the Adaptive egress scope resolves to (public / private / both) for the current primary engine. | APP |
+| `app.warnings.dismiss_cloud_embeddings` | `LDR_APP_WARNINGS_DISMISS_CLOUD_EMBEDDINGS` | `false` | Permanently hide the 'document chunks will be sent to OpenAI' (cloud embeddings) banner. Separate from the public-egress dismiss flag so acknowledging fresh-install noise never suppresses this critical warning. | APP |
+| `app.warnings.dismiss_cloud_llm` | `LDR_APP_WARNINGS_DISMISS_CLOUD_LLM` | `false` | Permanently hide the 'LLM provider is cloud-hosted' banner. Separate from the public-egress dismiss flag so acknowledging fresh-install noise never suppresses this critical warning. | APP |
+| `app.warnings.dismiss_context_reduced` | `LDR_APP_WARNINGS_DISMISS_CONTEXT_REDUCED` | `false` | Dismiss warnings based on historical context usage and truncation | APP |
+| `app.warnings.dismiss_egress_policy` | `LDR_APP_WARNINGS_DISMISS_EGRESS_POLICY` | `false` | Permanently hide the 'public search egress enabled' banner (the fresh-install informational one). The critical cloud-LLM and cloud-embeddings banners have their OWN dismiss flags so this can't silently hide them. | APP |
+| `app.warnings.dismiss_high_context` | `LDR_APP_WARNINGS_DISMISS_HIGH_CONTEXT` | `false` | Permanently hide the warning shown when your local LLM context window is set above 8,192 tokens, which may cause memory issues on some hardware. | APP |
+| `app.warnings.dismiss_legacy_config` | `LDR_APP_WARNINGS_DISMISS_LEGACY_CONFIG` | `false` | Dismiss warnings about the deprecated server_config.json file | APP |
+| `app.warnings.dismiss_model_mismatch` | `LDR_APP_WARNINGS_DISMISS_MODEL_MISMATCH` | `false` | Permanently hide the warning shown when using a large model (70B+ parameters) with a local provider and high context window, which may exceed available GPU memory. | APP |
+| `app.warnings.dismiss_searxng_recommendation` | `LDR_APP_WARNINGS_DISMISS_SEARXNG_RECOMMENDATION` | `false` | Dismiss recommendations about using more questions instead of iterations with SearXNG | APP |
+| `app.web_interface` | `LDR_APP_WEB_INTERFACE` | `true` | Enable the web interface for the application. When disabled, only CLI and API access is available. Requires server restart to take effect. | APP |
+| `backup.enabled` | `LDR_BACKUP_ENABLED` | `true` | Automatically backup database after successful login. Each backup uses disk space proportional to your database size. Disable this if disk space is limited. | APP |
+| `backup.max_age_days` | `LDR_BACKUP_MAX_AGE_DAYS` | `7` | Delete backups older than this many days | APP |
+| `backup.max_count` | `LDR_BACKUP_MAX_COUNT` | `1` | Maximum number of backup files to keep per user | APP |
+| `benchmark.evaluation.endpoint_url` | `LDR_BENCHMARK_EVALUATION_ENDPOINT_URL` | `https://openrouter.ai/api/v1` | Endpoint URL for evaluation model (when using OpenAI-compatible APIs) | APP |
+| `benchmark.evaluation.model` | `LDR_BENCHMARK_EVALUATION_MODEL` | `anthropic/claude-3.7-sonnet` | LLM model used as a judge to score benchmark answers against reference answers (e.g., 'anthropic/claude-3.7-sonnet' for OpenRouter, 'gpt-4o' for OpenAI). Must be compatible with the selected evaluation provider. | APP |
+| `benchmark.evaluation.provider` | `LDR_BENCHMARK_EVALUATION_PROVIDER` | `openai_endpoint` | LLM provider used to grade benchmark answers by comparing research outputs against reference answers. This is separate from the research LLM being tested. Options: openai_endpoint (recommended, supports OpenRouter), openai, anthropic, ollama. | APP |
+| `benchmark.evaluation.temperature` | `LDR_BENCHMARK_EVALUATION_TEMPERATURE` | `0` | Sampling temperature for the benchmark judge model (0.0-1.0). Use 0 for deterministic, reproducible grading results. Higher values introduce randomness in evaluation scores. | APP |
+| `chat.followup_context_mode` | `LDR_CHAT_FOLLOWUP_CONTEXT_MODE` | `summary` | What prior conversation context a follow-up research turn receives as its 'previous findings'. 'Focused summary' uses the LLM to summarize the whole conversation toward the new question (one extra LLM call per follow-up); 'Raw recent findings' reuses recent answers truncated; 'Full transcript' sends the entire conversation; 'None' sends no prior findings. | CHAT |
+| `chat.llm_title_generation` | `LDR_CHAT_LLM_TITLE_GENERATION` | `false` | Use LLM to generate descriptive chat session titles instead of simple query truncation. When disabled, titles are the first 100 characters of the query. | CHAT |
+| `chat.max_findings_to_include` | `LDR_CHAT_MAX_FINDINGS_TO_INCLUDE` | `5` | Maximum number of recent research findings to include in chat context. Higher values provide more background but increase token usage. | CHAT |
+| `chat.title_llm_timeout_seconds` | `LDR_CHAT_TITLE_LLM_TIMEOUT_SECONDS` | `30` | Hard wall-clock timeout (seconds) for the LLM call that generates a chat session title. Past this, the request thread falls back to truncation so a slow LLM endpoint cannot block title generation indefinitely. | CHAT |
+| `document_scheduler.download_pdfs` | `LDR_DOCUMENT_SCHEDULER_DOWNLOAD_PDFS` | `false` | Automatically download PDF files during scheduled document processing. Files will be stored on disk (requires storage space). WARNING: Downloaded files are stored unencrypted on disk. | APP |
+| `document_scheduler.enabled` | `LDR_DOCUMENT_SCHEDULER_ENABLED` | `true` | Enable automatic document processing from research history | APP |
+| `document_scheduler.extract_text` | `LDR_DOCUMENT_SCHEDULER_EXTRACT_TEXT` | `true` | Extract text content to database (minimal space usage) | APP |
+| `document_scheduler.generate_rag` | `LDR_DOCUMENT_SCHEDULER_GENERATE_RAG` | `false` | Index documents in the background for semantic search. LEGACY/DEPRECATED: this is now equivalent to "Auto-index documents in the background" below — both enable the unified reconciler that indexes every unindexed document (library uploads AND research downloads). Kept for backward compatibility so existing users keep their research downloads indexed; prefer the auto-index setting for new configurations. | APP |
+| `document_scheduler.interval_seconds` | `LDR_DOCUMENT_SCHEDULER_INTERVAL_SECONDS` | `1800` | How often to process new research (seconds) | APP |
+| `document_scheduler.sweep_library_collections` | `LDR_DOCUMENT_SCHEDULER_SWEEP_LIBRARY_COLLECTIONS` | `false` | Periodically index ALL unindexed documents in the background — both library uploads that weren't indexed immediately (e.g. because the immediate auto-index queue was full) AND research downloads that haven't been ingested into your library yet. Idempotent and processed in small batches per run; already-indexed documents are skipped. Off by default. Requires "Enable Document Scheduler" above to be on — the background sweep only runs while the document scheduler is enabled. | APP |
+| `embeddings.ollama.num_ctx` | `LDR_EMBEDDINGS_OLLAMA_NUM_CTX` | `8192` | Context window size (in tokens) used when generating Ollama embeddings. Must be at least as large as the longest chunk after tokenization; otherwise Ollama returns 'input length exceeds the context length' (HTTP 500) and indexing fails. The default of 8192 covers chunk sizes up to ~5000 characters for most models. Increase for very large chunks; decrease to save VRAM. qwen3-embedding supports up to 32768; nomic-embed-text supports 8192. | APP |
+| `embeddings.ollama.url` | `LDR_EMBEDDINGS_OLLAMA_URL` | `http://localhost:11434` | URL of the Ollama endpoint for embedding models. This setting allows you to use a different Ollama server for embeddings than for LLM operations. If not set, the system will fall back to the LLM Ollama URL. | APP |
+| `embeddings.openai.api_key` | `LDR_EMBEDDINGS_OPENAI_API_KEY` | `` | API key for the OpenAI embeddings endpoint. Required for the OpenAI cloud API. For OpenAI-compatible local servers (LM Studio, vLLM, llama.cpp), you can leave this blank — LDR will substitute a placeholder so the OpenAI client request succeeds against the keyless server. | APP |
+| `embeddings.openai.base_url` | `LDR_EMBEDDINGS_OPENAI_BASE_URL` | `` | Custom base URL for the embeddings endpoint. Leave blank to use the OpenAI cloud API (https://api.openai.com/v1). Set to your OpenAI-compatible local server URL to route embeddings there — examples: http://localhost:1234/v1 (LM Studio), http://localhost:8000/v1 (vLLM or llama.cpp server). When set, this overrides the cloud endpoint for embeddings without affecting the chat LLM provider. | APP |
+| `embeddings.openai.dimensions` | `LDR_EMBEDDINGS_OPENAI_DIMENSIONS` | `null` | Optional output vector dimensionality. Only applies to OpenAI text-embedding-3 models (which support truncation); ignored for other models including most local servers. Leave blank to use the model's native dimensionality. | APP |
+| `embeddings.openai.model` | `LDR_EMBEDDINGS_OPENAI_MODEL` | `text-embedding-3-small` | Embedding model identifier. For OpenAI cloud: text-embedding-3-small (default), text-embedding-3-large, or text-embedding-ada-002. For OpenAI-compatible local servers, use the exact model name your server reports (e.g. nomic-ai/nomic-embed-text-v1.5 on LM Studio). | APP |
+| `embeddings.require_local` | `LDR_EMBEDDINGS_REQUIRE_LOCAL` | `false` | When enabled, the embeddings provider must be local (sentence_transformers, or Ollama pointed at a local URL). Indexing with cloud embeddings (OpenAI) is blocked. Indexing a corpus with cloud embeddings sends every chunk off-machine. | LLM |
+| `focused_iteration.adaptive_questions` | `LDR_FOCUSED_ITERATION_ADAPTIVE_QUESTIONS` | `false` | When enabled (1), the system monitors recent search success and warns the LLM to broaden queries if 3+ searches in a recent window returned no results. When disabled (0), queries are generated without feedback from previous result counts. | SEARCH |
+| `focused_iteration.knowledge_summary_limit` | `LDR_FOCUSED_ITERATION_KNOWLEDGE_SUMMARY_LIMIT` | `10` | Limits the number of top search results included in the knowledge summary shown to the LLM during question generation. Set to 10 to show only the top 10 results, reducing token usage. Set to 0 for unlimited. Note: This only affects question generation prompts — final answer synthesis always uses all results. | SEARCH |
+| `focused_iteration.previous_searches_limit` | `LDR_FOCUSED_ITERATION_PREVIOUS_SEARCHES_LIMIT` | `10` | Maximum number of recent search iterations to show the LLM when generating follow-up questions. Helps the LLM avoid duplicate queries and reduces token usage. Set to 0 for unlimited (show all previous searches). When active, also caps displayed queries per iteration to 3. | SEARCH |
+| `focused_iteration.prompt_knowledge_truncate` | `LDR_FOCUSED_ITERATION_PROMPT_KNOWLEDGE_TRUNCATE` | `1500` | Maximum characters of accumulated knowledge to include in the LLM prompt for follow-up question generation. Applied after 'Results Shown for Question Generation' selects the top N results — this further truncates that combined text to a character limit. Set to 0 for unlimited. | SEARCH |
+| `focused_iteration.question_generator` | `LDR_FOCUSED_ITERATION_QUESTION_GENERATOR` | `browsecomp` | Algorithm for generating follow-up search queries. 'browsecomp' (default): Systematically extracts and combines entities (names, dates, locations) for predictable, focused searches. 'flexible': Allows the LLM more freedom to explore diverse search strategies — less rigid but may be less consistent. | SEARCH |
+| `focused_iteration.snippet_truncate` | `LDR_FOCUSED_ITERATION_SNIPPET_TRUNCATE` | `200` | Maximum characters per snippet when showing search results to LLM for question generation. Shorter snippets (e.g., 200) reduce token usage and keep context focused. Set to 0 for full snippets. NOTE: This only affects question generation - the final answer synthesis sees full snippets. | SEARCH |
+| `general.enable_fact_checking` | `LDR_GENERAL_ENABLE_FACT_CHECKING` | `false` | Enable LLM-based fact verification for research findings. Adds additional validation pass to check claims against sources. | APP |
+| `general.knowledge_accumulation` | `LDR_GENERAL_KNOWLEDGE_ACCUMULATION` | `ITERATION` | How to compress accumulated knowledge between search iterations. 'ITERATION': Compress and carry forward knowledge after each iteration completes (default). 'QUESTION': Compress after every individual question within an iteration (more aggressive, saves tokens). 'NO_KNOWLEDGE': Start fresh each iteration with no carried-forward context. | APP |
+| `general.knowledge_accumulation_context_limit` | `LDR_GENERAL_KNOWLEDGE_ACCUMULATION_CONTEXT_LIMIT` | `2000000` | Maximum character count for accumulated knowledge context. Reserved for future use — not currently enforced at runtime. | APP |
+| `general.output_instructions` | `LDR_GENERAL_OUTPUT_INSTRUCTIONS` | `` | Customize how research outputs are formatted: language, tone, style, formatting preferences, audience level, etc. Examples: 'Respond in Spanish with formal tone' \| 'Use simple language for beginners' \| 'Be concise with bullet points'. Leave empty for default English output. | APP |
+| `langgraph_agent.include_sub_research` | `LDR_LANGGRAPH_AGENT_INCLUDE_SUB_RESEARCH` | `true` | Enable parallel subagent research. When enabled, the agent can decompose complex questions into subtopics and research them simultaneously using separate subagents. | SEARCH |
+| `langgraph_agent.max_iterations` | `LDR_LANGGRAPH_AGENT_MAX_ITERATIONS` | `50` | Maximum number of agent reasoning cycles (each cycle = one LLM call + one tool call). Higher values let the agent search more thoroughly but use more tokens. The agent typically needs 10-20 cycles for simple queries and 30-50 for complex multi-faceted research. | SEARCH |
+| `langgraph_agent.max_sub_iterations` | `LDR_LANGGRAPH_AGENT_MAX_SUB_ITERATIONS` | `8` | Maximum reasoning cycles per subagent when researching subtopics in parallel. Each subagent investigates one aspect of the question independently. | SEARCH |
+| `llm.allowed_local_hostnames` | `LDR_LLM_ALLOWED_LOCAL_HOSTNAMES` | ``[]`` | Hostnames the policy will treat as local in addition to loopback / RFC1918 / link-local. Use for self-hosted services on custom hostnames (e.g. 'ollama.lan'). Public hostnames added here are rejected at save time. | LLM |
+| `llm.anthropic.api_key` | `LDR_LLM_ANTHROPIC_API_KEY` | `` | Your Anthropic API key for Claude models. Get one at console.anthropic.com. Required when using the Anthropic provider. | LLM |
+| `llm.anthropic_endpoint.api_key` | `LDR_LLM_ANTHROPIC_ENDPOINT_API_KEY` | `` | Optional API key for your Anthropic-compatible endpoint. Leave blank for self-hosted servers that don't require authentication. | LLM |
+| `llm.anthropic_endpoint.url` | `LDR_LLM_ANTHROPIC_ENDPOINT_URL` | `` | Base URL of a self-hosted service implementing the Anthropic Messages API (/v1/messages). The client appends the API path. Example: http://localhost:9090. Required when using the Anthropic-Compatible Endpoint provider. | LLM |
+| `llm.context_window_size` | `LDR_LLM_CONTEXT_WINDOW_SIZE` | `128000` | Maximum context window size in tokens for cloud LLMs. Only used when unrestricted context is disabled. | LLM |
+| `llm.context_window_unrestricted` | `LDR_LLM_CONTEXT_WINDOW_UNRESTRICTED` | `true` | Let cloud provider APIs manage their own token limits without a hard cap (recommended). Uncheck to enforce a specific limit via the Context Window Size setting. | LLM |
+| `llm.deepseek.api_key` | `LDR_LLM_DEEPSEEK_API_KEY` | `null` | API key to use for the DeepSeek provider. | LLM |
+| `llm.google.api_key` | `LDR_LLM_GOOGLE_API_KEY` | `null` | API key to use for the Google Gemini provider. | SEARCH |
+| `llm.ionos.api_key` | `LDR_LLM_IONOS_API_KEY` | `null` | API key to use for the IONOS AI Model Hub provider. | SEARCH |
+| `llm.llamacpp.api_key` | `LDR_LLM_LLAMACPP_API_KEY` | `` | Optional API key for `llama-server` if it's behind an auth proxy. Leave empty if your llama-server has no authentication. | LLM |
+| `llm.llamacpp.url` | `LDR_LLM_LLAMACPP_URL` | `http://localhost:8080/v1` | HTTP endpoint URL where llama.cpp's `llama-server` is running. Include the full path with /v1 (default: http://localhost:8080/v1). Run `llama-server -m <model.gguf>` to start it. | LLM |
+| `llm.lmstudio.api_key` | `LDR_LLM_LMSTUDIO_API_KEY` | `` | Optional API key for LM Studio if you've enabled authentication. Leave empty for the default no-auth setup. | LLM |
+| `llm.lmstudio.url` | `LDR_LLM_LMSTUDIO_URL` | `http://localhost:1234/v1` | HTTP endpoint URL where LM Studio is running locally. Include the full path (e.g., http://localhost:1234/v1 for OpenAI-compatible API format). | LLM |
+| `llm.local_context_window_size` | `LDR_LLM_LOCAL_CONTEXT_WINDOW_SIZE` | `20480` | Context window size in tokens for local LLMs (Ollama, LlamaCpp, LM Studio). The default of 20480 tokens fits on systems with limited VRAM while still leaving room for the synthesis prompt. If your synthesis prompts get truncated, increase this; if you run out of VRAM, reduce it and switch to the source-based strategy. | LLM |
+| `llm.max_tokens` | `LDR_LLM_MAX_TOKENS` | `30000` | Maximum tokens in model responses. Automatically capped at 80% of the context window size to leave room for prompt tokens. | LLM |
+| `llm.model` | `LDR_LLM_MODEL` | `` | The language model to use. Available models depend on the selected provider (e.g., 'gpt-4o' for OpenAI, 'gemma:7b' for Ollama). You can type any model name supported by your provider. | LLM |
+| `llm.ollama.api_key` | `LDR_LLM_OLLAMA_API_KEY` | `` | Optional API key for Ollama if you've put it behind a proxy that requires authentication. Leave empty for the default no-auth setup. | LLM |
+| `llm.ollama.enable_thinking` | `LDR_LLM_OLLAMA_ENABLE_THINKING` | `true` | Enable thinking/reasoning mode for models like deepseek-r1 and qwen2.5. When enabled (recommended), the model performs internal reasoning for more accurate answers, but the reasoning content is automatically separated and excluded from the final response. When disabled, the model gives faster but potentially less accurate direct answers without reasoning. | LLM |
+| `llm.ollama.url` | `LDR_LLM_OLLAMA_URL` | `http://localhost:11434` | HTTP endpoint URL where Ollama is running. Used to connect to local or remote Ollama instances for model inference. | LLM |
+| `llm.openai.api_key` | `LDR_LLM_OPENAI_API_KEY` | `` | Your OpenAI API key for GPT models. Get one at platform.openai.com. Required when using the OpenAI provider. | LLM |
+| `llm.openai_endpoint.api_key` | `LDR_LLM_OPENAI_ENDPOINT_API_KEY` | `` | API key for your custom OpenAI-compatible endpoint. For cloud services (OpenRouter, Groq), use your actual API key. For local servers (llama.cpp, vLLM), you can leave this blank — LDR will substitute a placeholder automatically. | LLM |
+| `llm.openai_endpoint.stream_usage` | `LDR_LLM_OPENAI_ENDPOINT_STREAM_USAGE` | `false` | Request token usage statistics on streamed responses (sends stream_options.include_usage). Enable this if the token metrics dashboard shows zero token counts for streamed calls. Supported by LM Studio (0.3.18+), llama.cpp server, vLLM, and OpenRouter. Leave disabled if your endpoint rejects requests with a 'stream_options' error (e.g. some API gateways). | LLM |
+| `llm.openai_endpoint.url` | `LDR_LLM_OPENAI_ENDPOINT_URL` | `https://openrouter.ai/api/v1` | URL of a custom OpenAI-compatible API endpoint. Use for llama.cpp server, vLLM, Ollama, OpenRouter, Groq, Together, or any service that implements the OpenAI API format. Example for local llama.cpp: http://localhost:8000/v1 | LLM |
+| `llm.openrouter.api_key` | `LDR_LLM_OPENROUTER_API_KEY` | `null` | API key to use for the OpenRouter provider. | SEARCH |
+| `llm.provider` | `LDR_LLM_PROVIDER` | `ollama` | The LLM service provider. Choose local providers (Ollama, LM Studio, LlamaCpp) for free on-device inference, or cloud APIs (OpenAI, Anthropic, Google, OpenRouter) for hosted models. | LLM |
+| `llm.require_local_endpoint` | `LDR_LLM_REQUIRE_LOCAL_ENDPOINT` | `false` | When enabled, the LLM provider's resolved endpoint must be on a local network (loopback, RFC1918, link-local). Cloud-only providers (OpenAI, Anthropic, Google, OpenRouter) are blocked. Independent of egress_scope. | LLM |
+| `llm.supports_max_tokens` | `LDR_LLM_SUPPORTS_MAX_TOKENS` | `true` | Feature flag for max_tokens parameter. Disable if your LLM API returns errors when max_tokens is set. When disabled, the parameter is not passed to the API. | LLM |
+| `llm.temperature` | `LDR_LLM_TEMPERATURE` | `0.7` | Controls randomness in model outputs. Lower values (0.0-0.3) produce deterministic, factual responses; higher values (0.7-1.0) produce more creative and varied outputs. | LLM |
+| `llm.xai.api_key` | `LDR_LLM_XAI_API_KEY` | `null` | API key to use for the xAI Grok provider. | LLM |
+| `local_search_chunk_overlap` | `LDR_LOCAL_SEARCH_CHUNK_OVERLAP` | `200` | Characters shared between adjacent chunks. Prevents important information from being cut at chunk boundaries. Typically 10-20% of chunk size. Affects new indexing only. | APP |
+| `local_search_chunk_size` | `LDR_LOCAL_SEARCH_CHUNK_SIZE` | `1000` | Number of characters per chunk when splitting documents for embedding. Larger chunks (1000+) keep more context but search is less precise. Smaller chunks (<500) are more granular but may cut off important context. Affects new indexing only. | APP |
+| `local_search_distance_metric` | `LDR_LOCAL_SEARCH_DISTANCE_METRIC` | `cosine` | How chunk vectors are compared during retrieval. Cosine similarity is the standard recommendation for text embeddings. L2 (Euclidean) and dot product are alternatives whose suitability depends on whether the embedding model is trained to produce normalized vectors. | APP |
+| `local_search_embedding_model` | `LDR_LOCAL_SEARCH_EMBEDDING_MODEL` | `all-MiniLM-L6-v2` | Embedding model used for the local-document search index. Free-form text because available models vary by provider (e.g. Sentence Transformers ships a fixed list, Ollama/OpenAI-compatible servers expose whatever is installed). Defaults to all-MiniLM-L6-v2 (384d) which is fast and good for general use. | APP |
+| `local_search_embedding_provider` | `LDR_LOCAL_SEARCH_EMBEDDING_PROVIDER` | `sentence_transformers` | Embedding provider used for the local-document search index. Drives which models the Embeddings page lists and which provider runs when documents are indexed. The model dropdown is rebuilt to match this provider. | APP |
+| `local_search_index_type` | `LDR_LOCAL_SEARCH_INDEX_TYPE` | `flat` | FAISS index type used for the local-document search index. Flat: exact search, recommended for <10K documents. HNSW: approximate but ~50x faster, recommended for large collections (>10K docs). IVF: balanced for collections that are updated frequently. Changing this requires re-indexing existing collections. | APP |
+| `local_search_normalize_vectors` | `LDR_LOCAL_SEARCH_NORMALIZE_VECTORS` | `true` | L2-normalize embedding vectors after generation. Recommended: ensures fair similarity comparison regardless of source document length. Some embedding models already produce normalized output; in that case this is a no-op. | APP |
+| `local_search_splitter_type` | `LDR_LOCAL_SEARCH_SPLITTER_TYPE` | `recursive` | How to divide documents into chunks. Recursive uses natural breaks (paragraphs, sentences). Token-based respects model context limits. Sentence-based produces clean sentence boundaries. Semantic uses an embedding model to group related content (smarter but slower). | APP |
+| `local_search_text_separators` | `LDR_LOCAL_SEARCH_TEXT_SEPARATORS` | ``["\n\n", "\n", ". ", " ", ""]`` | JSON array of separator strings tried in order by the recursive splitter. The default works for general prose. For code, add language-specific separators like "\nclass", "\ndef". | APP |
+| `news.display.default_headline_max_length` | `LDR_NEWS_DISPLAY_DEFAULT_HEADLINE_MAX_LENGTH` | `100` | Default maximum headline length | APP |
+| `news.display.max_query_length` | `LDR_NEWS_DISPLAY_MAX_QUERY_LENGTH` | `50` | Maximum query display length in UI | APP |
+| `news.feed.default_limit` | `LDR_NEWS_FEED_DEFAULT_LIMIT` | `20` | Default number of news items returned per feed page. Applied when user does not specify a limit in the request. Also used as the default limit for subscription history. | APP |
+| `news.preferences.max_stored` | `LDR_NEWS_PREFERENCES_MAX_STORED` | `100` | Maximum number of stored preferences per user | APP |
+| `news.progress.complete` | `LDR_NEWS_PROGRESS_COMPLETE` | `100` | Progress percentage when complete | APP |
+| `news.progress.generating_searches` | `LDR_NEWS_PROGRESS_GENERATING_SEARCHES` | `50` | Progress percentage when generating searches | APP |
+| `news.refresh.default_hours` | `LDR_NEWS_REFRESH_DEFAULT_HOURS` | `4` | Default refresh interval in hours | APP |
+| `news.scheduler.activity_check_interval` | `LDR_NEWS_SCHEDULER_ACTIVITY_CHECK_INTERVAL` | `5` | How often (in minutes) the scheduler scans for subscriptions that are due to run. More frequent checks ensure timely updates but increase CPU usage. Range: 1-60 minutes. | APP |
+| `news.scheduler.allow_api_control` | `LDR_NEWS_SCHEDULER_ALLOW_API_CONTROL` | `false` | Allow authenticated users to start, stop, and trigger the global news scheduler via API. This setting cannot be changed from the UI for security reasons. Disable in multi-user deployments by setting the environment variable LDR_NEWS_SCHEDULER_ALLOW_API_CONTROL=false. | APP |
+| `news.scheduler.batch_size` | `LDR_NEWS_SCHEDULER_BATCH_SIZE` | `5` | Number of due subscriptions to pick up and schedule in each check cycle. Larger batches are more efficient but use more memory. Range: 1-20. | APP |
+| `news.scheduler.cleanup_interval_hours` | `LDR_NEWS_SCHEDULER_CLEANUP_INTERVAL_HOURS` | `1` | How often (in hours) the scheduler checks for and removes expired user sessions. Shorter intervals reduce memory usage but add slight overhead. Range: 0.25-24 hours. | APP |
+| `news.scheduler.enabled` | `LDR_NEWS_SCHEDULER_ENABLED` | `true` | Enable the background scheduler that automatically runs news subscriptions on their configured intervals. When disabled, no automatic subscription processing occurs — subscriptions must be refreshed manually. | APP |
+| `news.scheduler.max_concurrent_jobs` | `LDR_NEWS_SCHEDULER_MAX_CONCURRENT_JOBS` | `10` | Maximum number of subscription refresh tasks that can run simultaneously across all users. Lower values reduce server load; higher values process subscriptions faster. Range: 1-50. | APP |
+| `news.scheduler.max_jitter_seconds` | `LDR_NEWS_SCHEDULER_MAX_JITTER_SECONDS` | `300` | Maximum random delay (in seconds) added to each subscription's next refresh time. Prevents all subscriptions from running simultaneously, reducing server load spikes. Range: 0-3600 seconds. | APP |
+| `news.scheduler.retention_hours` | `LDR_NEWS_SCHEDULER_RETENTION_HOURS` | `48` | Hours to keep a user's session active in the scheduler after their last interaction. After this period, the scheduler stops processing that user's subscriptions until they log in again. Range: 1-168 hours. | APP |
+| `news.storage.default_limit` | `LDR_NEWS_STORAGE_DEFAULT_LIMIT` | `100` | Default limit for storing news items | APP |
+| `news.subscription.default_type` | `LDR_NEWS_SUBSCRIPTION_DEFAULT_TYPE` | `search` | Default subscription type | APP |
+| `news.subscription.refresh_minutes` | `LDR_NEWS_SUBSCRIPTION_REFRESH_MINUTES` | `360` | Default interval (in minutes) for how often new subscriptions check for updates. Users can override this per subscription. Default is 240 minutes (4 hours). Range: 60-2880 minutes. | APP |
+| `news.trending.lookback_hours` | `LDR_NEWS_TRENDING_LOOKBACK_HOURS` | `24` | Hours to look back for trending analysis | APP |
+| `notifications.enabled` | `LDR_NOTIFICATIONS_ENABLED` | `false` | Enable external notifications for research events (email, Discord, Slack, etc.). When disabled, no notifications are sent regardless of individual event settings below. Requires service URL to be configured. | APP |
+| `notifications.on_api_quota_warning` | `LDR_NOTIFICATIONS_ON_API_QUOTA_WARNING` | `false` | Send notification when API quota or rate limits are approaching or exceeded. Helps prevent service interruptions from unexpected API usage limits. | APP |
+| `notifications.on_auth_issue` | `LDR_NOTIFICATIONS_ON_AUTH_ISSUE` | `false` | Send notification when authentication fails for external API services (e.g., invalid or expired API keys). Alerts you to broken integrations that need attention. | APP |
+| `notifications.on_research_completed` | `LDR_NOTIFICATIONS_ON_RESEARCH_COMPLETED` | `true` | Send notification when a research task completes successfully. This is the primary completion alert for tracking long-running research. | APP |
+| `notifications.on_research_failed` | `LDR_NOTIFICATIONS_ON_RESEARCH_FAILED` | `true` | Send notification when a research task fails or encounters an error. Enables rapid problem detection without monitoring the web interface. | APP |
+| `notifications.on_research_queued` | `LDR_NOTIFICATIONS_ON_RESEARCH_QUEUED` | `false` | Send notification when research is added to the processing queue. Useful in queue mode to confirm your request was accepted and is waiting to be processed. | APP |
+| `notifications.on_subscription_error` | `LDR_NOTIFICATIONS_ON_SUBSCRIPTION_ERROR` | `false` | Send notification when a subscription refresh fails or encounters errors. Helps identify broken subscriptions that need attention. | APP |
+| `notifications.on_subscription_update` | `LDR_NOTIFICATIONS_ON_SUBSCRIPTION_UPDATE` | `true` | Send notification when news subscriptions detect new content matching your configured topics. | APP |
+| `notifications.rate_limit_per_day` | `LDR_NOTIFICATIONS_RATE_LIMIT_PER_DAY` | `50` | Maximum number of notifications per day per user. Daily cap prevents notification fatigue. Default of 50 balances staying informed with avoiding excessive alerts. | APP |
+| `notifications.rate_limit_per_hour` | `LDR_NOTIFICATIONS_RATE_LIMIT_PER_HOUR` | `10` | Maximum number of notifications per hour per user. Prevents notification spam during burst activity. Default of 10 allows important notifications through while limiting noise. | APP |
+| `notifications.service_url` | `LDR_NOTIFICATIONS_SERVICE_URL` | `` | Notification service URL(s). Supports multiple comma-separated URLs to send notifications to multiple destinations (e.g., discord://YOUR_WEBHOOK_ID/YOUR_TOKEN, mailto://YOUR_USER:YOUR_PASS@smtp.example.com?to=recipient@example.com). See <a href="https://github.com/LearningCircuit/local-deep-research/blob/main/docs/NOTIFICATIONS.md" target="_blank">documentation</a> for supported services. | APP |
+| `policy.egress_scope` | `LDR_POLICY_EGRESS_SCOPE` | `adaptive` | EXPERIMENTAL — this egress boundary is new defense-in-depth, not an absolute guarantee yet; don't rely on it as your only protection for highly sensitive data. Restrict which search engines a research run may use. ADAPTIVE (default) = follow your primary engine: a private primary keeps everything local, a public primary allows public engines. STRICT = only the primary engine you picked; PUBLIC_ONLY = any public web/academic engine; PRIVATE_ONLY = any local engine (library, paperless, etc.); UNPROTECTED = escape hatch that disables egress protection entirely (not recommended; hard SSRF/cloud-metadata blocking still applies). | SEARCH |
+| `policy.trusted_inference_providers` | `LDR_POLICY_TRUSTED_INFERENCE_PROVIDERS` | ``[]`` | EXPERIMENTAL. LLM/embeddings providers you explicitly trust to receive sensitive content — listing one treats it as a 'contained' inference sink even though it is off-machine (e.g. a zero-retention enterprise Anthropic endpoint). Relaxes the two-axis egress classification; use only for providers you have vetted. | LLM |
+| `policy.trusted_search_engines` | `LDR_POLICY_TRUSTED_SEARCH_ENGINES` | ``[]`` | EXPERIMENTAL. Search engines whose endpoint you explicitly trust — listing one (e.g. a self-hosted Elasticsearch/Paperless on a public hostname) treats it as 'contained' rather than an exposing sink. Relaxes the two-axis egress classification; use only for endpoints you control. | SEARCH |
+| `rag.indexing_batch_size` | `LDR_RAG_INDEXING_BATCH_SIZE` | `15` | Number of documents to process in parallel during RAG indexing. Higher values are faster but use more memory. | APP |
+| `rate_limiting.decay_per_day` | `LDR_RATE_LIMITING_DECAY_PER_DAY` | `0.95` | How much to trust old rate limit data as it ages (0.5-0.99). At 0.95, data retains 95% reliability per day. Lower values discard old data faster, forcing the system to re-learn optimal wait times. | APP |
+| `rate_limiting.enabled` | `LDR_RATE_LIMITING_ENABLED` | `true` | Enable adaptive rate limiting that learns optimal wait times for each search engine to avoid hitting API rate limits. When disabled, requests are sent with minimal delay (0.1s), which may trigger rate limit errors from some APIs. | APP |
+| `rate_limiting.exploration_rate` | `LDR_RATE_LIMITING_EXPLORATION_RATE` | `0.1` | Fraction of requests that test shorter wait times to check if the API allows faster access (0.0-1.0). Higher values adapt faster to relaxed limits but risk more rate limit errors. | APP |
+| `rate_limiting.learning_rate` | `LDR_RATE_LIMITING_LEARNING_RATE` | `0.45` | How much weight to give new timing data vs. existing estimates (0.0-1.0). Higher values react faster to changes in API rate limits but produce less stable wait times. | APP |
+| `rate_limiting.llm_enabled` | `LDR_RATE_LIMITING_LLM_ENABLED` | `false` | Enable adaptive rate limiting for LLM API calls, separate from search engine rate limiting. Useful for free-tier or rate-limited LLM APIs that impose request quotas. | APP |
+| `rate_limiting.memory_window` | `LDR_RATE_LIMITING_MEMORY_WINDOW` | `100` | Number of recent requests to remember per search engine for calculating optimal wait times. At least 3 successful requests are needed before wait time estimates start updating. | APP |
+| `rate_limiting.profile` | `LDR_RATE_LIMITING_PROFILE` | `balanced` | Overall rate limiting behavior. 'Conservative' uses longer waits and adapts slowly (fewer errors). 'Balanced' is the default. 'Aggressive' uses shorter waits and adapts quickly (faster but riskier). | APP |
+| `report.citation_format` | `LDR_REPORT_CITATION_FORMAT` | `number_hyperlinks` | Citation formatting style for reports. Options include numbered hyperlinks [1], domain-based hyperlinks [arxiv.org], smart domain numbering [arxiv.org-1], source-tagged citations with the global citation number preserved (e.g. [arxiv-1], [openai.com-2], [arxiv-3]), and plain numbered references without links. | REPORT |
+| `report.enable_file_backup` | `LDR_REPORT_ENABLE_FILE_BACKUP` | `false` | Also save reports as files on disk for external access (reports are always stored in the database regardless). Enable this if you need to access reports from outside the web interface. | REPORT |
+| `report.export_formats` | `LDR_REPORT_EXPORT_FORMATS` | ``["markdown", "latex", "quarto", "ris"]`` | Available export format options displayed in the user interface. Supported formats include Markdown (.md), LaTeX (.tex), Quarto (.qmd), and RIS bibliographic format. | REPORT |
+| `report.max_context_chars` | `LDR_REPORT_MAX_CONTEXT_CHARS` | `4000` | Maximum character limit for previous section context when generating report sections. If accumulated context exceeds this, it is truncated at sentence boundaries. Lower values are safer for smaller local models. Default is 4000. | REPORT |
+| `report.max_context_sections` | `LDR_REPORT_MAX_CONTEXT_SECTIONS` | `3` | Number of previously generated report sections to include as context when writing new sections. Helps the LLM avoid repeating content. Higher values increase context size but may exceed smaller model limits. Default is 3. | REPORT |
+| `report.searches_per_section` | `LDR_REPORT_SEARCHES_PER_SECTION` | `2` | Number of web searches to perform per report section. Higher values increase research depth for each section but also increase search engine queries and processing time. Default is 2. | REPORT |
+| `research_library.auto_index_enabled` | `LDR_RESEARCH_LIBRARY_AUTO_INDEX_ENABLED` | `false` | Automatically index documents for RAG search when they are added to collections (via upload or download). Indexing runs in the background without blocking uploads. Disable to defer indexing and improve upload speed for large batches. | APP |
+| `research_library.confirm_deletions` | `LDR_RESEARCH_LIBRARY_CONFIRM_DELETIONS` | `true` | Show confirmation dialogs before delete operations. Recommended to keep enabled to prevent accidental data loss. | APP |
+| `research_library.max_pdf_size_mb` | `LDR_RESEARCH_LIBRARY_MAX_PDF_SIZE_MB` | `3072` | Maximum allowed PDF file size in megabytes for the research library. PDFs larger than this will not be stored. Uploads are also gated by the per-file upload cap (LDR_SECURITY_UPLOAD_MAX_FILE_SIZE_MB, default 3072 MB / 3 GB) — raising this value above that cap has no effect because uploads are rejected before they reach storage. | APP |
+| `research_library.pdf_storage_mode` | `RESEARCH_LIBRARY_PDF_STORAGE_MODE` | `database` | Choose how PDFs are stored. Each user has their own encrypted database. Database mode stores PDFs encrypted within it (secure, portable, isolated per-user). Filesystem mode stores unencrypted files on disk (faster for large files). None stores only extracted text (smallest footprint). | APP |
+| `research_library.shared_library` | `RESEARCH_LIBRARY_SHARED_LIBRARY` | `false` | ⚠️ WARNING: When enabled, all users share the same library directory. Documents from all users will be visible to each other. Only enable in trusted environments. | APP |
+| `research_library.storage_path` | `RESEARCH_LIBRARY_STORAGE_PATH` | `~/Documents/LocalDeepResearch/Library` | Base path for storing research library documents. Each user will have their own subdirectory unless shared library is enabled. | APP |
+| `research_library.upload_pdf_storage` | `RESEARCH_LIBRARY_UPLOAD_PDF_STORAGE` | `none` | Choose how user-uploaded PDFs are stored. Database mode stores PDFs encrypted in your personal database (increases database size but allows viewing/downloading later). Text Only extracts and stores only the text content (smallest footprint). Note: Filesystem storage is not available for uploads for security reasons. | APP |
+| `search.cross_engine_max_context_items` | `LDR_SEARCH_CROSS_ENGINE_MAX_CONTEXT_ITEMS` | `30` | Maximum number of search result previews sent to the LLM for relevance ranking during cross-engine filtering. Higher values let the model evaluate more candidates, improving recall at the cost of increased prompt size and latency. | SEARCH |
+| `search.cross_engine_max_results` | `LDR_SEARCH_CROSS_ENGINE_MAX_RESULTS` | `100` | Maximum number of search results to keep after cross-engine filtering. When results from multiple search engines are combined, this limits how many total results are displayed. Higher values show more comprehensive results. | SEARCH |
+| `search.engine.DEFAULT_SEARCH_ENGINE` | `LDR_SEARCH_ENGINE_DEFAULT_SEARCH_ENGINE` | `wikipedia` | Fallback search engine used when the configured engine is unavailable or has errors. | SEARCH |
+| `search.engine.web.arxiv.default_params.max_results` | `LDR_SEARCH_ENGINE_WEB_ARXIV_DEFAULT_PARAMS_MAX_RESULTS` | `20` | Maximum number of papers to retrieve from arXiv per search query. | SEARCH |
+| `search.engine.web.arxiv.default_params.sort_by` | `LDR_SEARCH_ENGINE_WEB_ARXIV_DEFAULT_PARAMS_SORT_BY` | `relevance` | How to sort arXiv results: 'relevance' for best match, 'lastUpdatedDate' for recently updated, 'submittedDate' for newest submissions. | SEARCH |
+| `search.engine.web.arxiv.default_params.sort_order` | `LDR_SEARCH_ENGINE_WEB_ARXIV_DEFAULT_PARAMS_SORT_ORDER` | `descending` | Sort order for arXiv results: 'descending' (newest/most relevant first) or 'ascending' (oldest/least relevant first). | SEARCH |
+| `search.engine.web.arxiv.description` | `LDR_SEARCH_ENGINE_WEB_ARXIV_DESCRIPTION` | `Search papers uploaded to ArXiv.` | Human-readable description of the search engine. | SEARCH |
+| `search.engine.web.arxiv.display_name` | `LDR_SEARCH_ENGINE_WEB_ARXIV_DISPLAY_NAME` | `ArXiv` | Display name to use in the U.I. for this search engine. | SEARCH |
+| `search.engine.web.arxiv.journal_reputation.enabled` | `LDR_SEARCH_ENGINE_WEB_ARXIV_JOURNAL_REPUTATION_ENABLED` | `true` | Enable journal quality filtering for this search engine. | SEARCH |
+| `search.engine.web.arxiv.reliability` | `LDR_SEARCH_ENGINE_WEB_ARXIV_RELIABILITY` | `0.9` | Reliability rating (0-1) for arXiv as a source. Higher values indicate more trustworthy academic content. | SEARCH |
+| `search.engine.web.arxiv.requires_api_key` | `LDR_SEARCH_ENGINE_WEB_ARXIV_REQUIRES_API_KEY` | `false` | Whether arXiv requires an API key. arXiv is free and open access, so this is false. | SEARCH |
+| `search.engine.web.arxiv.strengths` | `LDR_SEARCH_ENGINE_WEB_ARXIV_STRENGTHS` | ``["scientific papers", "academic research", "physics", "computer science", "mathematics", "statistics", "machine learning", "preprints"]`` | Key advantages of arXiv: Free access to preprints, covers physics/math/CS/biology, fast publication of cutting-edge research. | SEARCH |
+| `search.engine.web.arxiv.use_in_auto_search` | `LDR_SEARCH_ENGINE_WEB_ARXIV_USE_IN_AUTO_SEARCH` | `true` | Include ArXiv in auto search mode | SEARCH |
+| `search.engine.web.arxiv.weaknesses` | `LDR_SEARCH_ENGINE_WEB_ARXIV_WEAKNESSES` | ``["non-academic topics", "consumer products", "news", "general information"]`` | Limitations of arXiv: Preprints aren't peer-reviewed, limited to STEM fields, keyword-based search may miss semantic matches. | SEARCH |
+| `search.engine.web.brave.api_key` | `LDR_SEARCH_ENGINE_WEB_BRAVE_API_KEY` | `` | The Brave API key to use. | SEARCH |
+| `search.engine.web.brave.default_params.region` | `LDR_SEARCH_ENGINE_WEB_BRAVE_DEFAULT_PARAMS_REGION` | `US` | Geographic region code for Brave search results (e.g., 'US', 'UK', 'DE'). | SEARCH |
+| `search.engine.web.brave.default_params.safe_search` | `LDR_SEARCH_ENGINE_WEB_BRAVE_DEFAULT_PARAMS_SAFE_SEARCH` | `true` | Enable safe search filtering for Brave to exclude adult content. | SEARCH |
+| `search.engine.web.brave.default_params.search_language` | `LDR_SEARCH_ENGINE_WEB_BRAVE_DEFAULT_PARAMS_SEARCH_LANGUAGE` | `English` | Preferred language for Brave search results. | SEARCH |
+| `search.engine.web.brave.default_params.time_period` | `LDR_SEARCH_ENGINE_WEB_BRAVE_DEFAULT_PARAMS_TIME_PERIOD` | `y` | Time range filter for Brave results (e.g., 'd' for day, 'w' for week, 'm' for month, 'y' for year). | SEARCH |
+| `search.engine.web.brave.description` | `LDR_SEARCH_ENGINE_WEB_BRAVE_DESCRIPTION` | `Search the web using the Brave search engine.` | Human-readable description of the search engine. | SEARCH |
+| `search.engine.web.brave.display_name` | `LDR_SEARCH_ENGINE_WEB_BRAVE_DISPLAY_NAME` | `Brave` | Display name to use in the U.I. for this search engine. | SEARCH |
+| `search.engine.web.brave.reliability` | `LDR_SEARCH_ENGINE_WEB_BRAVE_RELIABILITY` | `0.7` | Reliability rating (0-1) for Brave Search. Good general-purpose search with privacy focus. | SEARCH |
+| `search.engine.web.brave.requires_api_key` | `LDR_SEARCH_ENGINE_WEB_BRAVE_REQUIRES_API_KEY` | `true` | Whether Brave Search requires an API key. Yes - get one at brave.com/search/api. | SEARCH |
+| `search.engine.web.brave.strengths` | `LDR_SEARCH_ENGINE_WEB_BRAVE_STRENGTHS` | ``["privacy-focused web search", "product information", "reviews", "recent content", "news", "broad coverage"]`` | Key advantages of Brave: Privacy-focused, good semantic search, reasonable pricing, full-text retrieval available. | SEARCH |
+| `search.engine.web.brave.supports_full_search` | `LDR_SEARCH_ENGINE_WEB_BRAVE_SUPPORTS_FULL_SEARCH` | `true` | Whether Brave supports fetching full webpage content beyond snippets. | SEARCH |
+| `search.engine.web.brave.use_in_auto_search` | `LDR_SEARCH_ENGINE_WEB_BRAVE_USE_IN_AUTO_SEARCH` | `false` | Include Brave search in auto search mode | SEARCH |
+| `search.engine.web.brave.weaknesses` | `LDR_SEARCH_ENGINE_WEB_BRAVE_WEAKNESSES` | ``["requires API key with usage limits", "smaller index than Google"]`` | Limitations of Brave: Smaller index than Google, requires paid API for production use. | SEARCH |
+| `search.engine.web.elasticsearch.default_params.api_key` | `LDR_SEARCH_ENGINE_WEB_ELASTICSEARCH_DEFAULT_PARAMS_API_KEY` | `` | API key for Elasticsearch authentication (optional) | SEARCH |
+| `search.engine.web.elasticsearch.default_params.cloud_id` | `LDR_SEARCH_ENGINE_WEB_ELASTICSEARCH_DEFAULT_PARAMS_CLOUD_ID` | `` | Elastic Cloud ID for cloud-hosted Elasticsearch (optional) | SEARCH |
+| `search.engine.web.elasticsearch.default_params.highlight_fields` | `LDR_SEARCH_ENGINE_WEB_ELASTICSEARCH_DEFAULT_PARAMS_HIGHLIGHT_FIELDS` | ``["content", "title"]`` | Fields to highlight in search results (JSON array) | SEARCH |
+| `search.engine.web.elasticsearch.default_params.hosts` | `LDR_SEARCH_ENGINE_WEB_ELASTICSEARCH_DEFAULT_PARAMS_HOSTS` | ``["http://localhost:9200"]`` | Elasticsearch server URLs (JSON array format) | SEARCH |
+| `search.engine.web.elasticsearch.default_params.index_name` | `LDR_SEARCH_ENGINE_WEB_ELASTICSEARCH_DEFAULT_PARAMS_INDEX_NAME` | `documents` | Name of the Elasticsearch index to search | SEARCH |
+| `search.engine.web.elasticsearch.default_params.max_results` | `LDR_SEARCH_ENGINE_WEB_ELASTICSEARCH_DEFAULT_PARAMS_MAX_RESULTS` | `10` | Maximum number of search results to return | SEARCH |
+| `search.engine.web.elasticsearch.default_params.password` | `LDR_SEARCH_ENGINE_WEB_ELASTICSEARCH_DEFAULT_PARAMS_PASSWORD` | `` | Password for Elasticsearch authentication (optional) | SEARCH |
+| `search.engine.web.elasticsearch.default_params.search_fields` | `LDR_SEARCH_ENGINE_WEB_ELASTICSEARCH_DEFAULT_PARAMS_SEARCH_FIELDS` | ``["content", "title", "description", "text"]`` | Fields to search in (JSON array) | SEARCH |
+| `search.engine.web.elasticsearch.default_params.username` | `LDR_SEARCH_ENGINE_WEB_ELASTICSEARCH_DEFAULT_PARAMS_USERNAME` | `` | Username for Elasticsearch authentication (optional) | SEARCH |
+| `search.engine.web.elasticsearch.description` | `LDR_SEARCH_ENGINE_WEB_ELASTICSEARCH_DESCRIPTION` | `Search engine for Elasticsearch databases. Efficient for searching document collections and structured data.` | Human-readable description of the search engine. | SEARCH |
+| `search.engine.web.elasticsearch.display_name` | `LDR_SEARCH_ENGINE_WEB_ELASTICSEARCH_DISPLAY_NAME` | `Elasticsearch` | Display name to use in the U.I. for this search engine. | SEARCH |
+| `search.engine.web.elasticsearch.reliability` | `LDR_SEARCH_ENGINE_WEB_ELASTICSEARCH_RELIABILITY` | `0.95` | Reliability of the Elasticsearch search engine | SEARCH |
+| `search.engine.web.elasticsearch.requires_api_key` | `LDR_SEARCH_ENGINE_WEB_ELASTICSEARCH_REQUIRES_API_KEY` | `false` | Whether this search engine requires an API key | SEARCH |
+| `search.engine.web.elasticsearch.requires_llm` | `LDR_SEARCH_ENGINE_WEB_ELASTICSEARCH_REQUIRES_LLM` | `true` | Whether this search engine requires an LLM for relevance filtering | SEARCH |
+| `search.engine.web.elasticsearch.strengths` | `LDR_SEARCH_ENGINE_WEB_ELASTICSEARCH_STRENGTHS` | ``["Fast full-text search", "Scalable and distributed", "Rich query DSL", "Supports aggregations", "Real-time search"]`` | Strengths of this search engine | SEARCH |
+| `search.engine.web.elasticsearch.supports_full_search` | `LDR_SEARCH_ENGINE_WEB_ELASTICSEARCH_SUPPORTS_FULL_SEARCH` | `true` | Whether this search engine supports full document search | SEARCH |
+| `search.engine.web.elasticsearch.use_in_auto_search` | `LDR_SEARCH_ENGINE_WEB_ELASTICSEARCH_USE_IN_AUTO_SEARCH` | `false` | Include Elasticsearch in auto search mode | SEARCH |
+| `search.engine.web.elasticsearch.weaknesses` | `LDR_SEARCH_ENGINE_WEB_ELASTICSEARCH_WEAKNESSES` | ``["Requires running Elasticsearch instance", "Needs proper index configuration", "Data must be pre-indexed"]`` | Weaknesses of this search engine | SEARCH |
+| `search.engine.web.exa.api_key` | `LDR_SEARCH_ENGINE_WEB_EXA_API_KEY` | `` | The Exa API key to use. Get one at exa.ai. | SEARCH |
+| `search.engine.web.exa.default_params.include_full_content` | `LDR_SEARCH_ENGINE_WEB_EXA_DEFAULT_PARAMS_INCLUDE_FULL_CONTENT` | `true` | Include full webpage content in results | SEARCH |
+| `search.engine.web.exa.default_params.search_type` | `LDR_SEARCH_ENGINE_WEB_EXA_DEFAULT_PARAMS_SEARCH_TYPE` | `auto` | Search type - auto (default), neural (semantic), fast, or deep (comprehensive multi-query) | SEARCH |
+| `search.engine.web.exa.description` | `LDR_SEARCH_ENGINE_WEB_EXA_DESCRIPTION` | `Neural search engine built for AI with semantic understanding and high-quality results optimized for LLM consumption.` | Human-readable description of the search engine. | SEARCH |
+| `search.engine.web.exa.display_name` | `LDR_SEARCH_ENGINE_WEB_EXA_DISPLAY_NAME` | `Exa` | Display name to use in the U.I. for this search engine. | SEARCH |
+| `search.engine.web.exa.reliability` | `LDR_SEARCH_ENGINE_WEB_EXA_RELIABILITY` | `0.85` | Reliability score (0-1). Exa rated high (0.85) - neural search with semantic understanding designed for AI applications. | SEARCH |
+| `search.engine.web.exa.requires_api_key` | `LDR_SEARCH_ENGINE_WEB_EXA_REQUIRES_API_KEY` | `true` | Indicates this engine requires an Exa API key. Get one at exa.ai. | SEARCH |
+| `search.engine.web.exa.strengths` | `LDR_SEARCH_ENGINE_WEB_EXA_STRENGTHS` | ``["neural search with semantic understanding", "results optimized for LLM consumption", "high-quality, clean content", "supports deep search for comprehensive results", "built-in highlights and summaries", "date filtering and domain controls"]`` | Advantages: Neural search with semantic understanding, results optimized for LLM consumption, high-quality content, supports deep search mode. | SEARCH |
+| `search.engine.web.exa.supports_full_search` | `LDR_SEARCH_ENGINE_WEB_EXA_SUPPORTS_FULL_SEARCH` | `true` | Indicates this engine can fetch full page content after initial search, not just snippets. | SEARCH |
+| `search.engine.web.exa.use_in_auto_search` | `LDR_SEARCH_ENGINE_WEB_EXA_USE_IN_AUTO_SEARCH` | `false` | Include Exa in auto search mode | SEARCH |
+| `search.engine.web.exa.weaknesses` | `LDR_SEARCH_ENGINE_WEB_EXA_WEAKNESSES` | ``["requires API key with usage limits", "newer service still evolving", "costs scale with usage"]`` | Limitations: Requires paid API key with usage limits, newer service with evolving features. | SEARCH |
+| `search.engine.web.github.api_key` | `LDR_SEARCH_ENGINE_WEB_GITHUB_API_KEY` | `` | GitHub Personal Access Token for API authentication. Without this, you're limited to 10 requests/minute for search API. With authentication, you get 30 requests/minute for search API. | SEARCH |
+| `search.engine.web.github.default_params.include_issues` | `LDR_SEARCH_ENGINE_WEB_GITHUB_DEFAULT_PARAMS_INCLUDE_ISSUES` | `false` | Include recent GitHub issues in repository search results. | SEARCH |
+| `search.engine.web.github.default_params.include_readme` | `LDR_SEARCH_ENGINE_WEB_GITHUB_DEFAULT_PARAMS_INCLUDE_README` | `true` | Fetch and include README content when searching repositories. | SEARCH |
+| `search.engine.web.github.default_params.max_results` | `LDR_SEARCH_ENGINE_WEB_GITHUB_DEFAULT_PARAMS_MAX_RESULTS` | `15` | Maximum number of GitHub results to retrieve per search. | SEARCH |
+| `search.engine.web.github.default_params.search_type` | `LDR_SEARCH_ENGINE_WEB_GITHUB_DEFAULT_PARAMS_SEARCH_TYPE` | `repositories` | What to search on GitHub: 'repositories', 'code', 'issues', or 'users'. | SEARCH |
+| `search.engine.web.github.description` | `LDR_SEARCH_ENGINE_WEB_GITHUB_DESCRIPTION` | `Search projects on Github.` | Human-readable description of the search engine. | SEARCH |
+| `search.engine.web.github.display_name` | `LDR_SEARCH_ENGINE_WEB_GITHUB_DISPLAY_NAME` | `Github` | Display name to use in the U.I. for this search engine. | SEARCH |
+| `search.engine.web.github.reliability` | `LDR_SEARCH_ENGINE_WEB_GITHUB_RELIABILITY` | `0.99` | Reliability rating (0-1) for GitHub as a source. Great for code and technical projects. | SEARCH |
+| `search.engine.web.github.requires_api_key` | `LDR_SEARCH_ENGINE_WEB_GITHUB_REQUIRES_API_KEY` | `true` | Whether GitHub requires a personal access token. Optional but recommended for higher rate limits (10→30 req/min). | SEARCH |
+| `search.engine.web.github.strengths` | `LDR_SEARCH_ENGINE_WEB_GITHUB_STRENGTHS` | ``["code repositories", "software documentation", "open source projects", "programming issues", "developer information", "technical documentation"]`` | Key advantages of GitHub: Excellent for code/technical queries, includes README content, issue discussions, works without API key. | SEARCH |
+| `search.engine.web.github.supports_full_search` | `LDR_SEARCH_ENGINE_WEB_GITHUB_SUPPORTS_FULL_SEARCH` | `true` | Whether GitHub supports fetching full content (README, code files) beyond basic search results. | SEARCH |
+| `search.engine.web.github.use_in_auto_search` | `LDR_SEARCH_ENGINE_WEB_GITHUB_USE_IN_AUTO_SEARCH` | `true` | Include GitHub in auto search mode | SEARCH |
+| `search.engine.web.github.weaknesses` | `LDR_SEARCH_ENGINE_WEB_GITHUB_WEAKNESSES` | ``["non-technical content", "content outside GitHub", "rate limits without API key"]`` | Limitations of GitHub: Limited to code/technical content, strict rate limits without authentication. | SEARCH |
+| `search.engine.web.google_pse.api_key` | `LDR_SEARCH_ENGINE_WEB_GOOGLE_PSE_API_KEY` | `` | The Google PSE API key to use. | SEARCH |
+| `search.engine.web.google_pse.default_params.region` | `LDR_SEARCH_ENGINE_WEB_GOOGLE_PSE_DEFAULT_PARAMS_REGION` | `us` | Geographic region for search results (e.g., 'us', 'uk', 'de'). Results prioritized from this region. | SEARCH |
+| `search.engine.web.google_pse.default_params.safe_search` | `LDR_SEARCH_ENGINE_WEB_GOOGLE_PSE_DEFAULT_PARAMS_SAFE_SEARCH` | `true` | Enable Google SafeSearch to filter explicit content from results. | SEARCH |
+| `search.engine.web.google_pse.default_params.search_language` | `LDR_SEARCH_ENGINE_WEB_GOOGLE_PSE_DEFAULT_PARAMS_SEARCH_LANGUAGE` | `English` | Language for search queries and results (e.g., 'English', 'German'). Filters results by language. | SEARCH |
+| `search.engine.web.google_pse.description` | `LDR_SEARCH_ENGINE_WEB_GOOGLE_PSE_DESCRIPTION` | `Search the web using Google's Programmable Search Engine API.` | Human-readable description of the search engine. | SEARCH |
+| `search.engine.web.google_pse.display_name` | `LDR_SEARCH_ENGINE_WEB_GOOGLE_PSE_DISPLAY_NAME` | `Google PSE` | Display name to use in the U.I. for this search engine. | SEARCH |
+| `search.engine.web.google_pse.reliability` | `LDR_SEARCH_ENGINE_WEB_GOOGLE_PSE_RELIABILITY` | `0.9` | Reliability score (0-1) indicating result quality. Google PSE is highly reliable (0.9) due to Google's indexing quality. | SEARCH |
+| `search.engine.web.google_pse.requires_api_key` | `LDR_SEARCH_ENGINE_WEB_GOOGLE_PSE_REQUIRES_API_KEY` | `true` | Indicates this engine requires a Google API key and Custom Search Engine ID. Get credentials from Google Cloud Console. | SEARCH |
+| `search.engine.web.google_pse.strengths` | `LDR_SEARCH_ENGINE_WEB_GOOGLE_PSE_STRENGTHS` | ``["custom search scope", "high-quality results", "domain-specific search", "configurable search experience", "control over search index"]`` | Advantages of Google PSE: Custom search scope, high-quality results, domain-specific filtering, configurable search experience. | SEARCH |
+| `search.engine.web.google_pse.supports_full_search` | `LDR_SEARCH_ENGINE_WEB_GOOGLE_PSE_SUPPORTS_FULL_SEARCH` | `true` | Indicates this engine can fetch full page content after initial search, not just snippets. | SEARCH |
+| `search.engine.web.google_pse.use_in_auto_search` | `LDR_SEARCH_ENGINE_WEB_GOOGLE_PSE_USE_IN_AUTO_SEARCH` | `false` | Include Google PSE in auto search mode | SEARCH |
+| `search.engine.web.google_pse.weaknesses` | `LDR_SEARCH_ENGINE_WEB_GOOGLE_PSE_WEAKNESSES` | ``["requires API key with usage limits", "limited to 10,000 queries/day on free tier", "requires search engine configuration in Google Control Panel"]`` | Limitations: Requires API key, limited to 10,000 queries/day on free tier, requires search engine configuration in Google Control Panel. | SEARCH |
+| `search.engine.web.gutenberg` | `LDR_SEARCH_ENGINE_WEB_GUTENBERG` | ``{"requires_api_key": false, "requires_llm": true, "description": "Search 70,000+ free public domain books with full text via Gutendex.", "reliability": 0.95}`` | Project Gutenberg search engine configuration | SEARCH |
+| `search.engine.web.gutenberg.default_params.max_results` | `LDR_SEARCH_ENGINE_WEB_GUTENBERG_DEFAULT_PARAMS_MAX_RESULTS` | `20` | Maximum number of results to retrieve from Project Gutenberg. | SEARCH |
+| `search.engine.web.gutenberg.description` | `LDR_SEARCH_ENGINE_WEB_GUTENBERG_DESCRIPTION` | `Search 70,000+ free public domain books with full text via Gutendex.` | Human-readable description of the search engine. | SEARCH |
+| `search.engine.web.gutenberg.display_name` | `LDR_SEARCH_ENGINE_WEB_GUTENBERG_DISPLAY_NAME` | `Project Gutenberg` | Display name for Project Gutenberg search engine. | SEARCH |
+| `search.engine.web.gutenberg.reliability` | `LDR_SEARCH_ENGINE_WEB_GUTENBERG_RELIABILITY` | `0.95` | Reliability score for Project Gutenberg. | SEARCH |
+| `search.engine.web.gutenberg.requires_api_key` | `LDR_SEARCH_ENGINE_WEB_GUTENBERG_REQUIRES_API_KEY` | `false` | Project Gutenberg is free and does not require an API key. | SEARCH |
+| `search.engine.web.gutenberg.strengths` | `LDR_SEARCH_ENGINE_WEB_GUTENBERG_STRENGTHS` | ``["public domain", "full text", "classic literature", "free ebooks", "multiple formats"]`` | Key strengths of Project Gutenberg. | SEARCH |
+| `search.engine.web.gutenberg.weaknesses` | `LDR_SEARCH_ENGINE_WEB_GUTENBERG_WEAKNESSES` | ``["public domain only", "older works", "limited modern titles"]`` | Limitations of Project Gutenberg. | SEARCH |
+| `search.engine.web.mojeek.api_key` | `LDR_SEARCH_ENGINE_WEB_MOJEEK_API_KEY` | `` | The Mojeek API key to use. Get one at mojeek.com/services/api. | SEARCH |
+| `search.engine.web.mojeek.default_params.language` | `LDR_SEARCH_ENGINE_WEB_MOJEEK_DEFAULT_PARAMS_LANGUAGE` | `en` | Language code in ISO 639-1 format for Mojeek language boost (e.g., 'en', 'fr'). | SEARCH |
+| `search.engine.web.mojeek.default_params.region` | `LDR_SEARCH_ENGINE_WEB_MOJEEK_DEFAULT_PARAMS_REGION` | `` | Country code in ISO 3166-1 alpha-2 format for Mojeek region boost (e.g., 'GB', 'US'). | SEARCH |
+| `search.engine.web.mojeek.default_params.safe_search` | `LDR_SEARCH_ENGINE_WEB_MOJEEK_DEFAULT_PARAMS_SAFE_SEARCH` | `false` | Enable safe search filtering for Mojeek to exclude adult content. | SEARCH |
+| `search.engine.web.mojeek.description` | `LDR_SEARCH_ENGINE_WEB_MOJEEK_DESCRIPTION` | `Privacy-focused search engine with its own independent web index.` | Human-readable description of the search engine. | SEARCH |
+| `search.engine.web.mojeek.display_name` | `LDR_SEARCH_ENGINE_WEB_MOJEEK_DISPLAY_NAME` | `Mojeek` | Display name to use in the U.I. for this search engine. | SEARCH |
+| `search.engine.web.mojeek.reliability` | `LDR_SEARCH_ENGINE_WEB_MOJEEK_RELIABILITY` | `0.65` | Reliability rating (0-1) for Mojeek. Independent index, smaller than Google/Bing. | SEARCH |
+| `search.engine.web.mojeek.requires_api_key` | `LDR_SEARCH_ENGINE_WEB_MOJEEK_REQUIRES_API_KEY` | `true` | Whether Mojeek requires an API key. Yes - get one at mojeek.com/services/api. | SEARCH |
+| `search.engine.web.mojeek.strengths` | `LDR_SEARCH_ENGINE_WEB_MOJEEK_STRENGTHS` | ``["privacy-focused with own index", "no tracking or profiling", "independent results", "European-based"]`` | Key advantages of Mojeek: Privacy-focused with own independent web index, no tracking. | SEARCH |
+| `search.engine.web.mojeek.supports_full_search` | `LDR_SEARCH_ENGINE_WEB_MOJEEK_SUPPORTS_FULL_SEARCH` | `true` | Whether Mojeek supports fetching full webpage content beyond snippets. | SEARCH |
+| `search.engine.web.mojeek.use_in_auto_search` | `LDR_SEARCH_ENGINE_WEB_MOJEEK_USE_IN_AUTO_SEARCH` | `false` | Include Mojeek in auto search mode | SEARCH |
+| `search.engine.web.mojeek.weaknesses` | `LDR_SEARCH_ENGINE_WEB_MOJEEK_WEAKNESSES` | ``["smaller index than Google/Bing", "requires paid API key"]`` | Limitations of Mojeek: Smaller index than Google/Bing, requires paid API key. | SEARCH |
+| `search.engine.web.nasa_ads` | `LDR_SEARCH_ENGINE_WEB_NASA_ADS` | ``{"requires_api_key": true, "requires_llm": true, "description": "Search millions of astronomy and physics papers with natural language queries", "reliability": 0.95}`` | NASA Astrophysics Data System search engine configuration | SEARCH |
+| `search.engine.web.nasa_ads.api_key` | `LDR_SEARCH_ENGINE_WEB_NASA_ADS_API_KEY` | `` | NASA ADS API key (required). Get one at https://ui.adsabs.harvard.edu/user/settings/token | SEARCH |
+| `search.engine.web.nasa_ads.default_params.from_publication_date` | `LDR_SEARCH_ENGINE_WEB_NASA_ADS_DEFAULT_PARAMS_FROM_PUBLICATION_DATE` | `` | Only return papers published after this date (YYYY-MM-DD format, leave empty for all dates) | SEARCH |
+| `search.engine.web.nasa_ads.default_params.include_arxiv` | `LDR_SEARCH_ENGINE_WEB_NASA_ADS_DEFAULT_PARAMS_INCLUDE_ARXIV` | `true` | Include ArXiv preprints in search results | SEARCH |
+| `search.engine.web.nasa_ads.default_params.max_results` | `LDR_SEARCH_ENGINE_WEB_NASA_ADS_DEFAULT_PARAMS_MAX_RESULTS` | `25` | Maximum number of results to retrieve from NASA ADS | SEARCH |
+| `search.engine.web.nasa_ads.default_params.min_citations` | `LDR_SEARCH_ENGINE_WEB_NASA_ADS_DEFAULT_PARAMS_MIN_CITATIONS` | `0` | Minimum number of citations required (0 for no filter) | SEARCH |
+| `search.engine.web.nasa_ads.default_params.sort_by` | `LDR_SEARCH_ENGINE_WEB_NASA_ADS_DEFAULT_PARAMS_SORT_BY` | `relevance` | How to sort the search results | SEARCH |
+| `search.engine.web.nasa_ads.description` | `LDR_SEARCH_ENGINE_WEB_NASA_ADS_DESCRIPTION` | `NASA Astrophysics Data System - comprehensive database of astronomy, astrophysics, physics, and geophysics papers. Includes both ArXiv preprints and published papers.` | Human-readable description of the search engine. | SEARCH |
+| `search.engine.web.nasa_ads.display_name` | `LDR_SEARCH_ENGINE_WEB_NASA_ADS_DISPLAY_NAME` | `NASA ADS` | Display name to use in the U.I. for this search engine. | SEARCH |
+| `search.engine.web.nasa_ads.journal_reputation.enabled` | `LDR_SEARCH_ENGINE_WEB_NASA_ADS_JOURNAL_REPUTATION_ENABLED` | `true` | Enable journal quality filtering for this search engine. | SEARCH |
+| `search.engine.web.nasa_ads.reliability` | `LDR_SEARCH_ENGINE_WEB_NASA_ADS_RELIABILITY` | `0.95` | Reliability score for NASA ADS search results (0.0-1.0, higher is better) | SEARCH |
+| `search.engine.web.nasa_ads.requires_api_key` | `LDR_SEARCH_ENGINE_WEB_NASA_ADS_REQUIRES_API_KEY` | `true` | Whether NASA ADS requires an API key (always true for NASA ADS) | SEARCH |
+| `search.engine.web.nasa_ads.strengths` | `LDR_SEARCH_ENGINE_WEB_NASA_ADS_STRENGTHS` | ``["astronomy papers", "astrophysics research", "physics papers", "space science", "planetary science", "cosmology", "high energy physics", "historical papers", "citation tracking", "natural language queries"]`` | List of research areas where NASA ADS excels | SEARCH |
+| `search.engine.web.nasa_ads.weaknesses` | `LDR_SEARCH_ENGINE_WEB_NASA_ADS_WEAKNESSES` | ``["biomedical content", "computer science papers", "social sciences", "humanities", "requires API key for good performance"]`` | List of research areas where NASA ADS has limited coverage | SEARCH |
+| `search.engine.web.openalex` | `LDR_SEARCH_ENGINE_WEB_OPENALEX` | ``{"requires_api_key": false, "requires_llm": true, "description": "Search 250+ million academic papers with natural language queries", "reliability": 0.95}`` | OpenAlex academic search engine configuration | SEARCH |
+| `search.engine.web.openalex.default_params.filter_open_access` | `LDR_SEARCH_ENGINE_WEB_OPENALEX_DEFAULT_PARAMS_FILTER_OPEN_ACCESS` | `false` | Only return open access papers that are freely available | SEARCH |
+| `search.engine.web.openalex.default_params.from_publication_date` | `LDR_SEARCH_ENGINE_WEB_OPENALEX_DEFAULT_PARAMS_FROM_PUBLICATION_DATE` | `` | Only return papers published after this date (YYYY-MM-DD format, leave empty for all dates) | SEARCH |
+| `search.engine.web.openalex.default_params.max_results` | `LDR_SEARCH_ENGINE_WEB_OPENALEX_DEFAULT_PARAMS_MAX_RESULTS` | `25` | Maximum number of results to retrieve from OpenAlex | SEARCH |
+| `search.engine.web.openalex.default_params.min_citations` | `LDR_SEARCH_ENGINE_WEB_OPENALEX_DEFAULT_PARAMS_MIN_CITATIONS` | `0` | Minimum number of citations required (0 for no filter) | SEARCH |
+| `search.engine.web.openalex.default_params.sort_by` | `LDR_SEARCH_ENGINE_WEB_OPENALEX_DEFAULT_PARAMS_SORT_BY` | `relevance` | How to sort the search results | SEARCH |
+| `search.engine.web.openalex.description` | `LDR_SEARCH_ENGINE_WEB_OPENALEX_DESCRIPTION` | `Search 250+ million academic papers with natural language queries. Comprehensive scholarly database covering all disciplines.` | Human-readable description of the search engine. | SEARCH |
+| `search.engine.web.openalex.display_name` | `LDR_SEARCH_ENGINE_WEB_OPENALEX_DISPLAY_NAME` | `OpenAlex` | Display name to use in the U.I. for this search engine. | SEARCH |
+| `search.engine.web.openalex.email` | `LDR_SEARCH_ENGINE_WEB_OPENALEX_EMAIL` | `` | Email address for polite pool access (optional but recommended). Provides faster response times. | SEARCH |
+| `search.engine.web.openalex.journal_reputation.enabled` | `LDR_SEARCH_ENGINE_WEB_OPENALEX_JOURNAL_REPUTATION_ENABLED` | `true` | Enable journal quality filtering for this search engine. | SEARCH |
+| `search.engine.web.openalex.reliability` | `LDR_SEARCH_ENGINE_WEB_OPENALEX_RELIABILITY` | `0.95` | Reliability score for OpenAlex search results (0.0-1.0, higher is better) | SEARCH |
+| `search.engine.web.openalex.requires_api_key` | `LDR_SEARCH_ENGINE_WEB_OPENALEX_REQUIRES_API_KEY` | `false` | Whether OpenAlex requires an API key (always false for OpenAlex) | SEARCH |
+| `search.engine.web.openalex.strengths` | `LDR_SEARCH_ENGINE_WEB_OPENALEX_STRENGTHS` | ``["academic papers", "scholarly research", "natural language queries", "all disciplines", "citation analysis", "open access content", "comprehensive coverage", "no API key required", "high rate limits"]`` | List of research areas where OpenAlex excels | SEARCH |
+| `search.engine.web.openalex.use_in_auto_search` | `LDR_SEARCH_ENGINE_WEB_OPENALEX_USE_IN_AUTO_SEARCH` | `true` | Include OpenAlex in auto search mode | SEARCH |
+| `search.engine.web.openalex.weaknesses` | `LDR_SEARCH_ENGINE_WEB_OPENALEX_WEAKNESSES` | ``["non-academic content", "news articles", "general web content", "real-time information"]`` | List of content types where OpenAlex has limited coverage | SEARCH |
+| `search.engine.web.openlibrary` | `LDR_SEARCH_ENGINE_WEB_OPENLIBRARY` | ``{"requires_api_key": false, "requires_llm": true, "description": "Search 2M+ books with metadata, covers, and full text from Internet Archive.", "reliability": 0.9}`` | Open Library search engine configuration | SEARCH |
+| `search.engine.web.openlibrary.default_params.max_results` | `LDR_SEARCH_ENGINE_WEB_OPENLIBRARY_DEFAULT_PARAMS_MAX_RESULTS` | `20` | Maximum number of results to retrieve from Open Library. | SEARCH |
+| `search.engine.web.openlibrary.description` | `LDR_SEARCH_ENGINE_WEB_OPENLIBRARY_DESCRIPTION` | `Search 2M+ books with metadata, covers, and full text from Internet Archive.` | Human-readable description of the search engine. | SEARCH |
+| `search.engine.web.openlibrary.display_name` | `LDR_SEARCH_ENGINE_WEB_OPENLIBRARY_DISPLAY_NAME` | `Open Library` | Display name for Open Library search engine. | SEARCH |
+| `search.engine.web.openlibrary.reliability` | `LDR_SEARCH_ENGINE_WEB_OPENLIBRARY_RELIABILITY` | `0.9` | Reliability score for Open Library. | SEARCH |
+| `search.engine.web.openlibrary.requires_api_key` | `LDR_SEARCH_ENGINE_WEB_OPENLIBRARY_REQUIRES_API_KEY` | `false` | Open Library is free and does not require an API key. | SEARCH |
+| `search.engine.web.openlibrary.strengths` | `LDR_SEARCH_ENGINE_WEB_OPENLIBRARY_STRENGTHS` | ``["books", "literature", "metadata", "covers", "full text", "Internet Archive"]`` | Key strengths of Open Library. | SEARCH |
+| `search.engine.web.openlibrary.weaknesses` | `LDR_SEARCH_ENGINE_WEB_OPENLIBRARY_WEAKNESSES` | ``["books only", "variable metadata quality", "English focus"]`` | Limitations of Open Library. | SEARCH |
+| `search.engine.web.paperless.api_key` | `PAPERLESS_API_TOKEN` | `` | API key for authentication (get from Paperless admin panel) | SEARCH |
+| `search.engine.web.paperless.default_params.api_url` | `LDR_SEARCH_ENGINE_WEB_PAPERLESS_DEFAULT_PARAMS_API_URL` | `http://localhost:8000` | URL of your Paperless-ngx instance (e.g., http://localhost:8000) | SEARCH |
+| `search.engine.web.paperless.default_params.include_content` | `LDR_SEARCH_ENGINE_WEB_PAPERLESS_DEFAULT_PARAMS_INCLUDE_CONTENT` | `true` | Include full document content (not just snippets) | SEARCH |
+| `search.engine.web.paperless.default_params.max_results` | `LDR_SEARCH_ENGINE_WEB_PAPERLESS_DEFAULT_PARAMS_MAX_RESULTS` | `10` | Maximum number of documents to return | SEARCH |
+| `search.engine.web.paperless.default_params.timeout` | `LDR_SEARCH_ENGINE_WEB_PAPERLESS_DEFAULT_PARAMS_TIMEOUT` | `30` | Request timeout in seconds | SEARCH |
+| `search.engine.web.paperless.default_params.verify_ssl` | `LDR_SEARCH_ENGINE_WEB_PAPERLESS_DEFAULT_PARAMS_VERIFY_SSL` | `true` | Verify SSL certificates (disable for self-signed certs) | SEARCH |
+| `search.engine.web.paperless.description` | `LDR_SEARCH_ENGINE_WEB_PAPERLESS_DESCRIPTION` | `Search your personal Paperless-ngx document archive.` | Human-readable description of the search engine. | SEARCH |
+| `search.engine.web.paperless.display_name` | `LDR_SEARCH_ENGINE_WEB_PAPERLESS_DISPLAY_NAME` | `Paperless` | Display name to use in the U.I. for this search engine. | SEARCH |
+| `search.engine.web.paperless.enabled` | `LDR_SEARCH_ENGINE_WEB_PAPERLESS_ENABLED` | `false` | Enable or disable this search engine | SEARCH |
+| `search.engine.web.paperless.reliability` | `LDR_SEARCH_ENGINE_WEB_PAPERLESS_RELIABILITY` | `0.95` | Reliability score for this search engine | SEARCH |
+| `search.engine.web.paperless.requires_api_key` | `LDR_SEARCH_ENGINE_WEB_PAPERLESS_REQUIRES_API_KEY` | `true` | Whether this engine requires an API key | SEARCH |
+| `search.engine.web.paperless.requires_llm` | `LDR_SEARCH_ENGINE_WEB_PAPERLESS_REQUIRES_LLM` | `true` | Whether this engine requires an LLM for operation | SEARCH |
+| `search.engine.web.paperless.strengths` | `LDR_SEARCH_ENGINE_WEB_PAPERLESS_STRENGTHS` | ``["personal documents", "OCR-processed content", "tagged and organized", "full-text search", "no rate limits", "private data"]`` | Strengths of this search engine | SEARCH |
+| `search.engine.web.paperless.supports_full_search` | `LDR_SEARCH_ENGINE_WEB_PAPERLESS_SUPPORTS_FULL_SEARCH` | `true` | Whether this engine supports full document search | SEARCH |
+| `search.engine.web.paperless.weaknesses` | `LDR_SEARCH_ENGINE_WEB_PAPERLESS_WEAKNESSES` | ``["keyword-based search only", "no semantic understanding", "limited query operators"]`` | Weaknesses of this search engine | SEARCH |
+| `search.engine.web.pubchem` | `LDR_SEARCH_ENGINE_WEB_PUBCHEM` | ``{"requires_api_key": false, "requires_llm": true, "description": "Search millions of chemical compounds with structures, properties, and bioactivity data.", "reliability": 0.95}`` | PubChem search engine configuration | SEARCH |
+| `search.engine.web.pubchem.default_params.include_synonyms` | `LDR_SEARCH_ENGINE_WEB_PUBCHEM_DEFAULT_PARAMS_INCLUDE_SYNONYMS` | `true` | Include compound synonyms in full content results. | SEARCH |
+| `search.engine.web.pubchem.default_params.max_results` | `LDR_SEARCH_ENGINE_WEB_PUBCHEM_DEFAULT_PARAMS_MAX_RESULTS` | `10` | Maximum number of results to retrieve from PubChem. | SEARCH |
+| `search.engine.web.pubchem.description` | `LDR_SEARCH_ENGINE_WEB_PUBCHEM_DESCRIPTION` | `Search millions of chemical compounds with structures, properties, and bioactivity data.` | Human-readable description of the search engine. | SEARCH |
+| `search.engine.web.pubchem.display_name` | `LDR_SEARCH_ENGINE_WEB_PUBCHEM_DISPLAY_NAME` | `PubChem` | Display name for PubChem search engine. | SEARCH |
+| `search.engine.web.pubchem.reliability` | `LDR_SEARCH_ENGINE_WEB_PUBCHEM_RELIABILITY` | `0.95` | Reliability score for PubChem. | SEARCH |
+| `search.engine.web.pubchem.requires_api_key` | `LDR_SEARCH_ENGINE_WEB_PUBCHEM_REQUIRES_API_KEY` | `false` | PubChem is free and does not require an API key. | SEARCH |
+| `search.engine.web.pubchem.strengths` | `LDR_SEARCH_ENGINE_WEB_PUBCHEM_STRENGTHS` | ``["chemistry", "compounds", "drug information", "molecular properties", "bioactivity"]`` | Key strengths of PubChem. | SEARCH |
+| `search.engine.web.pubchem.weaknesses` | `LDR_SEARCH_ENGINE_WEB_PUBCHEM_WEAKNESSES` | ``["chemistry only", "technical data", "requires chemical knowledge"]`` | Limitations of PubChem. | SEARCH |
+| `search.engine.web.pubmed.api_key` | `LDR_SEARCH_ENGINE_WEB_PUBMED_API_KEY` | `` | The PubMed API key to use. | SEARCH |
+| `search.engine.web.pubmed.default_params.days_limit` | `LDR_SEARCH_ENGINE_WEB_PUBMED_DEFAULT_PARAMS_DAYS_LIMIT` | `0` | Restrict results to articles published within the last N days. Leave empty for no time restriction. | SEARCH |
+| `search.engine.web.pubmed.default_params.full_text_limit` | `LDR_SEARCH_ENGINE_WEB_PUBMED_DEFAULT_PARAMS_FULL_TEXT_LIMIT` | `3` | Maximum number of full-text articles to download from PubMed Central when get_full_text is enabled. | SEARCH |
+| `search.engine.web.pubmed.default_params.get_abstracts` | `LDR_SEARCH_ENGINE_WEB_PUBMED_DEFAULT_PARAMS_GET_ABSTRACTS` | `true` | Fetch article abstracts for search results. Recommended for understanding paper content. | SEARCH |
+| `search.engine.web.pubmed.default_params.get_full_text` | `LDR_SEARCH_ENGINE_WEB_PUBMED_DEFAULT_PARAMS_GET_FULL_TEXT` | `false` | Attempt to retrieve full-text articles from PubMed Central (PMC). Only works for open-access papers. | SEARCH |
+| `search.engine.web.pubmed.default_params.include_authors_in_context` | `LDR_SEARCH_ENGINE_WEB_PUBMED_DEFAULT_PARAMS_INCLUDE_AUTHORS_IN_CONTEXT` | `false` | Include author names in context for analysis | SEARCH |
+| `search.engine.web.pubmed.default_params.include_citation_in_context` | `LDR_SEARCH_ENGINE_WEB_PUBMED_DEFAULT_PARAMS_INCLUDE_CITATION_IN_CONTEXT` | `false` | Include volume, issue, and pages in context for analysis | SEARCH |
+| `search.engine.web.pubmed.default_params.include_doi_in_context` | `LDR_SEARCH_ENGINE_WEB_PUBMED_DEFAULT_PARAMS_INCLUDE_DOI_IN_CONTEXT` | `false` | Include DOI (Digital Object Identifier) in context for analysis | SEARCH |
+| `search.engine.web.pubmed.default_params.include_full_date_in_context` | `LDR_SEARCH_ENGINE_WEB_PUBMED_DEFAULT_PARAMS_INCLUDE_FULL_DATE_IN_CONTEXT` | `false` | Include full publication date in context for analysis | SEARCH |
+| `search.engine.web.pubmed.default_params.include_journal_in_context` | `LDR_SEARCH_ENGINE_WEB_PUBMED_DEFAULT_PARAMS_INCLUDE_JOURNAL_IN_CONTEXT` | `true` | Include journal name in context for analysis | SEARCH |
+| `search.engine.web.pubmed.default_params.include_keywords_in_context` | `LDR_SEARCH_ENGINE_WEB_PUBMED_DEFAULT_PARAMS_INCLUDE_KEYWORDS_IN_CONTEXT` | `true` | Include article keywords in context for analysis | SEARCH |
+| `search.engine.web.pubmed.default_params.include_language_in_context` | `LDR_SEARCH_ENGINE_WEB_PUBMED_DEFAULT_PARAMS_INCLUDE_LANGUAGE_IN_CONTEXT` | `false` | Include language indicator in context for analysis | SEARCH |
+| `search.engine.web.pubmed.default_params.include_mesh_terms_in_context` | `LDR_SEARCH_ENGINE_WEB_PUBMED_DEFAULT_PARAMS_INCLUDE_MESH_TERMS_IN_CONTEXT` | `true` | Include MeSH terms (medical subject headings) in context for analysis | SEARCH |
+| `search.engine.web.pubmed.default_params.include_pmc_availability_in_context` | `LDR_SEARCH_ENGINE_WEB_PUBMED_DEFAULT_PARAMS_INCLUDE_PMC_AVAILABILITY_IN_CONTEXT` | `false` | Include PMC free full text availability in context for analysis | SEARCH |
+| `search.engine.web.pubmed.default_params.include_pmid_in_context` | `LDR_SEARCH_ENGINE_WEB_PUBMED_DEFAULT_PARAMS_INCLUDE_PMID_IN_CONTEXT` | `false` | Include PubMed ID in context for analysis | SEARCH |
+| `search.engine.web.pubmed.default_params.include_publication_type_in_context` | `LDR_SEARCH_ENGINE_WEB_PUBMED_DEFAULT_PARAMS_INCLUDE_PUBLICATION_TYPE_IN_CONTEXT` | `true` | Include publication type (e.g., Clinical Trial, Meta-Analysis) in context for analysis | SEARCH |
+| `search.engine.web.pubmed.default_params.include_year_in_context` | `LDR_SEARCH_ENGINE_WEB_PUBMED_DEFAULT_PARAMS_INCLUDE_YEAR_IN_CONTEXT` | `true` | Include publication year in context for analysis | SEARCH |
+| `search.engine.web.pubmed.default_params.max_keywords` | `LDR_SEARCH_ENGINE_WEB_PUBMED_DEFAULT_PARAMS_MAX_KEYWORDS` | `3` | Maximum number of keywords to show (0 = all) | SEARCH |
+| `search.engine.web.pubmed.default_params.max_mesh_terms` | `LDR_SEARCH_ENGINE_WEB_PUBMED_DEFAULT_PARAMS_MAX_MESH_TERMS` | `3` | Maximum number of MeSH terms to show (0 = all) | SEARCH |
+| `search.engine.web.pubmed.default_params.max_results` | `LDR_SEARCH_ENGINE_WEB_PUBMED_DEFAULT_PARAMS_MAX_RESULTS` | `20` | Maximum number of PubMed articles to retrieve per search query. | SEARCH |
+| `search.engine.web.pubmed.default_params.optimize_queries` | `LDR_SEARCH_ENGINE_WEB_PUBMED_DEFAULT_PARAMS_OPTIMIZE_QUERIES` | `true` | Automatically optimize natural language queries for PubMed's specialized syntax (MeSH terms, Boolean operators). | SEARCH |
+| `search.engine.web.pubmed.description` | `LDR_SEARCH_ENGINE_WEB_PUBMED_DESCRIPTION` | `Search papers indexed by PubMed.` | Human-readable description of the search engine. | SEARCH |
+| `search.engine.web.pubmed.display_name` | `LDR_SEARCH_ENGINE_WEB_PUBMED_DISPLAY_NAME` | `PubMed` | Display name to use in the U.I. for this search engine. | SEARCH |
+| `search.engine.web.pubmed.reliability` | `LDR_SEARCH_ENGINE_WEB_PUBMED_RELIABILITY` | `0.98` | Reliability rating (0-1) for PubMed. Peer-reviewed biomedical literature is highly reliable. | SEARCH |
+| `search.engine.web.pubmed.requires_api_key` | `LDR_SEARCH_ENGINE_WEB_PUBMED_REQUIRES_API_KEY` | `false` | Whether PubMed requires an NCBI API key. Optional but recommended for higher rate limits. | SEARCH |
+| `search.engine.web.pubmed.requires_llm` | `LDR_SEARCH_ENGINE_WEB_PUBMED_REQUIRES_LLM` | `true` | Whether PubMed query optimization uses LLM. Helps convert natural language to effective PubMed queries. | SEARCH |
+| `search.engine.web.pubmed.strengths` | `LDR_SEARCH_ENGINE_WEB_PUBMED_STRENGTHS` | ``["biomedical literature", "medical research", "clinical studies", "life sciences", "health information", "scientific papers"]`` | Key advantages of PubMed: Peer-reviewed biomedical literature, MeSH indexing, abstracts, citation data, free access. | SEARCH |
+| `search.engine.web.pubmed.use_in_auto_search` | `LDR_SEARCH_ENGINE_WEB_PUBMED_USE_IN_AUTO_SEARCH` | `true` | Include PubMed in auto search mode | SEARCH |
+| `search.engine.web.pubmed.weaknesses` | `LDR_SEARCH_ENGINE_WEB_PUBMED_WEAKNESSES` | ``["non-medical topics", "very recent papers may be missing", "limited to published research"]`` | Limitations of PubMed: Limited to biomedical literature, keyword-based search, full-text only for open-access papers. | SEARCH |
+| `search.engine.web.scaleserp.api_key` | `LDR_SEARCH_ENGINE_WEB_SCALESERP_API_KEY` | `` | The ScaleSerp API key to use. Get yours at https://scaleserp.com | SEARCH |
+| `search.engine.web.scaleserp.default_params.device` | `LDR_SEARCH_ENGINE_WEB_SCALESERP_DEFAULT_PARAMS_DEVICE` | `desktop` | Device type for search results | SEARCH |
+| `search.engine.web.scaleserp.default_params.enable_cache` | `LDR_SEARCH_ENGINE_WEB_SCALESERP_DEFAULT_PARAMS_ENABLE_CACHE` | `true` | Enable 1-hour caching to save API credits (cached searches are FREE within 1 hour) | SEARCH |
+| `search.engine.web.scaleserp.default_params.include_full_content` | `LDR_SEARCH_ENGINE_WEB_SCALESERP_DEFAULT_PARAMS_INCLUDE_FULL_CONTENT` | `false` | Include full webpage content in results (may slow down search) | SEARCH |
+| `search.engine.web.scaleserp.default_params.language` | `LDR_SEARCH_ENGINE_WEB_SCALESERP_DEFAULT_PARAMS_LANGUAGE` | `en` | Language code for search results (e.g., 'en', 'es', 'fr') | SEARCH |
+| `search.engine.web.scaleserp.default_params.location` | `LDR_SEARCH_ENGINE_WEB_SCALESERP_DEFAULT_PARAMS_LOCATION` | `United States` | Location for localized results (e.g., 'United States', 'London,England,United Kingdom') | SEARCH |
+| `search.engine.web.scaleserp.default_params.safe_search` | `LDR_SEARCH_ENGINE_WEB_SCALESERP_DEFAULT_PARAMS_SAFE_SEARCH` | `true` | Enable or disable safe search filtering | SEARCH |
+| `search.engine.web.scaleserp.description` | `LDR_SEARCH_ENGINE_WEB_SCALESERP_DESCRIPTION` | `Cost-effective Google Search API with 1-hour free caching that can reduce costs by 40-70% for repeated searches.` | Human-readable description of the search engine. | SEARCH |
+| `search.engine.web.scaleserp.display_name` | `LDR_SEARCH_ENGINE_WEB_SCALESERP_DISPLAY_NAME` | `ScaleSerp` | Display name to use in the U.I. for this search engine. | SEARCH |
+| `search.engine.web.scaleserp.reliability` | `LDR_SEARCH_ENGINE_WEB_SCALESERP_RELIABILITY` | `0.95` | Reliability of the ScaleSerp search engine | SEARCH |
+| `search.engine.web.scaleserp.requires_api_key` | `LDR_SEARCH_ENGINE_WEB_SCALESERP_REQUIRES_API_KEY` | `true` | Whether this search engine requires an API key | SEARCH |
+| `search.engine.web.scaleserp.requires_llm` | `LDR_SEARCH_ENGINE_WEB_SCALESERP_REQUIRES_LLM` | `false` | Whether this search engine requires an LLM for relevance filtering | SEARCH |
+| `search.engine.web.scaleserp.strengths` | `LDR_SEARCH_ENGINE_WEB_SCALESERP_STRENGTHS` | ``["1-hour free caching for repeated searches", "High-quality Google search results", "Real-time results with knowledge graph", "Cached searches don't count against quota", "Rich snippets and related questions"]`` | Strengths of this search engine | SEARCH |
+| `search.engine.web.scaleserp.supports_full_search` | `LDR_SEARCH_ENGINE_WEB_SCALESERP_SUPPORTS_FULL_SEARCH` | `true` | Whether this search engine supports full document search | SEARCH |
+| `search.engine.web.scaleserp.weaknesses` | `LDR_SEARCH_ENGINE_WEB_SCALESERP_WEAKNESSES` | ``["Requires API key with usage limits", "Not specialized for academic content", "Cache only lasts 1 hour"]`` | Weaknesses of this search engine | SEARCH |
+| `search.engine.web.searxng.default_params.categories` | `LDR_SEARCH_ENGINE_WEB_SEARXNG_DEFAULT_PARAMS_CATEGORIES` | ``["general"]`` | SearXNG search categories to use (general, images, videos, news, music, files, etc.). | SEARCH |
+| `search.engine.web.searxng.default_params.delay_between_requests` | `SEARXNG_DELAY` | `0` | Seconds to wait between SearXNG requests. Set higher values to avoid rate limiting on public instances. | SEARCH |
+| `search.engine.web.searxng.default_params.include_full_content` | `LDR_SEARCH_ENGINE_WEB_SEARXNG_DEFAULT_PARAMS_INCLUDE_FULL_CONTENT` | `true` | Fetch full webpage content for SearXNG results, not just snippets. | SEARCH |
+| `search.engine.web.searxng.default_params.instance_url` | `LDR_SEARCH_ENGINE_WEB_SEARXNG_DEFAULT_PARAMS_INSTANCE_URL` | `http://localhost:8080` | The SearXNG API endpoint URL. | SEARCH |
+| `search.engine.web.searxng.default_params.language` | `LDR_SEARCH_ENGINE_WEB_SEARXNG_DEFAULT_PARAMS_LANGUAGE` | `en` | Language code for SearXNG results (e.g., 'en', 'de', 'fr'). | SEARCH |
+| `search.engine.web.searxng.default_params.max_results` | `LDR_SEARCH_ENGINE_WEB_SEARXNG_DEFAULT_PARAMS_MAX_RESULTS` | `15` | Maximum number of results to retrieve from SearXNG per search. | SEARCH |
+| `search.engine.web.searxng.default_params.safe_search` | `LDR_SEARCH_ENGINE_WEB_SEARXNG_DEFAULT_PARAMS_SAFE_SEARCH` | `OFF` | Configure the safe search level | SEARCH |
+| `search.engine.web.searxng.description` | `LDR_SEARCH_ENGINE_WEB_SEARXNG_DESCRIPTION` | `A locally-hosted meta-search engine.` | Human-readable description of the search engine. | SEARCH |
+| `search.engine.web.searxng.display_name` | `LDR_SEARCH_ENGINE_WEB_SEARXNG_DISPLAY_NAME` | `SearXNG (Locally-hosted)` | Display name to use in the U.I. for this search engine. | SEARCH |
+| `search.engine.web.searxng.reliability` | `LDR_SEARCH_ENGINE_WEB_SEARXNG_RELIABILITY` | `1.0` | Reliability rating (0-1) for SearXNG. Aggregates multiple search engines for broad coverage. | SEARCH |
+| `search.engine.web.searxng.requires_api_key` | `LDR_SEARCH_ENGINE_WEB_SEARXNG_REQUIRES_API_KEY` | `false` | Whether SearXNG requires an API key. No - SearXNG is self-hosted and free. | SEARCH |
+| `search.engine.web.searxng.strengths` | `LDR_SEARCH_ENGINE_WEB_SEARXNG_STRENGTHS` | ``["comprehensive general information", "current events and news", "technical documentation", "factual queries", "historical information", "consumer products", "educational content", "multi-source aggregation", "real-time results", "combined results from major search engines"]`` | Key advantages of SearXNG: Free, self-hosted, privacy-respecting, aggregates multiple engines, no API limits. | SEARCH |
+| `search.engine.web.searxng.supports_full_search` | `LDR_SEARCH_ENGINE_WEB_SEARXNG_SUPPORTS_FULL_SEARCH` | `true` | Whether SearXNG supports fetching full webpage content beyond snippets. | SEARCH |
+| `search.engine.web.searxng.use_in_auto_search` | `LDR_SEARCH_ENGINE_WEB_SEARXNG_USE_IN_AUTO_SEARCH` | `true` | Include SearXNG in auto search mode | SEARCH |
+| `search.engine.web.searxng.weaknesses` | `LDR_SEARCH_ENGINE_WEB_SEARXNG_WEAKNESSES` | ``["requires self-hosting", "depends on other search engines", "may be rate limited by underlying engines"]`` | Limitations of SearXNG: Requires self-hosting, public instances may have rate limits, result quality varies. | SEARCH |
+| `search.engine.web.semantic_scholar` | `LDR_SEARCH_ENGINE_WEB_SEMANTIC_SCHOLAR` | ``{"requires_api_key": false, "requires_llm": true, "description": "Search millions of academic papers across all scientific fields with AI-powered features", "reliability": 0.9}`` | Semantic Scholar search engine configuration | SEARCH |
+| `search.engine.web.semantic_scholar.api_key` | `LDR_SEARCH_ENGINE_WEB_SEMANTIC_SCHOLAR_API_KEY` | `` | Semantic Scholar API key (optional but recommended for higher rate limits). Get one at https://www.semanticscholar.org/product/api | SEARCH |
+| `search.engine.web.semantic_scholar.default_params.citation_limit` | `LDR_SEARCH_ENGINE_WEB_SEMANTIC_SCHOLAR_DEFAULT_PARAMS_CITATION_LIMIT` | `10` | Maximum number of citations to fetch per paper | SEARCH |
+| `search.engine.web.semantic_scholar.default_params.get_abstracts` | `LDR_SEARCH_ENGINE_WEB_SEMANTIC_SCHOLAR_DEFAULT_PARAMS_GET_ABSTRACTS` | `true` | Fetch full abstracts for papers | SEARCH |
+| `search.engine.web.semantic_scholar.default_params.get_citations` | `LDR_SEARCH_ENGINE_WEB_SEMANTIC_SCHOLAR_DEFAULT_PARAMS_GET_CITATIONS` | `false` | Fetch citation information for papers | SEARCH |
+| `search.engine.web.semantic_scholar.default_params.get_references` | `LDR_SEARCH_ENGINE_WEB_SEMANTIC_SCHOLAR_DEFAULT_PARAMS_GET_REFERENCES` | `false` | Fetch reference information for papers | SEARCH |
+| `search.engine.web.semantic_scholar.default_params.get_tldr` | `LDR_SEARCH_ENGINE_WEB_SEMANTIC_SCHOLAR_DEFAULT_PARAMS_GET_TLDR` | `true` | Fetch AI-generated TLDR summaries for papers | SEARCH |
+| `search.engine.web.semantic_scholar.default_params.max_results` | `LDR_SEARCH_ENGINE_WEB_SEMANTIC_SCHOLAR_DEFAULT_PARAMS_MAX_RESULTS` | `20` | Maximum number of results to retrieve | SEARCH |
+| `search.engine.web.semantic_scholar.default_params.optimize_queries` | `LDR_SEARCH_ENGINE_WEB_SEMANTIC_SCHOLAR_DEFAULT_PARAMS_OPTIMIZE_QUERIES` | `true` | Use LLM to optimize natural language queries for better search results | SEARCH |
+| `search.engine.web.semantic_scholar.default_params.reference_limit` | `LDR_SEARCH_ENGINE_WEB_SEMANTIC_SCHOLAR_DEFAULT_PARAMS_REFERENCE_LIMIT` | `10` | Maximum number of references to fetch per paper | SEARCH |
+| `search.engine.web.semantic_scholar.description` | `LDR_SEARCH_ENGINE_WEB_SEMANTIC_SCHOLAR_DESCRIPTION` | `AI-powered search for scientific literature. Features include TLDR summaries, citation graphs, and semantic search across millions of papers.` | Human-readable description of the search engine. | SEARCH |
+| `search.engine.web.semantic_scholar.display_name` | `LDR_SEARCH_ENGINE_WEB_SEMANTIC_SCHOLAR_DISPLAY_NAME` | `Semantic Scholar` | Display name to use in the U.I. for this search engine. | SEARCH |
+| `search.engine.web.semantic_scholar.journal_reputation.enabled` | `LDR_SEARCH_ENGINE_WEB_SEMANTIC_SCHOLAR_JOURNAL_REPUTATION_ENABLED` | `true` | Enable journal quality filtering for this search engine. | SEARCH |
+| `search.engine.web.semantic_scholar.reliability` | `LDR_SEARCH_ENGINE_WEB_SEMANTIC_SCHOLAR_RELIABILITY` | `0.9` | Reliability rating for the search engine | SEARCH |
+| `search.engine.web.semantic_scholar.requires_api_key` | `LDR_SEARCH_ENGINE_WEB_SEMANTIC_SCHOLAR_REQUIRES_API_KEY` | `false` | Whether this search engine requires an API key | SEARCH |
+| `search.engine.web.semantic_scholar.strengths` | `LDR_SEARCH_ENGINE_WEB_SEMANTIC_SCHOLAR_STRENGTHS` | ``["AI-powered features", "TLDR summaries", "citation graphs", "semantic search", "all scientific fields", "no API key required", "computer science focus", "paper influence metrics"]`` | Strengths of this search engine | SEARCH |
+| `search.engine.web.semantic_scholar.use_in_auto_search` | `LDR_SEARCH_ENGINE_WEB_SEMANTIC_SCHOLAR_USE_IN_AUTO_SEARCH` | `true` | Include Semantic Scholar in auto search mode | SEARCH |
+| `search.engine.web.semantic_scholar.weaknesses` | `LDR_SEARCH_ENGINE_WEB_SEMANTIC_SCHOLAR_WEAKNESSES` | ``["rate limits without API key", "not all papers have TLDR", "coverage varies by field", "newer papers may be missing"]`` | Weaknesses of this search engine | SEARCH |
+| `search.engine.web.serpapi.api_key` | `LDR_SEARCH_ENGINE_WEB_SERPAPI_API_KEY` | `` | The Serp API key to use. | SEARCH |
+| `search.engine.web.serpapi.default_params.region` | `LDR_SEARCH_ENGINE_WEB_SERPAPI_DEFAULT_PARAMS_REGION` | `us` | Geographic region for search results (e.g., 'us', 'uk'). Results prioritized from this region. | SEARCH |
+| `search.engine.web.serpapi.default_params.safe_search` | `LDR_SEARCH_ENGINE_WEB_SERPAPI_DEFAULT_PARAMS_SAFE_SEARCH` | `true` | Enable safe search to filter explicit content from results. | SEARCH |
+| `search.engine.web.serpapi.default_params.search_language` | `LDR_SEARCH_ENGINE_WEB_SERPAPI_DEFAULT_PARAMS_SEARCH_LANGUAGE` | `English` | Language for search queries and results. Filters results by language. | SEARCH |
+| `search.engine.web.serpapi.default_params.time_period` | `LDR_SEARCH_ENGINE_WEB_SERPAPI_DEFAULT_PARAMS_TIME_PERIOD` | `y` | Time filter for results: 'd' (day), 'w' (week), 'm' (month), 'y' (year), or empty for all time. | SEARCH |
+| `search.engine.web.serpapi.description` | `LDR_SEARCH_ENGINE_WEB_SERPAPI_DESCRIPTION` | `Search the web with Google's search API.` | Human-readable description of the search engine. | SEARCH |
+| `search.engine.web.serpapi.display_name` | `LDR_SEARCH_ENGINE_WEB_SERPAPI_DISPLAY_NAME` | `SerpApi` | Display name to use in the U.I. for this search engine. | SEARCH |
+| `search.engine.web.serpapi.reliability` | `LDR_SEARCH_ENGINE_WEB_SERPAPI_RELIABILITY` | `0.6` | Reliability score (0-1). SerpAPI rated moderate (0.6) - good for general search but less authoritative than specialized sources. | SEARCH |
+| `search.engine.web.serpapi.requires_api_key` | `LDR_SEARCH_ENGINE_WEB_SERPAPI_REQUIRES_API_KEY` | `true` | Indicates this engine requires a SerpAPI key. Get one at serpapi.com. | SEARCH |
+| `search.engine.web.serpapi.strengths` | `LDR_SEARCH_ENGINE_WEB_SERPAPI_STRENGTHS` | ``["comprehensive web search", "product information", "reviews", "recent content", "news", "broad coverage"]`` | Advantages: Comprehensive web search via Google SERP, good for products, reviews, news, and broad coverage. | SEARCH |
+| `search.engine.web.serpapi.supports_full_search` | `LDR_SEARCH_ENGINE_WEB_SERPAPI_SUPPORTS_FULL_SEARCH` | `true` | Indicates this engine can fetch full page content after initial search, not just snippets. | SEARCH |
+| `search.engine.web.serpapi.use_in_auto_search` | `LDR_SEARCH_ENGINE_WEB_SERPAPI_USE_IN_AUTO_SEARCH` | `false` | Include SerpAPI in auto search mode | SEARCH |
+| `search.engine.web.serpapi.weaknesses` | `LDR_SEARCH_ENGINE_WEB_SERPAPI_WEAKNESSES` | ``["requires API key with usage limits", "not specialized for academic content"]`` | Limitations: Requires paid API key with usage limits, not specialized for academic or technical content. | SEARCH |
+| `search.engine.web.serper.api_key` | `LDR_SEARCH_ENGINE_WEB_SERPER_API_KEY` | `` | The Serper API key to use. Get yours at https://serper.dev | SEARCH |
+| `search.engine.web.serper.default_params.include_full_content` | `LDR_SEARCH_ENGINE_WEB_SERPER_DEFAULT_PARAMS_INCLUDE_FULL_CONTENT` | `false` | Include full webpage content in results (may slow down search) | SEARCH |
+| `search.engine.web.serper.default_params.region` | `LDR_SEARCH_ENGINE_WEB_SERPER_DEFAULT_PARAMS_REGION` | `us` | Country code for localized results (e.g., 'us', 'gb', 'fr') | SEARCH |
+| `search.engine.web.serper.default_params.safe_search` | `LDR_SEARCH_ENGINE_WEB_SERPER_DEFAULT_PARAMS_SAFE_SEARCH` | `true` | Enable or disable safe search filtering | SEARCH |
+| `search.engine.web.serper.default_params.search_language` | `LDR_SEARCH_ENGINE_WEB_SERPER_DEFAULT_PARAMS_SEARCH_LANGUAGE` | `en` | Language code for search results (e.g., 'en', 'es', 'fr') | SEARCH |
+| `search.engine.web.serper.default_params.time_period` | `LDR_SEARCH_ENGINE_WEB_SERPER_DEFAULT_PARAMS_TIME_PERIOD` | `null` | Filter results by time period ('day', 'week', 'month', 'year', or null for all time) | SEARCH |
+| `search.engine.web.serper.description` | `LDR_SEARCH_ENGINE_WEB_SERPER_DESCRIPTION` | `Fast and affordable Google Search API with real-time results.` | Human-readable description of the search engine. | SEARCH |
+| `search.engine.web.serper.display_name` | `LDR_SEARCH_ENGINE_WEB_SERPER_DISPLAY_NAME` | `Serper` | Display name to use in the U.I. for this search engine. | SEARCH |
+| `search.engine.web.serper.reliability` | `LDR_SEARCH_ENGINE_WEB_SERPER_RELIABILITY` | `0.95` | Reliability of the Serper search engine | SEARCH |
+| `search.engine.web.serper.requires_api_key` | `LDR_SEARCH_ENGINE_WEB_SERPER_REQUIRES_API_KEY` | `true` | Whether this search engine requires an API key | SEARCH |
+| `search.engine.web.serper.requires_llm` | `LDR_SEARCH_ENGINE_WEB_SERPER_REQUIRES_LLM` | `false` | Whether this search engine requires an LLM for relevance filtering | SEARCH |
+| `search.engine.web.serper.strengths` | `LDR_SEARCH_ENGINE_WEB_SERPER_STRENGTHS` | ``["Lightning-fast response times (1-2 seconds)", "Very affordable pricing", "High-quality Google search results", "Real-time results", "Knowledge graph and related searches", "People Also Ask data", "Simple pay-per-use pricing"]`` | Strengths of this search engine | SEARCH |
+| `search.engine.web.serper.supports_full_search` | `LDR_SEARCH_ENGINE_WEB_SERPER_SUPPORTS_FULL_SEARCH` | `true` | Whether this search engine supports full document search | SEARCH |
+| `search.engine.web.serper.weaknesses` | `LDR_SEARCH_ENGINE_WEB_SERPER_WEAKNESSES` | ``["Requires API key with usage limits", "Not specialized for academic content"]`` | Weaknesses of this search engine | SEARCH |
+| `search.engine.web.sofya.api_key` | `LDR_SEARCH_ENGINE_WEB_SOFYA_API_KEY` | `` | The Sofya API key to use. Get yours at https://sofya.co/dashboard (GitHub sign-up grants 1000 free credits/month). | SEARCH |
+| `search.engine.web.sofya.default_params.freshness` | `LDR_SEARCH_ENGINE_WEB_SOFYA_DEFAULT_PARAMS_FRESHNESS` | `null` | Filter results by recency ('day', 'week', 'month', 'year', or null for all time). | SEARCH |
+| `search.engine.web.sofya.default_params.include_full_content` | `LDR_SEARCH_ENGINE_WEB_SOFYA_DEFAULT_PARAMS_INCLUDE_FULL_CONTENT` | `true` | Fetch extracted page content. On uses Sofya 'basic' depth (content, 3 credits); off uses cheaper 'snippets' depth (1 credit). | SEARCH |
+| `search.engine.web.sofya.default_params.topic` | `LDR_SEARCH_ENGINE_WEB_SOFYA_DEFAULT_PARAMS_TOPIC` | `general` | Search topic. 'general' for broad web search, 'news' for current events. | SEARCH |
+| `search.engine.web.sofya.description` | `LDR_SEARCH_ENGINE_WEB_SOFYA_DESCRIPTION` | `Hosted web-search API for AI agents. Returns ranked results with SERP snippets and extracted page content (markdown) in a single call.` | Human-readable description of the search engine. | SEARCH |
+| `search.engine.web.sofya.display_name` | `LDR_SEARCH_ENGINE_WEB_SOFYA_DISPLAY_NAME` | `Sofya` | Display name to use in the U.I. for this search engine. | SEARCH |
+| `search.engine.web.sofya.reliability` | `LDR_SEARCH_ENGINE_WEB_SOFYA_RELIABILITY` | `0.85` | Reliability of the Sofya search engine | SEARCH |
+| `search.engine.web.sofya.requires_api_key` | `LDR_SEARCH_ENGINE_WEB_SOFYA_REQUIRES_API_KEY` | `true` | Whether this search engine requires an API key | SEARCH |
+| `search.engine.web.sofya.requires_llm` | `LDR_SEARCH_ENGINE_WEB_SOFYA_REQUIRES_LLM` | `false` | Whether this search engine requires an LLM for relevance filtering | SEARCH |
+| `search.engine.web.sofya.strengths` | `LDR_SEARCH_ENGINE_WEB_SOFYA_STRENGTHS` | ``["Returns SERP snippets and extracted page content (markdown) in one call", "Domain include/exclude filtering and recency filtering", "General and news topics", "1000 free credits/month for GitHub sign-ups", "Built for AI agents"]`` | Strengths of this search engine | SEARCH |
+| `search.engine.web.sofya.supports_full_search` | `LDR_SEARCH_ENGINE_WEB_SOFYA_SUPPORTS_FULL_SEARCH` | `true` | Whether this search engine supports full document search | SEARCH |
+| `search.engine.web.sofya.use_in_auto_search` | `LDR_SEARCH_ENGINE_WEB_SOFYA_USE_IN_AUTO_SEARCH` | `false` | Whether to include this engine in auto/meta search selection | SEARCH |
+| `search.engine.web.sofya.weaknesses` | `LDR_SEARCH_ENGINE_WEB_SOFYA_WEAKNESSES` | ``["Requires API key with credit-based usage limits", "Not specialized for academic content"]`` | Weaknesses of this search engine | SEARCH |
+| `search.engine.web.stackexchange` | `LDR_SEARCH_ENGINE_WEB_STACKEXCHANGE` | ``{"requires_api_key": false, "requires_llm": true, "description": "Search Stack Overflow and other Q&A sites for programming answers.", "reliability": 0.9}`` | Stack Exchange search engine configuration | SEARCH |
+| `search.engine.web.stackexchange.default_params.max_results` | `LDR_SEARCH_ENGINE_WEB_STACKEXCHANGE_DEFAULT_PARAMS_MAX_RESULTS` | `20` | Maximum number of results to retrieve from Stack Exchange. | SEARCH |
+| `search.engine.web.stackexchange.default_params.site` | `LDR_SEARCH_ENGINE_WEB_STACKEXCHANGE_DEFAULT_PARAMS_SITE` | `stackoverflow` | Stack Exchange site to search (stackoverflow, serverfault, superuser, etc.). | SEARCH |
+| `search.engine.web.stackexchange.description` | `LDR_SEARCH_ENGINE_WEB_STACKEXCHANGE_DESCRIPTION` | `Search Stack Overflow and other Q&A sites for programming answers.` | Human-readable description of the search engine. | SEARCH |
+| `search.engine.web.stackexchange.display_name` | `LDR_SEARCH_ENGINE_WEB_STACKEXCHANGE_DISPLAY_NAME` | `Stack Exchange` | Display name for Stack Exchange search engine. | SEARCH |
+| `search.engine.web.stackexchange.reliability` | `LDR_SEARCH_ENGINE_WEB_STACKEXCHANGE_RELIABILITY` | `0.9` | Reliability score for Stack Exchange. | SEARCH |
+| `search.engine.web.stackexchange.requires_api_key` | `LDR_SEARCH_ENGINE_WEB_STACKEXCHANGE_REQUIRES_API_KEY` | `false` | Stack Exchange is free without an API key (300 requests/day). | SEARCH |
+| `search.engine.web.stackexchange.strengths` | `LDR_SEARCH_ENGINE_WEB_STACKEXCHANGE_STRENGTHS` | ``["programming", "Q&A", "code examples", "community moderated", "multiple sites"]`` | Key strengths of Stack Exchange. | SEARCH |
+| `search.engine.web.stackexchange.weaknesses` | `LDR_SEARCH_ENGINE_WEB_STACKEXCHANGE_WEAKNESSES` | ``["technical focus", "rate limited", "Q&A format only"]`` | Limitations of Stack Exchange. | SEARCH |
+| `search.engine.web.tavily.api_key` | `LDR_SEARCH_ENGINE_WEB_TAVILY_API_KEY` | `` | The Tavily API key to use. | SEARCH |
+| `search.engine.web.tavily.default_params.include_full_content` | `LDR_SEARCH_ENGINE_WEB_TAVILY_DEFAULT_PARAMS_INCLUDE_FULL_CONTENT` | `true` | Include full webpage content in results | SEARCH |
+| `search.engine.web.tavily.default_params.search_depth` | `LDR_SEARCH_ENGINE_WEB_TAVILY_DEFAULT_PARAMS_SEARCH_DEPTH` | `basic` | Search depth - basic for speed, advanced for quality | SEARCH |
+| `search.engine.web.tavily.description` | `LDR_SEARCH_ENGINE_WEB_TAVILY_DESCRIPTION` | `AI-powered search engine optimized for research with built-in answer extraction.` | Human-readable description of the search engine. | SEARCH |
+| `search.engine.web.tavily.display_name` | `LDR_SEARCH_ENGINE_WEB_TAVILY_DISPLAY_NAME` | `Tavily` | Display name to use in the U.I. for this search engine. | SEARCH |
+| `search.engine.web.tavily.reliability` | `LDR_SEARCH_ENGINE_WEB_TAVILY_RELIABILITY` | `0.8` | Reliability score (0-1). Tavily rated high (0.8) - AI-optimized search designed for research applications. | SEARCH |
+| `search.engine.web.tavily.requires_api_key` | `LDR_SEARCH_ENGINE_WEB_TAVILY_REQUIRES_API_KEY` | `true` | Indicates this engine requires a Tavily API key. Get one at tavily.com. | SEARCH |
+| `search.engine.web.tavily.strengths` | `LDR_SEARCH_ENGINE_WEB_TAVILY_STRENGTHS` | ``["AI-powered search optimization", "built-in answer extraction", "research-focused results", "high-quality content filtering", "fast response times"]`` | Advantages: AI-powered search optimization, built-in answer extraction, research-focused results, fast responses. | SEARCH |
+| `search.engine.web.tavily.supports_full_search` | `LDR_SEARCH_ENGINE_WEB_TAVILY_SUPPORTS_FULL_SEARCH` | `true` | Indicates this engine can fetch full page content after initial search, not just snippets. | SEARCH |
+| `search.engine.web.tavily.use_in_auto_search` | `LDR_SEARCH_ENGINE_WEB_TAVILY_USE_IN_AUTO_SEARCH` | `false` | Include Tavily in auto search mode | SEARCH |
+| `search.engine.web.tavily.weaknesses` | `LDR_SEARCH_ENGINE_WEB_TAVILY_WEAKNESSES` | ``["requires API key with usage limits", "newer service with smaller historical data"]`` | Limitations: Requires paid API key with usage limits, newer service with less historical data coverage. | SEARCH |
+| `search.engine.web.tinyfish.api_key` | `LDR_SEARCH_ENGINE_WEB_TINYFISH_API_KEY` | `` | The TinyFish API key to use. Can also be supplied with LDR_SEARCH_ENGINE_WEB_TINYFISH_API_KEY. | SEARCH |
+| `search.engine.web.tinyfish.default_params.fetch_format` | `LDR_SEARCH_ENGINE_WEB_TINYFISH_DEFAULT_PARAMS_FETCH_FORMAT` | `markdown` | Output format for fetched page content. | SEARCH |
+| `search.engine.web.tinyfish.default_params.include_full_content` | `LDR_SEARCH_ENGINE_WEB_TINYFISH_DEFAULT_PARAMS_INCLUDE_FULL_CONTENT` | `true` | Fetch browser-rendered page content for search results. | SEARCH |
+| `search.engine.web.tinyfish.default_params.language` | `LDR_SEARCH_ENGINE_WEB_TINYFISH_DEFAULT_PARAMS_LANGUAGE` | `en` | Language code for search results. | SEARCH |
+| `search.engine.web.tinyfish.default_params.location` | `LDR_SEARCH_ENGINE_WEB_TINYFISH_DEFAULT_PARAMS_LOCATION` | `US` | Country code for geo-targeted results. | SEARCH |
+| `search.engine.web.tinyfish.description` | `LDR_SEARCH_ENGINE_WEB_TINYFISH_DESCRIPTION` | `Fast hosted web search with optional browser-rendered content extraction.` | Human-readable description of the search engine. | SEARCH |
+| `search.engine.web.tinyfish.display_name` | `LDR_SEARCH_ENGINE_WEB_TINYFISH_DISPLAY_NAME` | `TinyFish` | Display name to use in the U.I. for this search engine. | SEARCH |
+| `search.engine.web.tinyfish.reliability` | `LDR_SEARCH_ENGINE_WEB_TINYFISH_RELIABILITY` | `0.85` | Reliability score (0-1) for TinyFish search. | SEARCH |
+| `search.engine.web.tinyfish.requires_api_key` | `LDR_SEARCH_ENGINE_WEB_TINYFISH_REQUIRES_API_KEY` | `true` | Whether this search engine requires an API key. | SEARCH |
+| `search.engine.web.tinyfish.requires_llm` | `LDR_SEARCH_ENGINE_WEB_TINYFISH_REQUIRES_LLM` | `false` | Whether this search engine requires an LLM. | SEARCH |
+| `search.engine.web.tinyfish.strengths` | `LDR_SEARCH_ENGINE_WEB_TINYFISH_STRENGTHS` | ``["fast structured web search", "browser-rendered content extraction", "clean markdown output for LLM consumption", "batch fetches multiple URLs"]`` | Strengths of this search engine. | SEARCH |
+| `search.engine.web.tinyfish.supports_full_search` | `LDR_SEARCH_ENGINE_WEB_TINYFISH_SUPPORTS_FULL_SEARCH` | `true` | Whether this search engine can return extracted page content. | SEARCH |
+| `search.engine.web.tinyfish.use_in_auto_search` | `LDR_SEARCH_ENGINE_WEB_TINYFISH_USE_IN_AUTO_SEARCH` | `false` | Include TinyFish in auto search mode | SEARCH |
+| `search.engine.web.tinyfish.weaknesses` | `LDR_SEARCH_ENGINE_WEB_TINYFISH_WEAKNESSES` | ``["requires hosted API key", "external network requests leave the local environment", "fetch latency depends on target page rendering"]`` | Weaknesses of this search engine. | SEARCH |
+| `search.engine.web.wayback.default_params.closest_only` | `LDR_SEARCH_ENGINE_WEB_WAYBACK_DEFAULT_PARAMS_CLOSEST_ONLY` | `false` | When enabled, only return the snapshot closest to the target date instead of all available snapshots. | SEARCH |
+| `search.engine.web.wayback.default_params.language` | `LDR_SEARCH_ENGINE_WEB_WAYBACK_DEFAULT_PARAMS_LANGUAGE` | `English` | Preferred language for archived content when multiple versions are available. | SEARCH |
+| `search.engine.web.wayback.default_params.max_results` | `LDR_SEARCH_ENGINE_WEB_WAYBACK_DEFAULT_PARAMS_MAX_RESULTS` | `15` | Maximum number of archived snapshots to return per search query. | SEARCH |
+| `search.engine.web.wayback.default_params.max_snapshots_per_url` | `LDR_SEARCH_ENGINE_WEB_WAYBACK_DEFAULT_PARAMS_MAX_SNAPSHOTS_PER_URL` | `3` | Maximum historical snapshots to retrieve for each URL. Lower values are faster; higher shows more history. | SEARCH |
+| `search.engine.web.wayback.description` | `LDR_SEARCH_ENGINE_WEB_WAYBACK_DESCRIPTION` | `Search the Internet Archive's Wayback Machine.` | Human-readable description of the search engine. | SEARCH |
+| `search.engine.web.wayback.display_name` | `LDR_SEARCH_ENGINE_WEB_WAYBACK_DISPLAY_NAME` | `Wayback` | Display name to use in the U.I. for this search engine. | SEARCH |
+| `search.engine.web.wayback.reliability` | `LDR_SEARCH_ENGINE_WEB_WAYBACK_RELIABILITY` | `0.5` | Reliability score (0-1). Wayback rated moderate (0.5) - great for historical content but archives may be incomplete. | SEARCH |
+| `search.engine.web.wayback.requires_api_key` | `LDR_SEARCH_ENGINE_WEB_WAYBACK_REQUIRES_API_KEY` | `false` | Wayback Machine is free and does not require an API key. | SEARCH |
+| `search.engine.web.wayback.strengths` | `LDR_SEARCH_ENGINE_WEB_WAYBACK_STRENGTHS` | ``["historical web content", "archived websites", "content verification", "deleted or changed web pages", "website evolution tracking"]`` | Advantages: Access to historical web content, archived/deleted pages, content verification, and website evolution. | SEARCH |
+| `search.engine.web.wayback.supports_full_search` | `LDR_SEARCH_ENGINE_WEB_WAYBACK_SUPPORTS_FULL_SEARCH` | `true` | Indicates this engine can fetch full archived page content, not just metadata. | SEARCH |
+| `search.engine.web.wayback.use_in_auto_search` | `LDR_SEARCH_ENGINE_WEB_WAYBACK_USE_IN_AUTO_SEARCH` | `false` | Include Wayback in auto search mode | SEARCH |
+| `search.engine.web.wayback.weaknesses` | `LDR_SEARCH_ENGINE_WEB_WAYBACK_WEAKNESSES` | ``["limited to previously archived content", "may miss recent changes", "archiving quality varies"]`` | Limitations: Only finds previously archived content, may miss recent changes, archive quality varies by site. | SEARCH |
+| `search.engine.web.wikinews.adaptive_search` | `LDR_SEARCH_ENGINE_WEB_WIKINEWS_ADAPTIVE_SEARCH` | `true` | Automatically adjust the search period based on the query type (recent or historical) to improve news search relevance | SEARCH |
+| `search.engine.web.wikinews.description` | `LDR_SEARCH_ENGINE_WEB_WIKINEWS_DESCRIPTION` | `Search news articles written by volunteers with verified sources` | Human-readable description of the search engine. | SEARCH |
+| `search.engine.web.wikinews.display_name` | `LDR_SEARCH_ENGINE_WEB_WIKINEWS_DISPLAY_NAME` | `Wikinews` | Display name to use in the U.I. for this search engine. | SEARCH |
+| `search.engine.web.wikinews.reliability` | `LDR_SEARCH_ENGINE_WEB_WIKINEWS_RELIABILITY` | `0.85` | Reliability score (0-1). Wikinews rated moderately high (0.85) - volunteer-written but with verified sources. | SEARCH |
+| `search.engine.web.wikinews.requires_api_key` | `LDR_SEARCH_ENGINE_WEB_WIKINEWS_REQUIRES_API_KEY` | `false` | Wikinews is free and does not require an API key. | SEARCH |
+| `search.engine.web.wikinews.strengths` | `LDR_SEARCH_ENGINE_WEB_WIKINEWS_STRENGTHS` | ``["recent and historical news", "verified sources", "general coverage", "quick overviews"]`` | Advantages: Recent and historical news coverage, verified sources, general topic coverage, quick overviews. | SEARCH |
+| `search.engine.web.wikinews.use_in_auto_search` | `LDR_SEARCH_ENGINE_WEB_WIKINEWS_USE_IN_AUTO_SEARCH` | `true` | Include Wikinews in auto search mode | SEARCH |
+| `search.engine.web.wikinews.weaknesses` | `LDR_SEARCH_ENGINE_WEB_WIKINEWS_WEAKNESSES` | ``["less coverage of specialized topics", "limited editorial oversight", "variable article quality", "not ideal for breaking news"]`` | Limitations: Less coverage of specialized topics, variable article quality, not ideal for breaking news. | SEARCH |
+| `search.engine.web.wikipedia.default_params.include_content` | `LDR_SEARCH_ENGINE_WEB_WIKIPEDIA_DEFAULT_PARAMS_INCLUDE_CONTENT` | `true` | Include full article content in results, not just titles and summaries. | SEARCH |
+| `search.engine.web.wikipedia.default_params.max_results` | `LDR_SEARCH_ENGINE_WEB_WIKIPEDIA_DEFAULT_PARAMS_MAX_RESULTS` | `20` | Maximum number of Wikipedia articles to return per search query. | SEARCH |
+| `search.engine.web.wikipedia.description` | `LDR_SEARCH_ENGINE_WEB_WIKIPEDIA_DESCRIPTION` | `Search Wikipedia articles` | Human-readable description of the search engine. | SEARCH |
+| `search.engine.web.wikipedia.display_name` | `LDR_SEARCH_ENGINE_WEB_WIKIPEDIA_DISPLAY_NAME` | `Wikipedia` | Display name to use in the U.I. for this search engine. | SEARCH |
+| `search.engine.web.wikipedia.reliability` | `LDR_SEARCH_ENGINE_WEB_WIKIPEDIA_RELIABILITY` | `0.95` | Reliability score (0-1). Wikipedia rated very high (0.95) - well-curated, cited content with editorial review. | SEARCH |
+| `search.engine.web.wikipedia.requires_api_key` | `LDR_SEARCH_ENGINE_WEB_WIKIPEDIA_REQUIRES_API_KEY` | `false` | Wikipedia is free and does not require an API key. | SEARCH |
+| `search.engine.web.wikipedia.strengths` | `LDR_SEARCH_ENGINE_WEB_WIKIPEDIA_STRENGTHS` | ``["factual information", "general knowledge", "definitions", "historical facts", "biographies", "overview information"]`` | Advantages: Factual information, general knowledge, definitions, historical facts, biographies, and overviews. | SEARCH |
+| `search.engine.web.wikipedia.use_in_auto_search` | `LDR_SEARCH_ENGINE_WEB_WIKIPEDIA_USE_IN_AUTO_SEARCH` | `true` | Include Wikipedia in auto search mode | SEARCH |
+| `search.engine.web.wikipedia.weaknesses` | `LDR_SEARCH_ENGINE_WEB_WIKIPEDIA_WEAKNESSES` | ``["recent events", "specialized academic topics", "product comparisons"]`` | Limitations: May lag on recent events, less depth on specialized academic topics, no product comparisons. | SEARCH |
+| `search.engine.web.zenodo` | `LDR_SEARCH_ENGINE_WEB_ZENODO` | ``{"requires_api_key": false, "requires_llm": true, "description": "Search millions of research datasets, software, and publications with DOIs.", "reliability": 0.95}`` | Zenodo search engine configuration | SEARCH |
+| `search.engine.web.zenodo.default_params.max_results` | `LDR_SEARCH_ENGINE_WEB_ZENODO_DEFAULT_PARAMS_MAX_RESULTS` | `20` | Maximum number of results to retrieve from Zenodo. | SEARCH |
+| `search.engine.web.zenodo.description` | `LDR_SEARCH_ENGINE_WEB_ZENODO_DESCRIPTION` | `Search millions of research datasets, software, and publications with DOIs.` | Human-readable description of the search engine. | SEARCH |
+| `search.engine.web.zenodo.display_name` | `LDR_SEARCH_ENGINE_WEB_ZENODO_DISPLAY_NAME` | `Zenodo` | Display name for Zenodo search engine. | SEARCH |
+| `search.engine.web.zenodo.reliability` | `LDR_SEARCH_ENGINE_WEB_ZENODO_RELIABILITY` | `0.95` | Reliability score for Zenodo. | SEARCH |
+| `search.engine.web.zenodo.requires_api_key` | `LDR_SEARCH_ENGINE_WEB_ZENODO_REQUIRES_API_KEY` | `false` | Zenodo is free and does not require an API key for search. | SEARCH |
+| `search.engine.web.zenodo.strengths` | `LDR_SEARCH_ENGINE_WEB_ZENODO_STRENGTHS` | ``["research data", "datasets", "DOIs", "software", "open access", "CERN"]`` | Key strengths of Zenodo. | SEARCH |
+| `search.engine.web.zenodo.weaknesses` | `LDR_SEARCH_ENGINE_WEB_ZENODO_WEAKNESSES` | ``["research focus", "variable quality", "metadata only"]`` | Limitations of Zenodo. | SEARCH |
+| `search.favorites` | `LDR_SEARCH_FAVORITES` | ``["arxiv", "searxng", "library", "openalex"]`` | List of favorite search engine IDs that appear at the top of the dropdown. Edit as JSON array, e.g. ["arxiv", "brave"] | SEARCH |
+| `search.fetch.mode` | `LDR_SEARCH_FETCH_MODE` | `summary_focus_query` | Controls the LangGraph agent's fetch_content tool. 'summary_focus_query' (default) asks the LLM to extract only spans relevant to a focus question the agent supplies per call AND the original research query, so vague focus can be disambiguated — keeps small-model context clean. 'summary_focus' is the same but uses only the per-call focus question. 'full' returns the entire extracted page text (legacy behavior; can flood small-model context with boilerplate/metadata and degrade benchmark accuracy). 'disabled' removes the tool entirely so the agent works from search snippets only — useful for diagnosing whether fetching is helping or hurting accuracy. | SEARCH |
+| `search.final_max_results` | `LDR_SEARCH_FINAL_MAX_RESULTS` | `100` | Maximum unique sources to include in the final research output. Applied once after all search iterations complete, deduplicating and filtering across all accumulated results. | SEARCH |
+| `search.iterations` | `LDR_SEARCH_ITERATIONS` | `3` | Number of search-and-refine cycles to run. Each iteration generates follow-up questions based on previous results to progressively deepen the research. More iterations yield better coverage but increase search and LLM API usage. | SEARCH |
+| `search.journal_reputation.enable_llm_scoring` | `LDR_SEARCH_JOURNAL_REPUTATION_ENABLE_LLM_SCORING` | `false` | Enable Tier 4 LLM journal scoring via SearXNG. When disabled (default), only bundled data (OpenAlex, DOAJ, predatory lists) is used for scoring journal reputation. Tier 4 adds significant latency (1 SearXNG search + 1 LLM call per unknown journal) and only helps for journals outside the 217K-row reference DB. Requires SearXNG to be available. | SEARCH |
+| `search.journal_reputation.exclude_non_published` | `LDR_SEARCH_JOURNAL_REPUTATION_EXCLUDE_NON_PUBLISHED` | `false` | When enabled, excludes results without a formal journal publication reference. This ensures only sources with documented journal publications are included in analysis. | SEARCH |
+| `search.journal_reputation.max_context` | `LDR_SEARCH_JOURNAL_REPUTATION_MAX_CONTEXT` | `3000` | Maximum characters of source content to include when the LLM evaluates journal quality. Larger values provide more context but use more tokens. | SEARCH |
+| `search.journal_reputation.reanalysis_period` | `LDR_SEARCH_JOURNAL_REPUTATION_REANALYSIS_PERIOD` | `365` | How often to refresh cached journal quality assessments. After this period, the LLM will re-evaluate the journal's reputation. | SEARCH |
+| `search.journal_reputation.threshold` | `LDR_SEARCH_JOURNAL_REPUTATION_THRESHOLD` | `2` | Minimum journal quality score required for academic results. Sources below this threshold are excluded when the filter is enabled. Scores emitted: 1 predatory, 4 default/unknown, 5-6 moderate, 7-8 strong, 10 elite (values 2, 3, 9 are reserved but never assigned). Default 2 = drop only predatory; raise to 5 to also drop default/unknown venues; raise to 7+ to keep only strong/elite journals. | SEARCH |
+| `search.max_filtered_results` | `LDR_SEARCH_MAX_FILTERED_RESULTS` | `20` | Per-query limit on results kept after LLM relevance filtering. Raw results (controlled by 'Max Results') are filtered for relevance, then this setting caps how many are kept. Only applies when relevance filtering is active for the search engine. | SEARCH |
+| `search.max_results` | `LDR_SEARCH_MAX_RESULTS` | `50` | Maximum raw search results to retrieve from the search engine per query, before any relevance filtering is applied. | SEARCH |
+| `search.quality_check_urls` | `LDR_SEARCH_QUALITY_CHECK_URLS` | `true` | Validate that fetched webpage content matches the search result's title and snippet. Helps filter out broken links or mismatched content. | SEARCH |
+| `search.question_context_limit` | `LDR_SEARCH_QUESTION_CONTEXT_LIMIT` | `30` | Maximum number of previous search results provided as context when generating follow-up questions in subsequent iterations. Lower values reduce context size and focus on top results. Set to 0 for unlimited. | SEARCH |
+| `search.questions_per_iteration` | `LDR_SEARCH_QUESTIONS_PER_ITERATION` | `1` | Number of search queries to generate and execute per iteration. More questions increase coverage but also increase API calls and processing time. | SEARCH |
+| `search.rag.enable_relevance_filter` | `LDR_SEARCH_RAG_ENABLE_RELEVANCE_FILTER` | `true` | Apply an LLM relevance filter to local document / collection (RAG) results. Embeddings already rank by similarity, so this is optional; enabling it prunes semantically-near-but-irrelevant chunks at the cost of one extra LLM call per search. | SEARCH |
+| `search.rag.max_results` | `LDR_SEARCH_RAG_MAX_RESULTS` | `20` | Maximum results (embedding chunks) the local document / collection (RAG) search returns per query. Capped independently of 'Max Results' because vector search returns many near-duplicate chunks that can flood the agent's context across research iterations. | SEARCH |
+| `search.region` | `LDR_SEARCH_REGION` | `us` | Geographic region for search results localization. Affects which results are prioritized based on regional relevance. | SEARCH |
+| `search.safe_search` | `LDR_SEARCH_SAFE_SEARCH` | `true` | Filter out adult and inappropriate content from search results. Passed to search engines that support content filtering. | SEARCH |
+| `search.search_language` | `LDR_SEARCH_SEARCH_LANGUAGE` | `English` | Preferred language for search results. Search engines will attempt to return and prioritize results in this language when available. | SEARCH |
+| `search.search_strategy` | `LDR_SEARCH_SEARCH_STRATEGY` | `langgraph-agent` | Research methodology. 'source-based': Finds and extracts from sources (good for comprehensive research). 'focused-iteration': Uses entity-based progressive exploration (optimized for factual Q&A, ~95% SimpleQA accuracy, requires larger context windows). 'focused-iteration-standard': Comprehensive variant of focused-iteration with broader exploration. 'topic-organization': Organizes research around discovered topic clusters. 'langgraph-agent': Autonomous agentic research where the LLM decides what to search and when to synthesize. | SEARCH |
+| `search.searches_per_section` | `LDR_SEARCH_SEARCHES_PER_SECTION` | `2` | Number of independent searches per report section when generating structured reports. Increases coverage but also increases API usage. | SEARCH |
+| `search.skip_relevance_filter` | `LDR_SEARCH_SKIP_RELEVANCE_FILTER` | `true` | When checked, LLM-based relevance filtering is suppressed for general-purpose search engines that do not require it. Keyword-based engines (arXiv, PubMed, Wikipedia, GitHub, etc.) always run the filter regardless of this checkbox; per-engine settings override both. | SEARCH |
+| `search.snippets_only` | `LDR_SEARCH_SNIPPETS_ONLY` | `true` | Only retrieve search snippets instead of full webpage content. Reduces API usage and processing time, but provides less context for analysis. | SEARCH |
+| `search.time_period` | `LDR_SEARCH_TIME_PERIOD` | `y` | Time range filter for search results. Limits results to content published within the specified period (day, week, month, year, or all time). | SEARCH |
+| `search.tool` | `LDR_SEARCH_TOOL` | `searxng` | Search engine to use for research queries. The default langgraph-agent strategy can additionally pick specialized engines per query as tools. Each engine may require its own API key or service to be configured separately. | SEARCH |
+| `security.account_lockout_duration_minutes` | `LDR_SECURITY_ACCOUNT_LOCKOUT_DURATION_MINUTES` | `15` | Duration in minutes that an account remains locked after exceeding the failed login threshold. Only configurable via environment variable (LDR_SECURITY_ACCOUNT_LOCKOUT_DURATION_MINUTES). | APP |
+| `security.account_lockout_threshold` | `LDR_SECURITY_ACCOUNT_LOCKOUT_THRESHOLD` | `10` | Number of consecutive failed login attempts before an account is temporarily locked out. Only configurable via environment variable (LDR_SECURITY_ACCOUNT_LOCKOUT_THRESHOLD). | APP |
+| `security.rate_limit_default` | `LDR_SECURITY_RATE_LIMIT_DEFAULT` | `5000 per hour;50000 per day` | Default rate limit for all HTTP endpoints. Configured via LDR_SECURITY_RATE_LIMIT_DEFAULT environment variable. Multiple limits can be separated by semicolons (e.g., '5000 per hour;50000 per day'). Format: 'N per hour/minute/day'. Requires server restart. | APP |
+| `security.rate_limit_login` | `LDR_SECURITY_RATE_LIMIT_LOGIN` | `5 per 15 minutes` | Rate limit for login attempts to prevent brute force attacks. Configured via LDR_SECURITY_RATE_LIMIT_LOGIN environment variable (e.g., '5 per 15 minutes'). Requires server restart. | APP |
+| `security.rate_limit_registration` | `LDR_SECURITY_RATE_LIMIT_REGISTRATION` | `3 per hour` | Rate limit for registration attempts to prevent spam. Configured via LDR_SECURITY_RATE_LIMIT_REGISTRATION environment variable (e.g., '3 per hour'). Requires server restart. | APP |
+| `security.rate_limit_settings` | `LDR_SECURITY_RATE_LIMIT_SETTINGS` | `30 per minute` | Rate limit for settings modification endpoints. Configured via LDR_SECURITY_RATE_LIMIT_SETTINGS environment variable. Requires server restart. | APP |
+| `security.rate_limit_upload_ip` | `LDR_SECURITY_RATE_LIMIT_UPLOAD_IP` | `60 per minute;1000 per hour` | Per-IP rate limit for file upload endpoints (applies to anonymous and authenticated requests). Configured via LDR_SECURITY_RATE_LIMIT_UPLOAD_IP environment variable. Multiple limits separated by semicolons. Requires server restart. | APP |
+| `security.rate_limit_upload_user` | `LDR_SECURITY_RATE_LIMIT_UPLOAD_USER` | `60 per minute;1000 per hour` | Per-authenticated-user rate limit for file upload endpoints (e.g., RAG collection uploads). Configured via LDR_SECURITY_RATE_LIMIT_UPLOAD_USER environment variable. Multiple limits separated by semicolons (e.g., '60 per minute;1000 per hour'). Requires server restart. | APP |
+| `security.session_remember_me_days` | `LDR_SECURITY_SESSION_REMEMBER_ME_DAYS` | `30` | Number of days a 'Remember Me' session remains valid before requiring re-login. | APP |
+| `security.session_timeout_hours` | `LDR_SECURITY_SESSION_TIMEOUT_HOURS` | `2` | Session timeout in hours for sessions without 'Remember Me' checked. Only applies to internal session validation (browser sessions expire on close). | APP |
+| `web.enable_javascript_rendering` | `LDR_WEB_ENABLE_JAVASCRIPT_RENDERING` | `false` | When enabled, falls back to a headless browser (Crawl4AI/Playwright) for pages that need JavaScript to render. Requires Chromium to be installed - the default Docker production image does NOT include it. To enable: install Chromium in the runtime environment (e.g. 'playwright install --with-deps chromium'), then toggle this setting. Note: in our limited (mostly accidental) benchmark comparisons - between dev instances that happened to have Chromium installed and routine Docker runs that did not - JS rendering did not measurably improve research quality, and most of our regular benchmark runs are on Docker without Chromium anyway. Leave disabled unless you have a specific need to scrape JavaScript-only sites. | APP |
+| `web.host` | `LDR_WEB_HOST` | `0.0.0.0` | The host address to listen on. Configured via LDR_WEB_HOST environment variable. Requires server restart. | APP |
+| `web.port` | `LDR_WEB_PORT` | `5000` | The port to listen on. Configured via LDR_WEB_PORT environment variable. Requires server restart. | APP |
+| `web.use_https` | `LDR_WEB_USE_HTTPS` | `true` | Whether to enable HTTPS for the web interface. Configured via LDR_WEB_USE_HTTPS environment variable. Requires server restart. | APP |
+| `zotero.api_key` | `LDR_ZOTERO_API_KEY` | `` | Your Zotero API key. Create one at https://www.zotero.org/settings/keys (it needs read access to the library you want to import). Not needed if 'Use Local Zotero' is enabled. | APP |
+| `zotero.auto_sync_enabled` | `LDR_ZOTERO_AUTO_SYNC_ENABLED` | `false` | Periodically re-sync the configured collections in the background (downloads new items, updates changed ones, and removes items deleted in Zotero). Requires the Zotero integration to be enabled. | APP |
+| `zotero.collection_keys` | `LDR_ZOTERO_COLLECTION_KEYS` | `` | Comma-separated Zotero collection keys to import (e.g. ABCD1234). Leave empty to import the entire library. Use the 'List collections' button on the Zotero page to find keys. | APP |
+| `zotero.enabled` | `LDR_ZOTERO_ENABLED` | `true` | Enable the Zotero integration (on by default; nothing runs until an API key is configured). Turn off to pause all Zotero syncing without removing your credentials. | APP |
+| `zotero.import_annotations` | `LDR_ZOTERO_IMPORT_ANNOTATIONS` | `false` | Also import your Zotero PDF annotations (highlights + comments) and child notes, appended to the document's searchable text. Adds extra API calls per item, so it's off by default. Note: annotation/note edits that don't change the parent item are picked up only when the item itself changes. | APP |
+| `zotero.import_items_without_pdf` | `LDR_ZOTERO_IMPORT_ITEMS_WITHOUT_PDF` | `false` | Import items that have no attached PDF, using only their title/abstract/metadata as a text-only document. Off by default and generally discouraged: without full text, research over these documents is shallow — prefer attaching the PDF in Zotero. If enabled later, applies to already-skipped items on the next manual Sync now. | APP |
+| `zotero.import_tags` | `LDR_ZOTERO_IMPORT_TAGS` | `` | Only import items tagged with ALL of these comma-separated Zotero tags (case-insensitive). Leave empty to import everything. This filters what is imported; changing it does not remove items already imported. | APP |
+| `zotero.library_id` | `LDR_ZOTERO_LIBRARY_ID` | `` | Numeric library ID. Optional for personal libraries — it is detected automatically from the API key (a pasted username is resolved too). Required for group libraries: the group's numeric ID (use the group picker, or the number in the group's URL). | APP |
+| `zotero.library_type` | `LDR_ZOTERO_LIBRARY_TYPE` | `user` | Whether to read from your personal user library or a shared group library. | APP |
+| `zotero.pdf_storage_mode` | `LDR_ZOTERO_PDF_STORAGE_MODE` | `none` | How to store imported PDFs. 'none' (default) stores only the extracted, indexed text — all that research needs, and much smaller. 'database' additionally keeps the full PDF inside your encrypted database so you can open the original. | APP |
+| `zotero.sync_interval_minutes` | `LDR_ZOTERO_SYNC_INTERVAL_MINUTES` | `360` | How often the background auto-sync runs (minutes). | APP |
+| `zotero.use_local_api` | `LDR_ZOTERO_USE_LOCAL_API` | `false` | Sync from the Zotero desktop app running on this machine (local API at localhost:23119) instead of the cloud Web API. No API key needed, and it can read locally-stored / linked-file attachments the cloud API can't. Requires Zotero 7+ with the local API enabled in its preferences, and Zotero running during sync. Caution on shared/multi-user hosts: the desktop app's local API is unauthenticated, so any user or process on the server able to reach localhost:23119 can read that Zotero library. | APP |
+
+*Generated by scripts/generate_config_docs.py*

docs/CUSTOM_LLM_INTEGRATION.md

@@ -0,0 +1,276 @@
+# Custom LLM Integration Guide
+
+Local Deep Research now supports seamless integration with custom LangChain LLMs, allowing you to use your own language models, specialized wrappers, or third-party LLM providers alongside the built-in options.
+
+## Overview
+
+Similar to the custom retriever support, LDR allows you to register any LangChain-compatible LLM and use it throughout the system. This enables:
+
+- Using proprietary or fine-tuned models
+- Implementing custom retry logic or preprocessing
+- Integrating with LLM providers not built into LDR
+- Testing with mock LLMs
+- Creating specialized model configurations
+
+## Quick Start
+
+```python
+from local_deep_research.api import quick_summary
+
+# Option 1: Pass an LLM instance
+result = quick_summary(
+    query="Your research question",
+    llms={"my_model": your_llm_instance},
+    provider="my_model"  # Use your custom LLM
+)
+
+# Option 2: Pass a factory function
+def create_llm(model_name=None, temperature=0.7, **kwargs):
+    return YourCustomLLM(model=model_name, temp=temperature)
+
+result = quick_summary(
+    query="Your research question",
+    llms={"custom": create_llm},
+    provider="custom",
+    model_name="gpt-turbo",  # Passed to factory
+    temperature=0.5
+)
+```
+
+## Requirements
+
+Your custom LLM must:
+1. Inherit from `langchain_core.language_models.BaseChatModel`
+2. Implement the required methods (`_generate`, `_llm_type`)
+3. Handle the standard LangChain message formats
+
+## Example Implementation
+
+```python
+from langchain_core.language_models import BaseChatModel
+from langchain_core.messages import AIMessage
+from langchain_core.outputs import ChatResult, ChatGeneration
+from typing import List, Optional, Any
+
+class CustomLLM(BaseChatModel):
+    """Example custom LLM implementation."""
+
+    def __init__(self, api_key: str, model_name: str = "custom-v1", **kwargs):
+        super().__init__(**kwargs)
+        self.api_key = api_key
+        self.model_name = model_name
+
+    def _generate(
+        self,
+        messages: List[BaseMessage],
+        stop: Optional[List[str]] = None,
+        run_manager: Optional[Any] = None,
+        **kwargs: Any
+    ) -> ChatResult:
+        """Generate a response from your model."""
+        # Call your API/model here
+        response = your_api_call(messages, self.model_name, self.api_key)
+
+        # Convert to LangChain format
+        message = AIMessage(content=response.text)
+        generation = ChatGeneration(message=message)
+
+        return ChatResult(generations=[generation])
+
+    @property
+    def _llm_type(self) -> str:
+        """Return identifier for this LLM."""
+        return "custom"
+```
+
+## Using with Different Research Modes
+
+### Quick Summary
+```python
+from local_deep_research.api import quick_summary
+
+result = quick_summary(
+    query="Explain quantum computing",
+    llms={"quantum_expert": quantum_llm},
+    provider="quantum_expert",
+    search_tool="arxiv"  # Search scientific papers
+)
+```
+
+### Detailed Research
+```python
+from local_deep_research.api import detailed_research
+
+result = detailed_research(
+    query="Climate change impacts",
+    llms={"climate_model": climate_specialized_llm},
+    provider="climate_model",
+    iterations=3
+)
+```
+
+### Report Generation
+```python
+from local_deep_research.api import generate_report
+
+report = generate_report(
+    query="AI in healthcare",
+    llms={"medical_ai": medical_llm},
+    provider="medical_ai",
+    output_file="healthcare_ai_report.md"
+)
+```
+
+## Advanced Usage
+
+### Multiple Custom LLMs
+
+Register multiple LLMs for different purposes:
+
+```python
+llms = {
+    "technical": TechnicalWriterLLM(temperature=0.2),
+    "creative": CreativeWriterLLM(temperature=0.9),
+    "fact_checker": FactCheckingLLM(temperature=0.0)
+}
+
+# Use technical LLM for precise analysis
+result = quick_summary(
+    query="How do transformers work?",
+    llms=llms,
+    provider="technical"
+)
+```
+
+### Factory Functions with Configuration
+
+```python
+def create_configured_llm(model_name=None, temperature=0.7, max_retries=3, **kwargs):
+    """Factory that creates LLM with retry logic."""
+    base_llm = YourLLM(model=model_name, temperature=temperature)
+    return RetryWrapper(base_llm, max_retries=max_retries)
+
+result = quick_summary(
+    query="Your question",
+    llms={"retry_llm": create_configured_llm},
+    provider="retry_llm",
+    model_name="your-model-v2",
+    max_retries=5  # Custom parameter
+)
+```
+
+### Combining Custom LLMs and Retrievers
+
+```python
+result = quick_summary(
+    query="Internal policy on remote work",
+    llms={"company_llm": company_fine_tuned_llm},
+    retrievers={"company_docs": company_retriever},
+    provider="company_llm",
+    search_tool="company_docs"
+)
+```
+
+## Implementation Details
+
+### How It Works
+
+1. **Registration**: When you pass LLMs via the `llms` parameter, they are registered in a global registry
+2. **Provider Check**: When creating an LLM, the system first checks if the provider name matches a registered custom LLM
+3. **Factory Support**: If the registered LLM is callable, it's treated as a factory and called with the provided parameters
+4. **Wrapping**: All LLMs (custom and built-in) are wrapped with think-tag removal and token counting
+
+### Thread Safety
+
+The LLM registry is thread-safe, allowing concurrent usage in multi-threaded applications.
+
+### Scope
+
+Registered LLMs are available globally within the Python process. They persist until explicitly unregistered or the process ends.
+
+## Best Practices
+
+1. **Consistent Naming**: Use clear, descriptive names for your custom LLMs
+2. **Error Handling**: Implement proper error handling in your LLM's `_generate` method
+3. **Token Counting**: If your LLM supports token counting, implement the appropriate methods
+4. **Temperature Handling**: Respect the temperature parameter for consistency
+5. **Async Support**: Implement async methods if your LLM supports asynchronous operation
+
+## Common Use Cases
+
+### Fine-tuned Models
+```python
+# Use your fine-tuned model for domain-specific research
+fine_tuned_llm = CustomLLM(
+    model_path="/path/to/fine-tuned-model",
+    domain="medical"
+)
+
+result = quick_summary(
+    query="Latest treatments for condition X",
+    llms={"medical_expert": fine_tuned_llm},
+    provider="medical_expert"
+)
+```
+
+### Mock LLMs for Testing
+```python
+class MockLLM(BaseChatModel):
+    """Returns predefined responses for testing."""
+
+    def _generate(self, messages, **kwargs):
+        # Return test data
+        return ChatResult(generations=[
+            ChatGeneration(message=AIMessage(content="Test response"))
+        ])
+
+# Use in tests
+result = quick_summary(
+    query="Test query",
+    llms={"mock": MockLLM()},
+    provider="mock",
+    search_tool="none"  # Disable search for pure testing
+)
+```
+
+### Rate-Limited Wrapper
+```python
+class RateLimitedLLM(BaseChatModel):
+    """Adds rate limiting to any LLM."""
+
+    def __init__(self, base_llm, requests_per_minute=10):
+        super().__init__()
+        self.base_llm = base_llm
+        self.rate_limiter = RateLimiter(requests_per_minute)
+
+    def _generate(self, messages, **kwargs):
+        self.rate_limiter.wait_if_needed()
+        return self.base_llm._generate(messages, **kwargs)
+```
+
+## Troubleshooting
+
+### LLM Not Found
+If you get "Invalid provider" errors:
+- Ensure you're passing the `llms` parameter to the API function
+- Check that the provider name matches exactly (case-insensitive)
+- Verify your LLM instance is properly initialized
+
+### Parameter Passing
+When using factory functions:
+- Standard parameters (model_name, temperature) are passed automatically
+- Custom parameters can be passed via kwargs
+- The factory receives all parameters from the API call
+
+### Compatibility Issues
+Ensure your LLM:
+- Inherits from `BaseChatModel`
+- Returns proper `ChatResult` objects
+- Handles the LangChain message format
+
+## Related Documentation
+
+- [API Documentation](api-quickstart.md)
+- [Configuration Guide](env_configuration.md)
+- [Full Configuration Reference](CONFIGURATION.md)
+- [LangChain Retriever Integration](LANGCHAIN_RETRIEVER_INTEGRATION.md)

docs/FRONTEND_BUILD_SYSTEM.md

@@ -0,0 +1,302 @@
+# Frontend Build System Guide
+
+## Overview
+
+Local Deep Research uses a modern frontend build system with **npm** for package management and **Vite** for bundling. This ensures all JavaScript libraries and CSS frameworks are served locally without any external CDN dependencies.
+
+## Quick Start
+
+### Development Mode
+
+1. **Install dependencies** (first time only):
+   ```bash
+   npm install
+   ```
+
+2. **Start the Flask server**:
+   ```bash
+   cd src
+   python -m local_deep_research.web.app
+   ```
+
+3. **Start Vite dev server** (in a separate terminal):
+   ```bash
+   npm run dev
+   ```
+
+   Vite will start on http://localhost:5173 with Hot Module Replacement (HMR) for instant updates.
+
+### Production Mode
+
+1. **Build production assets**:
+   ```bash
+   npm run build
+   ```
+
+   This creates optimized bundles in `src/local_deep_research/web/static/dist/`
+
+2. **Start Flask normally**:
+   ```bash
+   cd src
+   python -m local_deep_research.web.app
+   ```
+
+   Flask will automatically serve the built assets from the `dist/` folder.
+
+## Architecture
+
+### Why No CDNs?
+
+- **Security**: All code is audited and served from your server
+- **Privacy**: No user data leaks to third-party CDNs
+- **Reliability**: Works offline, no dependency on external services
+- **Performance**: Assets are optimized and cached locally
+- **Compliance**: Important for enterprise/government deployments
+
+### Technology Stack
+
+- **npm**: Package manager for JavaScript dependencies
+- **Vite**: Fast build tool with instant HMR in development
+- **Flask**: Python web framework serving the application
+
+## Dependencies
+
+All frontend dependencies are managed in `package.json`:
+
+| Library | Purpose | License |
+|---------|---------|---------|
+| `@fortawesome/fontawesome-free` | Icons throughout the UI | Font Awesome Free |
+| `bootstrap` | CSS framework for some pages | MIT |
+| `bootstrap-icons` | Additional icons | MIT |
+| `chart.js` | Analytics charts | MIT |
+| `highlight.js` | Code syntax highlighting | BSD-3-Clause |
+| `marked` | Markdown rendering | MIT |
+| `socket.io-client` | Real-time updates | MIT |
+| `jspdf` & `html2canvas` | PDF export | MIT |
+
+## File Structure
+
+```
+.
+├── package.json                 # npm dependencies and scripts
+├── vite.config.js              # Vite configuration
+├── node_modules/               # Downloaded packages (git-ignored)
+└── src/local_deep_research/web/
+    ├── static/
+    │   ├── dist/              # Production build output (git-ignored)
+    │   │   ├── css/           # Bundled CSS
+    │   │   ├── fonts/         # Font files (Font Awesome, Bootstrap Icons)
+    │   │   └── js/            # Bundled JavaScript
+    │   ├── js/
+    │   │   ├── app.js         # Main entry point importing all dependencies
+    │   │   └── ...            # Other application JavaScript
+    │   └── css/
+    │       └── styles.css     # Application styles
+    ├── templates/
+    │   └── base.html          # Uses Vite helper for asset loading
+    └── utils/
+        └── vite_helper.py     # Flask-Vite integration
+
+```
+
+## Common Tasks
+
+### Update Dependencies
+
+```bash
+# Update all packages to latest versions
+npm update
+
+# Check for security vulnerabilities
+npm audit
+
+# Auto-fix vulnerabilities (if possible)
+npm audit fix
+
+# Rebuild after updates
+npm run build
+```
+
+### Add a New Library
+
+1. Install the package:
+   ```bash
+   npm install library-name
+   ```
+
+2. Import in `src/local_deep_research/web/static/js/app.js`:
+   ```javascript
+   import 'library-name/dist/library.css';  // If it has CSS
+   import Library from 'library-name';      // Import JS
+
+   // Make available globally if needed
+   window.Library = Library;
+   ```
+
+3. Rebuild:
+   ```bash
+   npm run build
+   ```
+
+### Debug Build Issues
+
+```bash
+# Clean install (removes node_modules and reinstalls)
+rm -rf node_modules package-lock.json
+npm install
+
+# Verbose build output
+npm run build -- --debug
+
+# Check what's included in the bundle
+npm run build -- --sourcemap
+```
+
+## Security
+
+### Automated Checks
+
+- **Pre-commit Hook**: `.pre-commit-hooks/check-external-resources.py` prevents CDN references
+- **GitHub Dependabot**: Enable it to get automated PRs for security updates
+- **npm audit**: Run in CI/CD pipeline to catch vulnerabilities
+
+### Manual Security Audit
+
+```bash
+# Check for known vulnerabilities
+npm audit
+
+# See dependency tree
+npm list
+
+# Check outdated packages
+npm outdated
+```
+
+## Troubleshooting
+
+### Icons Not Showing
+
+**Problem**: Font Awesome or Bootstrap icons appear as squares
+
+**Solution**:
+1. Ensure fonts were built: Check `src/local_deep_research/web/static/dist/fonts/`
+2. Rebuild if missing: `npm run build`
+3. Clear browser cache: Ctrl+F5 (or Cmd+Shift+R on Mac)
+
+### JavaScript Not Loading
+
+**Problem**: Libraries like marked or Chart.js not working
+
+**Solution**:
+1. Check browser console for errors
+2. Verify npm packages installed: `npm install`
+3. Rebuild assets: `npm run build`
+4. Check Flask is serving from dist: Look for `.vite/manifest.json`
+
+### Vite Dev Server Issues
+
+**Problem**: HMR not working or connection refused
+
+**Solution**:
+1. Ensure Vite is running: `npm run dev`
+2. Check port 5173 is free: `lsof -i :5173`
+3. Verify Flask debug mode is on for development
+
+### Build Errors
+
+**Problem**: `npm run build` fails
+
+**Solution**:
+1. Check Node.js version: `node --version` (should be 16+ or 18+)
+2. Clear cache and reinstall:
+   ```bash
+   rm -rf node_modules package-lock.json
+   npm cache clean --force
+   npm install
+   ```
+3. Check for conflicting global packages: `npm list -g --depth=0`
+
+## Development Tips
+
+### Using Vite Dev Server
+
+When developing, Vite provides:
+- **Instant updates**: Changes appear immediately without page refresh
+- **Better errors**: Build errors shown in browser
+- **Fast startup**: No bundling needed during development
+
+### Production Optimization
+
+Vite automatically:
+- Minifies JavaScript and CSS
+- Tree-shakes unused code
+- Splits code into chunks
+- Generates sourcemaps for debugging
+- Optimizes images and fonts
+- Creates cache-busting hashes
+
+### Flask Integration
+
+The `ViteHelper` class (`src/local_deep_research/web/utils/vite_helper.py`) handles:
+- Loading from Vite dev server in development
+- Loading built assets in production
+- Fallback if build hasn't run yet
+
+## CI/CD Integration
+
+Add to your CI pipeline:
+
+```yaml
+# Example GitHub Actions
+- name: Setup Node.js
+  uses: actions/setup-node@v3
+  with:
+    node-version: '24'
+    cache: 'npm'
+
+- name: Install dependencies
+  run: npm ci
+
+- name: Security audit
+  run: npm audit --audit-level=high
+
+- name: Build assets
+  run: npm run build
+
+- name: Check for external resources
+  run: python .pre-commit-hooks/check-external-resources.py
+```
+
+## Migration from CDNs
+
+If you're updating from an older version that used CDNs:
+
+1. **Pull latest changes**
+2. **Install npm dependencies**: `npm install`
+3. **Build assets**: `npm run build`
+4. **Clear browser cache**: CDN resources may be cached
+5. **Test thoroughly**: Ensure all features work offline
+
+## Contributing
+
+When adding frontend features:
+
+1. **No external resources**: All assets must be in npm packages
+2. **Update package.json**: Add new dependencies properly
+3. **Import in app.js**: Ensure libraries are imported
+4. **Test the build**: Run `npm run build` before committing
+5. **Document changes**: Update this guide if needed
+
+## Support
+
+- **Build issues**: Check Node.js version and reinstall packages
+- **Runtime issues**: Check browser console and Flask logs
+- **Security concerns**: Run `npm audit` and update packages
+- **Performance**: Vite automatically optimizes; check bundle size with `npm run build`
+
+---
+
+*Last updated: August 2024*
+*Vite version: 5.x*
+*npm version: 10.x*

docs/LANGCHAIN_RETRIEVER_INTEGRATION.md

@@ -0,0 +1,164 @@
+# LangChain Retriever Integration
+
+LDR now supports using any LangChain retriever as a search engine. This allows you to use vector stores, databases, or any custom retriever implementation with LDR's research capabilities.
+
+## Quick Start
+
+```python
+from langchain.vectorstores import FAISS
+from langchain.embeddings import OpenAIEmbeddings
+from local_deep_research.api import quick_summary
+
+# Create your retriever (any LangChain retriever works)
+embeddings = OpenAIEmbeddings()
+vectorstore = FAISS.from_documents(documents, embeddings)
+retriever = vectorstore.as_retriever()
+
+# Use with LDR
+result = quick_summary(
+    query="What are our deployment procedures?",
+    retrievers={"company_kb": retriever},
+    search_tool="company_kb"
+)
+```
+
+## How It Works
+
+1. Pass retrievers as a dictionary to any research function
+2. Each retriever gets a name (the dictionary key)
+3. Use the retriever by setting `search_tool` to its name
+4. Retrievers work exactly like built-in search engines
+
+## Usage Examples
+
+### Single Retriever
+
+```python
+result = quick_summary(
+    query="Your question",
+    retrievers={"my_kb": retriever},
+    search_tool="my_kb"  # Use only this retriever
+)
+```
+
+### Multiple Retrievers
+
+```python
+result = detailed_research(
+    query="Complex question",
+    retrievers={
+        "vector_db": vector_retriever,
+        "graph_db": graph_retriever,
+        "sql_db": sql_retriever
+    },
+    search_tool="vector_db"  # Primary retriever; with the default
+    # langgraph-agent strategy, every registered retriever is also
+    # exposed to the research agent as a search tool
+)
+```
+
+### Hybrid Search (Retriever + Web)
+
+```python
+result = quick_summary(
+    query="Compare internal and external practices",
+    retrievers={"internal": internal_retriever},
+    search_tool="searxng"  # Web engine as primary; the langgraph-agent
+    # strategy can also query the registered "internal" retriever
+)
+```
+
+### Selective Usage
+
+```python
+# Pass multiple retrievers but use only one
+all_retrievers = {
+    "tech_docs": tech_retriever,
+    "legal_docs": legal_retriever,
+    "hr_docs": hr_retriever
+}
+
+# Use only tech docs for this query
+result = quick_summary(
+    query="Technical question",
+    retrievers=all_retrievers,
+    search_tool="tech_docs"  # Select specific retriever
+)
+```
+
+## Supported Retrievers
+
+Any LangChain `BaseRetriever` implementation works:
+
+- **Vector Stores**: FAISS, Chroma, Pinecone, Weaviate, Qdrant, etc.
+- **Cloud Services**: Vertex AI, AWS Bedrock, Azure Cognitive Search
+- **Databases**: PostgreSQL, MongoDB, Elasticsearch
+- **Custom**: Any class inheriting from `BaseRetriever`
+
+## API Reference
+
+### Parameters
+
+All research functions (`quick_summary`, `detailed_research`, `generate_report`) accept:
+
+- `retrievers`: Optional[Dict[str, BaseRetriever]] - Dictionary of retrievers
+- `search_tool`: str - Name of the retriever (or built-in engine) to use as the primary search source
+
+### Example with Complex Setup
+
+```python
+from langchain.vectorstores import VertexAIVectorSearch
+from langchain.embeddings import VertexAIEmbeddings
+import os
+
+# User handles proxy/auth setup
+os.environ["HTTP_PROXY"] = "http://proxy.company.com:8080"
+
+# Create retriever with complex configuration
+embeddings = VertexAIEmbeddings(
+    project="my-project",
+    location="us-central1"
+)
+vectorstore = VertexAIVectorSearch(
+    project="my-project",
+    location="us-central1",
+    index="my-index",
+    endpoint="my-endpoint",
+    embeddings=embeddings
+)
+retriever = vectorstore.as_retriever()
+
+# Use with LDR - all complexity is hidden
+result = quick_summary(
+    query="Internal knowledge query",
+    retrievers={"vertex_ai": retriever},
+    search_tool="vertex_ai"
+)
+```
+
+## Benefits
+
+1. **Zero Coupling**: LDR doesn't need to know retriever internals
+2. **Full Compatibility**: Works with all LDR features (strategies, agentic engine selection, etc.)
+3. **Clean API**: Just pass a dictionary of retrievers
+4. **Flexible**: Mix retrievers with web search seamlessly
+
+## Testing Your Integration
+
+```python
+from langchain.schema import Document, BaseRetriever
+
+class TestRetriever(BaseRetriever):
+    def get_relevant_documents(self, query: str):
+        return [Document(page_content=f"Test doc about {query}")]
+
+    async def aget_relevant_documents(self, query: str):
+        return self.get_relevant_documents(query)
+
+# Test it
+result = quick_summary(
+    query="test",
+    retrievers={"test": TestRetriever()},
+    search_tool="test"
+)
+```

docs/MIGRATION_GUIDE_v1.md

@@ -0,0 +1,261 @@
+# Migration Guide: LDR v0.x to v1.0
+
+## Overview
+
+Local Deep Research v1.0 introduces significant security and architectural improvements:
+
+- **User Authentication**: All access now requires authentication
+- **Per-User Encrypted Databases**: Each user has their own encrypted SQLCipher database
+- **Settings Snapshots**: Thread-safe settings management for concurrent operations
+- **New API Structure**: Reorganized endpoints under blueprint prefixes
+
+## Breaking Changes
+
+### 1. Authentication Required
+
+**v0.x**: Open access, no authentication
+```python
+# Direct API access
+from local_deep_research.api import quick_summary
+result = quick_summary("query")
+```
+
+**v1.0**: Authentication required
+```python
+from local_deep_research.api import quick_summary
+from local_deep_research.settings import SettingsManager
+from local_deep_research.database.session_context import get_user_db_session
+
+# Must authenticate first
+with get_user_db_session(username="user", password="pass") as session:
+    settings_manager = SettingsManager(session)
+    settings_snapshot = settings_manager.get_all_settings()
+
+    result = quick_summary(
+        "query",
+        settings_snapshot=settings_snapshot  # Required parameter
+    )
+```
+
+### 2. HTTP API Changes
+
+#### Endpoint Structure
+- **v0.x**: `/api/v1/quick_summary`
+- **v1.0**: `/api/start_research`
+
+#### Authentication Flow
+```python
+import requests
+
+# v1.0 requires session-based authentication
+session = requests.Session()
+
+# 1. Login
+session.post(
+    "http://localhost:5000/auth/login",
+    json={"username": "user", "password": "pass"}
+)
+
+# 2. Get CSRF token for state-changing operations
+csrf = session.get("http://localhost:5000/auth/csrf-token").json()["csrf_token"]
+
+# 3. Make API requests with CSRF token
+response = session.post(
+    "http://localhost:5000/api/start_research",
+    json={"query": "test"},
+    headers={"X-CSRF-Token": csrf}
+)
+```
+
+### 3. Database Changes
+
+#### v0.x
+- Single shared database: `ldr.db`
+- No encryption
+- Direct database access from any thread
+
+#### v1.0
+- Per-user databases: `encrypted_databases/{username}.db`
+- SQLCipher encryption with user passwords
+- Thread-local session management
+- In-memory queue tracking (no more service_db)
+
+### 4. Settings Management
+
+#### v0.x
+```python
+# Direct settings access
+from local_deep_research.config import get_db_setting
+value = get_db_setting("llm.provider")
+```
+
+#### v1.0
+```python
+# Settings require context
+from local_deep_research.settings import SettingsManager
+
+# Within authenticated session
+settings_manager = SettingsManager(session)
+value = settings_manager.get_setting("llm.provider")
+
+# Or use settings snapshot for thread safety
+settings_snapshot = settings_manager.get_all_settings()
+```
+
+## Migration Steps
+
+### 1. Update Dependencies
+
+```bash
+pip install --upgrade local-deep-research
+```
+
+### 2. Create User Accounts
+
+Users must create accounts through the web interface:
+
+1. Start the server: `python -m local_deep_research.web.app`
+2. Open http://localhost:5000
+3. Click "Register" and create an account
+4. Configure LLM providers and API keys in Settings
+
+### 3. Update Programmatic Code
+
+#### Before (v0.x):
+```python
+from local_deep_research.api import (
+    quick_summary,
+    detailed_research,
+    generate_report
+)
+
+# Direct usage
+result = quick_summary("What is AI?")
+```
+
+#### After (v1.0):
+```python
+from local_deep_research.api import quick_summary
+from local_deep_research.settings import SettingsManager
+from local_deep_research.database.session_context import get_user_db_session
+
+def run_research(username, password, query):
+    with get_user_db_session(username, password) as session:
+        settings_manager = SettingsManager(session)
+        settings_snapshot = settings_manager.get_all_settings()
+
+        return quick_summary(
+            query=query,
+            settings_snapshot=settings_snapshot,
+            # Other parameters remain the same
+            iterations=2,
+            questions_per_iteration=3
+        )
+```
+
+### 4. Update HTTP API Calls
+
+Create a wrapper for authenticated requests:
+
+```python
+class LDRClient:
+    def __init__(self, base_url="http://localhost:5000"):
+        self.base_url = base_url
+        self.session = requests.Session()
+        self.csrf_token = None
+
+    def login(self, username, password):
+        response = self.session.post(
+            f"{self.base_url}/auth/login",
+            json={"username": username, "password": password}
+        )
+        if response.status_code == 200:
+            self.csrf_token = self.session.get(
+                f"{self.base_url}/auth/csrf-token"
+            ).json()["csrf_token"]
+        return response
+
+    def start_research(self, query, **kwargs):
+        return self.session.post(
+            f"{self.base_url}/api/start_research",
+            json={"query": query, **kwargs},
+            headers={"X-CSRF-Token": self.csrf_token}
+        )
+
+# Usage
+client = LDRClient()
+client.login("user", "pass")
+result = client.start_research("What is quantum computing?")
+```
+
+### 5. Update Configuration
+
+#### API Keys
+API keys are now stored encrypted in per-user databases. Users must:
+1. Login to the web interface
+2. Go to Settings
+3. Re-enter API keys for LLM providers
+
+#### Custom LLMs
+Custom LLM registrations now require settings context:
+
+```python
+# v1.0 custom LLM with settings support
+def create_custom_llm(model_name=None, temperature=None, settings_snapshot=None):
+    # Access settings from snapshot if needed
+    api_key = settings_snapshot.get("llm.custom.api_key", {}).get("value")
+    return CustomLLM(api_key=api_key, model=model_name, temperature=temperature)
+```
+
+## Common Issues and Solutions
+
+### Issue: "No settings context available in thread"
+**Solution**: Pass `settings_snapshot` parameter to all API calls
+
+### Issue: "Encrypted database requires password"
+**Solution**: Ensure you're using `get_user_db_session()` with credentials
+
+### Issue: CSRF token errors
+**Solution**: Get fresh CSRF token before state-changing requests
+
+### Issue: Old endpoints return 404
+**Solution**: Update to new endpoint structure (see mapping above)
+
+### Issue: Rate limiting not working
+**Solution**: Rate limits are now per-user; ensure proper authentication
+
+## Backward Compatibility
+
+For temporary backward compatibility, you can:
+
+1. Set environment variable: `LDR_USE_SHARED_DB=1` (not recommended)
+2. Create a compatibility wrapper for your existing code
+
+```python
+# compatibility.py
+import os
+os.environ["LDR_USE_SHARED_DB"] = "1"  # Use at your own risk
+
+def quick_summary_compat(query, **kwargs):
+    # Minimal compatibility wrapper
+    # Note: This bypasses security features!
+    from local_deep_research.api import quick_summary
+    return quick_summary(query, settings_snapshot={}, **kwargs)
+```
+
+⚠️ **Warning**: Compatibility mode bypasses security features and is not recommended for production use.
+
+## Benefits of v1.0
+
+1. **Security**: Encrypted databases protect sensitive API keys and research data
+2. **Multi-User**: Multiple users can work simultaneously without conflicts
+3. **Performance**: Cached settings and thread-local sessions improve speed
+4. **Reliability**: Thread-safe operations prevent race conditions
+5. **Privacy**: User data is completely isolated
+
+## Getting Help
+
+- Check the [API Quick Start Guide](api-quickstart.md)
+- See [examples/api_usage](../examples/api_usage/) for updated examples
+- Join our [Discord](https://discord.gg/ttcqQeFcJ3) for migration support
+- Report issues on [GitHub](https://github.com/LearningCircuit/local-deep-research/issues)

docs/NOTIFICATIONS.md

@@ -0,0 +1,323 @@
+# Notifications System
+
+The Local Deep Research (LDR) notifications system provides a flexible way to send notifications to various services when important events occur, such as research completion, failures, or subscription updates.
+
+## Overview
+
+The notification system uses [Apprise](https://github.com/caronc/apprise) to support multiple notification services with a unified API. It allows users to configure comma-separated service URLs to receive notifications for different events.
+
+## Server-Side Opt-In Required
+
+> **Outbound notifications are disabled by default.** The deployment operator must explicitly enable them by setting an environment variable on the server:
+>
+> ```bash
+> LDR_NOTIFICATIONS_ALLOW_OUTBOUND=true
+> ```
+>
+> Without this, every `send_notification` call returns `False` and the "Send Test Notification" button returns an error. This applies to **all** users on the deployment.
+
+### Why?
+
+Notification webhooks have a known **DNS-rebinding TOCTOU window** that cannot be closed in code: LDR validates the URL once when it is configured, but the underlying Apprise library resolves the hostname *again* at send time, and Apprise exposes no DNS/Session hook to pin the resolved IP. A logged-in user with a controllable domain can serve a public IP at validation and a private IP at send time, causing the LDR server to make outbound HTTP requests to its own internal services (e.g. `127.0.0.1:<internal-port>`) or the local network.
+
+Because LDR is multi-user (per-user encrypted SQLCipher databases behind `@login_required`), the right default is to keep this feature off until the operator explicitly opts in — flipping the env var is the operator's acknowledgement of the residual risk. See [SECURITY.md](../SECURITY.md#notification-webhook-ssrf) for the full rationale and operator-side mitigations (prefer plugin schemes over raw `http(s)://`, restrict egress).
+
+### Symptoms when the gate is closed
+
+If you've configured a notification URL and aren't receiving messages, check the server logs first. You should see lines like:
+
+```
+WARNING  Notification refused: outbound notifications are disabled at the
+         server level. Set LDR_NOTIFICATIONS_ALLOW_OUTBOUND=true to enable.
+         See SECURITY.md 'Notification Webhook SSRF' for the rationale and
+         residual risk. (event=research_completed, user=...)
+```
+
+The "Send Test Notification" UI button returns the same message inline.
+
+## Supported Services
+
+The system supports all services that Apprise supports, including but not limited to:
+
+- Discord (via webhooks)
+- Slack (via webhooks)
+- Telegram
+- Email (SMTP)
+- Pushover
+- Gotify
+- Many more...
+
+For a complete list, refer to the [Apprise documentation](https://github.com/caronc/apprise/wiki).
+
+## Configuration
+
+Notifications are configured per-user via the settings system:
+
+### Service URL Setting
+- **Key**: `notifications.service_url`
+- **Type**: String (comma-separated list of service URLs)
+- **Example**: `discord://webhook_id/webhook_token,mailto://user:password@smtp.gmail.com`
+- **Security**: Service URLs containing credentials are encrypted at rest using SQLCipher (AES-256) in your per-user encrypted database. The encryption key is derived from your login password, ensuring zero-knowledge security.
+
+### Event-Specific Settings
+- `notifications.on_research_completed` - Enable notifications for completed research (default: true)
+- `notifications.on_research_failed` - Enable notifications for failed research (default: true)
+- `notifications.on_research_queued` - Enable notifications when research is queued (default: false)
+- `notifications.on_subscription_update` - Enable notifications for subscription updates (default: true)
+- `notifications.on_subscription_error` - Enable notifications for subscription errors (default: false)
+- `notifications.on_api_quota_warning` - Enable notifications for API quota/rate limit warnings (default: false)
+- `notifications.on_auth_issue` - Enable notifications for authentication failures (default: false)
+
+### Rate Limiting Settings
+- `notifications.rate_limit_per_hour` - Max notifications per hour (per user, default: 10)
+- `notifications.rate_limit_per_day` - Max notifications per day (per user, default: 50)
+
+**Per-User Rate Limiting**: Each user can configure their own rate limits via their settings. Rate limits are enforced independently per user, so one user hitting their limit does not affect other users. This ensures fair resource allocation in multi-user deployments.
+
+**Note on Multi-Worker Deployments**: The current rate limiting implementation uses in-memory storage and is process-local. In multi-worker deployments (e.g., gunicorn with multiple workers), each worker process maintains its own rate limit counters. This means a user could potentially send up to `N × max_per_hour` notifications (where N = number of workers) by distributing requests across different workers. For single-worker deployments (the default for LDR), this is not a concern. If you're running a multi-worker production deployment, consider monitoring notification volumes or implementing Redis-based rate limiting.
+
+### URL Configuration
+- `app.external_url` - Public URL where your LDR instance is accessible (e.g., `https://ldr.example.com`). Used to generate clickable links in notifications. If not set, defaults to `http://localhost:5000` or auto-constructs from `app.host` and `app.port`.
+
+## Service URL Format
+
+Multiple service URLs can be configured by separating them with commas:
+
+```
+discord://webhook1_id/webhook1_token,slack://token1/token2/token3,mailto://user:password@smtp.gmail.com
+```
+
+Each URL follows the Apprise format for the specific service.
+
+## Available Event Types
+
+### Research Events
+- `research_completed` - When research completes successfully
+- `research_failed` - When research fails (error details are sanitized in notifications for security)
+- `research_queued` - When research is added to the queue
+
+### Subscription Events
+- `subscription_update` - When a subscription completes
+- `subscription_error` - When a subscription fails
+
+### System Events
+- `api_quota_warning` - When API quota or rate limits are exceeded
+- `auth_issue` - When authentication fails for API services
+
+## Testing Notifications
+
+Use the test function to verify notification configuration:
+
+```python
+from local_deep_research.notifications.manager import NotificationManager
+
+# Create manager for testing (user_id is required)
+notification_manager = NotificationManager(
+    settings_snapshot={},
+    user_id="test_user"
+)
+
+# Test a service URL
+result = notification_manager.test_service("discord://webhook_id/webhook_token")
+print(result)  # {'success': True, 'message': 'Test notification sent successfully'}
+```
+
+## Programmatic Usage
+
+For detailed code examples, see the source files in `src/local_deep_research/notifications/`.
+
+### Basic Notification
+
+```python
+from local_deep_research.notifications.manager import NotificationManager
+from local_deep_research.notifications.templates import EventType
+from local_deep_research.settings import SettingsManager
+from local_deep_research.database.session_context import get_user_db_session
+
+# Get settings snapshot
+username = "your_username"
+with get_user_db_session(username, password) as session:
+    settings_manager = SettingsManager(session)
+    settings_snapshot = settings_manager.get_settings_snapshot()
+
+# Create notification manager with user_id for per-user rate limiting
+notification_manager = NotificationManager(
+    settings_snapshot=settings_snapshot,
+    user_id=username  # Enables per-user rate limit configuration
+)
+
+# Send notification (user_id already set in manager)
+notification_manager.send_notification(
+    event_type=EventType.RESEARCH_COMPLETED,
+    context={"query": "...", "summary": "...", "url": "/research/123"},
+)
+```
+
+**Important**: The `user_id` parameter is **required** when creating a `NotificationManager`. This ensures the user's configured rate limits from their settings are properly applied and enforces per-user isolation.
+
+### Building Full URLs
+
+Use `build_notification_url()` to convert relative paths to full URLs for clickable links in notifications.
+
+## Architecture
+
+The notification system consists of three main components:
+
+1. **NotificationManager** - High-level manager that handles rate limiting, settings, and user preferences
+2. **NotificationService** - Low-level service that uses Apprise to send notifications
+3. **Settings Integration** - User-specific configuration for services and event preferences
+
+The system fetches service URLs from user settings when needed, rather than maintaining persistent channels, making it more efficient and secure.
+
+### Security & Privacy
+
+- **Encrypted Storage**: All notification service URLs (including credentials like SMTP passwords or webhook tokens) are stored encrypted at rest in your per-user SQLCipher database using AES-256 encryption.
+- **Zero-Knowledge Architecture**: The encryption key is derived from your login password using PBKDF2-SHA512. Your password is never stored, and notification settings cannot be recovered without it.
+- **URL Masking**: Service URLs are automatically masked in logs to prevent credential exposure (e.g., `discord://webhook_id/***`).
+- **Per-User Isolation**: Each user's notification settings are completely isolated in their own encrypted database.
+
+### Performance Optimizations
+
+- **Temporary Apprise Instances**: Temporary Apprise instances are created for each send operation and automatically garbage collected by Python. This simple approach avoids memory management complexity.
+- **Shared Rate Limiter with Per-User Limits**: A single rate limiter instance is shared across all NotificationManager instances for efficiency, while maintaining separate rate limit configurations and counters for each user. This provides both memory efficiency (~24 bytes per user for limit storage) and proper per-user isolation.
+- **Thread-Safe**: The rate limiter uses threading locks for safe concurrent access within a single process.
+- **Exponential Backoff Retry**: Failed notifications are retried up to 3 times with exponential backoff (0.5s → 1.0s → 2.0s) to handle transient network issues.
+- **Dynamic Limit Updates**: User rate limits can be updated at runtime when creating a new NotificationManager instance with updated settings.
+
+## Thread Safety & Background Tasks
+
+The notification system is designed to work safely from background threads (e.g., research queue processors). Use the **settings snapshot pattern** to avoid thread-safety issues with database sessions.
+
+### Settings Snapshot Pattern
+
+**Key Principle**: Capture settings once with a database session, then pass the snapshot (not the session) to `NotificationManager`.
+
+- ✅ **Correct**: `NotificationManager(settings_snapshot=settings_snapshot, user_id=username)`
+- ❌ **Wrong**: `NotificationManager(session=session)` - Not thread-safe!
+
+See the source code in `web/queue/processor_v2.py` and `error_handling/error_reporter.py` for implementation examples.
+
+## Advanced Usage
+
+### Multiple Service URLs
+
+Configure multiple comma-separated service URLs to send notifications to multiple services simultaneously (Discord, Slack, email, etc.).
+
+### Custom Retry Behavior
+
+Use `force=True` parameter to bypass rate limits and disabled settings for critical notifications.
+
+### Event-Specific Configuration
+
+Each event type can be individually enabled/disabled via settings (see Event-Specific Settings above).
+
+### Per-User Rate Limiting
+
+The notification system supports independent rate limiting for each user:
+
+**How It Works:**
+- Each user configures their own rate limits via settings (e.g., `notifications.rate_limit_per_hour`)
+- Rate limits are enforced per-user, not globally
+- One user hitting their limit does not affect other users
+- Rate limits can be different for each user based on their settings
+
+**Example:**
+```python
+# User A with conservative limits (5/hour)
+snapshot_a = {"notifications.rate_limit_per_hour": 5}
+manager_a = NotificationManager(snapshot_a, user_id="user_a")
+
+# User B with generous limits (20/hour)
+snapshot_b = {"notifications.rate_limit_per_hour": 20}
+manager_b = NotificationManager(snapshot_b, user_id="user_b")
+
+# User A can send 5 notifications per hour
+# User B can send 20 notifications per hour
+# They don't interfere with each other
+```
+
+**Technical Details:**
+- The rate limiter maintains separate counters for each user
+- Each user's limits are stored in memory (~24 bytes per user)
+- Limits can be updated dynamically by creating a new NotificationManager for that user
+- The `user_id` parameter is required when creating a NotificationManager
+
+### Rate Limit Handling
+
+Rate limit exceptions (`RateLimitError`) can be caught and handled gracefully. See `notifications/exceptions.py` for available exception types.
+
+**Example:**
+```python
+from local_deep_research.notifications.exceptions import RateLimitError
+
+try:
+    notification_manager.send_notification(
+        event_type=EventType.RESEARCH_COMPLETED,
+        context=context,
+    )
+except RateLimitError as e:
+    # The manager already knows the user_id from initialization
+    logger.warning(f"Rate limit exceeded: {e}")
+    # Handle rate limit (e.g., queue for later, notify user)
+```
+
+## Troubleshooting
+
+### Notifications Not Sending
+
+1. **Check service URL configuration**: Use `SettingsManager.get_setting("notifications.service_url")` to verify the service URL is configured
+2. **Test service connection**: Use `notification_manager.test_service(service_url)` to verify connectivity
+3. **Check event-specific settings**: Verify the specific event type is enabled (e.g., `notifications.on_research_completed`)
+4. **Check rate limits**: Look for "Rate limit exceeded for user {user_id}" messages in logs
+
+### Common Issues
+
+**Issue**: "No notification service URLs configured"
+- **Cause**: `notifications.service_url` setting is empty or not set
+- **Fix**: Configure service URL in settings dashboard or via API
+
+**Issue**: "Rate limit exceeded"
+- **Cause**: User has sent too many notifications within their configured time window (hourly or daily limit)
+- **Fix**: Wait for rate limit window to expire (1 hour for hourly, 1 day for daily), adjust rate limit settings, or use `force=True` for critical notifications
+- **Note**: Rate limits are enforced per-user, so this only affects the specific user who exceeded their limit
+
+**Issue**: "Failed to send notification after 3 attempts"
+- **Cause**: Service is unreachable or credentials are invalid
+- **Fix**: Verify service URL is correct, test with `test_service()`, check network connectivity
+
+**Issue**: Notifications work in main thread but fail in background thread
+- **Cause**: Using database session in background thread (not thread-safe)
+- **Fix**: Use settings snapshot pattern as shown in migration guide above
+
+### Webhook URL Was Rejected
+
+The "Test" button in the notifications settings page returns the validator's reason directly. Common categories and the fix:
+
+**"Blocked private/internal IP address: \<host\>"**
+- **Cause**: The URL resolves to a loopback (`127.0.0.1`, `::1`), RFC1918 (`10.x`, `172.16-31.x`, `192.168.x`), CGNAT (`100.64.0.0/10`), link-local (`169.254.0.0/16`), or IPv6 private (`fc00::/7`, `fe80::/10`) address. The default SSRF policy blocks these for outbound webhooks.
+- **Fix (operator-only, env-only)**: Set `LDR_NOTIFICATIONS_ALLOW_PRIVATE_IPS=true` in the server environment. This is intentionally not exposed in the user-writable settings API. Only enable it if the notification endpoints are on a trusted local network.
+- **Fix (IPv6-only deployments using NAT64)**: If the host wraps IPv4 through `64:ff9b::/96` (RFC 6052 well-known) or `64:ff9b:1::/48` (RFC 8215 local-use), additionally set `LDR_SECURITY_ALLOW_NAT64=true`. The opt-in is scoped strictly to those two prefixes — 6to4 (`2002::/16`), Teredo (`2001::/32`), the discard prefix (`100::/64`), and the deprecated IPv4-Compatible IPv6 form (`::/96`) remain blocked.
+- **Note (cloud-metadata IPs)**: If `\<host\>` is a cloud-metadata IP (see the next bullet), the env-var hint above is intentionally **not** surfaced by the "Test" button — neither flag re-opens metadata, so the hint would mislead. The user-visible error is just the bare `"Blocked private/internal IP address: 169.254.169.254"`.
+
+**"Blocked cloud-metadata IP address: \<host\>"**
+- **Cause**: The URL uses an Apprise plugin scheme (`discord://`, `signal://`, `ntfy://`, etc.) whose host resolves to a cloud-metadata endpoint. Plugin schemes bypass the http/https private-IP check (their endpoints are typically LAN-local) but still hit the absolute cloud-metadata block.
+- **Always blocked**: AWS IMDS / ECS, Azure, OCI, DigitalOcean, AlibabaCloud, Tencent — both as plain IPv4 and wrapped through any NAT64 prefix. No env var re-opens these. See [SECURITY.md](../SECURITY.md#cloud-metadata-endpoint-block-list).
+- **Fix**: Choose a different webhook destination. Metadata endpoints expose IAM/instance credentials and are never legitimate webhook targets.
+
+**"Blocked unsafe protocol: \<scheme\>"** / **"Unsupported protocol: \<scheme\>"**
+- **Cause**: The URL uses a scheme that is either denylisted (`file`, `ftp`, `data`, `javascript`, `vbscript`, `about`, `blob`) or not in the Apprise-supported allowlist.
+- **Fix**: Use one of the allowed schemes — `http`, `https`, `mailto`, `discord`, `slack`, `telegram`, `gotify`, `pushover`, `ntfy`, `ntfys`, `signal`, `matrix`, `mattermost`, `rocketchat`, `teams`, `json`, `xml`, `form`. Prefer Apprise plugin schemes (`discord://`, `slack://`, `ntfy://`, etc.) over raw `http(s)://` webhooks — they hardcode their endpoints and have no SSRF surface.
+
+**"URL contains characters that are not allowed (whitespace, backslash, or control bytes)"**
+- **Cause**: Layer-1 defense against parser-differential SSRF bypasses (GHSA-g23j-2vwm-5c25) — RFC 3986 forbids these characters in URLs.
+- **Fix**: Remove the whitespace / backslash / control bytes. Percent-encode if a legitimate use case requires them.
+
+**"Outbound notifications are disabled. The server administrator must set LDR_NOTIFICATIONS_ALLOW_OUTBOUND=true …"**
+- **Cause**: Server-level master switch is off. See the "Server-Side Opt-In Required" section at the top of this document for the rationale (DNS-rebinding TOCTOU window in Apprise).
+- **Fix (operator-only)**: Set `LDR_NOTIFICATIONS_ALLOW_OUTBOUND=true` after reviewing the residual risk.
+
+## See Also
+
+- [Full Configuration Reference](CONFIGURATION.md) - All notification settings, defaults, and environment variables
+- [News Subscriptions](news-subscriptions.md) - News subscription system
+- [Features](features.md) - Feature overview

docs/NOTIFICATION_FLOW.md

@@ -0,0 +1,619 @@
+# Notification Flow - Complete Trace
+
+This document traces the complete flow of notifications from research completion/failure through to delivery via Apprise.
+
+## Overview
+
+Notifications are sent when:
+1. **Research completes successfully** → `RESEARCH_COMPLETED` event
+2. **Research fails** → `RESEARCH_FAILED` event
+3. **Research is queued** → `RESEARCH_QUEUED` event
+4. **Subscription updates** → `SUBSCRIPTION_UPDATE` event
+5. **Subscription errors** → `SUBSCRIPTION_ERROR` event
+6. **API quota/rate limits exceeded** → `API_QUOTA_WARNING` event
+7. **Authentication fails** → `AUTH_ISSUE` event
+
+## Complete Flow: Research Completed
+
+### 1. Research Thread Completes (`research_service.py:1166-1168`)
+
+When research finishes successfully in the background thread:
+
+```python
+# src/local_deep_research/web/services/research_service.py:1166
+cleanup_research_resources(
+    research_id, active_research, termination_flags, username
+)
+```
+
+### 2. Cleanup Calls Queue Processor (`research_service.py:1727`)
+
+The cleanup function notifies the queue processor:
+
+```python
+# src/local_deep_research/web/services/research_service.py:1727
+queue_processor.notify_research_completed(username, research_id, user_password=user_password)
+```
+
+**Key Points:**
+- Called from background research thread
+- Passes `user_password` for secure database access
+- Uses `processor_v2` which handles encrypted per-user databases
+
+### 3. Queue Processor Updates Status (`processor_v2.py:278-306`)
+
+```python
+# src/local_deep_research/web/queue/processor_v2.py:278-306
+def notify_research_completed(self, username: str, research_id: str, user_password: str = None):
+    with get_user_db_session(username, user_password) as session:
+        # Update queue status
+        queue_service = UserQueueService(session)
+        queue_service.update_task_status(research_id, "completed")
+
+        # Send notification if enabled
+        self._send_research_notification(
+            session=session,
+            username=username,
+            research_id=research_id,
+            event_type="RESEARCH_COMPLETED",
+        )
+```
+
+**Key Points:**
+- Opens encrypted user database with password
+- Updates queue status first
+- Delegates to `_send_research_notification` helper
+
+### 4. Build Notification Context (`processor_v2.py:357-243`)
+
+The helper method prepares the notification:
+
+```python
+# src/local_deep_research/web/queue/processor_v2.py:357-243
+def _send_research_notification(
+    self,
+    session,
+    username: str,
+    research_id: str,
+    event_type: str,
+    error_message: str = None,
+):
+    # Get settings snapshot for thread-safe notification sending
+    settings_manager = SettingsManager(session)
+    settings_snapshot = settings_manager.get_settings_snapshot()
+
+    # Lookup research details (with retry logic for timing issues)
+    research = session.query(ResearchHistory).filter_by(id=research_id).first()
+
+    if research:
+        # Create notification manager with settings snapshot and user_id
+        notification_manager = NotificationManager(
+            settings_snapshot=settings_snapshot,
+            user_id=username  # Enables per-user rate limiting
+        )
+
+        # Build full URL for notification
+        full_url = build_notification_url(
+            f"/research/{research_id}",
+            settings_manager=settings_manager,
+        )
+
+        # Build notification context
+        context = {
+            "query": research.query or "Unknown query",
+            "research_id": research_id,
+            "summary": report_content[:200] + "...",  # Truncated
+            "url": full_url,  # Full clickable URL
+        }
+
+        # Send notification (user_id already set in manager init)
+        result = notification_manager.send_notification(
+            event_type=EventType.RESEARCH_COMPLETED,
+            context=context,
+        )
+```
+
+**Key Points:**
+- **Settings Snapshot**: Captures settings at notification time (thread-safe)
+- **Research Lookup**: Queries database for research details with 3 retry attempts
+- **URL Building**: Constructs full URL using `app.external_url` or `app.host`/`app.port`
+- **Context Building**: Includes query, research_id, summary (truncated to 200 chars), full URL
+- **No Session Passed**: NotificationManager gets `settings_snapshot`, NOT session (thread-safe)
+
+### 5. NotificationManager Checks Settings (`manager.py:91-126`)
+
+```python
+# src/local_deep_research/notifications/manager.py:91-126
+def send_notification(
+    self,
+    event_type: EventType,
+    context: Dict[str, Any],
+    user_id: Optional[str] = None,
+    force: bool = False,
+) -> bool:
+    # Check if notifications are enabled for this event type
+    should_notify = self._should_notify(event_type, user_id)
+    if not force and not should_notify:
+        logger.debug(f"Notifications disabled for event type: {event_type.value}")
+        return False
+
+    # Check rate limit
+    rate_limit_ok = self._rate_limiter.allow(user_id or "default")
+    if not force and not rate_limit_ok:
+        raise RateLimitError("Notification rate limit exceeded")
+
+    # Get service URLs from settings snapshot
+    service_urls = self._get_setting("notifications.service_url", default="")
+
+    if not service_urls or not service_urls.strip():
+        logger.debug("No notification service URLs configured")
+        return False
+
+    # Send notification with service URLs
+    result = self.service.send_event(event_type, context, service_urls=service_urls)
+
+    return result
+```
+
+**Settings Checked (from `settings_snapshot`):**
+1. **`notifications.on_research_completed`** - Is this event type enabled? (default: False for most events)
+2. **Per-User Rate Limits** - Check shared rate limiter with user-specific limits:
+   - `notifications.rate_limit_per_hour` (default: 10) - Configured per user
+   - `notifications.rate_limit_per_day` (default: 50) - Configured per user
+   - Each user has independent rate limit counters
+3. **`notifications.service_url`** - Comma-separated list of Apprise URLs (required)
+
+**Key Points:**
+- All settings come from `settings_snapshot` (captured earlier)
+- Rate limiter is **shared singleton** with per-user limits and counters
+- Each user's rate limits are configured independently when `NotificationManager` is created with `user_id`
+- One user hitting their rate limit does NOT affect other users
+- `force=False` by default (respects settings and rate limits)
+
+### 6. NotificationService Formats Message (`service.py:198-235`)
+
+```python
+# src/local_deep_research/notifications/service.py:198-235
+def send_event(
+    self,
+    event_type: EventType,
+    context: Dict[str, Any],
+    service_urls: Optional[str] = None,
+    tag: Optional[str] = None,
+    custom_template: Optional[Dict[str, str]] = None,
+) -> bool:
+    # Format notification using template
+    message = NotificationTemplate.format(
+        event_type, context, custom_template
+    )
+
+    # Send notification
+    result = self.send(
+        title=message["title"],
+        body=message["body"],
+        service_urls=service_urls,
+        tag=tag,
+    )
+    return result
+```
+
+**Template Used (`templates.py`):**
+```python
+EventType.RESEARCH_COMPLETED: {
+    "title": "Research Completed: {query}",
+    "body": "Your research '{query}' has completed successfully.\n\n"
+            "Summary: {summary}\n\n"
+            "View results: {url}",
+}
+```
+
+**Context Variables:**
+- `{query}` - Research query text
+- `{summary}` - Truncated report content (max 200 chars)
+- `{url}` - Full clickable URL to view research
+- `{research_id}` - Research ID (available but not used in default template)
+
+### 7. NotificationService Sends via Apprise (`service.py:47-196`)
+
+```python
+# src/local_deep_research/notifications/service.py:47-196
+def send(
+    self,
+    title: str,
+    body: str,
+    service_urls: Optional[str] = None,
+    tag: Optional[str] = None,
+    attach: Optional[List[str]] = None,
+) -> bool:
+    # Retry logic with exponential backoff
+    retry_delay = INITIAL_RETRY_DELAY  # 0.5s
+
+    for attempt in range(1, MAX_RETRY_ATTEMPTS + 1):  # 3 attempts
+        try:
+            # Create temporary Apprise instance
+            # Automatically garbage collected by Python when out of scope
+            apprise_instance = apprise.Apprise()
+            apprise_instance.add(service_urls, tag=tag)
+
+            # Send notification
+            notify_result = apprise_instance.notify(
+                title=title,
+                body=body,
+                tag=tag,
+                attach=attach,
+            )
+
+            if notify_result:
+                return True
+
+            # Retry with exponential backoff
+            time.sleep(retry_delay)
+            retry_delay *= RETRY_BACKOFF_MULTIPLIER  # 2x
+
+        except Exception as e:
+            logger.error(f"Error sending notification: {e}")
+            time.sleep(retry_delay)
+            retry_delay *= RETRY_BACKOFF_MULTIPLIER
+
+    # All attempts failed
+    raise SendError("Failed to send notification after 3 attempts")
+```
+
+**Retry Strategy:**
+- **Attempt 1**: Immediate send
+- **Attempt 2**: Wait 0.5s, retry
+- **Attempt 3**: Wait 1.0s, retry
+- **After 3 attempts**: Raise `SendError`
+
+**Apprise Instances:**
+- Temporary instances created for each send operation
+- Automatically garbage collected by Python
+- Multiple service URLs supported (comma-separated)
+
+### 8. Apprise Delivers Notification
+
+Apprise handles the actual delivery to configured services:
+
+```python
+# User's settings: notifications.service_url
+# Security: placeholder example credentials below, not real secrets
+"discord://webhook_id/webhook_token,mailto://user:password@smtp.gmail.com"
+```
+
+**Supported Services** (via Apprise):
+- Discord, Slack, Telegram
+- Email (SMTP, Gmail, etc.)
+- Pushover, Gotify
+- Many more: https://github.com/caronc/apprise/wiki
+
+**Security:**
+- Service URLs encrypted at rest (AES-256 via SQLCipher)
+- Encryption key derived from user's login password (PBKDF2-SHA512)
+- URLs masked in logs (e.g., `discord://webhook_id/***`)
+
+## Complete Flow: Research Failed
+
+Similar to completed flow, but:
+
+1. **Entry Point**: `research_service.py:1642` (exception handler)
+2. **Queue Method**: `queue_processor.queue_error_update()` (processor.py)
+3. **Notification**: Sent from `processor.py:577-606` (error update handler)
+4. **Event Type**: `EventType.RESEARCH_FAILED`
+5. **Context**: Includes `error` field instead of `summary`
+
+**Template for RESEARCH_FAILED:**
+```python
+EventType.RESEARCH_FAILED: {
+    "title": "Research Failed: {query}",
+    "body": "Research on '{query}' failed.\n\n"
+            "Error: {error}\n\n"
+            "Please check the logs for more details.",
+}
+```
+
+**Note**: Error messages are sanitized for security to avoid exposing sensitive information in notifications.
+
+## Settings Snapshot Pattern
+
+**Why Settings Snapshot?**
+
+Notifications are sent from background threads that shouldn't access Flask `g` or SQLite sessions (not thread-safe). The solution is to **capture settings once** in the main thread and pass them as a dict.
+
+**How It Works:**
+
+```python
+# 1. In main thread (with database session)
+settings_manager = SettingsManager(session)
+settings_snapshot = settings_manager.get_settings_snapshot()
+# Returns dict like: {"notifications.service_url": "...", "notifications.on_research_completed": True, ...}
+
+# 2. Pass to NotificationManager (thread-safe - no session!)
+notification_manager = NotificationManager(
+    settings_snapshot=settings_snapshot,
+    user_id=username
+)
+
+# 3. NotificationManager reads from snapshot
+def _get_setting(self, key: str, default: Any = None) -> Any:
+    return self._settings_snapshot.get(key, default)
+```
+
+**Benefits:**
+- ✅ Thread-safe (no database access in background threads)
+- ✅ Consistent settings (captured at notification time, not changed mid-notification)
+- ✅ No Flask `g` context needed
+- ✅ Works from queue processors, schedulers, etc.
+
+## Rate Limiting
+
+**Implementation:** In-memory, per-user limits and counters, shared singleton
+
+```python
+class RateLimiter:
+    _lock = threading.Lock()
+    _user_limits: Dict[str, tuple[int, int]] = {}  # user_id -> (max_per_hour, max_per_day)
+    _hourly_counts: Dict[str, deque] = {}  # user_id -> timestamps
+    _daily_counts: Dict[str, deque] = {}   # user_id -> timestamps
+
+    def set_user_limits(self, user_id: str, max_per_hour: int, max_per_day: int):
+        """Configure rate limits for a specific user."""
+        with self._lock:
+            self._user_limits[user_id] = (max_per_hour, max_per_day)
+
+    def allow(self, user_id: str) -> bool:
+        with self._lock:  # Thread-safe
+            now = datetime.now(timezone.utc)
+
+            # Clean old entries (> 1 hour, > 1 day)
+            self._clean_old_entries(user_id, now)
+
+            # Get user-specific limits or defaults
+            max_per_hour, max_per_day = self._user_limits.get(
+                user_id, (self.max_per_hour, self.max_per_day)
+            )
+
+            # Check limits using user-specific values
+            if len(self._hourly_counts[user_id]) >= max_per_hour:
+                return False
+            if len(self._daily_counts[user_id]) >= max_per_day:
+                return False
+
+            # Record this notification
+            self._hourly_counts[user_id].append(now)
+            self._daily_counts[user_id].append(now)
+
+            return True
+```
+
+**Shared Singleton Pattern with Per-User Configuration:**
+```python
+class NotificationManager:
+    _shared_rate_limiter: Optional["RateLimiter"] = None
+    _rate_limiter_lock = threading.Lock()
+
+    def __init__(self, settings_snapshot: Dict[str, Any], user_id: str):
+        with NotificationManager._rate_limiter_lock:
+            if NotificationManager._shared_rate_limiter is None:
+                # Create shared rate limiter with defaults
+                NotificationManager._shared_rate_limiter = RateLimiter(
+                    max_per_hour=settings_snapshot.get("notifications.rate_limit_per_hour", 10),
+                    max_per_day=settings_snapshot.get("notifications.rate_limit_per_day", 50),
+                )
+
+            self._rate_limiter = NotificationManager._shared_rate_limiter
+
+            # Configure per-user limits (user_id is required)
+            max_per_hour = settings_snapshot.get("notifications.rate_limit_per_hour", 10)
+            max_per_day = settings_snapshot.get("notifications.rate_limit_per_day", 50)
+            self._rate_limiter.set_user_limits(user_id, max_per_hour, max_per_day)
+```
+
+**Key Points:**
+- **One rate limiter instance** across all NotificationManager instances (singleton)
+- **Per-user rate limits**: Each user can have different limits based on their settings
+- **Per-user counters**: Each user has independent notification counters
+- **User isolation**: One user hitting their limit does NOT affect others
+- **Thread-safe** with `threading.Lock()` for all operations
+- **Automatic cleanup** of old entries (> 1 hour, > 1 day)
+- **Periodic cleanup** of inactive users (every 24 hours)
+- **Memory efficient**: ~24 bytes per user for limit storage
+
+## Configuration
+
+### Required Settings
+
+```python
+# Service URL (required) - comma-separated list
+# Security: placeholder example credentials below, not real secrets
+notifications.service_url = "discord://webhook_id/token,mailto://user:pass@smtp.gmail.com"
+
+# Event-specific toggles (default: False for most events)
+notifications.on_research_completed = True   # Default: True
+notifications.on_research_failed = True      # Default: True
+notifications.on_research_queued = False     # Default: False
+notifications.on_subscription_update = True  # Default: True
+notifications.on_subscription_error = False  # Default: False
+notifications.on_api_quota_warning = False   # Default: False
+notifications.on_auth_issue = False          # Default: False
+
+# Rate limits (per-user, configured independently for each user)
+notifications.rate_limit_per_hour = 10   # Max notifications per hour (per user)
+notifications.rate_limit_per_day = 50    # Max notifications per day (per user)
+
+# URL configuration (for clickable links)
+app.external_url = "https://ldr.example.com"  # Preferred
+# OR
+app.host = "localhost"
+app.port = 5000
+```
+
+### Testing Notifications
+
+```python
+from local_deep_research.notifications.manager import NotificationManager
+
+# Create manager with settings and required user_id
+notification_manager = NotificationManager(
+    settings_snapshot={},
+    user_id="test_user"  # Required for per-user rate limiting
+)
+
+# Test a service URL
+result = notification_manager.test_service("discord://webhook_id/webhook_token")
+print(result)  # {'success': True, 'message': 'Test notification sent successfully'}
+```
+
+## Error Handling
+
+### Rate Limit Exceeded
+
+```python
+try:
+    notification_manager.send_notification(
+        event_type=EventType.RESEARCH_COMPLETED,
+        context=context,
+    )
+except RateLimitError as e:
+    logger.warning(f"Rate limit exceeded: {e}")
+    # Notification not sent, user needs to wait
+    # Note: This only affects the user that the manager was created for
+```
+
+### Send Failure (After 3 Retries)
+
+```python
+try:
+    result = service.send(title="...", body="...", service_urls="...")
+except SendError as e:
+    logger.error(f"Failed to send notification after 3 attempts: {e}")
+    # All retry attempts exhausted
+```
+
+### No Service URLs Configured
+
+```python
+result = notification_manager.send_notification(...)
+# Returns: False (no error raised)
+# Log: "No notification service URLs configured for user {user_id}"
+```
+
+### Notifications Disabled for Event Type
+
+```python
+# settings_snapshot = {"notifications.on_research_completed": False}
+result = notification_manager.send_notification(
+    event_type=EventType.RESEARCH_COMPLETED,
+    context=context
+)
+# Returns: False (respects user preference)
+# Log: "Notifications disabled for event type: research_completed"
+```
+
+## Architecture Diagram
+
+```
+[Research Thread]
+      ↓
+cleanup_research_resources()
+      ↓
+queue_processor.notify_research_completed(username, research_id, password)
+      ↓
+[Queue Processor - Main Thread]
+      ↓
+get_user_db_session(username, password) → [Encrypted DB]
+      ↓
+SettingsManager(session).get_settings_snapshot() → settings_snapshot
+      ↓
+ResearchHistory.query.filter_by(id=research_id).first() → research details
+      ↓
+build_notification_url() → full_url
+      ↓
+NotificationManager(settings_snapshot=settings_snapshot, user_id=username)
+      ↓
+notification_manager.send_notification(
+    event_type=EventType.RESEARCH_COMPLETED,
+    context={query, research_id, summary, url}
+)
+      ↓
+[NotificationManager]
+      ├─ Check: notifications.on_research_completed
+      ├─ Check: Per-user rate limiter (hourly/daily)
+      └─ Get: notifications.service_url
+      ↓
+NotificationService.send_event(event_type, context, service_urls)
+      ↓
+NotificationTemplate.format(event_type, context) → {title, body}
+      ↓
+NotificationService.send(title, body, service_urls)
+      ↓
+[Retry Loop: 3 attempts with exponential backoff]
+      ├─ Attempt 1: Immediate
+      ├─ Attempt 2: Wait 0.5s
+      └─ Attempt 3: Wait 1.0s
+      ↓
+Apprise.add(service_urls)
+      ↓
+Apprise.notify(title=title, body=body)
+      ↓
+[Apprise - Delivery]
+      ├─ Discord webhook
+      ├─ SMTP email
+      └─ Other services...
+```
+
+## Key Design Decisions
+
+1. **Settings Snapshot Pattern**: Avoids thread-safety issues with database sessions
+2. **Shared Rate Limiter with Per-User Limits**: Single rate limiter instance ensures correct per-user enforcement while maintaining separate limits and counters for each user
+3. **User Isolation**: Each user's rate limits are independent - one user hitting their limit does not affect others
+4. **Temporary Apprise Instances**: Created per-send and automatically garbage collected
+5. **Exponential Backoff**: 3 retry attempts with increasing delays (0.5s → 1.0s → 2.0s)
+6. **Encrypted Storage**: Service URLs stored encrypted in per-user SQLCipher database
+7. **URL Masking**: Credentials hidden in logs (e.g., `discord://webhook_id/***`)
+8. **No Session in Manager**: NotificationManager never receives database session (thread-safe)
+9. **Error Sanitization**: Error messages sanitized in notifications to prevent information exposure
+
+## Additional Notification Events
+
+### Research Queued Notifications
+
+Sent when research is added to the queue:
+
+- **Event**: `EventType.RESEARCH_QUEUED`
+- **Triggered from**: `web/queue/manager.py` when adding research to queue
+- **Context**: `query`, `position`, `wait_time`
+- **Default**: Disabled (opt-in)
+
+### API Quota Warnings
+
+Sent when API rate limits are exceeded:
+
+- **Event**: `EventType.API_QUOTA_WARNING`
+- **Triggered from**: `error_handling/error_reporter.py` when detecting rate limit errors
+- **Context**: `service`, `current`, `limit`, `reset_time`
+- **Default**: Disabled (opt-in)
+
+### Authentication Issues
+
+Sent when API authentication fails:
+
+- **Event**: `EventType.AUTH_ISSUE`
+- **Triggered from**: `error_handling/error_reporter.py` when detecting auth errors
+- **Context**: `service`
+- **Default**: Disabled (opt-in)
+
+### Subscription Notifications
+
+Sent when subscriptions update or fail:
+
+- **Events**: `EventType.SUBSCRIPTION_UPDATE`, `EventType.SUBSCRIPTION_ERROR`
+- **Context**: `subscription_name`, `subscription_id`, `item_count`/`error`, `url`
+
+## Testing
+
+Run notification tests:
+```bash
+pdm run python -m pytest tests/notifications/ -v
+```
+
+All 98 tests passing ✓ (including 7 new per-user rate limiting tests)

docs/RELEASE_GUIDE.md

@@ -0,0 +1,290 @@
+# Release Guide
+
+## 🚀 Automated Release Process
+
+Releases are **fully automated** end-to-end whenever a PR that bumps
+`src/local_deep_research/__version__.py` is merged to `main`. No
+separate tag push, manual workflow trigger, or release-page click is
+required — the workflow detects the new version, runs all gates, cuts
+the GitHub release, and (after one approval click) publishes to PyPI
+and Docker Hub.
+
+PRs that don't touch `__version__.py` merge normally but skip the
+release pipeline (the version-check job sees the tag already exists
+and short-circuits everything downstream).
+
+## 📋 How Releases Work
+
+### 1. **Automatic Release Creation**
+- **Trigger**: Push to `main` whose `__version__.py` resolves to a tag
+  that does not yet exist as a GitHub release. In practice this means
+  "merge a PR that bumps `__version__.py`". Tag pushes (`v*.*.*`) and
+  manual `workflow_dispatch` runs also trigger the pipeline and bypass
+  the version-exists check.
+- **Version**: Read from `src/local_deep_research/__version__.py` by
+  the `version-check` job; the tag is `v<version>`.
+- **Release body**: Composed by `.github/workflows/release.yml` from three sources:
+  1. **AI narrative** generated by OpenRouter (`vars.AI_MODEL`, default
+     `moonshotai/kimi-k2-thinking`). The model receives the rendered
+     hand-written notes, the auto-generated PR list, every PR's title +
+     body (batched via one GraphQL call), and the diff between the
+     previous release tag and this one (filtered to drop lockfiles,
+     generated docs, SBOM, static assets, and binary patches; capped at
+     700k chars).
+  2. **Hand-written notes** from `docs/release_notes/<version>.md` —
+     rendered from contributor-supplied `changelog.d/*.md` fragments by
+     the workflow itself at release time (see
+     [Release-notes flow](#-release-notes-flow-towncrier-news-fragments)
+     below). No manual `pdm run towncrier build` is required before
+     merging the bump.
+  3. **Auto-generated PR list** from GitHub's generate-notes API,
+     label-categorized.
+- **No duplicates**: If a release for `v<version>` already exists, the
+  `version-check` job sets `should_release=false` and every downstream
+  job (security gate, CI gate, build, publish) is skipped.
+
+### 2. **Approval and Publishing**
+
+The release pipeline uses the `release` GitHub environment to gate the
+publish steps. `DOCKER_USERNAME` / `DOCKER_PASSWORD` are scoped to that
+environment, so any job that pushes to Docker Hub must declare
+`environment: release` and therefore goes through the approval gate.
+
+When you merge to `main` (or push a tag), the pipeline runs in this
+order:
+
+1. Security gates + CI gates run automatically.
+2. `build` job runs (version pin, SBOM, Sigstore bundles), then
+   `provenance` job generates SLSA provenance for those artifacts.
+3. **One `release` env approval prompt** in `release.yml`. Approving
+   unlocks all release-env jobs in the same run, which then execute
+   sequentially:
+   1. `prerelease-docker` — canonical multi-arch Docker build, cosign
+      sign, SBOM/SLSA attestations, push as `prerelease-v<ver>-<sha>`
+      and re-point the floating `:prerelease` tag.
+   2. `publish-docker` — retags the prerelease manifest as `:1.6.9`,
+      `:1.6`, `:latest` (no rebuild, digest-preserving), then re-verifies
+      digest + cosign + Trivy on the promoted tag.
+   3. `trigger-pypi` — dispatches `publish.yml` via `repository_dispatch`
+      (PyPI Trusted Publishing requires the publish step to run in a
+      top-level workflow, so this can't be a reusable workflow_call).
+   4. `monitor-pypi` — polls `publish.yml` for completion. The inner
+      polling loop times out at 40 minutes (after which the job fails);
+      the surrounding GH Actions `timeout-minutes` is 90 to leave a
+      safety margin around the poll budget.
+   5. `create-release` — publishes the GitHub Release with
+      SBOM/sig/provenance assets. Runs **last**, gated on all of the
+      above succeeding, so the public Release never points at missing
+      Docker tags or a missing PyPI version.
+
+If any of `prerelease-docker`, `publish-docker`, or `monitor-pypi` fails,
+`create-release` is skipped and no public GitHub Release is created. The
+`cleanup-on-rejection` job then handles failure-mode cleanup:
+
+- If `publish-docker` failed mid-retag (e.g., `:1.6.9` landed but
+  `:latest` failed), it rolls back any landed release tags BEFORE
+  deleting prerelease tags and cosign artifacts (deleting cosign
+  artifacts while release tags share the manifest digest would invalidate
+  release-tag signatures).
+- If `publish-docker` succeeded but a later step (PyPI or
+  create-release) failed, `cleanup-on-rejection` does NOT fire — Docker
+  release tags exist and their cosign artifacts must stay. See
+  "Recovery from PyPI failure" below.
+
+### Recovery from PyPI failure (atomicity hole)
+
+The one orphan state the pipeline cannot fully clean up: `publish-docker`
+succeeded, PyPI failed. At this point Docker `:1.6.9` / `:1.6` /
+`:latest` exist and are signed; PyPI has nothing; no GitHub Release.
+`monitor-pypi` opens a tracking issue labeled `ci-cd`. To recover:
+
+1. Inspect the `publish.yml` workflow run, fix the underlying cause.
+2. Manually re-dispatch PyPI publish. `client_payload[sha]` is REQUIRED
+   (publish.yml fails closed without it) and must be the full 40-hex
+   release commit — the commit the failed `release.yml` run built, NOT
+   current main HEAD (main may have moved during the approval wait).
+   Get it with `gh run view <release-run-id> --json headSha`:
+   ```bash
+   gh api repos/LearningCircuit/local-deep-research/dispatches \
+     -f event_type=publish-pypi \
+     -F 'client_payload[tag]=v<X.Y.Z>' \
+     -F 'client_payload[sha]=<40-hex release commit>'
+   ```
+3. Once PyPI publishes successfully, manually create the GitHub Release
+   from the existing tag (the SBOM/sig/provenance artifacts are still
+   uploaded as workflow artifacts on the failed `release.yml` run; you
+   can download them and attach manually, or re-run `create-release`
+   manually if the run is still re-runnable in the Actions UI).
+
+> Earlier iterations of this refactor described a single approval gate
+> with a pre-approval testing window. That design required
+> `DOCKER_USERNAME` / `DOCKER_PASSWORD` to be repo-level secrets so the
+> canonical build could run without env approval. They are env-scoped to
+> `release` instead, so the gate sits in front of the build. The
+> atomicity refactor preserves this single-approval model — one click
+> unlocks the whole chain, and create-release runs last so the
+> "published Release with broken artifacts" failure mode is closed.
+
+## 👥 Who Can Release
+
+Code owners (defined in `.github/CODEOWNERS`):
+- `@LearningCircuit`
+- `@hashedviking`
+- `@djpetti`
+
+## 📝 Release Workflow
+
+### For Regular Releases:
+
+1. **Bump version** in `src/local_deep_research/__version__.py` (or
+   merge the auto-bump PR opened by `.github/workflows/version_check.yml`).
+2. **Merge to main** → Release automatically created. The release
+   workflow renders fragments from `changelog.d/*.md` into
+   `docs/release_notes/<X.Y.Z>.md`, composes the body (AI narrative +
+   rendered changelog + auto PR list), and publishes the GitHub release.
+3. **Approve publishing** in GitHub Actions (PyPI/Docker).
+4. **Merge the cleanup PR** opened automatically by the
+   `cleanup-changelog` job (titled
+   `chore: clear changelog fragments for <X.Y.Z>`). It persists
+   `docs/release_notes/<X.Y.Z>.md` and removes the consumed fragments
+   from `changelog.d/`. Squash-merge — the diff has no review value
+   beyond a sanity check that the rendered notes look right.
+
+To preview the rendered notes locally before merging the bump:
+
+```bash
+pdm run towncrier build --draft --version <X.Y.Z>
+```
+
+(`--draft` writes nothing.) Skip if no fragments exist for this release
+— the workflow tolerates a missing per-version file (warns and proceeds
+with auto-notes plus AI summary only).
+
+### For Hotfixes:
+
+1. **Create hotfix branch** from main
+2. **Make minimal fix**
+3. **Bump patch version** (e.g., 0.4.3 → 0.4.4)
+4. **Fast-track review** by code owners
+5. **Merge to main** → Automatic release
+
+## 🔧 Manual Release Options
+
+### Option A: Manual Trigger
+- Go to Actions → "Create Release" → "Run workflow"
+- No inputs are required: the workflow reads the version from
+  `src/local_deep_research/__version__.py` at HEAD. To release an
+  older or different version, use Option B (push a version tag).
+
+### Option B: Version Tags
+- `git tag v0.4.3 && git push origin v0.4.3`
+- Automatically creates release; the workflow uses the tag's commit
+  SHA (not `main` HEAD), so this is the correct path for backporting.
+
+## 🛡️ Branch Protection
+
+- **Main branch** is protected
+- **Required reviews** from code owners
+- **No direct pushes** - only via approved PRs
+- **Status checks** must pass (CI tests)
+
+## 📦 Version Numbering
+
+Follow [Semantic Versioning](https://semver.org/):
+- **Major** (X.0.0): Breaking changes
+- **Minor** (0.X.0): New features, backward compatible
+- **Patch** (0.0.X): Bug fixes, backward compatible
+
+## 🚨 Emergency Procedures
+
+If automation fails, do NOT create a GitHub release through the UI as
+the first recovery step — under the atomicity refactor, a manually
+created GitHub release does NOT trigger `publish.yml` (it listens only
+on `repository_dispatch`) and does NOT trigger `docker-publish.yml`
+(workflow_call only). The downstream `release:` listeners that DO fire
+(`backwards-compatibility.yml`, `sbom.yml`) are observability-only.
+
+Recovery, in order of preference:
+1. **Check workflow logs** in GitHub Actions to identify which job
+   failed, and use the targeted recovery for that failure mode:
+   - PyPI failure with Docker already promoted: see
+     [Recovery from PyPI failure](#recovery-from-pypi-failure-atomicity-hole) above.
+   - Any other failure: re-run the failed job via the Actions UI if
+     it's still re-runnable (typically within 30 days).
+2. **Re-trigger the full pipeline** via `workflow_dispatch` if
+   re-running individual jobs isn't possible. Safe for digest-keyed
+   cosign verification — old digests remain valid because their cosign
+   artifacts persist; the new run produces a new digest with its own
+   signatures.
+3. **Contact code owners** if recovery requires manual Docker Hub or
+   PyPI intervention.
+
+## 📝 Release-notes flow (towncrier news fragments)
+
+Hand-written release notes are assembled from per-PR fragments using
+[towncrier](https://towncrier.readthedocs.io). This replaces the older
+shared `docs/release_notes/<version>.md` model, which broke down at
+LDR's PR throughput (multiple PRs/day racing for the same file).
+
+### Contributor side
+
+Each PR with user-visible behavior change drops one tiny markdown file:
+
+```
+changelog.d/<PR-number>.<category>.md
+```
+
+Categories: `breaking`, `security`, `feature`, `bugfix`, `removal`,
+`misc` (canonical list lives in `[[tool.towncrier.type]]` entries in
+`pyproject.toml`; the pre-commit hook reads it from there). Orphan
+fragments (no PR/issue number) use `changelog.d/+<slug>.<category>.md`.
+The pre-commit hook (`recommend-release-notes`) nudges contributors who
+add ≥20 source lines without a fragment, and validates filenames so a
+typo'd category doesn't silently vanish at render time. See
+[`changelog.d/README.md`](../changelog.d/README.md) for the full
+convention.
+
+### Maintainer side
+
+The release workflow handles the render. There is nothing to run
+manually before merging the version bump.
+
+`.github/workflows/release.yml` (`create-release` job):
+1. Sparse-checks-out `changelog.d/` + `pyproject.toml`, installs
+   `towncrier~=24.8`.
+2. Runs `towncrier build --yes --version <X.Y.Z>` against the runner's
+   throwaway workspace. Towncrier writes the rendered output to
+   `docs/release_notes/<X.Y.Z>.md` (per the `{version}`-templated
+   `filename`; `single_file = false` makes this a per-release file
+   rather than appending to a master CHANGELOG) and removes the
+   consumed fragments locally.
+3. Reads the rendered file as input to the AI summary and as the
+   "hand-written notes" section of the published GitHub release body.
+
+Persistence to `main` happens in the `cleanup-changelog` job, which
+re-runs the same render against the release commit (`github.sha`) and
+opens a `chore/post-release-cleanup-<X.Y.Z>` PR with the deletions and
+the rendered file. Squash-merge it.
+
+If `changelog.d/` is empty (maintenance release with no fragments), the
+render step is skipped cleanly and the release proceeds with the AI
+narrative + auto PR list only — no hard failure.
+
+### Preview without committing
+
+```bash
+pdm run towncrier build --draft --version <X.Y.Z>
+```
+
+`--draft` renders to stdout without touching any files or fragments.
+Useful while iterating on a fragment locally.
+
+## 📊 Release Checklist
+
+- [ ] Version updated in `__version__.py` (or auto-bump PR merged)
+- [ ] Code owner approval received
+- [ ] CI tests passing
+- [ ] Merge to main completed
+- [ ] Release automatically created
+- [ ] PyPI/Docker publishing approved
+- [ ] `chore: clear changelog fragments for <X.Y.Z>` PR merged

docs/SQLCIPHER_INSTALL.md

@@ -0,0 +1,173 @@
+# SQLCipher Installation Guide
+
+Local Deep Research uses SQLCipher to provide encrypted databases for each user. This ensures that all user data, including API keys and research results, are encrypted at rest.
+
+## Installation by Platform
+
+### Ubuntu/Debian Linux
+
+SQLCipher can be easily installed from the package manager:
+
+```bash
+sudo apt update
+sudo apt install sqlcipher libsqlcipher-dev
+```
+
+After installing SQLCipher, install the project with PDM:
+```bash
+pdm install
+```
+
+PDM will automatically select the correct Python binding for your platform:
+- x86_64 Linux: `sqlcipher3-binary` (pre-compiled wheel)
+- ARM64 Linux: `sqlcipher3` (builds from source)
+- Other platforms: `sqlcipher3`
+
+### macOS
+
+Install using Homebrew:
+
+```bash
+brew install sqlcipher
+```
+
+Then install the project with PDM:
+```bash
+# May need to set environment variables for building
+export LDFLAGS="-L$(brew --prefix sqlcipher)/lib"
+export CPPFLAGS="-I$(brew --prefix sqlcipher)/include"
+pdm install
+```
+
+### Windows
+
+As of `sqlcipher3` 0.6.2+, pre-built self-contained wheels are available for Windows
+(x86, x64, ARM64, Python 3.9–3.14). No compilation or system libraries needed:
+
+```bash
+pip install sqlcipher3
+```
+
+This is automatically included when you `pip install local-deep-research`.
+
+<details>
+<summary>Manual build (older versions / troubleshooting)</summary>
+
+If you need to build from source on older Python versions:
+
+1. Install Visual Studio 2015 or later (Community Edition works)
+2. Install the "Desktop Development with C++" workload
+3. Download SQLCipher source from https://github.com/sqlcipher/sqlcipher
+4. Build using Visual Studio Native Tools Command Prompt
+
+Alternatively, use WSL2 with Ubuntu.
+</details>
+
+## Alternative: Using Docker
+
+If you have difficulty installing SQLCipher, you can run Local Deep Research in a Docker container where SQLCipher is pre-installed:
+
+```dockerfile
+FROM python:3.12-slim
+
+# Install SQLCipher
+RUN apt-get update && apt-get install -y \
+    sqlcipher \
+    libsqlcipher-dev \
+    gcc \
+    && rm -rf /var/lib/apt/lists/*
+
+# Install Local Deep Research (SQLCipher binding selected automatically)
+RUN pip install local-deep-research
+
+CMD ["ldr", "serve"]
+```
+
+## Verifying Installation
+
+You can verify SQLCipher is installed correctly:
+
+```bash
+# Check command line tool
+sqlcipher --version
+
+# Test Python binding
+python -c "from local_deep_research.database.sqlcipher_compat import get_sqlcipher_module; get_sqlcipher_module(); print('SQLCipher is installed!')"
+```
+
+## Fallback Mode
+
+If SQLCipher is not available, Local Deep Research will fall back to using regular SQLite databases. However, this means your data will not be encrypted at rest. A warning will be displayed when running without encryption.
+
+## Security Notes
+
+- Each user's database is encrypted with their password
+- There is no password recovery mechanism - if a user forgets their password, their data cannot be recovered
+- The encryption uses SQLCipher's default settings with AES-256
+- API keys and sensitive data are only stored in the encrypted user databases
+
+## Troubleshooting
+
+### Linux: "Package not found"
+
+If your distribution doesn't have SQLCipher in its repositories, you may need to build from source or use a third-party repository.
+
+### macOS: "Library not loaded"
+
+Make sure you've set the LDFLAGS and CPPFLAGS environment variables as shown above.
+
+### Windows: Build errors
+
+Ensure you're using the Visual Studio Native Tools Command Prompt and have all required dependencies installed.
+
+### Python: "No module named sqlcipher3" or "pysqlcipher3"
+
+**Error variants:**
+- `ModuleNotFoundError: No module named 'sqlcipher3'`
+- `ModuleNotFoundError: No module named 'pysqlcipher3'`
+
+The project automatically selects the correct SQLCipher package based on your platform:
+- **x86_64 Linux**: Uses `sqlcipher3-binary` (pre-compiled wheel)
+- **ARM64 Linux**: Uses `sqlcipher3` (builds from source with system SQLCipher)
+- **Other platforms**: Uses `sqlcipher3`
+
+**Solution:**
+1. First, ensure you have the system SQLCipher library installed:
+   ```bash
+   # Debian/Ubuntu
+   sudo apt-get install libsqlcipher-dev
+
+   # macOS
+   brew install sqlcipher
+   ```
+
+2. Then reinstall the Python package:
+   ```bash
+   # If using PDM (recommended)
+   pdm install
+
+   # If NOT using PDM (e.g., pip-only setup):
+   pip install --force-reinstall sqlcipher3-binary  # x86_64 Linux
+   pip install --force-reinstall sqlcipher3         # Other platforms
+   ```
+
+## For Developers
+
+To add SQLCipher to an automated installation script:
+
+```bash
+#!/bin/bash
+# For Ubuntu/Debian
+if command -v apt-get &> /dev/null; then
+    sudo apt-get update
+    sudo apt-get install -y sqlcipher libsqlcipher-dev
+fi
+
+# For macOS with Homebrew
+if command -v brew &> /dev/null; then
+    brew install sqlcipher
+fi
+
+# Install Python package (PDM handles platform-specific dependencies automatically)
+pdm install
+```

docs/SearXNG-Setup.md

@@ -0,0 +1,139 @@
+# SearXNG Integration for Local Deep Research
+
+This document explains how to configure and use the SearXNG integration with Local Deep Research.
+
+## Configuring SearXNG Access
+
+The SearXNG search engine is **disabled by default** until you provide an instance URL. This ensures the system doesn't attempt to use public instances without explicit configuration.
+
+### Setting Up Access
+
+You have two ways to enable the SearXNG search engine:
+
+1. **Environment Variable (Recommended)**:
+   ```bash
+   # Add to your .env file or set in your environment
+   SEARXNG_INSTANCE=http://localhost:8080
+
+   # Optional: Set custom delay between requests (in seconds)
+   SEARXNG_DELAY=2.0
+   ```
+
+2. **Configuration Parameter**: Add to your `config.py`:
+   ```python
+   # In config.py
+   SEARXNG_CONFIG = {
+       "instance_url": "http://localhost:8080",
+       "delay_between_requests": 2.0
+   }
+   ```
+
+## Self-Hosting SearXNG (Recommended)
+
+For the most ethical usage, we strongly recommend self-hosting your own SearXNG instance:
+
+### Using Docker (easiest method)
+
+```bash
+# Pull the SearXNG Docker image
+docker pull searxng/searxng
+
+# Run SearXNG (will be available at http://localhost:8080)
+docker run -d -p 8080:8080 --name searxng searxng/searxng
+```
+
+### Using Docker Compose (recommended for production)
+
+1. Create a file named `docker-compose.yml` with the following content:
+
+```yaml
+version: '3'
+services:
+  searxng:
+    container_name: searxng
+    image: searxng/searxng
+    ports:
+      - "8080:8080"
+    volumes:
+      - ./searxng:/etc/searxng
+    environment:
+      - SEARXNG_BASE_URL=http://localhost:8080/
+    restart: unless-stopped
+```
+
+2. Run with Docker Compose:
+
+```bash
+docker-compose up -d
+```
+
+## Using Public Instances
+
+If you must use a public instance:
+
+1. **Get Permission**: Always contact the administrator of any public instance
+2. **Respect Resources**: Use a longer delay (4-5 seconds minimum) between requests
+3. **Limited Usage**: Keep your research volume reasonable
+
+Example configuration for a public instance:
+```bash
+SEARXNG_INSTANCE=https://instance.example.com
+SEARXNG_DELAY=5.0
+```
+
+## Checking Configuration
+
+To verify if SearXNG is properly configured:
+
+```python
+from web_search_engines.search_engine_factory import create_search_engine
+
+# Create the engine
+engine = create_search_engine("searxng")
+
+# Check if available
+if engine and hasattr(engine, 'is_available') and engine.is_available:
+    print(f"SearXNG configured with instance: {engine.instance_url}")
+    print(f"Delay between requests: {engine.delay_between_requests} seconds")
+else:
+    print("SearXNG is not properly configured or is disabled")
+```
+
+## Network Security
+
+SearXNG is designed for self-hosting, so Local Deep Research allows SearXNG to access **private network IPs** by default. This means you can run SearXNG on:
+
+- **Localhost**: `http://127.0.0.1:8080` or `http://localhost:8080`
+- **LAN IPs**: `http://192.168.1.100:8080`, `http://10.0.0.5:8080`, `http://172.16.0.2:8080`
+- **Docker networks**: `http://172.17.0.2:8080`
+- **Local hostnames**: `http://searxng.local:8080` (if configured in DNS/hosts)
+
+This is intentional and secure because:
+1. The SearXNG URL is **admin-configured**, not user input
+2. Private IPs are only accessible from your local network
+3. **Cloud metadata endpoints** (AWS IMDS / ECS, Azure, OCI, DigitalOcean, AlibabaCloud, Tencent Cloud — see `ssrf_validator.ALWAYS_BLOCKED_METADATA_IPS`) are always blocked to prevent credential theft in cloud environments
+
+### IPv6-only deployments (NAT64)
+
+The "private IPs allowed" exception above does **not** cover IPv6 transition prefixes. On IPv6-only Kubernetes / cloud deployments (AWS / GKE / Azure IPv6-only nodes) where outbound IPv4 traffic is synthesized through NAT64 (`64:ff9b::/96` RFC 6052 well-known or `64:ff9b:1::/48` RFC 8215 local-use), reaching a SearXNG instance through these prefixes is blocked by default. To opt in, set:
+
+```bash
+LDR_SECURITY_ALLOW_NAT64=true
+```
+
+The opt-in is scoped strictly to the two NAT64 prefixes — 6to4 (`2002::/16`), Teredo (`2001::/32`), the discard prefix (`100::/64`), and the deprecated IPv4-Compatible IPv6 form (`::/96`) remain blocked, and cloud-metadata IPs stay unreachable through any NAT64 wrap. See [SECURITY.md](../SECURITY.md#ipv6-transition-prefix-block-list) for the full rationale.
+
+## Troubleshooting
+
+If you encounter errors:
+
+1. Check that your instance is running
+2. Verify the URL is correct in your environment variables
+3. Ensure you can access the instance in your browser
+4. Check firewall settings and network connectivity
+
+## Resources
+
+- [SearXNG Documentation](https://searxng.github.io/searxng/)
+- [SearXNG GitHub Repository](https://github.com/searxng/searxng)
+- [SearXNG Docker Hub](https://hub.docker.com/r/searxng/searxng)

docs/analytics-dashboard.md

@@ -0,0 +1,246 @@
+# Metrics Dashboard Guide
+
+The Local Deep Research metrics dashboard provides insights into your research activities, costs, and system performance.
+
+> **Note**: This documentation is maintained by the community and may contain inaccuracies. While we strive to keep it up-to-date, please verify critical information and report any errors via [GitHub Issues](https://github.com/LearningCircuit/local-deep-research/issues).
+
+## Overview
+
+The metrics dashboard tracks:
+- Token usage and costs across LLM providers
+- Search engine performance and health
+- Research session statistics
+- User satisfaction ratings
+- Rate limiting status
+
+## Accessing Metrics
+
+- **Web Interface**: Navigate to `/metrics` in your browser
+- **Direct URL**: `http://localhost:5000/metrics`
+
+## Dashboard Components
+
+### System Overview Cards
+
+The main dashboard displays key metrics:
+
+- **Total Tokens Used**: Shows total token consumption with expandable breakdown by model
+- **Total Researches**: Number of research sessions conducted
+- **Average Response Time**: System performance metric
+- **Success Rate**: Percentage of successful operations
+- **User Satisfaction**: Average star rating from user feedback
+- **Estimated Costs**: Token-based cost estimation with provider breakdown
+
+### Time-based Filtering
+
+Filter analytics by time period:
+- Last 7 days
+- Last 30 days
+- Last 3 months
+- Last year
+- All time
+
+Additional filters:
+- Research mode (Quick Summary / Detailed / All)
+
+## Detailed Analytics Pages
+
+### Star Reviews Analytics
+
+Access via: `/metrics/star-reviews`
+
+**Features:**
+- 5-star rating distribution
+- Average ratings by time period
+- Rating trends visualization
+- Breakdown by model and search engine
+- User feedback analysis
+
+### Cost Analytics
+
+Access via: `/metrics/costs`
+
+**Tracked Metrics:**
+- Cost breakdown by provider (OpenAI, Anthropic, etc.)
+- Token usage details (input/output/total)
+- Cost trends over time
+- Model-specific cost analysis
+- Research type cost comparison
+
+### Rate Limiting Dashboard
+
+**Real-time monitoring of:**
+- Search engine rate limit status
+- Success/failure rates per engine
+- Wait time tracking
+- Engine health indicators:
+  - 🟢 Healthy: >95% success rate
+  - 🟡 Degraded: 70-95% success rate
+  - 🔴 Poor: <70% success rate
+
+## Metrics Tracked
+
+### Token Metrics
+- Total tokens (input + output)
+- Token usage by model
+- Average tokens per research
+- Token consumption trends
+
+### Search Metrics
+- Search engine usage frequency
+- Response times per engine
+- Success/failure rates
+- Results count statistics
+
+### Research Metrics
+- Total research sessions
+- Research duration
+- Completion status
+- Strategy usage
+- Query complexity
+
+### Performance Metrics
+- API response times
+- System latency
+- Error rates
+- Throughput statistics
+
+## Data Export
+
+### Research Reports
+Export individual research results as:
+- **PDF**: Formatted reports with citations
+- **Markdown**: Raw markdown with formatting
+- **JSON**: Structured data via API
+
+### Analytics Data
+Access analytics data via API:
+
+```bash
+# Get overall metrics
+curl http://localhost:5000/api/metrics
+
+# Get specific research metrics
+curl http://localhost:5000/api/metrics/research/<research_id>
+
+# Get enhanced tracking data
+curl http://localhost:5000/api/metrics/enhanced
+
+# Get rating analytics
+curl http://localhost:5000/api/star-reviews
+
+# Get cost analytics
+curl http://localhost:5000/api/cost-analytics
+
+# Get rate limiting status
+curl http://localhost:5000/api/rate-limiting
+```
+
+## Visualizations
+
+The dashboard uses Chart.js to display:
+
+- **Line Charts**: Token usage and search activity over time
+- **Bar Charts**: Model usage comparison, cost breakdown
+- **Pie Charts**: Provider distribution, search engine usage
+- **Progress Indicators**: Success rates, health status
+
+## Cost Tracking
+
+### Automatic Cost Calculation
+
+Costs are calculated based on:
+- Provider pricing (OpenAI, Anthropic, etc.)
+- Actual token usage
+- Model-specific rates
+- Both input and output tokens
+
+### Supported Providers
+- OpenAI (GPT-3.5, GPT-4)
+- Anthropic (Claude models)
+- Google (Gemini models)
+- Local models (shown as $0)
+
+## Rate Limiting Analytics
+
+### Monitoring Features
+- Real-time rate limit status
+- Historical rate limit events
+- Automatic wait time optimization
+- Per-engine performance tracking
+
+### Managing Rate Limits
+
+Via command line:
+```bash
+# Check status
+python -m local_deep_research.web_search_engines.rate_limiting status
+
+# Reset rate limits
+python -m local_deep_research.web_search_engines.rate_limiting reset
+```
+
+## Privacy and Data Storage
+
+- All analytics data is stored locally
+- No external analytics services used
+- Data stored in SQLite databases
+- Configurable data retention
+
+## Using Analytics for Optimization
+
+### Identify Cost Drivers
+1. Review high-token queries in cost analytics
+2. Compare model costs vs. quality ratings
+3. Optimize model selection for different tasks
+
+### Improve Search Performance
+1. Monitor search engine health status
+2. Identify frequently rate-limited engines
+3. Adjust search strategy based on success rates
+
+### Enhance Research Quality
+1. Analyze user ratings by research type
+2. Review low-rated sessions for patterns
+3. Adjust parameters based on feedback
+
+## Benchmarking Integration
+
+For advanced users, analytics integrates with the benchmarking system:
+
+- Track performance across different configurations
+- Visualize optimization results (when matplotlib available)
+- Compare quality vs. speed trade-offs
+- Export benchmark data for analysis
+
+## API Reference
+
+### Metrics Endpoints
+
+```bash
+# General metrics with optional time filter
+curl 'http://localhost:5000/api/metrics?days=30&mode=quick'
+
+# Research-specific metrics
+curl http://localhost:5000/api/metrics/research/<id>
+
+# Enhanced metrics (detailed tracking)
+curl http://localhost:5000/api/metrics/enhanced
+
+# Star ratings data
+curl 'http://localhost:5000/api/star-reviews?days=30'
+
+# Cost breakdown
+curl 'http://localhost:5000/api/cost-analytics?provider=openai'
+
+# Rate limit status
+curl http://localhost:5000/api/rate-limiting
+```
+
+## Related Documentation
+
+- [Features Documentation](features.md)
+- [Configuration Guide](env_configuration.md)
+- [Full Configuration Reference](CONFIGURATION.md)
+- [API Documentation](api-quickstart.md)
+- [Benchmarking Guide](BENCHMARKING.md)

docs/api-quickstart.md

@@ -0,0 +1,231 @@
+# API Quick Start
+
+## Overview
+
+Local Deep Research provides both HTTP REST API and programmatic Python API access. Since version 1.0, authentication is required for all API endpoints, and the system uses per-user encrypted databases.
+
+## Simplest Usage - Python Client
+
+The easiest way to use the API is with the built-in client that handles all authentication complexity:
+
+```python
+from local_deep_research.api import LDRClient, quick_query
+
+# One-liner for quick research
+summary = quick_query("username", "password", "What is DNA?")
+print(summary)
+
+# Or use the client for multiple operations
+client = LDRClient()
+client.login("username", "password")
+result = client.quick_research("What is machine learning?")
+print(result["summary"])
+```
+
+No need to worry about CSRF tokens, HTML parsing, or session management!
+
+## Authentication
+
+### Web UI Authentication
+
+The API requires authentication through the web interface first:
+
+1. Start the server:
+   ```bash
+   python -m local_deep_research.web.app
+   ```
+
+2. Open http://localhost:5000 in your browser
+3. Register a new account or login
+4. Your session cookie will be used for API authentication
+
+### HTTP API Authentication
+
+For HTTP API requests, you need to:
+
+1. First authenticate through the login endpoint
+2. Include the session cookie in subsequent requests
+3. Include CSRF token for state-changing operations
+
+Example authentication flow:
+
+```python
+import requests
+from bs4 import BeautifulSoup
+
+# Create a session to persist cookies
+session = requests.Session()
+
+# 1. Get login page and extract CSRF token for login
+login_page = session.get("http://localhost:5000/auth/login")
+soup = BeautifulSoup(login_page.text, 'html.parser')
+csrf_input = soup.find('input', {'name': 'csrf_token'})
+login_csrf = csrf_input.get('value') if csrf_input else None
+
+# 2. Login with form data (not JSON) including CSRF
+login_response = session.post(
+    "http://localhost:5000/auth/login",
+    data={
+        "username": "your_username",
+        "password": "your_password",
+        "csrf_token": login_csrf
+    }
+)
+
+if login_response.status_code in [200, 302]:
+    print("Login successful")
+    # Session cookie is automatically stored
+else:
+    print(f"Login failed: {login_response.text}")
+
+# 3. Get CSRF token for API requests
+csrf_response = session.get("http://localhost:5000/auth/csrf-token")
+csrf_token = csrf_response.json()["csrf_token"]
+
+# 4. Make API requests with CSRF header
+headers = {"X-CSRF-Token": csrf_token}
+api_response = session.post(
+    "http://localhost:5000/api/start_research",
+    json={
+        "query": "What is quantum computing?",
+        "model": "gpt-3.5-turbo",
+        "search_engines": ["searxng"],
+    },
+    headers=headers
+)
+```
+
+## Programmatic API Access
+
+The programmatic API now requires a settings snapshot for proper context:
+
+```python
+from local_deep_research.api import quick_summary
+from local_deep_research.settings import SettingsManager
+from local_deep_research.database.session_context import get_user_db_session
+
+# Get user session and settings
+with get_user_db_session(username="your_username", password="your_password") as session:
+    settings_manager = SettingsManager(session)
+    settings_snapshot = settings_manager.get_all_settings()
+
+    # Use the API with settings snapshot
+    result = quick_summary(
+        query="What is machine learning?",
+        settings_snapshot=settings_snapshot,
+        iterations=2,
+        questions_per_iteration=3
+    )
+
+    print(result["summary"])
+```
+
+## API Endpoints
+
+### Research Endpoints
+
+Research endpoints are under `/api/`:
+
+- `POST /api/start_research` - Start new research
+- `GET /api/research/{id}/status` - Check research status
+- `GET /api/report/{id}` - Get research results
+- `POST /api/terminate/{id}` - Stop running research
+
+### Settings Endpoints
+
+Settings endpoints are under `/settings/api/`:
+
+- `GET /settings/api` - Get all settings
+- `GET /settings/api/{key}` - Get specific setting
+- `PUT /settings/api/{key}` - Update setting
+- `GET /settings/api/available-models` - Get available LLM providers
+- `GET /settings/api/available-search-engines` - Get search engines
+
+### History Endpoints
+
+- `GET /history/api` - Get research history
+- `GET /history/api/{id}` - Get specific research details
+
+## Important Changes from v1.x
+
+1. **Authentication Required**: All API endpoints now require authentication
+2. **Settings Snapshot**: Programmatic API calls need settings_snapshot parameter
+3. **Per-User Databases**: Each user has their own encrypted database
+4. **CSRF Protection**: State-changing requests require CSRF token
+5. **New Endpoint Structure**: Research APIs are under `/api/` (e.g., `/api/start_research`)
+
+## Example: Complete Research Flow
+
+```python
+import requests
+import time
+
+# Setup session and login
+session = requests.Session()
+session.post(
+    "http://localhost:5000/auth/login",
+    json={"username": "user", "password": "pass"}
+)
+
+# Get CSRF token
+csrf = session.get("http://localhost:5000/auth/csrf-token").json()["csrf_token"]
+headers = {"X-CSRF-Token": csrf}
+
+# Start research
+research = session.post(
+    "http://localhost:5000/api/start_research",
+    json={
+        "query": "Latest advances in quantum computing",
+        "model": "gpt-3.5-turbo",
+        "search_engines": ["arxiv", "wikipedia"],
+        "iterations": 3
+    },
+    headers=headers
+).json()
+
+research_id = research["research_id"]
+
+# Poll for results
+while True:
+    status = session.get(
+        f"http://localhost:5000/api/research/{research_id}/status"
+    ).json()
+
+    if status["status"] in ["completed", "failed"]:
+        break
+
+    print(f"Progress: {status.get('progress', 'unknown')}")
+    time.sleep(5)
+
+# Get final results
+results = session.get(
+    f"http://localhost:5000/api/report/{research_id}"
+).json()
+
+print(f"Summary: {results['summary']}")
+print(f"Sources: {len(results['sources'])}")
+```
+
+## Rate Limiting
+
+The API includes adaptive rate limiting:
+- Default: 60 requests per minute per user
+- Automatic retry with exponential backoff
+- Rate limits are per-user, not per-IP
+
+## Error Handling
+
+Common error responses:
+- `401`: Not authenticated - login required
+- `403`: CSRF token missing or invalid
+- `404`: Resource not found
+- `429`: Rate limit exceeded
+- `500`: Server error
+
+Always check response status and handle errors appropriately.
+
+## Next Steps
+
+- See [examples/api_usage](../examples/api_usage/) for complete examples
+- Check [docs/CUSTOM_LLM_INTEGRATION.md](CUSTOM_LLM_INTEGRATION.md) for custom LLM setup
+- Read [docs/LANGCHAIN_RETRIEVER_INTEGRATION.md](LANGCHAIN_RETRIEVER_INTEGRATION.md) for custom retrievers

docs/architecture.md

@@ -0,0 +1,582 @@
+# Architecture Overview
+
+This document provides detailed technical diagrams of Local Deep Research's architecture.
+
+## System Architecture
+
+```mermaid
+flowchart TB
+    subgraph USER["👤 User Interface"]
+        WEB[Web Browser<br/>localhost:5000]
+    end
+
+    subgraph FLASK["🌐 Flask Backend"]
+        API[REST API + WebSocket]
+        AUTH[Authentication<br/>CSRF Protection]
+        ROUTES[Research Routes]
+    end
+
+    subgraph RESEARCH["🔬 Research Engine"]
+        STRAT[Strategy Selector]
+        QGEN[Question Generator]
+        EXEC[Search Executor]
+        SYNTH[Report Synthesizer]
+    end
+
+    subgraph LLM["🤖 LLM Providers"]
+        direction TB
+        LOCAL_LLM[/"🏠 Local LLMs"\]
+        OLLAMA[Ollama]
+        LMSTUDIO[LM Studio]
+        CLOUD_LLM[/"☁️ Cloud LLMs"\]
+        OPENAI[OpenAI]
+        ANTHROPIC[Anthropic]
+        GEMINI[Google Gemini]
+        OPENROUTER[OpenRouter<br/>100+ models]
+    end
+
+    subgraph SEARCH["🔍 Search Engines"]
+        direction TB
+        LOCAL_SEARCH[/"🏠 Local Search"\]
+        SEARXNG[SearXNG]
+        ELASTIC[Elasticsearch]
+        LIBRARY[Document Library]
+        WEB_SEARCH[/"🌐 Web Search"\]
+        TAVILY[Tavily]
+        BRAVE[Brave]
+        DDG[DuckDuckGo]
+        ACADEMIC[/"📚 Academic"\]
+        ARXIV[ArXiv]
+        PUBMED[PubMed]
+        SEMANTIC[Semantic Scholar]
+    end
+
+    subgraph STORAGE["💾 Storage Layer"]
+        SQLCIPHER[(SQLCipher DB<br/>AES-256 Encrypted)]
+        VECTORS[(Vector Store<br/>Embeddings)]
+        FILES[(File Storage<br/>PDFs & Docs)]
+    end
+
+    subgraph OUTPUT["📄 Output"]
+        MD[Markdown]
+        PDF[PDF Export]
+        LATEX[LaTeX]
+        QUARTO[Quarto]
+        RIS[RIS/BibTeX]
+    end
+
+    WEB <--> API
+    API --> AUTH
+    AUTH --> ROUTES
+    ROUTES --> STRAT
+
+    STRAT --> QGEN
+    QGEN --> EXEC
+    EXEC --> SYNTH
+
+    EXEC <--> OLLAMA & LMSTUDIO
+    EXEC <--> OPENAI & ANTHROPIC & GEMINI & OPENROUTER
+
+    EXEC <--> SEARXNG & ELASTIC & LIBRARY
+    EXEC <--> TAVILY & BRAVE & DDG
+    EXEC <--> ARXIV & PUBMED & SEMANTIC
+
+    SYNTH --> SQLCIPHER
+    LIBRARY <--> VECTORS
+    LIBRARY <--> FILES
+
+    SYNTH --> MD & PDF & LATEX & QUARTO & RIS
+
+    LOCAL_LLM ~~~ OLLAMA
+    CLOUD_LLM ~~~ OPENAI
+    LOCAL_SEARCH ~~~ SEARXNG
+    WEB_SEARCH ~~~ TAVILY
+    ACADEMIC ~~~ ARXIV
+
+    style USER fill:#e1f5fe
+    style FLASK fill:#fff3e0
+    style RESEARCH fill:#f3e5f5
+    style LLM fill:#e8f5e9
+    style SEARCH fill:#fce4ec
+    style STORAGE fill:#fff8e1
+    style OUTPUT fill:#e0f2f1
+```
+
+## Research Flow
+
+```mermaid
+flowchart LR
+    subgraph INPUT["1️⃣ Input"]
+        Q[Research Query]
+    end
+
+    subgraph ITERATE["2️⃣ Iterative Research"]
+        direction TB
+        GEN[Generate<br/>Questions]
+        SEARCH[Parallel<br/>Search]
+        ANALYZE[Analyze<br/>Results]
+        GEN --> SEARCH --> ANALYZE
+        ANALYZE -.->|"Need more info?"| GEN
+    end
+
+    subgraph SYNTHESIZE["3️⃣ Synthesis"]
+        REPORT[Generate<br/>Report]
+        CITE[Add<br/>Citations]
+        REPORT --> CITE
+    end
+
+    subgraph OUTPUT["4️⃣ Output"]
+        RESULT[Final Report<br/>with Sources]
+    end
+
+    Q --> GEN
+    ANALYZE --> REPORT
+    CITE --> RESULT
+
+    style INPUT fill:#e3f2fd
+    style ITERATE fill:#f3e5f5
+    style SYNTHESIZE fill:#e8f5e9
+    style OUTPUT fill:#fff3e0
+```
+
+## Deployment Options
+
+```mermaid
+flowchart TB
+    subgraph FULL_LOCAL["🏠 Fully Local (Maximum Privacy)"]
+        direction LR
+        L_LDR[LDR] <--> L_OLLAMA[Ollama]
+        L_LDR <--> L_SEARX[SearXNG]
+        L_LDR <--> L_DB[(Encrypted DB)]
+    end
+
+    subgraph HYBRID["⚡ Hybrid (Balanced)"]
+        direction LR
+        H_LDR[LDR] <--> H_OLLAMA[Ollama]
+        H_LDR <-->|"Web Search"| H_CLOUD[Cloud APIs]
+        H_LDR <--> H_DB[(Encrypted DB)]
+    end
+
+    subgraph CLOUD["☁️ Cloud-Powered (Maximum Speed)"]
+        direction LR
+        C_LDR[LDR] <--> C_OPENAI[OpenAI/Claude]
+        C_LDR <--> C_TAVILY[Tavily/Brave]
+        C_LDR <--> C_DB[(Encrypted DB)]
+    end
+
+    style FULL_LOCAL fill:#e8f5e9
+    style HYBRID fill:#fff3e0
+    style CLOUD fill:#e3f2fd
+```
+
+## Feature Map
+
+```mermaid
+mindmap
+  root((Local Deep<br/>Research))
+    Research
+      Quick Summary
+      Detailed Reports
+      Follow-up Questions
+      Research Strategies
+    Search Sources
+      Web Search
+        SearXNG
+        Tavily
+        Brave
+        DuckDuckGo
+      Academic
+        ArXiv
+        PubMed
+        Semantic Scholar
+      Local
+        Document Library
+        Collections
+        Elasticsearch
+    LLM Support
+      Local
+        Ollama
+        LM Studio
+      Cloud
+        OpenAI
+        Anthropic
+        Google
+        OpenRouter
+    Output
+      Markdown
+      PDF
+      LaTeX
+      Quarto
+      RIS/BibTeX
+    Features
+      News Subscriptions
+      Cost Analytics
+      Benchmarking
+      RAG Search
+      Per-User Encryption
+```
+
+## Component Details
+
+### LLM Providers
+
+| Provider | Type | Description |
+|----------|------|-------------|
+| Ollama | Local | Self-hosted open-source models |
+| LM Studio | Local | Desktop app for local models |
+| OpenAI | Cloud | GPT-4, GPT-3.5 |
+| Anthropic | Cloud | Claude 3 family |
+| Google | Cloud | Gemini models |
+| OpenRouter | Cloud | 100+ models via single API |
+
+### Search Engines
+
+| Engine | Type | Best For |
+|--------|------|----------|
+| SearXNG | Local/Self-hosted | Privacy, aggregated results |
+| Tavily | Cloud | AI-optimized search |
+| ArXiv | Academic | Physics, CS, Math papers |
+| PubMed | Academic | Biomedical research |
+| Semantic Scholar | Academic | Cross-discipline papers |
+| Wikipedia | Knowledge | General knowledge |
+| Your Documents | Local | Private document search |
+
+### Output Formats
+
+| Format | Use Case |
+|--------|----------|
+| Markdown | Default, web display |
+| PDF | Sharing, printing |
+| LaTeX | Academic papers |
+| Quarto | Reproducible documents |
+| RIS/BibTeX | Reference managers |
+
+## Knowledge Loop: Research → Library → Future Research
+
+One of LDR's powerful features is the ability to build a personal knowledge base that improves future research.
+
+```mermaid
+flowchart TB
+    subgraph RESEARCH["1️⃣ Research"]
+        Q[Your Question] --> Engine[Research Engine]
+        Engine --> Results[Results + Sources]
+    end
+
+    subgraph DOWNLOAD["2️⃣ Download"]
+        Results --> Track[Track Sources]
+        Track --> Queue[Download Queue]
+        Queue --> Extract[Download & Extract Text]
+    end
+
+    subgraph LIBRARY["3️⃣ Library"]
+        Extract --> Store[(Document Storage)]
+        Store --> Organize[Organize into Collections]
+    end
+
+    subgraph INDEX["4️⃣ Index"]
+        Organize --> Chunk[Chunk Documents]
+        Chunk --> Embed[Generate Embeddings]
+        Embed --> FAISS[(Vector Index)]
+    end
+
+    subgraph REUSE["5️⃣ Reuse"]
+        FAISS --> SearchEngine[Collection as Search Engine]
+        SearchEngine --> Q
+    end
+
+    style RESEARCH fill:#e3f2fd
+    style DOWNLOAD fill:#fff3e0
+    style LIBRARY fill:#e8f5e9
+    style INDEX fill:#f3e5f5
+    style REUSE fill:#fce4ec
+```
+
+### How It Works
+
+1. **Research Completes** → Sources are tracked in `ResearchResource` table
+2. **Download Sources** → Click "Get All Research PDFs" to queue downloads
+   - Smart downloaders for ArXiv, PubMed, Semantic Scholar, etc.
+   - Automatic text extraction from PDFs
+3. **Build Library** → Documents stored in encrypted database
+   - Deduplication via content hash
+   - Multiple storage modes: database (encrypted), filesystem, text-only
+4. **Create Collections** → Organize documents by topic/project
+   - Each collection can have different embedding settings
+   - Documents can belong to multiple collections
+5. **Index for Search** → Generate vector embeddings
+   - Configurable chunk size and overlap
+   - FAISS index for fast similarity search
+6. **Use in Future Research** → Select collection as search engine
+   - RAG search finds relevant passages
+   - Results cite back to your documents
+
+### Key Components
+
+| Component | Purpose |
+|-----------|---------|
+| `DownloadService` | Manages PDF downloads with source-specific strategies |
+| `LibraryService` | Queries and manages document library |
+| `LibraryRAGService` | Creates vector indices for semantic search |
+| `CollectionSearchEngine` | Searches collections using RAG |
+| `Document` | Stores text content, metadata, file references |
+| `DocumentChunk` | Stores indexed text chunks with embeddings |
+| `Collection` | Groups documents with shared embedding settings |
+
+### Storage Options
+
+| Mode | Security | Use Case |
+|------|----------|----------|
+| Database | AES-256 encrypted | Default, maximum security |
+| Filesystem | Unencrypted | Need external tool access |
+| Text Only | Encrypted text, no PDFs | Minimal storage |
+
+---
+
+## Technical Analysis & Project Health
+
+*Last updated: December 2024*
+
+This section provides a comprehensive technical analysis of the codebase, including quality metrics, architecture patterns, and project health indicators.
+
+### Project Statistics
+
+| Metric | Count |
+|--------|-------|
+| Test Classes | 809+ |
+| Search Engine Implementations | 25 |
+| LLM Provider Implementations | 9 |
+| Search Strategies | 5 |
+| Abstract Base Classes | 26 |
+| CI/CD Workflows | 57 |
+| Security Scanners in CI | 22+ |
+| Core Dependencies | 63 |
+
+### Architecture Patterns
+
+#### Extensibility Design
+
+The codebase follows a consistent pattern for extensibility:
+
+```
+┌─────────────────────────────────────────────────────────────┐
+│                    Abstract Base Classes                     │
+├─────────────────────────────────────────────────────────────┤
+│ BaseSearchEngine      │ Common interface for 25+ engines    │
+│ BaseSearchStrategy    │ Strategy pattern for research       │
+│ BaseCitationHandler   │ Citation processing abstraction     │
+│ BaseQuestionGenerator │ Question generation interface       │
+│ BaseExporter          │ Export format abstraction           │
+└─────────────────────────────────────────────────────────────┘
+```
+
+#### Search Engine Plugin System
+
+New search engines can be added by:
+1. Creating a class inheriting from `BaseSearchEngine`
+2. Placing it in `web_search_engines/engines/`
+3. Auto-discovery handles registration
+
+```python
+# Example: Adding a new search engine
+class SearchEngineCustom(BaseSearchEngine):
+    def run(self, query: str) -> List[Dict]:
+        # Implementation
+        pass
+```
+
+#### LLM Provider Integration
+
+Supports 9 LLM providers with auto-discovery:
+- Ollama (local)
+- LM Studio (local)
+- OpenAI
+- Anthropic
+- Google Gemini
+- OpenRouter (100+ models)
+- DeepSeek
+- Mistral
+- Groq
+
+### Quality Ratings
+
+```mermaid
+pie title Project Health Ratings
+    "Code Quality (86)" : 86
+    "Security (93)" : 93
+    "Extensibility (93)" : 93
+    "Performance (85)" : 85
+    "UX/DX (83)" : 83
+    "Maintenance (89)" : 89
+```
+
+#### Detailed Ratings
+
+| Category | Score | Highlights |
+|----------|-------|------------|
+| **Code Quality** | 86/100 | 809+ test classes, ruff/mypy enforcement, comprehensive pre-commit hooks |
+| **Security** | 93/100 | SQLCipher AES-256 encryption, full SHA pinning in CI, 22+ security scanners |
+| **Extensibility** | 93/100 | 26 abstract base classes, plugin architecture, strategy pattern throughout |
+| **Performance** | 85/100 | Adaptive rate limiting, cache stampede protection, parallel search execution |
+| **UX/Developer Experience** | 83/100 | Real-time WebSocket updates, in-tool documentation, comprehensive error handling |
+| **Maintenance** | 89/100 | 57 CI/CD workflows, automated security scanning, structured changelog |
+| **Overall** | 88/100 | Production-ready with excellent security and extensibility |
+
+### Security Architecture
+
+```mermaid
+flowchart TB
+    subgraph SECURITY["🔐 Security Layers"]
+        direction TB
+        AUTH[Flask Session Auth<br/>CSRF Protection]
+        ENCRYPT[SQLCipher<br/>AES-256 Encryption]
+        SCAN[22+ Security Scanners<br/>CodeQL, Semgrep, Bandit...]
+        PIN[Full SHA Pinning<br/>All 57 CI Workflows]
+    end
+
+    AUTH --> ENCRYPT
+    ENCRYPT --> SCAN
+    SCAN --> PIN
+
+    style SECURITY fill:#e8f5e9
+```
+
+**Security Features:**
+- Per-user encrypted databases (SQLCipher with AES-256)
+- Full GitHub Action SHA pinning (not tag-based)
+- Comprehensive CI security scanning:
+  - CodeQL (Python, JavaScript)
+  - Semgrep (custom rulesets)
+  - Bandit (Python security)
+  - Trivy (container scanning)
+  - Dependency review
+  - Secret scanning
+
+### Performance Optimizations
+
+| Feature | Implementation |
+|---------|----------------|
+| **Parallel Search** | `concurrent.futures.ThreadPoolExecutor` for multi-question search |
+| **Rate Limiting** | Adaptive system with `learning_rate=0.3` |
+| **Cache Protection** | `fetch_events` + `fetch_locks` for stampede prevention |
+| **Progress Streaming** | SocketIO for real-time UI updates |
+| **Cross-Engine Filtering** | LLM-powered relevance scoring and deduplication |
+
+#### Thread & Resource Lifecycle
+
+Unlike typical web apps that share a single database connection pool, this application maintains **separate database engines per user** because each user has their own encrypted SQLite file with a unique SQLCipher key ([SQLCipher](https://www.zetetic.net/sqlcipher/) is an encrypted extension of SQLite). This creates a threading challenge: every background thread needs its own engine for the user it serves.
+
+A single **QueuePool** engine is kept per user (`pool_size=20`,
+`max_overflow=40`, `pool_timeout=10`) in
+`DatabaseManager.connections[username]`. It is created with
+`check_same_thread=False`, so it is shared safely across Flask
+request-handler threads AND background threads (research workers,
+scheduler jobs, metric writers). All of these acquire sessions from the
+same pool, which keeps FD usage bounded by `pool_size + max_overflow`
+per user.
+
+> Earlier revisions of the code maintained a second, parallel system of
+> per-`(username, thread_id)` **NullPool** engines for background metric
+> writes. That system was removed because orphaned entries leaked
+> SQLCipher + WAL file handles (3 FDs per active connection) under load,
+> eventually exhausting the 1024 FD soft limit. Routing all work through
+> the single per-user QueuePool made FD usage bounded and let us delete
+> ~200 lines of thread-engine bookkeeping.
+
+##### Resource Cleanup Layers
+
+Cleanup is defense-in-depth with multiple layers:
+
+| Layer | Trigger | What it cleans |
+|-------|---------|----------------|
+| `@thread_cleanup` decorator | Thread function exit (normal or exception) | DB session (returned to per-user pool), settings context, search context |
+| `finally` blocks | Per-research in `run_research_process()` | Search engine HTTP sessions, strategy thread pool executors |
+| `teardown_appcontext` | After each HTTP request | QueuePool session, thread-local session, triggers dead-thread credential sweep |
+| Periodic pool dispose | Every 30 min | Calls `engine.dispose()` on per-user QueuePool engines to release SQLCipher+WAL file handles that accumulate from out-of-order connection closes |
+| Logout cascade | User logout | Scheduler unregister (removes password) → DB close (`engine.dispose()`) → session destroy |
+| Stale session cleanup | `before_request` (~1% of requests, sampled) | Clears Flask sessions for users whose DB connection is gone |
+
+##### Research Thread Lifecycle
+
+```mermaid
+flowchart TD
+    A["POST /api/start_research"] --> B{"Slots available?"}
+    B -- "Yes (direct)" --> C["start_research_process()"]
+    B -- "No (queued)" --> D["Database queue"]
+    D --> E["_process_queue_loop<br/>(daemon thread, 10s poll)"]
+    E --> C
+
+    C --> F["Research thread<br/>(daemon, semaphore-gated)"]
+    F --> G["@thread_cleanup wraps<br/>run_research_process()"]
+
+    G --> H["Strategy.find_relevant_information()"]
+    H --> I["ThreadPoolExecutor sub-tasks<br/>(within strategies only)"]
+    I --> J["Worker @thread_cleanup:<br/>close worker DB session"]
+
+    J --> K["finally: search_engine.close()<br/>system.close() → strategy.close()"]
+    K --> L["Main @thread_cleanup:<br/>close DB session, return connection to pool"]
+
+    subgraph sweep ["Defense-in-depth (independent timers)"]
+        M["Dead-thread credential sweep<br/>(~60s via processor_v2)"] -.-> N["Remove stale credentials<br/>for dead thread IDs"]
+        O["Pool dispose<br/>(30 min via connection_cleanup)"] -.-> P["engine.dispose() on all<br/>QueuePool engines"]
+    end
+
+    style sweep fill:#f0f0f0,stroke:#999
+```
+
+##### FD Budget
+
+Each SQLCipher connection in WAL (Write-Ahead Logging) mode uses **2 file descriptors** (main db + WAL file). All connections to the same database within a process share **1 SHM** (shared-memory) file descriptor. The formula per user database is: `connections × 2 + 1`.
+
+| Component | FD Formula | With defaults |
+|-----------|-----------|---------------|
+| QueuePool (steady state) | `logged_in_users × (pool_size × 2 + 1)` | `users × 41` FDs |
+| QueuePool (peak) | `logged_in_users × ((pool_size + max_overflow) × 2 + 1)` | `users × 121` FDs |
+
+Default Linux ulimit is 1024 soft (bare metal), which is tight for multi-user deployments. Docker's daemon default (typically 1M+) is adequate. QueuePool engines are created at login and disposed at logout, so only active users consume FDs.
+
+For more on diagnosing FD exhaustion, see [Troubleshooting - Resource Exhaustion](./troubleshooting.md#resource-exhaustion).
+
+##### Key Files
+
+| File | Role |
+|------|------|
+| `src/local_deep_research/database/encrypted_db.py` | `DatabaseManager`, engine lifecycle, pool management |
+| `src/local_deep_research/database/thread_local_session.py` | `@thread_cleanup` decorator, thread-local sessions, credential cleanup |
+| `src/local_deep_research/web/app_factory.py` | `teardown_appcontext` handler, cleanup orchestration |
+| `src/local_deep_research/web/services/research_service.py` | Research thread creation, `run_research_process()` |
+| `src/local_deep_research/web/queue/processor_v2.py` | Queue processing, credential cleanup trigger |
+
+### Areas for Improvement
+
+While the project scores highly overall, these areas have room for growth:
+
+1. **Integration Testing** - More end-to-end tests for full research workflows
+2. **API Documentation** - OpenAPI/Swagger spec for REST endpoints
+3. **Metrics Dashboard** - Prometheus/Grafana integration for monitoring
+4. **Resource Observability** - Expose FD count, thread count, and connection pool stats in /api/v1/health; add periodic sweep logging
+5. **Async Architecture** - Migration to async/await for I/O-bound operations
+
+### Key Source Files
+
+| Component | Location | Purpose |
+|-----------|----------|---------|
+| Research Engine | `src/local_deep_research/search_system.py` | Main `AdvancedSearchSystem` class |
+| Strategies | `src/local_deep_research/advanced_search_system/strategies/` | Research strategy implementations |
+| Search Engines | `src/local_deep_research/web_search_engines/engines/` | 25 search engine implementations |
+| Report Generation | `src/local_deep_research/report_generator.py` | `IntegratedReportGenerator` |
+| Web API | `src/local_deep_research/web/routes/` | Flask routes and WebSocket handlers |
+| Database | `src/local_deep_research/web/database/` | SQLCipher models and migrations |
+| Encrypted DB | `src/local_deep_research/database/encrypted_db.py` | Per-user SQLCipher engine lifecycle |
+| Thread Sessions | `src/local_deep_research/database/thread_local_session.py` | Thread-safe session management and cleanup |
+| Settings | `src/local_deep_research/config/` | Configuration and LLM setup |
+
+### Contributing to Architecture
+
+When extending the system:
+
+1. **Adding Search Engines**: Inherit from `BaseSearchEngine`, implement `run()` method
+2. **Adding Strategies**: Inherit from `BaseSearchStrategy`, implement `analyze_topic()` method
+3. **Adding LLM Providers**: Add to `config/llm_config.py` with proper initialization
+4. **Adding Export Formats**: Inherit from base exporter pattern in `utilities/`
+
+See [CONTRIBUTING.md](../CONTRIBUTING.md) for detailed guidelines.

docs/architecture/DATABASE_SCHEMA.md

@@ -0,0 +1,608 @@
+# Database Schema
+
+This document describes the database models and their relationships in Local Deep Research.
+
+## Table of Contents
+
+- [Overview](#overview)
+- [Entity Relationship Diagram](#entity-relationship-diagram)
+- [Model Groups](#model-groups)
+  - [Research Domain](#research-domain)
+  - [Authentication](#authentication)
+  - [Settings](#settings)
+  - [Library & Documents](#library--documents)
+  - [Queue Management](#queue-management)
+  - [Metrics & Analytics](#metrics--analytics)
+  - [News System](#news-system)
+  - [Benchmarking](#benchmarking)
+  - [Rate Limiting](#rate-limiting)
+  - [File Integrity](#file-integrity)
+
+---
+
+## Overview
+
+Local Deep Research uses **SQLAlchemy ORM** with **SQLCipher** for encryption.
+
+**Key Characteristics:**
+- **Per-user databases**: Each user has their own encrypted SQLite database
+- **AES-256 encryption**: User password derives the encryption key
+- **HMAC verification**: Ensures database integrity
+- **Central auth database**: Only stores usernames (no passwords)
+
+**Location:** `src/local_deep_research/database/models/`
+
+---
+
+## Entity Relationship Diagram
+
+```mermaid
+erDiagram
+    %% Research Domain
+    ResearchTask ||--o{ SearchQuery : contains
+    ResearchTask ||--o{ SearchResult : produces
+    ResearchTask ||--o{ Report : generates
+    SearchQuery ||--o{ SearchResult : returns
+
+    Research ||--o{ ResearchHistory : tracks
+    Research ||--o{ ResearchResource : uses
+
+    Report ||--o{ ReportSection : contains
+
+    %% Library
+    Document ||--o{ DocumentChunk : splits_into
+    Document }o--o{ Collection : belongs_to
+    Collection ||--o{ RAGIndex : indexes
+
+    %% News
+    NewsSubscription ||--o{ NewsCard : produces
+    NewsCard ||--o{ UserRating : receives
+
+    %% Benchmarks
+    BenchmarkRun ||--o{ BenchmarkResult : contains
+    BenchmarkRun ||--o{ BenchmarkProgress : tracks
+
+    %% Queue
+    QueuedResearch ||--o| TaskMetadata : has
+
+    %% Metrics
+    Research ||--o{ TokenUsage : tracks
+    Research ||--o{ SearchCall : logs
+```
+
+---
+
+## Model Groups
+
+### Research Domain
+
+The core models for conducting research.
+
+#### ResearchTask
+
+Top-level research container.
+
+| Column | Type | Description |
+|--------|------|-------------|
+| `id` | Integer | Primary key |
+| `title` | String(500) | Research title |
+| `description` | Text | Detailed description |
+| `status` | String(50) | pending, in_progress, completed, failed |
+| `priority` | Integer | Priority level (higher = more urgent) |
+| `tags` | JSON | List of categorization tags |
+| `research_metadata` | JSON | Flexible metadata storage |
+| `created_at` | DateTime | Creation timestamp |
+| `updated_at` | DateTime | Last update timestamp |
+| `started_at` | DateTime | When research started |
+| `completed_at` | DateTime | When research completed |
+
+**Relationships:** `searches`, `results`, `reports`
+
+#### SearchQuery
+
+Individual search queries within a research task.
+
+| Column | Type | Description |
+|--------|------|-------------|
+| `id` | Integer | Primary key |
+| `research_task_id` | Integer | FK to ResearchTask |
+| `query` | Text | The search query text |
+| `search_engine` | String(50) | Engine used (duckduckgo, arxiv, etc.) |
+| `search_type` | String(50) | Type (web, academic, news) |
+| `parameters` | JSON | Additional search parameters |
+| `status` | String(50) | pending, executing, completed, failed |
+| `error_message` | Text | Error details if failed |
+| `retry_count` | Integer | Number of retry attempts |
+| `executed_at` | DateTime | When query was executed |
+
+**Indexes:** `idx_research_task_status`, `idx_search_engine`
+
+#### SearchResult
+
+Individual results from search queries.
+
+| Column | Type | Description |
+|--------|------|-------------|
+| `id` | Integer | Primary key |
+| `research_task_id` | Integer | FK to ResearchTask |
+| `search_query_id` | Integer | FK to SearchQuery |
+| `title` | String(500) | Result title |
+| `url` | Text | Result URL (indexed) |
+| `snippet` | Text | Brief preview |
+| `content` | Text | Full fetched content |
+| `content_type` | String(50) | html, pdf, text, etc. |
+| `content_hash` | String(64) | For deduplication |
+| `relevance_score` | Float | Calculated relevance |
+| `position` | Integer | Position in results |
+| `domain` | String(255) | Source domain (indexed) |
+| `language` | String(10) | Content language |
+| `published_date` | DateTime | Publication date |
+| `fetch_status` | String(50) | pending, fetched, failed, skipped |
+
+#### Research
+
+Simplified research record (alternative to ResearchTask).
+
+| Column | Type | Description |
+|--------|------|-------------|
+| `id` | Integer | Primary key |
+| `query` | Text | Original query |
+| `mode` | Enum | ResearchMode value |
+| `strategy` | Enum | ResearchStrategy value |
+| `status` | Enum | ResearchStatus value |
+| `result` | Text | Final result/summary |
+| `iterations` | Integer | Iterations completed |
+| `created_at` | DateTime | Creation timestamp |
+
+**Enums:**
+- `ResearchMode`: quick, detailed, report
+- `ResearchStatus`: pending, queued, in_progress, completed, suspended, failed, error, cancelled
+- `ResearchStrategy`: source-based, focused-iteration, etc.
+
+#### ResearchHistory
+
+Tracks research iterations and progress.
+
+| Column | Type | Description |
+|--------|------|-------------|
+| `id` | Integer | Primary key |
+| `research_id` | Integer | FK to Research |
+| `iteration` | Integer | Iteration number |
+| `questions` | JSON | Questions asked |
+| `findings` | JSON | Findings discovered |
+| `created_at` | DateTime | When recorded |
+
+#### Report / ReportSection
+
+Generated research reports.
+
+| Column (Report) | Type | Description |
+|-----------------|------|-------------|
+| `id` | Integer | Primary key |
+| `research_task_id` | Integer | FK to ResearchTask |
+| `title` | String(500) | Report title |
+| `format` | String(50) | markdown, pdf, latex |
+| `content` | Text | Full report content |
+| `created_at` | DateTime | Generation time |
+
+| Column (ReportSection) | Type | Description |
+|------------------------|------|-------------|
+| `id` | Integer | Primary key |
+| `report_id` | Integer | FK to Report |
+| `title` | String(255) | Section title |
+| `content` | Text | Section content |
+| `order` | Integer | Display order |
+
+---
+
+### Authentication
+
+User management with per-user encrypted databases.
+
+#### User
+
+Central user registry (stored in auth database, not user database).
+
+| Column | Type | Description |
+|--------|------|-------------|
+| `id` | Integer | Primary key |
+| `username` | String(80) | Unique username (indexed) |
+| `created_at` | DateTime | Registration date |
+| `last_login` | DateTime | Last login time |
+| `database_version` | Integer | Schema version |
+
+**Note:** Passwords are NEVER stored. They derive encryption keys.
+
+#### APIKey
+
+API keys for programmatic access.
+
+| Column | Type | Description |
+|--------|------|-------------|
+| `id` | Integer | Primary key |
+| `key_hash` | String(64) | Hashed API key |
+| `name` | String(100) | Key description |
+| `created_at` | DateTime | Creation date |
+| `last_used` | DateTime | Last usage |
+| `expires_at` | DateTime | Expiration date |
+| `is_active` | Boolean | Whether key is valid |
+
+---
+
+### Settings
+
+Configuration storage.
+
+#### Setting
+
+Global application settings.
+
+| Column | Type | Description |
+|--------|------|-------------|
+| `id` | Integer | Primary key |
+| `key` | String(255) | Setting key (unique, indexed) |
+| `value` | Text | Setting value |
+| `type` | Enum | SettingType (string, int, bool, json) |
+| `category` | String(100) | Setting category |
+| `description` | Text | Human-readable description |
+| `updated_at` | DateTime | Last update |
+
+#### UserSettings
+
+Per-user setting overrides.
+
+| Column | Type | Description |
+|--------|------|-------------|
+| `id` | Integer | Primary key |
+| `key` | String(255) | Setting key |
+| `value` | Text | User's value |
+| `updated_at` | DateTime | Last update |
+
+---
+
+### Library & Documents
+
+Document management for RAG.
+
+#### Document
+
+Documents in the research library.
+
+| Column | Type | Description |
+|--------|------|-------------|
+| `id` | Integer | Primary key |
+| `title` | String(500) | Document title |
+| `source_type` | FK → SourceType | Source type (see SourceType table) |
+| `source_url` | Text | Original source URL |
+| `file_path` | Text | Local file path |
+| `file_hash` | String(64) | Content hash |
+| `mime_type` | String(100) | MIME type |
+| `file_size` | Integer | Size in bytes |
+| `text_content` | Text | Extracted text |
+| `metadata` | JSON | Additional metadata |
+| `created_at` | DateTime | When added |
+| `indexed_at` | DateTime | When indexed for RAG |
+
+**SourceType** (normalized table): research_download, user_upload, manual_entry, research_report, research_source
+
+#### Collection
+
+Document collections for organization.
+
+| Column | Type | Description |
+|--------|------|-------------|
+| `id` | Integer | Primary key |
+| `name` | String(255) | Collection name |
+| `description` | Text | Description |
+| `is_default` | Boolean | Default collection flag |
+| `created_at` | DateTime | Creation date |
+
+#### DocumentCollection
+
+Junction table for document-collection relationship.
+
+| Column | Type | Description |
+|--------|------|-------------|
+| `document_id` | Integer | FK to Document |
+| `collection_id` | Integer | FK to Collection |
+
+#### DocumentChunk
+
+Text chunks for RAG indexing.
+
+| Column | Type | Description |
+|--------|------|-------------|
+| `id` | Integer | Primary key |
+| `document_id` | Integer | FK to Document |
+| `chunk_index` | Integer | Position in document |
+| `content` | Text | Chunk text |
+| `embedding` | BLOB | Vector embedding |
+| `metadata` | JSON | Chunk metadata |
+
+#### RAGIndex
+
+Vector index metadata.
+
+| Column | Type | Description |
+|--------|------|-------------|
+| `id` | Integer | Primary key |
+| `collection_id` | Integer | FK to Collection |
+| `status` | Enum | RAGIndexStatus |
+| `embedding_model` | String(100) | Model used |
+| `chunk_count` | Integer | Number of chunks |
+| `created_at` | DateTime | Creation time |
+| `updated_at` | DateTime | Last update |
+
+---
+
+### Queue Management
+
+Background task processing.
+
+#### QueuedResearch
+
+Research waiting to be processed.
+
+| Column | Type | Description |
+|--------|------|-------------|
+| `id` | Integer | Primary key |
+| `query` | Text | Research query |
+| `mode` | String(50) | Research mode |
+| `strategy` | String(50) | Strategy name |
+| `status` | Enum | QueueStatus |
+| `priority` | Integer | Queue priority |
+| `created_at` | DateTime | When queued |
+| `started_at` | DateTime | When started |
+| `completed_at` | DateTime | When finished |
+| `error` | Text | Error message if failed |
+
+**Enum QueueStatus:** pending, running, completed, failed, cancelled
+
+#### TaskMetadata
+
+Additional task information.
+
+| Column | Type | Description |
+|--------|------|-------------|
+| `id` | Integer | Primary key |
+| `queued_research_id` | Integer | FK to QueuedResearch |
+| `key` | String(255) | Metadata key |
+| `value` | Text | Metadata value |
+
+---
+
+### Metrics & Analytics
+
+Usage tracking and analytics.
+
+#### TokenUsage
+
+LLM token consumption.
+
+| Column | Type | Description |
+|--------|------|-------------|
+| `id` | Integer | Primary key |
+| `research_id` | Integer | FK to Research |
+| `model` | String(100) | Model name |
+| `provider` | String(50) | Provider name |
+| `input_tokens` | Integer | Input token count |
+| `output_tokens` | Integer | Output token count |
+| `cost` | Float | Estimated cost |
+| `created_at` | DateTime | When recorded |
+
+#### SearchCall
+
+Search API call logging.
+
+| Column | Type | Description |
+|--------|------|-------------|
+| `id` | Integer | Primary key |
+| `research_id` | Integer | FK to Research |
+| `engine` | String(50) | Search engine |
+| `query` | Text | Query text |
+| `result_count` | Integer | Results returned |
+| `duration_ms` | Integer | Request duration |
+| `success` | Boolean | Whether succeeded |
+| `created_at` | DateTime | When called |
+
+#### ModelUsage
+
+Aggregated model usage statistics.
+
+| Column | Type | Description |
+|--------|------|-------------|
+| `id` | Integer | Primary key |
+| `model` | String(100) | Model name |
+| `provider` | String(50) | Provider name |
+| `total_input_tokens` | Integer | Cumulative input |
+| `total_output_tokens` | Integer | Cumulative output |
+| `total_cost` | Float | Cumulative cost |
+| `request_count` | Integer | Number of requests |
+| `date` | Date | Aggregation date |
+
+#### ResearchRating
+
+User ratings for research quality.
+
+| Column | Type | Description |
+|--------|------|-------------|
+| `id` | Integer | Primary key |
+| `research_id` | Integer | FK to Research |
+| `rating` | Integer | 1-5 rating |
+| `feedback` | Text | Optional feedback |
+| `created_at` | DateTime | When rated |
+
+---
+
+### News System
+
+News subscription and recommendation.
+
+#### NewsSubscription
+
+User news subscriptions.
+
+| Column | Type | Description |
+|--------|------|-------------|
+| `id` | Integer | Primary key |
+| `topic` | String(255) | Subscription topic |
+| `type` | Enum | SubscriptionType |
+| `status` | Enum | SubscriptionStatus |
+| `frequency` | String(50) | Update frequency |
+| `last_fetched` | DateTime | Last fetch time |
+| `created_at` | DateTime | Creation date |
+
+#### NewsCard
+
+Individual news items.
+
+| Column | Type | Description |
+|--------|------|-------------|
+| `id` | Integer | Primary key |
+| `subscription_id` | Integer | FK to NewsSubscription |
+| `title` | String(500) | News title |
+| `summary` | Text | News summary |
+| `url` | Text | Source URL |
+| `source` | String(100) | Source name |
+| `published_at` | DateTime | Publication date |
+| `card_type` | Enum | CardType |
+| `created_at` | DateTime | When fetched |
+
+#### UserRating / UserPreference / NewsInterest
+
+User interaction tracking for recommendations.
+
+---
+
+### Benchmarking
+
+Performance benchmarking system.
+
+#### BenchmarkRun
+
+Benchmark execution record.
+
+| Column | Type | Description |
+|--------|------|-------------|
+| `id` | Integer | Primary key |
+| `name` | String(255) | Run name |
+| `dataset_type` | Enum | DatasetType (SimpleQA, BrowseComp) |
+| `strategy` | String(100) | Strategy tested |
+| `status` | Enum | BenchmarkStatus |
+| `config` | JSON | Configuration used |
+| `started_at` | DateTime | Start time |
+| `completed_at` | DateTime | End time |
+
+#### BenchmarkResult
+
+Individual benchmark results.
+
+| Column | Type | Description |
+|--------|------|-------------|
+| `id` | Integer | Primary key |
+| `run_id` | Integer | FK to BenchmarkRun |
+| `question` | Text | Test question |
+| `expected_answer` | Text | Expected answer |
+| `actual_answer` | Text | Model's answer |
+| `is_correct` | Boolean | Whether correct |
+| `score` | Float | Quality score |
+| `latency_ms` | Integer | Response time |
+| `tokens_used` | Integer | Tokens consumed |
+
+#### BenchmarkProgress
+
+Progress tracking during runs.
+
+| Column | Type | Description |
+|--------|------|-------------|
+| `id` | Integer | Primary key |
+| `run_id` | Integer | FK to BenchmarkRun |
+| `completed` | Integer | Questions completed |
+| `total` | Integer | Total questions |
+| `current_accuracy` | Float | Running accuracy |
+| `updated_at` | DateTime | Last update |
+
+---
+
+### Rate Limiting
+
+Adaptive rate limiting data.
+
+#### RateLimitAttempt
+
+Individual rate limit events.
+
+| Column | Type | Description |
+|--------|------|-------------|
+| `id` | Integer | Primary key |
+| `engine` | String(50) | Search engine |
+| `wait_time` | Float | Wait time used |
+| `success` | Boolean | Whether request succeeded |
+| `created_at` | DateTime | When occurred |
+
+#### RateLimitEstimate
+
+Learned rate limit estimates.
+
+| Column | Type | Description |
+|--------|------|-------------|
+| `id` | Integer | Primary key |
+| `engine` | String(50) | Search engine |
+| `estimated_wait` | Float | Optimal wait time |
+| `confidence` | Float | Estimate confidence |
+| `sample_count` | Integer | Data points used |
+| `updated_at` | DateTime | Last update |
+
+---
+
+### File Integrity
+
+File verification for security.
+
+#### FileIntegrityRecord
+
+File hash records.
+
+| Column | Type | Description |
+|--------|------|-------------|
+| `id` | Integer | Primary key |
+| `file_path` | Text | File path |
+| `sha256_hash` | String(64) | SHA256 hash |
+| `blake3_hash` | String(64) | BLAKE3 hash |
+| `file_size` | Integer | Size in bytes |
+| `verified_at` | DateTime | Last verification |
+| `created_at` | DateTime | First recorded |
+
+#### FileVerificationFailure
+
+Failed verification attempts.
+
+| Column | Type | Description |
+|--------|------|-------------|
+| `id` | Integer | Primary key |
+| `file_path` | Text | File path |
+| `expected_hash` | String(64) | Expected hash |
+| `actual_hash` | String(64) | Computed hash |
+| `failure_type` | String(50) | Type of failure |
+| `created_at` | DateTime | When detected |
+
+---
+
+## Database Location
+
+```
+~/.local/share/local-deep-research/
+├── auth.db                    # Central auth database (unencrypted)
+└── users/
+    └── <username>/
+        └── research.db        # User's encrypted database
+```
+
+---
+
+## See Also
+
+- [Architecture Overview](./OVERVIEW.md) - System architecture
+- [Semantic Search](./SEMANTIC_SEARCH.md) - How Document/Collection models enable semantic search
+- [Extension Guide](../developing/EXTENDING.md) - Adding custom components
+- [Troubleshooting](../troubleshooting.md) - Common issues

docs/architecture/OVERVIEW.md

@@ -0,0 +1,467 @@
+# Architecture Overview
+
+This document provides a comprehensive overview of Local Deep Research's system architecture.
+
+## Table of Contents
+
+- [System Components](#system-components)
+- [Entry Points](#entry-points)
+- [Research Execution Flow](#research-execution-flow)
+  - [Research Status Lifecycle](#research-status-lifecycle)
+- [Module Responsibilities](#module-responsibilities)
+- [Threading Model](#threading-model)
+- [Configuration System](#configuration-system)
+- [Key Interfaces](#key-interfaces)
+
+---
+
+## System Components
+
+```mermaid
+graph TB
+    subgraph "Entry Points"
+        CLI[ldr CLI]
+        WEB[ldr-web Flask App]
+        API[REST API /api/v1]
+    end
+
+    subgraph "Core Research Engine"
+        SS[SearchSystem<br/>search_system.py]
+        SSF[StrategyFactory<br/>search_system_factory.py]
+        RG[ReportGenerator<br/>report_generator.py]
+    end
+
+    subgraph "Search Layer"
+        STRAT[32 Search Strategies<br/>advanced_search_system/strategies/]
+        ENG[30+ Search Engines<br/>web_search_engines/engines/]
+        RL[Rate Limiter<br/>rate_limiting/]
+    end
+
+    subgraph "Data Layer"
+        DB[(SQLCipher DB<br/>Per-User Encrypted)]
+        MODELS[20+ ORM Models<br/>database/models/]
+        CACHE[Memory Cache]
+    end
+
+    subgraph "LLM Layer"
+        PROV[LLM Providers<br/>Ollama, OpenAI, etc.]
+        EMB[Embeddings<br/>embeddings/]
+        RERANK[Reranker<br/>reranker/]
+    end
+
+    CLI --> SS
+    WEB --> SS
+    API --> SS
+
+    SS --> SSF
+    SS --> RG
+    SSF --> STRAT
+    STRAT --> ENG
+    ENG --> RL
+
+    SS --> PROV
+    SS --> DB
+    MODELS --> DB
+
+    RG --> PROV
+    ENG --> CACHE
+```
+
+---
+
+## Entry Points
+
+### Web Application (`ldr-web`)
+
+**Location:** `src/local_deep_research/web/app.py`
+
+The primary user interface. Launches a Flask server with SocketIO for real-time updates.
+
+```mermaid
+graph LR
+    A[Browser] -->|HTTP/WS| B[Flask App]
+    B --> C[Blueprints]
+    C --> D[research_routes]
+    C --> E[api_routes]
+    C --> F[settings_routes]
+    C --> G[auth_routes]
+    B -->|Real-time| H[SocketIO]
+```
+
+**Key files:**
+- `web/app.py` - Main entry, starts server
+- `web/app_factory.py` - Flask app creation with middleware
+- `web/routes/` - Blueprint route handlers
+- `web/services/` - Business logic services
+
+### REST API (`/api/v1`)
+
+**Location:** `src/local_deep_research/web/api.py`
+
+Programmatic access for integrations.
+
+| Endpoint | Method | Purpose |
+|----------|--------|---------|
+| `/api/v1/quick_summary` | POST | Quick research summary |
+| `/api/v1/generate_report` | POST | Full research report |
+| `/api/v1/analyze_documents` | POST | Search local collections |
+| `/api/v1/health` | GET | Health check |
+
+---
+
+## Research Execution Flow
+
+```mermaid
+sequenceDiagram
+    participant User
+    participant Web as Flask App
+    participant SS as SearchSystem
+    participant SF as StrategyFactory
+    participant Strat as Strategy
+    participant Eng as SearchEngine
+    participant LLM as LLM Provider
+    participant DB as Database
+
+    User->>Web: Submit Query
+    Web->>SS: start_research()
+    SS->>SF: create_strategy(name)
+    SF-->>SS: Strategy instance
+
+    loop Research Iterations
+        SS->>Strat: analyze_topic(query)
+        Strat->>LLM: Generate questions
+        LLM-->>Strat: Questions
+
+        loop Per Question
+            Strat->>Eng: search(question)
+            Eng-->>Strat: Results
+        end
+
+        Strat->>LLM: Synthesize findings
+        LLM-->>Strat: Synthesis
+        Strat-->>SS: Findings
+    end
+
+    SS->>DB: Save results
+    SS-->>Web: Research complete
+    Web-->>User: Results via SocketIO
+```
+
+### Research Status Lifecycle
+
+```mermaid
+stateDiagram-v2
+    [*] --> QUEUED : Concurrency limit reached
+    [*] --> IN_PROGRESS : Slots available
+
+    QUEUED --> IN_PROGRESS : Worker picks up task
+    QUEUED --> SUSPENDED : User terminates
+
+    IN_PROGRESS --> COMPLETED : Research succeeds
+    IN_PROGRESS --> FAILED : Unrecoverable error
+    IN_PROGRESS --> SUSPENDED : User terminates
+
+    COMPLETED --> [*]
+    FAILED --> [*]
+    SUSPENDED --> [*]
+
+    note right of FAILED
+        Set by processor_v2 and
+        research_service on errors
+    end note
+
+    note left of SUSPENDED
+        Set when user clicks
+        terminate/stop
+    end note
+```
+
+> **Unused statuses:** `PENDING` (declared as a model default but never set by any creation
+> path), `ERROR` (never set; predates `FAILED`), and `CANCELLED` (unused by research; used
+> by benchmarks) exist in `ResearchStatus` for backward compatibility.
+
+---
+
+## Module Responsibilities
+
+### Core Modules
+
+| Module | Location | Responsibility |
+|--------|----------|----------------|
+| **SearchSystem** | `search_system.py` | Orchestrates research, coordinates strategies and engines |
+| **StrategyFactory** | `search_system_factory.py` | Creates strategy instances based on configuration |
+| **ReportGenerator** | `report_generator.py` | Generates structured reports from research findings |
+| **CitationHandler** | `citation_handler.py` | Processes and validates citations |
+
+### Search System
+
+| Module | Location | Responsibility |
+|--------|----------|----------------|
+| **BaseSearchEngine** | `web_search_engines/search_engine_base.py` | Abstract base for all search engines |
+| **SearchEngineFactory** | `web_search_engines/search_engine_factory.py` | Creates engine instances |
+| **RateLimitTracker** | `web_search_engines/rate_limiting/tracker.py` | Adaptive rate limiting |
+| **RetrieverRegistry** | `web_search_engines/retriever_registry.py` | LangChain retriever integration |
+
+### Strategy System
+
+| Module | Location | Responsibility |
+|--------|----------|----------------|
+| **BaseSearchStrategy** | `advanced_search_system/strategies/base_strategy.py` | Abstract base for strategies |
+| **FindingsRepository** | `advanced_search_system/findings/` | Accumulates research findings |
+| **QuestionGenerator** | `advanced_search_system/questions/` | Generates research questions |
+
+### Web Application
+
+| Module | Location | Responsibility |
+|--------|----------|----------------|
+| **SocketIOService** | `web/services/socket_service.py` | Real-time communication |
+| **ResearchService** | `web/services/research_service.py` | Research execution |
+| **QueueManager** | `web/queue/` | Background task queue |
+| **SessionManager** | `web/auth/session_manager.py` | User session handling |
+
+### Data Layer
+
+| Module | Location | Responsibility |
+|--------|----------|----------------|
+| **Models** | `database/models/` | SQLAlchemy ORM models |
+| **SessionContext** | `database/session_context.py` | Thread-safe DB sessions |
+| **EncryptedDB** | `database/encrypted_db.py` | SQLCipher integration |
+
+### LLM Integration
+
+| Module | Location | Responsibility |
+|--------|----------|----------------|
+| **LLM Providers** | `llm/providers/implementations/` | Provider-specific LLM wrappers |
+| **AutoDiscovery** | `llm/providers/auto_discovery.py` | Dynamic provider detection |
+| **LLMRegistry** | `llm/llm_registry.py` | Custom LLM registration |
+
+---
+
+## Threading Model
+
+```mermaid
+graph TB
+    subgraph "Main Thread"
+        FLASK[Flask Server]
+        SOCKETIO[SocketIO Handler]
+    end
+
+    subgraph "Research Threads"
+        RT1[Research Thread 1]
+        RT2[Research Thread 2]
+        RTN[Research Thread N]
+    end
+
+    subgraph "Queue Processor"
+        QP[Queue Processor Thread]
+    end
+
+    subgraph "Thread-Local Storage"
+        TC[Thread Context<br/>Settings Snapshot]
+        DBS[DB Session<br/>Per-User]
+    end
+
+    FLASK --> RT1
+    FLASK --> RT2
+    FLASK --> RTN
+
+    RT1 --> TC
+    RT2 --> TC
+    RTN --> TC
+
+    QP --> RT1
+
+    RT1 -.-> SOCKETIO
+    RT2 -.-> SOCKETIO
+```
+
+**Key Threading Concepts:**
+
+1. **Thread Context** (`config/thread_settings.py`)
+   - Each research thread has its own settings snapshot
+   - Prevents race conditions on configuration changes
+
+2. **Per-User DB Sessions** (`database/session_context.py`)
+   - Each user has an isolated SQLCipher database
+   - Sessions are thread-local via context manager
+
+3. **Queue Processing** (`web/queue/`)
+   - Background queue for long-running research
+   - Processes items from `QueuedResearch` table
+
+4. **SocketIO Updates**
+   - Research threads emit progress via SocketIO
+   - Uses threading async mode (not asyncio)
+
+---
+
+## Configuration System
+
+```mermaid
+graph TB
+    subgraph "Configuration Sources"
+        ENV[Environment Variables]
+        DB[(Database Settings<br/>per-user)]
+        JSON[Default Settings<br/>default_settings.json]
+    end
+
+    subgraph "Settings Management"
+        SM[SettingsManager<br/>manager.py]
+    end
+
+    subgraph "Runtime Access"
+        SNAP[Settings Snapshot<br/>Thread-safe copy]
+        TC[Thread Context<br/>thread_settings.py]
+    end
+
+    ENV --> SM
+    JSON --> SM
+    DB --> SM
+    SM --> SNAP
+    SNAP --> TC
+```
+
+**Configuration Flow:**
+
+1. **Defaults** - Default settings loaded from JSON
+2. **Environment** - Environment variables override defaults
+3. **Database** - User settings loaded from encrypted per-user DB
+4. **Snapshot** - Thread-safe copy created for each research
+5. **Access** - Code reads from snapshot via thread context
+
+**Key Settings Categories:**
+
+| Category | Examples |
+|----------|----------|
+| `llm.*` | Provider, model, temperature, API keys |
+| `search.*` | Engine selection, max results, rate limits |
+| `app.*` | Debug mode, logging, UI preferences |
+| `notifications.*` | Email, webhook configurations |
+
+---
+
+## Key Interfaces
+
+### Search Engine Interface
+
+All search engines implement `BaseSearchEngine`:
+
+```python
+class BaseSearchEngine(ABC):
+    # Classification flags
+    is_public: bool = True
+    is_generic: bool = True
+    is_scientific: bool = False
+    is_local: bool = False
+    is_news: bool = False
+    is_code: bool = False
+    is_lexical: bool = False
+    needs_llm_relevance_filter: bool = False
+
+    @abstractmethod
+    def run(self, query: str) -> List[Dict[str, Any]]:
+        """Execute search and return results."""
+        # Returns: [{"title": ..., "link": ..., "snippet": ...}]
+```
+
+### Strategy Interface
+
+All strategies implement `BaseSearchStrategy`:
+
+```python
+class BaseSearchStrategy(ABC):
+    def __init__(self, search, model, all_links_of_system,
+                 settings_snapshot, **kwargs):
+        ...
+
+    @abstractmethod
+    def analyze_topic(self, query: str) -> Dict:
+        """Execute research strategy."""
+        # Returns: {
+        #     "findings": [...],
+        #     "iterations": int,
+        #     "questions": {...},
+        #     "formatted_findings": str,
+        #     "current_knowledge": {...}
+        # }
+```
+
+### LLM Provider Interface
+
+All providers extend `BaseLLMProvider` (most extend `OpenAICompatibleProvider`):
+
+```python
+class OpenAICompatibleProvider(BaseLLMProvider):
+    provider_name: str
+    api_key_setting: str | None      # Settings key, or None for no-key providers
+    api_key_optional: bool = False   # If True, missing key uses placeholder
+    url_setting: str
+    default_base_url: str
+    default_model: str
+
+    @classmethod
+    def create_llm(cls, model_name, temperature, **kwargs) -> BaseChatModel:
+        """Create LangChain LLM instance."""
+```
+
+---
+
+## Directory Structure
+
+```
+src/local_deep_research/
+├── search_system.py           # Main orchestrator
+├── search_system_factory.py   # Strategy factory
+├── report_generator.py        # Report generation
+├── citation_handler.py        # Citation processing
+│
+├── web/                       # Flask application
+│   ├── app.py                # Entry point
+│   ├── app_factory.py        # App creation
+│   ├── routes/               # Blueprint handlers
+│   ├── services/             # Business logic
+│   ├── queue/                # Task queue
+│   └── auth/                 # Authentication
+│
+├── advanced_search_system/    # Search strategies
+│   ├── strategies/           # 32 strategy implementations
+│   ├── questions/            # Question generation
+│   ├── findings/             # Findings management
+│   └── ...
+│
+├── web_search_engines/        # Search engines
+│   ├── engines/              # 30+ engine implementations
+│   ├── search_engine_base.py # Abstract base
+│   ├── search_engine_factory.py
+│   └── rate_limiting/        # Adaptive rate limiting
+│
+├── database/                  # Data layer
+│   ├── models/               # 20+ ORM models
+│   ├── session_context.py    # Session management
+│   └── encrypted_db.py       # SQLCipher
+│
+├── llm/                       # LLM integration
+│   ├── providers/            # Provider implementations
+│   └── llm_registry.py       # Custom LLM registration
+│
+├── config/                    # Configuration
+│   ├── llm_config.py         # LLM setup
+│   ├── search_config.py      # Search setup
+│   └── thread_settings.py    # Thread context
+│
+├── settings/                  # Settings management
+│   └── manager.py            # SettingsManager
+│
+└── api/                       # Programmatic API
+    ├── client.py             # HTTP client
+    └── research_functions.py # Direct functions
+```
+
+---
+
+## See Also
+
+- [Database Schema](./DATABASE_SCHEMA.md) - Detailed data model documentation
+- [Semantic Search](./SEMANTIC_SEARCH.md) - Indexing pipeline, search modes, and three-tier merge algorithm
+- [Extension Guide](../developing/EXTENDING.md) - How to add custom components
+- [Troubleshooting](../troubleshooting.md) - Common issues and solutions

docs/architecture/SEMANTIC_SEARCH.md

@@ -0,0 +1,167 @@
+# Semantic Search
+
+Semantic search lets users find research by meaning, not just title keywords. It indexes completed research reports and their sources into FAISS vector stores, then offers three search modes -- Hybrid (default), Text-Only, and AI-Only -- with a three-tier ranked merge algorithm for hybrid results.
+
+## Table of Contents
+
+- [Indexing Pipeline](#indexing-pipeline)
+- [Search Pipeline](#search-pipeline)
+- [Three-Tier Merge Algorithm](#three-tier-merge-algorithm)
+- [File Structure](#file-structure)
+- [API Routes](#api-routes)
+- [Reusing on Other Pages](#reusing-on-other-pages)
+- [See Also](#see-also)
+
+---
+
+## Indexing Pipeline
+
+```mermaid
+flowchart LR
+    RH[ResearchHistory<br/>report + sources] --> RHI[ResearchHistoryIndexer<br/>converts to Documents]
+    RHI --> DOC[Document rows<br/>+ DocumentCollection links]
+    DOC --> FACTORY[RAGServiceFactory<br/>resolves settings]
+    FACTORY --> RAG[LibraryRAGService<br/>chunk + embed]
+    RAG --> FAISS[(FAISS Index<br/>on disk)]
+```
+
+**Steps:**
+
+1. `ResearchHistoryIndexer` reads completed `ResearchHistory` rows and their `ResearchResource` sources.
+2. Each report becomes a `Document` (source\_type `research_report`). Each source with sufficient content becomes a `Document` (source\_type `research_source`).
+3. Documents are linked to the History `Collection` via `DocumentCollection`.
+4. `RAGServiceFactory` creates a `LibraryRAGService` with the collection's embedding settings.
+5. `LibraryRAGService` chunks each document, generates embeddings, and writes the FAISS index to disk.
+
+Bulk indexing uses **SSE streaming** -- the `/index` endpoint yields progress events (`start`, `progress`, `complete`, `error`) so the frontend can show a real-time progress bar.
+
+---
+
+## Search Pipeline
+
+```mermaid
+flowchart LR
+    Q[User query] --> MODE{Search Mode}
+    MODE -->|Text-Only| TF[Title filter<br/>instant, client-side]
+    MODE -->|AI-Only| API
+    MODE -->|Hybrid| TF & API
+
+    API[POST /library/api/collections/:id/search] --> CSE[CollectionSearchEngine]
+    CSE --> FAISS[(FAISS Index)]
+    FAISS --> ENRICH[Enrich metadata<br/>report/source type]
+    ENRICH --> JSON[JSON response]
+
+    TF --> RENDER
+    JSON --> MERGE[buildTieredResults<br/>three-tier merge]
+    MERGE --> RENDER[Render cards]
+```
+
+**Text filter** runs instantly on the client against cached history items. **Semantic search** is an async `POST` to the collection search endpoint, which delegates to `CollectionSearchEngine` for FAISS similarity search. In **Hybrid mode**, text results render immediately while the semantic call runs in the background; once it resolves, `buildTieredResults` merges both result sets into three tiers and re-renders.
+
+---
+
+## Three-Tier Merge Algorithm
+
+Implemented in `semantic_search.js :: buildTieredResults()`.
+
+| Tier | Contents | Sort Order |
+|------|----------|------------|
+| **Tier 1** | Matched both text filter AND semantic search | Similarity score DESC |
+| **Tier 2** | Text-only matches (no semantic hit) | Original order (recency) |
+| **Tier 3** | Semantic-only matches (below a visual divider) | Similarity score DESC |
+
+The merge groups semantic results by `research_id` (keeping the best similarity per research), then classifies each text result as Tier 1 or Tier 2 based on whether a corresponding semantic match exists. Remaining semantic-only results become Tier 3.
+
+---
+
+## File Structure
+
+### Backend
+
+```
+research_library/
+├── search/                              # Semantic search subpackage
+│   ├── __init__.py                      # Exports search_bp, ResearchHistoryIndexer
+│   ├── routes/
+│   │   └── search_routes.py             # 4 endpoints + _enrich helper
+│   └── services/
+│       └── research_history_indexer.py   # Converts ResearchHistory -> Documents -> RAG
+├── services/
+│   ├── rag_service_factory.py           # Creates LibraryRAGService with collection settings
+│   └── library_rag_service.py           # FAISS indexing, chunking, embedding management
+```
+
+### Frontend
+
+```
+js/components/
+├── semantic_search.js     # Shared module (window.SemanticSearch):
+│                          #   renderSnippet, buildTieredResults,
+│                          #   createSemanticResultCard, isSafeExternalUrl
+├── history.js             # Page controller: mode switching, hybrid merge, rendering
+└── history_search.js      # Indexing UI, semantic API calls, collection ID caching
+
+js/config/
+└── constants.js           # LDR_CONSTANTS.SEARCH_MODE: HYBRID | TEXT | SEMANTIC
+
+css/components/
+└── semantic-search.css    # All semantic search styles (badges, cards, dividers)
+```
+
+---
+
+## API Routes
+
+All routes require `@login_required`. Blueprint prefix: `/library`.
+
+| Method | Path | Purpose |
+|--------|------|---------|
+| `GET` | `/api/research-history/collection` | Get collection ID and indexing status (auto-converts unconverted entries) |
+| `POST` | `/api/research-history/convert-all` | Convert all completed research to Documents |
+| `POST` | `/api/research/<id>/add-to-collection` | Add research to a custom collection |
+| `POST` | `/api/collections/<id>/search` | Semantic search any collection (generic) |
+
+The search endpoint is **collection-agnostic** -- it works for any collection type. For `research_history` collections, results are enriched with report/source type metadata.
+
+---
+
+## Reusing on Other Pages
+
+The `semantic_search.js` shared module is designed for reuse on library or collection pages.
+
+1. **Load the shared assets** in your template (order matters):
+   ```html
+   <link rel="stylesheet" href="/static/css/components/semantic-search.css">
+   <script defer src="/static/js/components/semantic_search.js"></script>
+   ```
+
+2. **Call the search endpoint** with `POST /library/api/collections/<collection_id>/search`:
+   ```javascript
+   const response = await fetch(`/library/api/collections/${collectionId}/search`, {
+       method: 'POST',
+       headers: { 'Content-Type': 'application/json' },
+       body: JSON.stringify({ query: 'your search', limit: 10 }),
+   });
+   const { results } = await response.json();
+   ```
+
+3. **Render results** using the shared utilities:
+   ```javascript
+   for (const result of results) {
+       container.appendChild(SemanticSearch.createSemanticResultCard(result));
+   }
+
+   // Or for hybrid mode with text + semantic merge:
+   const tiered = SemanticSearch.buildTieredResults(textResults, semanticResults);
+   // tiered.tier1, tiered.tier2, tiered.tier3
+   ```
+
+No backend changes needed -- the search route and `CollectionSearchEngine` already support any collection type.
+
+---
+
+## See Also
+
+- [Architecture Overview](./OVERVIEW.md) -- System architecture
+- [Database Schema](./DATABASE_SCHEMA.md) -- Document, Collection, and DocumentCollection models
+- [Library & RAG Guide](../library-and-rag.md) -- User-facing guide to library and search features

docs/ci/workflow-status.md

@@ -0,0 +1,142 @@
+# Workflow Status
+
+> **Live status of every GitHub Actions workflow in this repo.**
+> Auto-generated by [`scripts/generate_workflow_status.py`](../../scripts/generate_workflow_status.py).
+> Do not edit between the generated markers — regenerate with
+> `pdm run python scripts/generate_workflow_status.py`. Anything outside
+> the markers is preserved on regeneration.
+
+## How to read this page
+
+- **Live badges** (right column on active gates) re-render on every page
+  view and reflect the current head-of-default-branch status from
+  GitHub. Click one to land on that workflow's runs page. The badge is
+  the source of truth for current status — there is intentionally no
+  static status column, because that would flip every regeneration as
+  the most-recent run cycles through `success → skipped → in_progress`.
+- **Last activity** uses coarse calendar buckets — `last 30 days`,
+  `1-3 months ago`, `3-6 months ago`, `long ago`, `never`. Exact dates
+  would change every regeneration; buckets only change when a workflow
+  drifts, which is the signal worth seeing in version-bump diffs.
+- **Disabled** = a caller has the `uses:` line commented out, or the
+  workflow is disabled in the GitHub UI. **Stale** = scheduled trigger
+  but no successful run within 2× its cron cadence (and ≥60 days). The
+  three top sections are the action items.
+- Reusable workflows (those triggered only by `workflow_call:`) show
+  their **gated** run — the most recent run of their parent (release.yml,
+  release-gate.yml, ci-gate.yml) that included them — not their own
+  empty direct-run history.
+
+<!-- BEGIN GENERATED -->
+
+**64 workflows:** 1 disabled · 1 manual-only · 62 active
+
+## ⚠ Disabled workflows
+
+| Workflow | Disabled where | Last direct run |
+|---|---|---|
+| `nuclei.yml` | `release-gate.yml:195` (commented) | never |
+
+## ⚠ Stale (scheduled but no recent successful run)
+
+_None._
+
+## ℹ Manual-only by design
+
+| Workflow | Last manual run | Trigger |
+|---|---|---|
+| `check-config-docs.yml` | 3-6 months ago | manual |
+
+## Release-blocking gates — daily (release-gate cron 02:00 UTC)
+
+| Workflow | Last activity | Trigger | Live badge |
+|---|---|---|---|
+| `backwards-compatibility.yml` | last 30 days | workflow_call, PR, push:main, release, schedule(0 2 * * 0), manual | [![status](https://github.com/LearningCircuit/local-deep-research/actions/workflows/backwards-compatibility.yml/badge.svg)](https://github.com/LearningCircuit/local-deep-research/actions/workflows/backwards-compatibility.yml) |
+| `bearer.yml` | last 30 days | manual, workflow_call, schedule(0 4 * * *) | [![status](https://github.com/LearningCircuit/local-deep-research/actions/workflows/bearer.yml/badge.svg)](https://github.com/LearningCircuit/local-deep-research/actions/workflows/bearer.yml) |
+| `checkov.yml` | last 30 days | manual, workflow_call | [![status](https://github.com/LearningCircuit/local-deep-research/actions/workflows/checkov.yml/badge.svg)](https://github.com/LearningCircuit/local-deep-research/actions/workflows/checkov.yml) |
+| `codeql.yml` | last 30 days | push:main, PR, schedule(45 5 * * 0), workflow_call, manual | [![status](https://github.com/LearningCircuit/local-deep-research/actions/workflows/codeql.yml/badge.svg)](https://github.com/LearningCircuit/local-deep-research/actions/workflows/codeql.yml) |
+| `container-security.yml` | last 30 days | workflow_call, manual | [![status](https://github.com/LearningCircuit/local-deep-research/actions/workflows/container-security.yml/badge.svg)](https://github.com/LearningCircuit/local-deep-research/actions/workflows/container-security.yml) |
+| `devskim.yml` | last 30 days | manual, workflow_call, schedule(0 10 * * *) | [![status](https://github.com/LearningCircuit/local-deep-research/actions/workflows/devskim.yml/badge.svg)](https://github.com/LearningCircuit/local-deep-research/actions/workflows/devskim.yml) |
+| `docker-multiarch-test.yml` | last 30 days | workflow_call, manual | [![status](https://github.com/LearningCircuit/local-deep-research/actions/workflows/docker-multiarch-test.yml/badge.svg)](https://github.com/LearningCircuit/local-deep-research/actions/workflows/docker-multiarch-test.yml) |
+| `dockle.yml` | last 30 days | manual, workflow_call, schedule(0 10 * * 2) | [![status](https://github.com/LearningCircuit/local-deep-research/actions/workflows/dockle.yml/badge.svg)](https://github.com/LearningCircuit/local-deep-research/actions/workflows/dockle.yml) |
+| `gitleaks-main.yml` | last 30 days | workflow_call, manual | [![status](https://github.com/LearningCircuit/local-deep-research/actions/workflows/gitleaks-main.yml/badge.svg)](https://github.com/LearningCircuit/local-deep-research/actions/workflows/gitleaks-main.yml) |
+| `grype.yml` | last 30 days | workflow_call, manual | [![status](https://github.com/LearningCircuit/local-deep-research/actions/workflows/grype.yml/badge.svg)](https://github.com/LearningCircuit/local-deep-research/actions/workflows/grype.yml) |
+| `hadolint.yml` | last 30 days | PR, manual, workflow_call, schedule(0 9 * * 2) | [![status](https://github.com/LearningCircuit/local-deep-research/actions/workflows/hadolint.yml/badge.svg)](https://github.com/LearningCircuit/local-deep-research/actions/workflows/hadolint.yml) |
+| `journal-data-integration.yml` | last 30 days | workflow_call, manual, schedule(0 4 * * 1) | [![status](https://github.com/LearningCircuit/local-deep-research/actions/workflows/journal-data-integration.yml/badge.svg)](https://github.com/LearningCircuit/local-deep-research/actions/workflows/journal-data-integration.yml) |
+| `npm-audit.yml` | last 30 days | manual, workflow_call | [![status](https://github.com/LearningCircuit/local-deep-research/actions/workflows/npm-audit.yml/badge.svg)](https://github.com/LearningCircuit/local-deep-research/actions/workflows/npm-audit.yml) |
+| `owasp-zap-scan.yml` | last 30 days | workflow_call, manual | [![status](https://github.com/LearningCircuit/local-deep-research/actions/workflows/owasp-zap-scan.yml/badge.svg)](https://github.com/LearningCircuit/local-deep-research/actions/workflows/owasp-zap-scan.yml) |
+| `retirejs.yml` | last 30 days | manual, workflow_call, schedule(0 4 * * 1) | [![status](https://github.com/LearningCircuit/local-deep-research/actions/workflows/retirejs.yml/badge.svg)](https://github.com/LearningCircuit/local-deep-research/actions/workflows/retirejs.yml) |
+| `security-headers-validation.yml` | last 30 days | workflow_call, schedule(0 3 * * *), manual | [![status](https://github.com/LearningCircuit/local-deep-research/actions/workflows/security-headers-validation.yml/badge.svg)](https://github.com/LearningCircuit/local-deep-research/actions/workflows/security-headers-validation.yml) |
+| `security-tests.yml` | last 30 days | workflow_call, manual | [![status](https://github.com/LearningCircuit/local-deep-research/actions/workflows/security-tests.yml/badge.svg)](https://github.com/LearningCircuit/local-deep-research/actions/workflows/security-tests.yml) |
+| `semgrep.yml` | last 30 days | workflow_call, manual | [![status](https://github.com/LearningCircuit/local-deep-research/actions/workflows/semgrep.yml/badge.svg)](https://github.com/LearningCircuit/local-deep-research/actions/workflows/semgrep.yml) |
+| `zizmor-security.yml` | last 30 days | manual, workflow_call, schedule(0 9 * * 1) | [![status](https://github.com/LearningCircuit/local-deep-research/actions/workflows/zizmor-security.yml/badge.svg)](https://github.com/LearningCircuit/local-deep-research/actions/workflows/zizmor-security.yml) |
+
+## Release gates — release-time only
+
+| Workflow | Last activity | Trigger | Live badge |
+|---|---|---|---|
+| `check-env-vars.yml` | 1-3 months ago | PR, workflow_call, manual | [![status](https://github.com/LearningCircuit/local-deep-research/actions/workflows/check-env-vars.yml/badge.svg)](https://github.com/LearningCircuit/local-deep-research/actions/workflows/check-env-vars.yml) |
+| `ci-gate.yml` | 1-3 months ago | workflow_call, manual | [![status](https://github.com/LearningCircuit/local-deep-research/actions/workflows/ci-gate.yml/badge.svg)](https://github.com/LearningCircuit/local-deep-research/actions/workflows/ci-gate.yml) |
+| `compose-integration-test.yml` | 1-3 months ago | workflow_call, manual | [![status](https://github.com/LearningCircuit/local-deep-research/actions/workflows/compose-integration-test.yml/badge.svg)](https://github.com/LearningCircuit/local-deep-research/actions/workflows/compose-integration-test.yml) |
+| `docker-publish.yml` | 1-3 months ago | workflow_call | [![status](https://github.com/LearningCircuit/local-deep-research/actions/workflows/docker-publish.yml/badge.svg)](https://github.com/LearningCircuit/local-deep-research/actions/workflows/docker-publish.yml) |
+| `docker-tests.yml` | 1-3 months ago | PR, push:main, workflow_call, manual | [![status](https://github.com/LearningCircuit/local-deep-research/actions/workflows/docker-tests.yml/badge.svg)](https://github.com/LearningCircuit/local-deep-research/actions/workflows/docker-tests.yml) |
+| `file-whitelist-check.yml` | 1-3 months ago | PR, workflow_call, manual | [![status](https://github.com/LearningCircuit/local-deep-research/actions/workflows/file-whitelist-check.yml/badge.svg)](https://github.com/LearningCircuit/local-deep-research/actions/workflows/file-whitelist-check.yml) |
+| `mypy-type-check.yml` | 1-3 months ago | PR, workflow_call, manual | [![status](https://github.com/LearningCircuit/local-deep-research/actions/workflows/mypy-type-check.yml/badge.svg)](https://github.com/LearningCircuit/local-deep-research/actions/workflows/mypy-type-check.yml) |
+| `playwright-webkit-tests.yml` | last 30 days | workflow_call, manual, schedule(0 2 * * *) | [![status](https://github.com/LearningCircuit/local-deep-research/actions/workflows/playwright-webkit-tests.yml/badge.svg)](https://github.com/LearningCircuit/local-deep-research/actions/workflows/playwright-webkit-tests.yml) |
+| `pre-commit.yml` | 1-3 months ago | PR, workflow_call, manual | [![status](https://github.com/LearningCircuit/local-deep-research/actions/workflows/pre-commit.yml/badge.svg)](https://github.com/LearningCircuit/local-deep-research/actions/workflows/pre-commit.yml) |
+| `prerelease-docker.yml` | 1-3 months ago | workflow_call | [![status](https://github.com/LearningCircuit/local-deep-research/actions/workflows/prerelease-docker.yml/badge.svg)](https://github.com/LearningCircuit/local-deep-research/actions/workflows/prerelease-docker.yml) |
+| `puppeteer-e2e-tests.yml` | last 30 days | PR, workflow_call, manual, schedule(0 2 * * 0) | [![status](https://github.com/LearningCircuit/local-deep-research/actions/workflows/puppeteer-e2e-tests.yml/badge.svg)](https://github.com/LearningCircuit/local-deep-research/actions/workflows/puppeteer-e2e-tests.yml) |
+| `responsive-ui-tests-enhanced.yml` | 1-3 months ago | workflow_call, manual | [![status](https://github.com/LearningCircuit/local-deep-research/actions/workflows/responsive-ui-tests-enhanced.yml/badge.svg)](https://github.com/LearningCircuit/local-deep-research/actions/workflows/responsive-ui-tests-enhanced.yml) |
+| `security-file-write-check.yml` | 1-3 months ago | PR, workflow_call, manual | [![status](https://github.com/LearningCircuit/local-deep-research/actions/workflows/security-file-write-check.yml/badge.svg)](https://github.com/LearningCircuit/local-deep-research/actions/workflows/security-file-write-check.yml) |
+| `validate-image-pinning.yml` | 1-3 months ago | PR, workflow_call, manual | [![status](https://github.com/LearningCircuit/local-deep-research/actions/workflows/validate-image-pinning.yml/badge.svg)](https://github.com/LearningCircuit/local-deep-research/actions/workflows/validate-image-pinning.yml) |
+| `vulture-dead-code.yml` | 3-6 months ago | workflow_call, manual | [![status](https://github.com/LearningCircuit/local-deep-research/actions/workflows/vulture-dead-code.yml/badge.svg)](https://github.com/LearningCircuit/local-deep-research/actions/workflows/vulture-dead-code.yml) |
+
+## Scheduled (own cron)
+
+| Workflow | Last activity | Trigger | Live badge |
+|---|---|---|---|
+| `compose-published-smoke.yml` | last 30 days | manual, schedule(0 5 * * 1) | [![status](https://github.com/LearningCircuit/local-deep-research/actions/workflows/compose-published-smoke.yml/badge.svg?event=schedule)](https://github.com/LearningCircuit/local-deep-research/actions/workflows/compose-published-smoke.yml?query=event%3Aschedule) |
+| `fuzz.yml` | last 30 days | schedule(0 0 * * 0), manual, PR | [![status](https://github.com/LearningCircuit/local-deep-research/actions/workflows/fuzz.yml/badge.svg?event=schedule)](https://github.com/LearningCircuit/local-deep-research/actions/workflows/fuzz.yml?query=event%3Aschedule) |
+| `gitleaks.yml` | last 30 days | PR, manual, schedule(0 3 * * *) | [![status](https://github.com/LearningCircuit/local-deep-research/actions/workflows/gitleaks.yml/badge.svg?event=schedule)](https://github.com/LearningCircuit/local-deep-research/actions/workflows/gitleaks.yml?query=event%3Aschedule) |
+| `ossf-scorecard.yml` | last 30 days | branch_protection_rule, schedule(0 8 * * 1), manual, push:main | [![status](https://github.com/LearningCircuit/local-deep-research/actions/workflows/ossf-scorecard.yml/badge.svg?event=schedule)](https://github.com/LearningCircuit/local-deep-research/actions/workflows/ossf-scorecard.yml?query=event%3Aschedule) |
+| `osv-scanner-scheduled.yml` | last 30 days | push:main, schedule(41 21 * * 1), manual | [![status](https://github.com/LearningCircuit/local-deep-research/actions/workflows/osv-scanner-scheduled.yml/badge.svg?event=schedule)](https://github.com/LearningCircuit/local-deep-research/actions/workflows/osv-scanner-scheduled.yml?query=event%3Aschedule) |
+| `osv-scanner.yml` | last 30 days | PR, merge_group, schedule(39 12 * * 1), manual | [![status](https://github.com/LearningCircuit/local-deep-research/actions/workflows/osv-scanner.yml/badge.svg?event=schedule)](https://github.com/LearningCircuit/local-deep-research/actions/workflows/osv-scanner.yml?query=event%3Aschedule) |
+| `release-gate.yml` | last 30 days | workflow_call, manual, schedule(0 2 * * *) | [![status](https://github.com/LearningCircuit/local-deep-research/actions/workflows/release-gate.yml/badge.svg?event=schedule)](https://github.com/LearningCircuit/local-deep-research/actions/workflows/release-gate.yml?query=event%3Aschedule) |
+| `sbom.yml` | last 30 days | manual, schedule(0 10 * * 3), release | [![status](https://github.com/LearningCircuit/local-deep-research/actions/workflows/sbom.yml/badge.svg?event=schedule)](https://github.com/LearningCircuit/local-deep-research/actions/workflows/sbom.yml?query=event%3Aschedule) |
+| `update-dependencies.yml` | last 30 days | workflow_call, manual, schedule(0 8 * * 3) | [![status](https://github.com/LearningCircuit/local-deep-research/actions/workflows/update-dependencies.yml/badge.svg?event=schedule)](https://github.com/LearningCircuit/local-deep-research/actions/workflows/update-dependencies.yml?query=event%3Aschedule) |
+| `update-npm-dependencies.yml` | last 30 days | workflow_call, manual, schedule(0 8 * * 4) | [![status](https://github.com/LearningCircuit/local-deep-research/actions/workflows/update-npm-dependencies.yml/badge.svg?event=schedule)](https://github.com/LearningCircuit/local-deep-research/actions/workflows/update-npm-dependencies.yml?query=event%3Aschedule) |
+| `update-precommit-hooks.yml` | last 30 days | manual, schedule(0 8 * * 5) | [![status](https://github.com/LearningCircuit/local-deep-research/actions/workflows/update-precommit-hooks.yml/badge.svg?event=schedule)](https://github.com/LearningCircuit/local-deep-research/actions/workflows/update-precommit-hooks.yml?query=event%3Aschedule) |
+
+## PR / push checks
+
+| Workflow | Last activity | Trigger | Live badge |
+|---|---|---|---|
+| `advanced-search-reminder.yml` | last 30 days | PR | [![status](https://github.com/LearningCircuit/local-deep-research/actions/workflows/advanced-search-reminder.yml/badge.svg)](https://github.com/LearningCircuit/local-deep-research/actions/workflows/advanced-search-reminder.yml) |
+| `ai-code-reviewer.yml` | last 30 days | PR-target | [![status](https://github.com/LearningCircuit/local-deep-research/actions/workflows/ai-code-reviewer.yml/badge.svg)](https://github.com/LearningCircuit/local-deep-research/actions/workflows/ai-code-reviewer.yml) |
+| `check-workflow-status.yml` | last 30 days | PR, manual | [![status](https://github.com/LearningCircuit/local-deep-research/actions/workflows/check-workflow-status.yml/badge.svg)](https://github.com/LearningCircuit/local-deep-research/actions/workflows/check-workflow-status.yml) |
+| `claude-code-review.yml` | last 30 days | PR | [![status](https://github.com/LearningCircuit/local-deep-research/actions/workflows/claude-code-review.yml/badge.svg)](https://github.com/LearningCircuit/local-deep-research/actions/workflows/claude-code-review.yml) |
+| `danger-zone-alert.yml` | last 30 days | PR | [![status](https://github.com/LearningCircuit/local-deep-research/actions/workflows/danger-zone-alert.yml/badge.svg)](https://github.com/LearningCircuit/local-deep-research/actions/workflows/danger-zone-alert.yml) |
+| `dependency-review.yml` | last 30 days | PR, manual | [![status](https://github.com/LearningCircuit/local-deep-research/actions/workflows/dependency-review.yml/badge.svg)](https://github.com/LearningCircuit/local-deep-research/actions/workflows/dependency-review.yml) |
+| `e2e-research-test.yml` | last 30 days | PR | [![status](https://github.com/LearningCircuit/local-deep-research/actions/workflows/e2e-research-test.yml/badge.svg)](https://github.com/LearningCircuit/local-deep-research/actions/workflows/e2e-research-test.yml) |
+| `labels-sync.yml` | last 30 days | push:main, manual | [![status](https://github.com/LearningCircuit/local-deep-research/actions/workflows/labels-sync.yml/badge.svg)](https://github.com/LearningCircuit/local-deep-research/actions/workflows/labels-sync.yml) |
+| `mcp-tests.yml` | last 30 days | push:main, PR, manual | [![status](https://github.com/LearningCircuit/local-deep-research/actions/workflows/mcp-tests.yml/badge.svg)](https://github.com/LearningCircuit/local-deep-research/actions/workflows/mcp-tests.yml) |
+| `pr-triage.yml` | last 30 days | PR, pull_request_review | [![status](https://github.com/LearningCircuit/local-deep-research/actions/workflows/pr-triage.yml/badge.svg)](https://github.com/LearningCircuit/local-deep-research/actions/workflows/pr-triage.yml) |
+| `release.yml` | last 30 days | push:main, manual | [![status](https://github.com/LearningCircuit/local-deep-research/actions/workflows/release.yml/badge.svg)](https://github.com/LearningCircuit/local-deep-research/actions/workflows/release.yml) |
+| `ui-full-shards.yml` | last 30 days | PR, manual | [![status](https://github.com/LearningCircuit/local-deep-research/actions/workflows/ui-full-shards.yml/badge.svg)](https://github.com/LearningCircuit/local-deep-research/actions/workflows/ui-full-shards.yml) |
+| `version_check.yml` | last 30 days | push:main, manual | [![status](https://github.com/LearningCircuit/local-deep-research/actions/workflows/version_check.yml/badge.svg)](https://github.com/LearningCircuit/local-deep-research/actions/workflows/version_check.yml) |
+| `welcome-first-time.yml` | last 30 days | PR-target | [![status](https://github.com/LearningCircuit/local-deep-research/actions/workflows/welcome-first-time.yml/badge.svg)](https://github.com/LearningCircuit/local-deep-research/actions/workflows/welcome-first-time.yml) |
+
+## Repository-dispatch publishers
+
+| Workflow | Last activity | Trigger | Live badge |
+|---|---|---|---|
+| `publish.yml` | last 30 days | repo_dispatch | [![status](https://github.com/LearningCircuit/local-deep-research/actions/workflows/publish.yml/badge.svg)](https://github.com/LearningCircuit/local-deep-research/actions/workflows/publish.yml) |
+
+## Other
+
+| Workflow | Last activity | Trigger | Live badge |
+|---|---|---|---|
+| `issue-research.yml` | last 30 days | issues | [![status](https://github.com/LearningCircuit/local-deep-research/actions/workflows/issue-research.yml/badge.svg)](https://github.com/LearningCircuit/local-deep-research/actions/workflows/issue-research.yml) |
+| `ldr-research-reusable.yml` | never | workflow_call | [![status](https://github.com/LearningCircuit/local-deep-research/actions/workflows/ldr-research-reusable.yml/badge.svg)](https://github.com/LearningCircuit/local-deep-research/actions/workflows/ldr-research-reusable.yml) |
+
+<!-- END GENERATED -->

docs/cli-tools.md

@@ -0,0 +1,306 @@
+# CLI Tools Reference
+
+Local Deep Research includes command-line tools for benchmarking and rate limit management.
+
+## Table of Contents
+
+- [Benchmarking CLI](#benchmarking-cli)
+- [Rate Limiting CLI](#rate-limiting-cli)
+- [MCP Server CLI](#mcp-server-cli)
+
+---
+
+## Benchmarking CLI
+
+Run benchmarks to evaluate search quality and compare configurations.
+
+### Basic Usage
+
+```bash
+python -m local_deep_research.benchmarks.cli.benchmark_commands <command> [options]
+```
+
+### Commands
+
+#### `simpleqa` - Run SimpleQA Benchmark
+
+Tests factual question answering accuracy.
+
+```bash
+python -m local_deep_research.benchmarks.cli.benchmark_commands simpleqa [options]
+```
+
+**Options:**
+
+| Option | Default | Description |
+|--------|---------|-------------|
+| `--examples` | 100 | Number of questions to test |
+| `--iterations` | 3 | Search iterations per question |
+| `--questions` | 3 | Questions per iteration |
+| `--search-tool` | searxng | Search engine to use |
+| `--search-strategy` | source_based | Strategy (source-based, focused-iteration, focused-iteration-standard, topic-organization) |
+| `--search-model` | (default) | LLM model for research |
+| `--search-provider` | (default) | LLM provider |
+| `--eval-model` | (default) | Model for answer evaluation |
+| `--eval-provider` | (default) | Provider for evaluation |
+| `--output-dir` | ~/.local-deep-research/benchmark_results | Results directory |
+| `--human-eval` | false | Use human evaluation |
+| `--no-eval` | false | Skip evaluation phase |
+| `--custom-dataset` | - | Path to custom dataset |
+
+**Example:**
+
+```bash
+# Run 50 examples with Ollama
+python -m local_deep_research.benchmarks.cli.benchmark_commands simpleqa \
+  --examples 50 \
+  --search-provider ollama \
+  --search-model llama3.2
+```
+
+#### `browsecomp` - Run BrowseComp Benchmark
+
+Tests complex reasoning and multi-step research.
+
+```bash
+python -m local_deep_research.benchmarks.cli.benchmark_commands browsecomp [options]
+```
+
+Same options as `simpleqa`.
+
+**Example:**
+
+```bash
+# Run BrowseComp with focused-iteration strategy
+python -m local_deep_research.benchmarks.cli.benchmark_commands browsecomp \
+  --examples 20 \
+  --search-strategy focused-iteration \
+  --iterations 5
+```
+
+#### `compare` - Compare Configurations
+
+Compare multiple search configurations on the same dataset.
+
+```bash
+python -m local_deep_research.benchmarks.cli.benchmark_commands compare [options]
+```
+
+**Options:**
+
+| Option | Default | Description |
+|--------|---------|-------------|
+| `--dataset` | simpleqa | Dataset to use (simpleqa, browsecomp) |
+| `--examples` | 20 | Examples per configuration |
+| `--output-dir` | ~/.local-deep-research/benchmark_results/comparison | Results directory |
+
+**Example:**
+
+```bash
+# Compare configurations
+python -m local_deep_research.benchmarks.cli.benchmark_commands compare \
+  --dataset simpleqa \
+  --examples 30
+```
+
+#### `list` - List Available Benchmarks
+
+```bash
+python -m local_deep_research.benchmarks.cli.benchmark_commands list
+```
+
+Shows available benchmark datasets and their descriptions.
+
+---
+
+## Rate Limiting CLI
+
+Monitor and manage the adaptive rate limiting system.
+
+### Basic Usage
+
+```bash
+python -m local_deep_research.web_search_engines.rate_limiting.cli <command> [options]
+```
+
+### Commands
+
+#### `status` - Show Rate Limit Statistics
+
+View current rate limit data for search engines.
+
+```bash
+# All engines
+python -m local_deep_research.web_search_engines.rate_limiting.cli status
+
+# Specific engine
+python -m local_deep_research.web_search_engines.rate_limiting.cli status --engine DuckDuckGoSearchEngine
+```
+
+**Output columns:**
+
+| Column | Description |
+|--------|-------------|
+| Engine | Search engine name |
+| Base Wait | Current wait time in seconds |
+| Range | Min-max wait times |
+| Success | Success rate percentage |
+| Attempts | Total request attempts |
+| Updated | Last update timestamp |
+
+**Example output:**
+
+```
+Rate Limit Statistics:
+--------------------------------------------------------------------------------
+Engine               Base Wait    Range                Success    Attempts   Updated
+--------------------------------------------------------------------------------
+DuckDuckGoSearchEngine 2.50        1.0s - 5.0s         95.2%      150        12-26 14:30
+ArXivSearchEngine      0.50        0.5s - 1.0s         99.8%      85         12-26 12:15
+```
+
+#### `reset` - Reset Engine Rate Limits
+
+Clear learned rate limit data for an engine.
+
+```bash
+python -m local_deep_research.web_search_engines.rate_limiting.cli reset --engine <engine_name>
+```
+
+**Example:**
+
+```bash
+# Reset DuckDuckGo rate limits
+python -m local_deep_research.web_search_engines.rate_limiting.cli reset --engine DuckDuckGoSearchEngine
+```
+
+Use this when:
+- Rate limits are too conservative
+- After API changes
+- When switching environments
+
+#### `export` - Export Rate Limit Data
+
+Export rate limit statistics in various formats.
+
+```bash
+# Table format (default)
+python -m local_deep_research.web_search_engines.rate_limiting.cli export
+
+# CSV format
+python -m local_deep_research.web_search_engines.rate_limiting.cli export --format csv
+
+# JSON format
+python -m local_deep_research.web_search_engines.rate_limiting.cli export --format json
+```
+
+**Formats:**
+
+| Format | Use Case |
+|--------|----------|
+| `table` | Human-readable display |
+| `csv` | Spreadsheet import |
+| `json` | Programmatic processing |
+
+**Example CSV output:**
+
+```csv
+engine_type,base_wait_seconds,min_wait_seconds,max_wait_seconds,last_updated,total_attempts,success_rate
+DuckDuckGoSearchEngine,2.5,1.0,5.0,1703612400,150,0.952
+```
+
+#### `cleanup` - Remove Old Data
+
+Clean up rate limit data older than a specified number of days.
+
+```bash
+python -m local_deep_research.web_search_engines.rate_limiting.cli cleanup --days <days>
+```
+
+**Example:**
+
+```bash
+# Remove data older than 30 days
+python -m local_deep_research.web_search_engines.rate_limiting.cli cleanup --days 30
+
+# Remove data older than 7 days
+python -m local_deep_research.web_search_engines.rate_limiting.cli cleanup --days 7
+```
+
+---
+
+## MCP Server CLI
+
+Run the MCP server for Claude Desktop integration.
+
+### Basic Usage
+
+```bash
+# Via entry point
+ldr-mcp
+
+# Via module
+python -m local_deep_research.mcp
+```
+
+The server communicates over STDIO (stdin/stdout for JSON-RPC, stderr for logs). It is designed to be launched by Claude Desktop, not run interactively.
+
+### Environment Variables
+
+Set via Claude Desktop config `env` block or shell environment:
+
+| Variable | Description | Example |
+|----------|-------------|---------|
+| `LDR_LLM_PROVIDER` | LLM provider | `openai`, `ollama`, `anthropic` |
+| `LDR_LLM_MODEL` | Model name | `gpt-4`, `llama3:8b` |
+| `LDR_LLM_OPENAI_API_KEY` | OpenAI API key | `sk-...` |
+| `LDR_SEARCH_TOOL` | Default search engine | `searxng`, `arxiv`, `wikipedia` |
+| `LDR_SEARCH_SEARCH_STRATEGY` | Default strategy | `source-based`, `focused-iteration` |
+
+See [MCP Server Guide](mcp-server.md) for full documentation.
+
+---
+
+## Common Engine Names
+
+When using the rate limiting CLI, use these engine class names:
+
+| Engine | Class Name |
+|--------|------------|
+| DuckDuckGo | `DuckDuckGoSearchEngine` |
+| SearXNG | `SearXNGSearchEngine` |
+| Brave | `BraveSearchEngine` |
+| arXiv | `ArXivSearchEngine` |
+| PubMed | `PubMedSearchEngine` |
+| Semantic Scholar | `SemanticScholarSearchEngine` |
+| Wikipedia | `WikipediaSearchEngine` |
+| GitHub | `GitHubSearchEngine` |
+
+---
+
+## Troubleshooting
+
+### Benchmark Not Starting
+
+- Verify LLM provider is configured
+- Check search engine is available
+- Ensure sufficient disk space for results
+
+### Rate Limit Data Missing
+
+- Run some searches first to generate data
+- Check database file exists
+- Try `status` without `--engine` flag
+
+### Export Permission Error
+
+- Check write permissions on output directory
+- Use a different output directory
+
+---
+
+## See Also
+
+- [Architecture Overview](architecture/OVERVIEW.md) - System architecture
+- [Troubleshooting](troubleshooting.md) - Common issues
+- [BENCHMARKING.md](BENCHMARKING.md) - Detailed benchmark documentation

docs/decisions/0001-remove-detect-secrets.md

@@ -0,0 +1,60 @@
+# ADR-0001: Remove detect-secrets in favor of gitleaks
+
+**Date:** 2026-02-28
+**Status:** Accepted
+
+## Context
+
+The repository used two pre-commit secret scanners:
+
+1. **detect-secrets** (Yelp) — entropy + pattern analysis, tracking false positives
+   in a `.secrets.baseline` JSON file (173 KB, ~5200 lines)
+2. **gitleaks** — pattern-based detection with path/regex allowlists in
+   `.gitleaks.toml`
+
+### Problems with detect-secrets
+
+detect-secrets tracks known false positives using **line numbers** in
+`.secrets.baseline`. When any file changes above a flagged line, the baseline
+shifts and must be regenerated. This caused:
+
+- **Constant merge conflicts** on `.secrets.baseline` across branches
+- **Manual re-staging** of the baseline on every commit touching flagged files
+- **173 KB of PR churn** unrelated to actual secret scanning
+
+All 34 entropy-only detections in the baseline were verified as false positives
+(git SHAs, GPG fingerprints, base64 character sets, test fixtures).
+
+### Why gitleaks is sufficient
+
+- gitleaks uses **path-based and regex-based** allowlists (`.gitleaks.toml`)
+  that are stable across line changes — no baseline regeneration needed
+- gitleaks covers every service-specific detector that detect-secrets provides
+  (AWS, GitHub, Stripe, Slack, etc.)
+- gitleaks runs as both a **pre-commit hook** and in **CI workflows**
+  (`gitleaks.yml` on PRs, `gitleaks-main.yml` as release gate)
+- Additional CI coverage from **Semgrep** (`p/secrets`) and **Bearer**
+  (`scanner: sast,secrets`) in the release gate
+
+## Decision
+
+Remove detect-secrets entirely:
+
+- Delete the `Yelp/detect-secrets` hook from `.pre-commit-config.yaml`
+- Delete `.secrets.baseline`
+- Clean up references in `.gitleaks.toml`, `.gitleaksignore`, `CODEOWNERS`,
+  `danger-zone-alert.yml`, `SECURITY_REVIEW_PROCESS.md`, `SECURITY.md`,
+  and `.file-whitelist.txt`
+
+**Do not re-add detect-secrets.** Use `.gitleaks.toml` allowlists for managing
+false positives instead.
+
+## Consequences
+
+- Eliminates merge conflicts and churn from `.secrets.baseline`
+- Simplifies the pre-commit pipeline (one secret scanner instead of two)
+- False positives are managed via `.gitleaks.toml` path/regex allowlists and
+  `.gitleaksignore` commit fingerprints, both of which are stable across
+  line-number changes
+- No loss of detection coverage — gitleaks + Semgrep + Bearer provide
+  equivalent or better coverage

docs/decisions/0002-pre-commit-hook-reviews.md

@@ -0,0 +1,102 @@
+# ADR-0002: Pre-commit hook batch review
+
+**Date:** 2026-03-28
+**Status:** Accepted
+
+## Context
+
+Nine PRs proposing new or modified pre-commit hooks were evaluated
+simultaneously. The repository already has 43 pre-commit hooks; new hooks must
+clear a high bar for signal-to-noise ratio, speed, and non-duplication with CI.
+
+Review was conducted using 86 automated review agents across 4 rounds,
+cross-validating findings and verifying correctness of each recommendation.
+
+## Decision
+
+### Accepted (4 PRs)
+
+| PR | Hook | Fix Applied |
+|----|------|-------------|
+| #3220 | fix(security): escape server data in innerHTML | None — merged as-is |
+| #3221 | fix(config): sync ruff version (v0.14.10 → v0.15.8) | Full `ruff format .` pass to reformat all old-style lambdas |
+| #3225 | chore(hooks): `raise ... from` enforcement in except blocks | None — merged as-is |
+| #3219 | chore(hooks): layer-import boundary enforcement | Added `exclude: ^tests/` to prevent false positives on test files |
+
+### Rejected (5 PRs)
+
+**#3218 — utcnow callable check:** Directly contradicts the existing
+`check-utcnow-parens` hook. `sqlalchemy_utc.utcnow` is a `FunctionElement`
+class, so `utcnow()` creates a SQL expression rendered per-INSERT as
+`CURRENT_TIMESTAMP`. The existing hook correctly requires parentheses. Having
+both hooks would block every commit.
+
+**#3222 — codespell (typo detection):** False-positives on camelCase identifiers
+are a known, unsolved problem
+([codespell-project/codespell#196](https://github.com/codespell-project/codespell/issues/196),
+open since 2017). The tool treats `hasTable`, `doesnt`, `crasher`, and similar
+code identifiers as misspellings, requiring an ever-growing ignore-words-list.
+
+**#3227 — innerHTML regex scanner:** The core regex (`INTERPOLATION_RE` using
+`[^}]+`) truncates at the first `}`, failing on nested braces in template
+literals. Multiple real JS files are affected: `settings.js:80`,
+`collection_details.js:210`, `history.js:421,501,503`,
+`semantic_search.js:197-205`. This causes both false positives and false
+negatives — unacceptable for a security tool. The project already has ESLint and
+Bearer for JavaScript security analysis, both of which use proper AST-based
+parsing. Use ESLint `no-unsanitized` rule instead.
+
+**#3230 — vulture (dead code detection):** The hook used `language: system`,
+which requires vulture on the system PATH and breaks in CI (the pre-commit
+workflow does not install project dev dependencies). The larger issue is policy:
+vulture is advisory-only in CI (release gate, not PR gate). Promoting it to a
+blocking pre-commit hook is a policy escalation that should be a deliberate team
+decision. Use the existing `vulture-dead-code.yml` CI workflow instead.
+
+**#3231 — mypy (type checking):** Most production modules have
+`ignore_errors = true` in `pyproject.toml` (`web.*`, `database.*`, `settings.*`,
+`utilities.*`, `security.*`, `api.*`, and 8+ more). mypy already blocks PRs via
+`mypy-type-check.yml` + `ci-gate.yml`. Adding it as a pre-commit hook would add
+10-30 seconds per commit for near-zero detection value on the majority of the
+codebase.
+
+## Consequences
+
+- Four new hooks strengthen the pre-commit pipeline (XSS prevention, ruff
+  consistency, exception chaining, architectural boundaries)
+- Five proposals are documented as rejected with specific technical reasons,
+  preventing re-proposals without new information
+- Rejected hooks are listed in `.pre-commit-config.yaml` comments with a link
+  to this document
+- The total hook count increases from 43 to 45, with negligible performance
+  impact (~100-200ms per commit for the new hooks)
+
+## Principles for future hook proposals
+
+1. **Fast**: target under 3 seconds for a typical staged-file set.
+2. **Scoped appropriately**: file-scoped hooks are preferred. Hooks requiring
+   full-repo analysis need a narrow `files:` trigger and team consensus.
+3. **High local value**: don't add slow CI checks to pre-commit when the local
+   feedback value is low (e.g., mypy with most modules ignored).
+4. **Validated**: run the hook against the existing codebase before proposing.
+5. **Low false-positive rate**: if suppressions are needed from day one,
+   reconsider whether the hook belongs here.
+6. **Low false-positive on identifiers**: tools designed for natural language
+   may over-fire on camelCase identifiers, abbreviations, and domain terms.
+7. **Self-contained**: hooks must work in pre-commit's CI workflow without
+   requiring system-level dependencies (`language: python` with
+   `additional_dependencies`, not `language: system`).
+
+## Related PRs
+
+| PR | Title | Decision |
+|----|-------|----------|
+| [#3218](https://github.com/LearningCircuit/local-deep-research/pull/3218) | chore(hooks): prevent frozen utcnow() | Rejected — conflicts with existing hook |
+| [#3219](https://github.com/LearningCircuit/local-deep-research/pull/3219) | chore(hooks): layer-import boundary enforcement | Accepted (fix: `exclude: ^tests/`) |
+| [#3220](https://github.com/LearningCircuit/local-deep-research/pull/3220) | fix(security): escape server data in innerHTML | Accepted |
+| [#3221](https://github.com/LearningCircuit/local-deep-research/pull/3221) | fix(config): sync ruff version | Accepted (fix: full `ruff format .`) |
+| [#3222](https://github.com/LearningCircuit/local-deep-research/pull/3222) | chore(hooks): add codespell typo detection | Rejected — CamelCase false positives |
+| [#3225](https://github.com/LearningCircuit/local-deep-research/pull/3225) | chore(hooks): `raise ... from` in except blocks | Accepted |
+| [#3227](https://github.com/LearningCircuit/local-deep-research/pull/3227) | chore(hooks): detect unescaped innerHTML | Rejected — regex bug |
+| [#3230](https://github.com/LearningCircuit/local-deep-research/pull/3230) | chore(hooks): vulture dead code detection | Rejected — policy escalation |
+| [#3231](https://github.com/LearningCircuit/local-deep-research/pull/3231) | chore(hooks): mypy type checking | Rejected — already in CI |

docs/decisions/0003-reject-raise-without-from-hook.md

@@ -0,0 +1,88 @@
+# ADR-0003: Reject universal raise-without-from enforcement
+
+**Date:** 2026-03-28
+**Status:** Accepted
+
+## Context
+
+PR #3225 proposed a pre-commit hook (`check-raise-without-from`) to enforce
+`raise X(...) from e` inside all `except` blocks, citing PEP 3134 (explicit
+exception chaining). The hook would flag any `raise NewException(...)` inside
+an except handler that omits the `from` clause.
+
+### The conflict with existing security hooks
+
+The codebase already enforces **exception sanitization** to prevent PII
+leakage through three mechanisms:
+
+1. **`check-sensitive-logging.py`** blocks exception variables in
+   `logger.warning/error/critical()` calls and forbids `exc_info=True` on
+   production log levels. Only `logger.exception()` and
+   `logger.debug(..., exc_info=True)` are permitted.
+
+2. **`fix-exception-logging.py`** auto-removes exception variable references
+   from f-strings and `exc_info=True` from non-debug logs.
+
+3. **Intentional chain-breaking pattern** used across the codebase: catch a
+   broad exception, log full details with `logger.exception()`, then raise a
+   sanitized application exception *without* `from e`:
+
+   ```python
+   except Exception as e:
+       logger.exception("Error getting news feed")    # full details logged
+       raise NewsFeedGenerationException(str(e), ...)  # no "from e"
+   ```
+
+   This pattern appears in `news/api.py`, `notifications/service.py`,
+   `utilities/es_utils.py`, and others. Flask error handlers then return only
+   `error_code` and `message` to the client — never stack traces.
+
+### Why `raise X from e` is harmful here
+
+`raise X from e` preserves the full original exception chain at the Python
+runtime level. Even if logs are sanitized, the chain propagates to:
+
+- Error tracking services (Sentry, DataDog) that capture `__cause__`
+- Any downstream handler that logs with `exc_info=True`
+- Test output and development tooling that prints full tracebacks
+
+If the original exception contains PII (SQL errors with user data, API
+responses with auth tokens, file paths with usernames), preserving the chain
+defeats the sanitization strategy.
+
+### When `from e` IS appropriate
+
+Explicit chaining is valuable in infrastructure code where:
+
+- The original exception contains no user data (e.g., config parsing errors)
+- Both exceptions will be caught and handled before reaching any boundary
+- Debugging requires understanding the root cause chain
+
+The seven existing `from e` usages in the codebase (`mcp/client.py`,
+`config/llm_config.py`, `security/path_validator.py`, `exporters/`) follow
+this pattern correctly.
+
+## Decision
+
+Do not add a universal `raise-without-from` enforcement hook. The existing
+pattern of intentional chain-breaking for PII protection takes precedence
+over PEP 3134 compliance.
+
+Developers should use their judgment:
+
+- **Use `raise X from e`** when the original exception is safe to propagate
+  (no user data, internal infrastructure errors)
+- **Use `raise X from None`** when you want to explicitly suppress the chain
+  and document the intent
+- **Omit `from`** when wrapping user-facing exceptions to sanitize error
+  details (the current default pattern)
+
+## Consequences
+
+- The allowlisted files in the rejected PR do not need remediation
+- Exception chain decisions remain a code review concern, not an automated
+  enforcement
+- The existing `check-sensitive-logging` and `fix-exception-logging` hooks
+  remain the primary defense against PII leakage through exceptions
+- `raise X from None` is available for cases where developers want to
+  explicitly document chain suppression, but is not required

docs/decisions/0004-nullpool-for-sqlcipher.md

@@ -0,0 +1,103 @@
+# ADR-0004: QueuePool (not NullPool) for SQLCipher databases
+
+**Date:** 2026-04-13
+**Status:** Accepted
+
+## Context
+
+Each user has their own SQLCipher-encrypted SQLite database, opened at
+login and closed at logout. Background threads (research workers,
+metric writers, news scheduler jobs) need database sessions for the
+same user concurrently with Flask request handlers.
+
+### Why QueuePool
+
+SQLCipher's `PRAGMA key` adds ~0.2 ms per connection open. With 20–30
+queries per page load, NullPool (new connection per checkout) adds a
+noticeable 4–6 ms overhead vs QueuePool's ~1.5 ms for pool-resident
+connections.
+
+### Why not per-thread NullPool engines
+
+An earlier design maintained a second engine system: one NullPool
+engine per `(username, thread_id)` in `_thread_engines`, used by
+background threads for metric writes. This was removed in PR #3441
+because:
+
+1. **FD leak.** Each SQLCipher + WAL connection holds 3 file
+   descriptors (main db + WAL + SHM). Orphaned thread engines — left
+   behind when `@thread_cleanup` did not fire — accumulated FDs
+   unboundedly, eventually exhausting the 1024 soft limit and crashing
+   the server with `OSError: [Errno 24] Too many open files`.
+
+2. **Architectural redundancy.** The per-user QueuePool engine is
+   already created with `check_same_thread=False`, making it safe for
+   background threads. Routing all work through one bounded pool per
+   user keeps FD usage at `pool_size + max_overflow` (currently 60)
+   instead of scaling with background-thread count.
+
+### SQLCipher + WAL handle leak workaround
+
+SQLCipher in WAL mode leaks file handles when pooled connections close
+out of open-order (a known issue with WAL-mode SQLite engines under
+connection pooling). The cleanup scheduler in `connection_cleanup.py`
+calls `engine.dispose()` on all per-user engines every 30 minutes,
+closing all idle pooled connections and resetting handle state. This is
+a workaround, not a root fix — it limits accumulation to a 30-minute
+window.
+
+### Current pool sizing
+
+```
+pool_size     = 20
+max_overflow  = 40
+pool_timeout  = 10   # seconds; fail fast rather than queue
+pool_recycle  = 3600 # seconds; recycle stale connections
+pool_pre_ping = True
+```
+
+Peak FD usage per user: `(20 + 40) × 2 + 1 = 121` (WAL mode).
+
+## Decision
+
+Use a single shared QueuePool engine per user for all threads (request
+handlers and background workers). Do not maintain per-thread engines.
+Periodic `dispose()` mitigates the SQLCipher+WAL handle leak.
+
+## Consequences
+
+- FD usage is bounded and predictable.
+- `pool_timeout=10` makes pool exhaustion a loud error rather than a
+  silent deadlock.
+- The `ParallelConstrainedStrategy` (`max_workers=100`) could
+  theoretically spike past 60 simultaneous checkouts. Sessions are
+  short-lived (millisecond metric writes), so sustained contention is
+  unlikely. Flagged as a known follow-up.
+
+## Addendum — PR #3487 investigation (2026-04-16)
+
+PR #3487 proposed skipping the 30-min `engine.dispose()` for any engine
+with `pool.checkedout() > 0`, citing a claim that dispose orphans
+checked-out connections and causes torn writes in the post-login bulk
+settings import. Investigation found:
+
+- **SA 2.0 source disagrees.** `QueuePool.dispose` only drains idle
+  queue entries (`_pool.get(False)`); `Engine.dispose` replaces the
+  pool via `pool.recreate()`. SA docs: *"Connections that are still
+  checked out will not be closed."* A thread holding a checked-out
+  connection keeps using it until return.
+- **Real root cause was elsewhere.** The sticky-loop symptom came
+  from the post-login path committing twice
+  (`load_from_defaults_file(commit=True)` then `update_db_version()`
+  with its own commit), which left `app.version` unwritten on any
+  inter-commit failure. Fix: one session, one terminal commit, both
+  calls use `commit=False`. See `web/auth/routes.py` ATOMICITY
+  INVARIANT comment.
+- **Regression guard:** `tests/database/test_post_login_settings_atomicity.py`
+  locks in both properties — the atomic all-or-nothing write and the
+  SA 2.0 checked-out-survives-dispose contract — so neither can
+  silently regress.
+
+Future refactors of the cleanup cycle should preserve the
+checked-out-survival property; do not add a `checkedout()` skip guard
+without a real reproducer.

docs/decisions/0005-reject-inapp-response-compression.md

@@ -0,0 +1,83 @@
+# ADR-0005: Reject in-app HTTP response compression
+
+**Date:** 2026-06-28
+**Status:** Accepted
+
+## Context
+
+LDR serves its front-end as Vite-built static bundles (~2 MB JS + ~435 KB CSS)
+through the custom `app_serve_static` route in `web/app_factory.py`. In a
+single-process deployment with no reverse proxy these assets are sent
+**uncompressed**, so a proposal (PR #3158, then a re-scoped PR #4832) added
+[`flask-compress`](https://github.com/colour-science/flask-compress) to compress
+them in-process with zstd/brotli/deflate, restricted to static MIME types.
+
+The change was implemented, adversarially reviewed, and ultimately **rejected**.
+Both PRs are closed; nothing was merged.
+
+### Why the benefit is marginal
+
+- **Localhost is the default and the point of the tool.** Over loopback,
+  transfer is effectively free; compression only spends CPU. Content-hashed
+  bundles are already `immutable`-cached, so a browser fetches them once.
+- The win only materialises for users self-hosting LDR **remotely, for multiple
+  users, with no reverse proxy** — a small slice of a "Local" research tool.
+- Those same users should run a reverse proxy anyway (for TLS), and **nginx /
+  Caddy then provide compression *and* a disk-cached compressed artifact for
+  free** — strictly better than re-compressing on every cold request in-process
+  (`flask-compress`'s `COMPRESS_CACHE_BACKEND` defaults to `None`, so there is no
+  compressed-output cache).
+
+### Why the cost/risk is real
+
+In-app compression bolts an HTTP-correctness surface onto the app. Adversarial
+review found, and empirically reproduced:
+
+- **Malformed `Range`/`206` responses.** `flask-compress` has no range guard:
+  a `Range` request with a compressible `Accept-Encoding` yields a `206` whose
+  `Content-Range` describes the *identity* (uncompressed) coordinates while the
+  body is compressed (e.g. `Content-Range: bytes 0-1023/14000` with a 23-byte
+  zstd body). This is malformed per RFC 9110 and **corrupts conformant slicing
+  intermediaries**: nginx's `slice` module resets the connection
+  ("unexpected range in slice response"), CloudFront caches corrupt byte math,
+  and `curl -C -` / download-manager resumes break.
+- **Varnish mis-serve.** Varnish (default `http_gzip_support=on`) deliberately
+  ignores `Vary: Accept-Encoding`; with the standard "brotli through Varnish"
+  VCL it can cache a brotli body and serve it to a gzip-only client.
+- **CDN re-opens BREACH.** Cloudflare and Fastly compress `text/html` themselves
+  by default, so behind them the CSRF-token HTML is compressed regardless of an
+  in-app MIME allow-list. (LDR's CSRF token is already per-render masked by
+  Flask-WTF, which is the durable BREACH mitigation — not declining to compress
+  in-process.)
+- **CPU amplification.** `app_serve_static` is public and `@limiter.exempt`; with
+  no compressed-output cache, a client can force repeated compression of the
+  ~2 MB bundle. Pre-change, static serving was a near-zero-CPU file send.
+
+The benefit accrues to a minority; several of the risks affect *more* setups
+(anyone behind a slicing CDN, Varnish, or a TLS-terminating CDN). That is a poor
+trade for a local-first tool.
+
+## Decision
+
+**Do not add in-app HTTP response compression to the Flask application.** Do not
+re-introduce `flask-compress` or an equivalent runtime compressor.
+
+For remote / multi-user deployments, the guidance is to **front LDR with a
+reverse proxy** (nginx / Caddy), which handles TLS, compression, and caching —
+see [Deploying behind a reverse proxy](../deployment/reverse-proxy.md).
+
+If asset compression is ever genuinely wanted *without* a proxy, use
+**build-time pre-compression** instead: have Vite emit `.br` / `.gz` artifacts
+served as plain static files. That has zero per-request CPU cost, adds no runtime
+dependency, and — because the files are served plainly — avoids the
+content-negotiation, conditional-request, and byte-range edge cases above.
+
+## Consequences
+
+- The codebase keeps a near-zero-CPU static file path and no new dependency
+  (`flask-compress` + transitive `backports-zstd`).
+- Remote operators get compression from their reverse proxy/CDN, which is where
+  it belongs.
+- Two of PR #3158's original items were unrelated and **already shipped**:
+  static `Cache-Control` headers (#3185 / #3207) and removal of the duplicate
+  `styles.css` link (#3207). Only the compression idea is rejected here.

docs/decisions/0006-focused-iteration-scope-and-final-max-results.md

@@ -0,0 +1,336 @@
+# ADR-0006: Keep `focused-iteration` minimal — accept the state-leak/citation fixes, defer the `final_max_results` truncation
+
+**Date:** 2026-06-28 (updated 2026-07-03)
+**Status:** Accepted
+**Relates to:** PR #4850 (merged 2026-06-28), issue #4851, #2001 (closed), #2067 (closed), #2716/#2724/#2799 (open), #4420 (merged)
+
+> **Outcome:** this decision was enacted the same day it was drafted. The
+> contributor dropped the truncation and added tests for both fixes; #4850
+> merged on 2026-06-28 as "fix(focused-iteration): reset per-call state and
+> offset citations across subsections". See the Decision section at the end.
+
+## Context
+
+PR #4850 (originally "fix(focused-iteration): resolve state leak, add
+truncation ceiling and correct citation offset") bundled three independent
+changes to `advanced_search_system/strategies/focused_iteration_strategy.py`.
+Two are genuine bug fixes; the third re-introduces a change we have already
+rejected twice. This ADR records the per-change verdict and the surrounding
+history so we do not re-litigate it a fourth time.
+
+`focused-iteration` is one of the five production strategies that remain after
+#4420 removed ~22 experimental ones (and #4548 later removed `mcp`). It is the documented 96.51%-SimpleQA strategy
+(see the module docstring), but it is **not** the primary direction anymore: the
+default strategy is now **`langgraph-agent`**. langgraph-agent is considered the
+more flexible and stronger-performing option — it can drive multiple tools and
+adapt its approach per query — and it is the recommended path when there is
+enough VRAM to run a capable model (or a hosted model is used). `focused-iteration`
+remains supported and strong for SimpleQA-style factual lookups, but it is a
+secondary path. So the bar for changing it is "fix real bugs, change behaviour as
+little as possible" — not "make it converge with `source-based`."
+
+## The three changes in PR #4850
+
+### 1. State-leak reset — **ACCEPT**
+
+```python
+# top of analyze_topic()
+self.all_search_results = []
+self.results_by_iteration = {}
+```
+
+In report ("detailed") mode the **same** strategy instance handles every
+subsection sequentially (`report_generator.py:476-493`, which also pins
+`max_iterations=1` per subsection). On `main`, `all_search_results` was never
+reset, so subsection *N* synthesised over subsections 1..*N* combined →
+unbounded growth → context-window overflow (issue #4851).
+
+Why we want it: it is the actual root cause. Three independent reviewers
+confirmed the reset fixes the leak and introduces no new cross-subsection
+corruption. Residual un-reset state (`explorer.progress`,
+`findings_repository.documents`) is pre-existing and inert under report mode's
+1-iteration pin — a possible follow-up, not a blocker. The sibling
+`source_based_strategy` avoids the class of bug structurally by using a *local*
+accumulator instead of a `self.` attribute; that is the cleaner long-term shape,
+but the reset is correct as-is.
+
+**Completeness check (does the reset drop sources?).** A natural worry is that
+this change — which also moves the bibliography collection from a per-iteration
+`all_links_of_system.extend(iteration_results)` *inside* the loop to a single
+post-loop `all_links_of_system.extend(self.all_search_results)` — could leave the
+source list incomplete. It does not, because `all_links_of_system` is **not**
+reset: it is the cumulative, shared list (the *same* object as
+`search_system.all_links_of_system`, search_system.py:229; the `id()` guard at
+search_system.py:472-475 prevents the back-compat re-extend from doubling it).
+Each `analyze_topic` call appends only *its own* accumulated results to that
+shared list, so across report subsections it accumulates correctly:
+
+| Step | `all_search_results` | `all_links_of_system` (shared) | offset |
+|------|----------------------|--------------------------------|--------|
+| Sub1 | reset→`[A,B,C]`      | `[A,B,C]`                      | 0 → [1,2,3] |
+| Sub2 | reset→`[D,E]`        | `[A,B,C,D,E]`                  | 3 → [4,5] |
+| Sub3 | reset→`[F]`          | `[A,B,C,D,E,F]`                | 5 → [6] |
+
+On `main` the same scenario also ends with `[A…F]` in `all_links` — the `main`
+bug was that Sub3's *synthesis* re-ingested A–E (overflow), not that sources went
+missing. So **on the success path**, with the truncation ceiling (change #2) held
+high enough never to fire, the PR's `all_links_of_system` contains the same
+sources as `main` (verified for single-research mode, report mode, and the
+follow-up delegate path, which propagates the delegate's sources via
+`result["all_links_of_system"]` in `enhanced_contextual_followup.py:208-244`).
+Nothing reads `all_links_of_system` mid-loop, so removing the per-iteration extend
+breaks no incremental reader.
+
+**One error-path caveat (adversarially verified).** Because the PR's *only*
+`all_links_of_system.extend(...)` now lives *after* the loop, inside the same
+`try` that wraps the loop (the catch-all `except Exception` in
+`analyze_topic`; line 382 as of the #4850 merge), an **uncaught exception mid-loop** — e.g.
+the unguarded `model.invoke` in question generation (`browsecomp_question.py:96`
+and `:282`), which fails routinely on LLM rate-limit/timeout — skips that extend.
+On `main`, the per-iteration extend had already recorded the completed iterations'
+sources, so a degraded/failed single-research run still surfaces them in its
+Sources list; on the PR, that failed run surfaces none. This is a real divergence,
+but only on the *error* path (the run already returns `_create_error_response`),
+and the PR's all-or-nothing behaviour is arguably *preferable* (no half-populated
+bibliography from a run that errored out). In report mode it does not arise
+(`max_iterations=1`: a mid-loop throw means iteration 1 never completed, so
+neither side accumulated anything). It is **not** the "source list suddenly
+incomplete" symptom on successful research — that one is change #2 alone (below).
+
+### 2. `final_max_results` truncation ceiling — **DROP FROM THIS PR (cap is open for separate, benchmark-gated debate)**
+
+The objection is to the *blind insertion-order slice as shipped in #4850*, not to
+the idea of capping results in principle. A relevance-aware cap may have merit;
+the maintainer is open to it as its own PR, decided with a benchmark (see the
+"standing position" note below).
+
+```python
+max_results = int(self.get_setting("search.final_max_results", 100))
+if len(self.all_search_results) > max_results:
+    self.all_search_results = self.all_search_results[:max_results]   # blind, insertion-order
+```
+
+Why we do **not** want this:
+
+- **It keeps the *first* 100, not the *best* 100.** Results accumulate
+  iteration 1→8; the strategy's highest-value work (verification searches gated
+  on `iteration > 3`) lands at the *tail* and is the first thing dropped. The
+  proven sibling applies the *same* setting via
+  `cross_engine_filter.filter_results(..., reorder=True, reindex=True)` —
+  relevance-rank + dedup, then top-N (`source_based_strategy.py:421-431`). A raw
+  `[:100]` slice does neither; the setting's own description even promises
+  "deduplicating and filtering."
+- **It runs unconditionally on the protected SimpleQA path** (8 iterations),
+  which the docstring asks us to preserve. `main` applies **no** truncation
+  here, so the 96.51% number was measured without it. At default
+  `search.max_results=50` × 5 questions, the 100 ceiling is hit by ~iteration 2,
+  discarding iterations 3–8 from synthesis — an unbenchmarked accuracy risk.
+- **It is mostly unnecessary for the bug it targets.** Report mode already
+  forces `max_iterations=1`, and change #1 (the reset) already fixes the
+  overflow. The truncation's cost falls almost entirely on the *non-report* path
+  it was not meant to touch.
+- **It is the one change that actually shrinks the source list.** Per the
+  completeness check under #1, `all_links_of_system` is otherwise complete. But
+  in single-research mode (8 iterations producing, say, 250 results) the slice
+  caps `all_search_results` at 100 *before* the `all_links` extend, so the
+  returned `sources` / Sources section silently drops the other 150 (and keeps
+  the *first* 100 by insertion order, not the best). On `main` all 250 are
+  returned. This is exactly the "source list suddenly incomplete" failure mode —
+  and it is caused by #2, not #1. Dropping #2 restores a complete, correctly
+  numbered source list.
+- **Minor robustness smell:** `int(get_setting(..., 100))` raises on a
+  present-but-null/non-numeric value, and the surrounding `except Exception`
+  converts that into a generic "Search iteration failed" *after* all search work
+  (reachable via direct DB / programmatic `settings_snapshot`, not the validated
+  UI).
+
+This is **the third time** this exact change has been proposed:
+
+- **#2001** — "apply final_max_results filtering in focused_iteration_strategy"
+  (CrossEngineFilter, matching source-based). **Closed.** Maintainer:
+  *"I am not so sure about this change. It is not what the strategy is supposed
+  to be. Not every strategy should do the same thing."* and *"Outdated due to
+  langraph strategy."*
+- **#2067** — same idea, simpler. **Closed.**
+- **#4850 (#2 of 3)** — same idea, blunter (`[:N]` instead of the filter).
+
+**Standing position (open, not a flat no).** The decision for *this PR* is to drop
+#2; the broader question of whether `focused-iteration` should cap results at all
+is open for separate debate. Guardrails for that debate: (a) not every strategy
+needs the same final-cap behaviour; (b) any cap must preserve the *best* sources
+(relevance-rank/dedup, e.g. harvested from the V2/V3 work below), not drop them by
+position; and (c) it should be gated on a SimpleQA benchmark proving no
+regression. Note also that an earlier informal evaluation (maintainer's
+recollection, details unverified) did not show a clear performance gain from a
+results cap — a reason to benchmark before committing, not a settled conclusion.
+
+**Post-merge evidence (2026-06-29).** After #4850 merged (with the truncation
+dropped), the contributor reported still hitting the context-length error on
+report mode: the reset fixes the *cross-subsection* leak, but nothing in the
+strategy accounts for the model's context window within a single call, so a
+sufficiently large single run can still overflow. That confirms #4851 had two
+components — the state leak (fixed) and missing context-length awareness (not
+fixed) — and it strengthens the case for having the capping debate. It does
+*not* change the verdict on the blind head-slice: a context-aware or
+relevance-aware mechanism under guardrails (a)–(c) above is the follow-up to
+propose, not `[:100]`.
+
+### 3. Citation offset — **ACCEPT**
+
+```python
+nr_of_links = total_citation_count_before_this_search  # = len(all_links_of_system), was hardcoded 0
+```
+
+In report mode each subsection previously re-numbered citations from `[1]`
+(`nr_of_links=0`), so `[1]` meant different sources in different subsections and
+desynced from the single shared bibliography that `format_links_to_markdown`
+builds from `all_links_of_system`. Offsetting by the running link count makes
+inline citations contiguous and collision-free across subsections, matching the
+proven `source_based_strategy` invariant (verified by walking subsections 1→3:
+no off-by-one, gap, or orphan). For single (non-report) research the offset is 0,
+so citations still start at `[1]`.
+
+Test coverage: an early revision of #4850 only tested change #1; the merged
+version covers everything we asked for —
+`test_does_not_accumulate_results_across_calls` (#1),
+`test_citation_offset_uses_existing_all_links_count` and
+`test_report_mode_indices_stay_sequential_across_sections` (#3).
+
+## Cross-strategy comparison — #1 and #3 were focused-only oversights
+
+The two fixes are not a cross-cutting change; they bring `focused-iteration` in
+line with what the other production strategies already do. The contested
+truncation (#2) matches *neither* sibling.
+
+| Concern | focused (pre-#4850) | focused (post-#4850) | `langgraph-agent` (default) | `source-based` |
+|---|---|---|---|---|
+| **#1** Reset per-subsection working set | ❌ never reset (the leak) | ✅ resets `all_search_results` | ✅ `self.collector.reset()` (`langgraph_agent_strategy.py:788`) | ✅ *local* accumulator — no shared state to leak |
+| **#3** Citation offset | ❌ `nr_of_links=0` | ✅ `len(all_links_of_system)` | ✅ `nr_of_links = len(self.all_links_of_system)` (`:789`) | ✅ `:178` / `:481` |
+| **#2** `final_max_results` cap | none | blind `[:100]` | **none** (agent manages its own context) | proper: cross-engine filter `reorder=True, reindex=True` |
+
+Takeaways:
+
+- **#1 and #3 fix bugs unique to `focused-iteration`.** `langgraph-agent` and
+  `source-based` already reset per subsection and already offset citations, so the
+  fixes match two independent, already-shipping implementations — strong evidence
+  they are correct.
+- **#2 is an outlier in both directions.** Only `source-based` caps, and it does so
+  *properly* (relevance reorder + reindex); `langgraph-agent` deliberately doesn't
+  cap. A blind head-slice matches neither — extra evidence to drop it.
+- Because `langgraph-agent` is the **default**, the #4851 overflow this PR fixes
+  mainly bites users who switched to `focused-iteration` for report mode — which
+  fits the "not the primary investment area" framing.
+
+## Cross-section context & citation resolution (report mode)
+
+Resetting the per-subsection working set raises a fair question: does a later
+subsection still "know" what earlier ones used, and do in-text citations still
+resolve? Both hold, because the relevant state is **not** what gets reset:
+
+- **Coherence channel is independent of the reset.** Report generation feeds each
+  subsection a `=== CONTENT ALREADY WRITTEN (DO NOT REPEAT) ===` block built from
+  the previous sections' *generated content* (`report_generator.py:261-297`,
+  `_build_previous_context`; default last 3 sections, char-capped), injected into
+  the subsection query that drives both question generation and synthesis. This
+  survives change #1 untouched. The `main` leak added *raw search results* on top
+  of that — which is the overflow bug, not the coherence mechanism.
+- **In-text citations still resolve.** Inline `[N]` markers are hyperlinked by
+  index-matching against the structured source list
+  (`text_optimization/citation_formatter.py` `apply_inline_hyperlinks`:
+  `{str(s["index"]): (title, url)}`), and that list is the cumulative, un-reset
+  `all_links_of_system`. With #3 giving each subsection a distinct index range,
+  every `[N]` maps to exactly one bibliography entry. The reset removes sources
+  from the per-subsection *synthesis input*, not from the *source list citations
+  resolve against* — only #2's truncation removes anything from the latter.
+
+**Known soft nuance (separate follow-up, not a blocker for #4850).**
+`_build_previous_context` injects prior prose *with its old `[N]` markers intact*
+(no stripping). After the reset, the current subsection's synthesis LLM sees those
+numbers but no longer holds their source content (only its own offset-numbered
+sources). They still resolve globally in the assembled report, and the "do not
+repeat" instruction discourages echoing them, so this is a quality wrinkle rather
+than broken links — and it is strictly *less* bad than `main`, where `nr_of_links=0`
+made the numbers collide across sections. The clean fix (strip inline markers from
+the context block, or wire a real `previous_knowledge` in place of the hardcoded
+`""` in `focused_iteration_strategy.py`'s `analyze_followup` call, line 325 as of
+the #4850 merge) belongs in its own PR.
+
+## History of focused-iteration (for future readers)
+
+- **#917** — introduced the strategy (standard + adaptive search).
+- **#1277** — configurable options (the A/B knobs in `__init__`).
+- **#1258 / #1282** — robustness (empty-questions ThreadPool crash; instruction
+  injection when adaptive questions disabled).
+- **#2293** — docs: suggest 10–20 iterations.
+- **#2001 / #2067** — `final_max_results` truncation. **Both closed** (see above).
+- **V2/V3/V4 A/B era (#2716 = V2, #2724 = V2+V3 combined, #2799 = V4 — all
+  open):** separate, selectable strategies for benchmarking, leaving V1
+  untouched. V2 adds **URL dedup**; V3 adds **semantic dedup + relevance
+  sorting** (places most-relevant results last for context-window use); V4 swaps
+  in a task-oriented question generator. Merged review-fix PRs from this era:
+  #2797, #2980, #2996.
+- **#4420 (merged):** removed ~22 experimental strategies + the
+  `show_all_strategies` toggle. Survivors: `source-based`, `focused-iteration`,
+  `focused-iteration-standard`, `topic-organization`, `mcp`, `langgraph-agent`
+  (default). `mcp` was subsequently removed by #4548 (the factory now redirects
+  it to `langgraph-agent`). The V2/V3/V4 strategy files are **not** on `main`.
+- **#4711 / #4717 / #4729** — DRY refactors (base error-response + citation
+  helpers, `run_parallel_searches`, post-merge cleanups).
+
+## Open V2/V3/V4 PRs (#2716, #2724, #2799) — keep open, revisit later
+
+**Status: undecided — left open on purpose.** This ADR records the context for a
+future revisit rather than recommending closure now.
+
+Factors to weigh when we come back to them:
+
+- **Against merging as-is:** exploratory A/B artifacts (created March 2026;
+  785 / 2,406 / 3,687 lines; `mergeable` UNKNOWN — likely heavy conflicts after
+  #4711/#4717/#4729 and #4420); **never benchmarked** against V1 (#2716's
+  SimpleQA checkbox is unchecked; #2724/#2799 don't mention a benchmark at
+  all); and as standalone strategies they would re-add
+  experimental options that #4420 deliberately removed, against the
+  langgraph-first direction.
+- **Worth keeping for:** the *ideas*, not the parallel strategies. V2's URL dedup
+  and V3's relevance-sorting-before-cap are exactly the "keep the best, not the
+  first" mechanism a future `focused-iteration` cap would need (see change #2).
+  If a cap is ever justified, the cleanest path is to port that — small, targeted,
+  into the single strategy — and benchmark it, rather than reviving three
+  separate strategies.
+
+No action on these PRs for now; this ADR just makes the trade-offs explicit so
+the eventual decision is a quick one.
+
+## Decision
+
+We asked the contributor to **split**: keep changes #1 (reset) and #3 (citation
+offset) — both correct, valuable bug fixes — and **drop #2 (truncation)**, plus
+add the missing tests for #3. The contributor did exactly that, and #4850
+merged on 2026-06-28 with #1 + #3 and tests for both. That shipped the
+state-leak half of #4851 without reshaping the strategy's synthesis behaviour
+or re-opening a question we had already closed twice.
+
+## Consequences
+
+- `focused-iteration` now resets its per-call working set and numbers citations
+  contiguously across report subsections, matching `langgraph-agent` and
+  `source-based`.
+- No truncation was added; the 96.51%-SimpleQA synthesis path is
+  behaviourally unchanged.
+- The context-length half of #4851 remains open (see the post-merge evidence
+  under change #2): any future cap must follow guardrails (a)–(c) —
+  per-strategy justification, keep-the-best (not head-slice), SimpleQA
+  benchmark gate.
+- Follow-ups (separate PRs):
+  1. **Cross-section context quality** — strip inline `[N]` markers from
+     `_build_previous_context`, and/or wire a real `previous_knowledge` in
+     place of the hardcoded `""` in `focused_iteration_strategy.py`'s
+     `analyze_followup` call; verify with an LLM-judged before/after on a
+     report-mode query (not unit-testable).
+  2. **Optional refactor** — move `focused-iteration` to a *local* accumulator
+     like `source-based`, so the per-call working set cannot leak by
+     construction (removes the need to remember the reset).
+  3. **If a result cap is ever wanted on `focused-iteration`** — port V2/V3's
+     relevance-rank + dedup (not a blind slice), consider context-length
+     awareness per the post-merge evidence, and gate it behind a SimpleQA
+     benchmark.

docs/decisions/0007-egress-two-axis-data-classification.md

@@ -0,0 +1,302 @@
+# ADR-0007: Restructure egress around a two-axis data-classification model
+
+**Date:** 2026-06-28
+**Status:** Accepted (incremental rollout)
+
+## The model at a glance
+
+Every model a run touches (search engine, collection / store, LLM, embeddings)
+is labelled on two **orthogonal** axes — **Sensitivity** (of the data it
+*sources*) and **Exposure** (of the *sink* it is). The matrix reads: *a
+component with this Sensitivity (row) and this Exposure (column) is treated as
+follows.*
+
+|  | **Contained** sink | **Exposing** sink |
+|---|---|---|
+| **Non-sensitive** source | Combines with **anything** — *public collection, local Ollama* | Only alongside **non-sensitive** sources — *public web engine, cloud LLM* |
+| **Sensitive** source | Only with other **contained** sources; never an exposing sink — *private collection + local LLM* | **By itself only**, with contained (local) inference — *remote monitored store of private data* |
+
+**The rule in one line:** a run may never let a *sensitive* source reach an
+*exposing* sink — unless it is explicitly switched to **Permissive** mode (which
+warns but never blocks). The rest of this document explains why this is the
+right model, how each component is classified, and how we roll it out.
+
+## Out of scope: the query text itself
+
+This model classifies **sources** and **sinks** — not the **question the user
+types**. A run's query is sent verbatim to whichever search sinks the run uses,
+so a sensitive question typed against an exposing engine leaves the machine
+*regardless* of how the sources are labelled. The guardrail cannot inspect or
+sanitise query intent; that stays the **user's responsibility**. The UI must say
+so plainly — *we can't protect the content of your questions, so choose sources
+appropriate to how sensitive your question is.*
+
+## Context
+
+The egress guardrail (`src/local_deep_research/security/egress/`, see its
+`README.md`) currently expresses policy on a **single axis**: a per-run
+`policy.egress_scope` enum (`adaptive` / `both` / `public_only` /
+`private_only` / `strict`), refined by a per-collection `is_public` flag and
+two `require_local` inference toggles.
+
+This works for the common cases, but it collapses two genuinely independent
+properties into one word, and that conflation is now blocking us.
+
+### Problem 1 — sensitivity and exposure are different things, fused into one word
+
+"public ↔ private" is treated as a single spectrum. Two distinct questions hide
+inside it:
+
+- **Is the *data* sensitive?** — a property of a **source** (a collection, a
+  document store).
+- **Does the *destination* expose data outward?** — a property of a **sink** (a
+  web search engine you send a query to; a cloud LLM you send chunks to).
+
+A collection marked `is_public` is the clearest symptom. The word implies the
+data is *published*, but a collection is **always a local store** — searching it
+never pushes its contents to a search engine. What `is_public` actually
+authorizes is *cloud inference* on that data. "Public" overstates the exposure
+axis to describe the sensitivity axis.
+
+### Problem 2 — dual-risk sources can't be expressed
+
+Elasticsearch and Paperless break the single axis outright. An Elasticsearch
+instance can be:
+
+- a **sensitive data store** (your private corpus) → must not be combined with
+  exposing sinks, **and/or**
+- a **monitored / external service** (a sink that itself exposes the queries you
+  send it).
+
+These are opposite risks, and a single `is_local` / `is_public` flag can only
+pick one. Today the only safety for them is the asymmetric URL fail-up. We
+genuinely "cannot guarantee anything" for these engines under the current model,
+so for now they should not be auto-combined with other sources.
+
+### Problem 3 — `both` is a silent blanket
+
+`both` overrides every per-source classification at once, with no visual or
+conceptual signal that protection has been dropped. It is the muddy middle
+between "classify each source" and "turn the policy off."
+
+### This is a known model
+
+The two-axis framing is the standard **Data Loss Prevention (DLP) /
+information-flow-control** model: classify data by **sensitivity**, classify
+destinations by **trust / exposure**, forbid sensitive→exposing flows. The
+enforcement-mode vocabulary is **SELinux's** (`enforcing` / `permissive` /
+`disabled`). The egress package already borrows XACML / zero-trust PDP–PEP
+vocabulary, so this is a continuation, not a foreign import.
+
+## Decision
+
+Re-base the egress model on **two orthogonal labels**:
+
+| Axis | Applies to | Values | Question |
+|---|---|---|---|
+| **Sensitivity** | sources | `sensitive` / `non-sensitive` | Must this data never leave the machine? |
+| **Exposure** | sinks | `exposing` / `contained` | Does sending data here put it off-machine? |
+
+Some components play **both roles** — a search engine is a *sink* for your query
+and a *source* of results; Elasticsearch is a *source* of data and a *sink* for
+your query. Each gets a label on every axis it participates in. This is exactly
+what the single-axis model cannot represent and is the crux of the change.
+
+**Core invariant:** *sensitive* data must not flow to an *exposing* sink —
+unless the run is explicitly switched to the escape-hatch mode.
+
+### Every model carries both labels — classification is per-component
+
+The classification is computed **per component**, and "component" means *every*
+model the run touches: each search engine, each collection / store, the LLM, and
+the embeddings model. A single function —
+`classify(component, settings) -> (Sensitivity, Exposure)` — becomes the one
+home for logic scattered today across `_engine_bucket`, the
+`_CLOUD_LLM_PROVIDERS` set, the per-collection `is_public` lookup, and the URL
+fail-up.
+
+| Component (example) | Sensitivity | Exposure |
+|---|---|---|
+| Public collection / public web results | non-sensitive | contained |
+| Private collection (default) | **sensitive** | contained |
+| Public web / academic engine (Google, arXiv) | non-sensitive | **exposing** (query egress) |
+| Cloud LLM / embeddings (Anthropic, OpenAI) | non-sensitive | **exposing** |
+| Local LLM / embeddings (Ollama, sentence-transformers) | non-sensitive | contained |
+| Paperless / Elasticsearch — local, private | **sensitive** | contained |
+| Elasticsearch — remote / monitored, private | **sensitive** | **exposing** |
+
+### The four quadrants — the combination rule
+
+A run touches a *set* of components. Whether the set is permitted follows
+directly from the two labels:
+
+| # | A component that is… | may be combined with | Example |
+|---|---|---|---|
+| 1 | non-sensitive, contained | **anything** | public collection; local Ollama |
+| 2 | **sensitive**, contained | other **contained** sources only (sensitive or not) — **no exposing sink** | private collection + local LLM |
+| 3 | non-sensitive, **exposing** | **non-sensitive** sources only | cloud LLM; public web engine |
+| 4 | **sensitive + exposing** | the **only sensitive source** (non-sensitive contained companions are fine), with **contained (local) inference** | remote monitored store holding private data |
+
+Underlying invariant: **a run may not let a sensitive source reach an exposing
+sink.** Because an agentic run can turn one engine's results into another
+engine's query (the LangGraph silent-expansion class already in our threat
+model), this is enforced over the whole run's component set, not per call.
+Equivalent phrasing: a run must be **either** all-non-sensitive **or** free of
+exposing sinks — with a lone quadrant-4 component allowed to run as the **only
+sensitive source** (non-sensitive contained companions are still fine), since
+returning its own data to itself is not a *new* leak and its inference is forced
+contained.
+
+The escape hatch (**Permissive / "Unprotected"**) means "suspend this invariant
+for this run, keep warning."
+
+### Enforcement modes (SELinux naming)
+
+- **Enforcing** — current behaviour: the invariant is enforced; violating
+  engines/sinks are blocked.
+- **Permissive ("Unprotected")** — the escape hatch: the policy is still
+  *evaluated* so the warning banners fire, but nothing is blocked. Replaces
+  `both`, with an honest, loud, light-red UI. (User-facing label is an open
+  question — "Unprotected" / "Unrestricted" / "Permissive".)
+
+### Mapping the current model onto the two axes (migration, not rewrite)
+
+- collection `is_public=False` → Sensitivity **sensitive**; `is_public=True` →
+  **non-sensitive** (relabel away from "public" — it never *publishes*).
+- public web / academic engines → Exposure **exposing**.
+- cloud LLM / cloud embeddings → Exposure **exposing** sinks;
+  `require_local_*` = "forbid exposing inference sinks."
+- local collections / local LLM (Ollama) → Exposure **contained**.
+- scopes re-expressed: `private_only` ≈ "no exposing sinks"; `public_only` ≈
+  "exposing sinks allowed, sensitive sources excluded"; `strict` ≈ single
+  source; `adaptive` ≈ infer the run's posture from the primary.
+- `both` → removed, superseded by per-source classification + Permissive mode.
+- Elasticsearch / Paperless → declared **sensitive + contained** by default
+  (usable with other contained / local sources — quadrant 2), with the URL
+  fail-up flipping exposure to **exposing** (quadrant 4) when the configured
+  endpoint resolves to a public host. A per-destination trust entry
+  (`policy.trusted_search_engines`) can re-contain a self-hosted instance that
+  happens to sit on a public hostname.
+
+### Per-destination trust (the "I trust Anthropic" case)
+
+A user-managed override that re-labels a specific sink **contained** (e.g. a
+zero-retention Anthropic endpoint, a self-hosted Paperless on a public
+hostname). This is the Exposure-axis counterpart of promoting a collection on
+the Sensitivity axis. Out of scope for the first steps; recorded here as its
+principled home.
+
+## Consequences
+
+**Positive**
+- One coherent model that matches industry DLP / IFC vocabulary.
+- Dual-risk sources (Elasticsearch / Paperless) become expressible.
+- Honest labels: promoting a collection is clearly about *inference
+  authorization*, not *publication*.
+- The escape hatch is explicit and visible, not a silent scope.
+
+**Negative / risks**
+- The egress guardrail is security-critical and has already been through two
+  adversarial review rounds. A big-bang rewrite is high blast radius.
+- Every engine / source / sink needs labels on both axes; some are genuinely
+  ambiguous.
+
+**Therefore: rewrite incrementally, core-first, behind a test net.** The model
+is the chosen direction, but the guardrail is security-critical and has already
+survived two adversarial review rounds — so it lands in stages that each keep
+the egress test suite green:
+
+A. **Classification core** (`security/egress/classification.py`): the
+   `Sensitivity` / `Exposure` types, `classify(component, settings)`, and
+   `evaluate_run(components)` implementing the four-quadrant rule — pure, fully
+   unit-tested against the truth table above, **not yet wired in** (zero
+   behaviour change).
+B. **Wire the PEPs** to the core; re-express the existing scopes in its terms;
+   keep behaviour parity (existing egress tests stay green) while the new
+   capabilities (quadrant 4, dual-risk sources) become reachable.
+C. **UI**: surface the two axes, add **Permissive / "Unprotected"** mode, remove
+   `both`, make collection labels honest.
+D. **Per-destination trust** (the Exposure override — the "trust Anthropic"
+   case) and explicit classification for Elasticsearch / Paperless.
+
+**Status (PR #4882).** All four stages are implemented.
+
+- **A / B** — classification core (`classification.py`) + resolver
+  (`run_classification.py`) + explicit per-element labels on every engine and
+  provider.
+- **C** — the `UNPROTECTED` escape hatch (with the SSRF / cloud-metadata
+  invariant preserved *after* the hatch gate), `both` **fully retired** — removed
+  from the selector, existing rows migrated to `adaptive` by migration 0019, and
+  any residual value (env var / queued snapshot / un-migrated DB) coerced to
+  `adaptive` at read time; `EgressScope.BOTH` survives only as the internal
+  resolution result for an unclassifiable ADAPTIVE primary — a
+  non-dismissible "protection disabled" banner, and the **enforcement flip**:
+  the run-start precheck rejects a two-axis denial as defense-in-depth over the
+  scope PEPs (`UNPROTECTED` evaluates permissive; an uncomputable decision
+  **fails closed** — a denying `audit_error` — not open). This runs at the web
+  `/api/start_research` precheck **and** at the shared worker chokepoint in
+  `run_research_process`, so follow-up / chat / queue runs are covered too; only
+  CLI / programmatic callers that bypass both get the scope PEPs alone (which
+  still force local inference under `private_only` / adaptive-private). The
+  precheck classifies the **primary** engine plus the inference providers;
+  widening to the full resolved engine set is a follow-up (see residuals).
+- **D** — per-destination trust (`policy.trusted_inference_providers` /
+  `policy.trusted_search_engines`) relaxing a trusted off-machine sink to
+  contained, a trust banner, and honest collection-label copy.
+
+**Before it enforces in a shipped release it must pass a fresh adversarial
+review** — the enforcement flip is a behaviour change to a security boundary.
+
+### Known residuals (tracked in issue #4951)
+
+Four rounds of adversarial review established that the run-start audit is
+best-effort defense-in-depth, not a completeness guarantee: it re-derives each
+sink's classification from settings and can diverge from what the run actually
+connects to. The **default `adaptive` config is protected by the scope PEPs**
+(a private primary forces local inference), and retiring `both` removes the
+scope where the audit was the sole guard. These edges remain and are tracked as
+issues rather than blockers:
+
+- **Search axis — Elasticsearch `cloud_id`.** The exposure fail-up inspects
+  `hosts`, not `cloud_id`; a sensitive ES store reachable via `cloud_id` on a
+  permissive scope is classified contained. The engine's own
+  `_cloud_id_forbidden_by_scope` covers `private_only` / `strict`.
+- **Full engine set / agentic expansion.** Enforcement (audit and the scope
+  PEPs' local-inference coupling) keys on the run's *primary* engine; a
+  non-primary sensitive engine pulled in mid-run by expansion is not reflected
+  back into `require_local`.
+- **Dual-risk store on a public host.** A self-hosted store whose URL resolves
+  public classifies PUBLIC_ONLY, so the scope-level local-inference coupling
+  does not fire (the run-start audit still flags it on web paths).
+- **Name-keyed trust drift.** `trusted_inference_providers` is keyed by provider
+  name, not the vetted endpoint URL, and is honored only in the audit, not the
+  boundary PEP.
+- **`LibraryRAGService` direct construction** resolves its primary from the
+  global `search.tool`, so a direct (non-factory) construction can miss the
+  local-embeddings coupling.
+- **Query text** is out of scope by design (see above).
+
+## Open questions — resolution
+
+1. **Deferred.** User-facing axis vocabulary. Internally settled as
+   `Sensitivity{sensitive, non_sensitive}` / `Exposure{contained, exposing}`.
+   The two axes are *not* surfaced as independent UI controls yet — they are
+   expressed through the existing egress-scope selector, the per-collection
+   public flag, and the trust lists. A dedicated two-axis UI is a follow-up.
+2. **Resolved — Unprotected.** Setting value / label / `EgressScope.UNPROTECTED`
+   (the internal enforcement *mode* is separately `Mode.PERMISSIVE`).
+3. **Resolved.** Elasticsearch / Paperless default to **sensitive + contained**;
+   the URL fail-up flips exposure to exposing (quadrant 4) on a public endpoint;
+   `policy.trusted_search_engines` is the explicit override.
+4. **Resolved — no.** The aggregate `library` is always sensitive
+   (`_resolve_collection_is_public` hard-returns private for it).
+5. **Resolved — settings lists.** `policy.trusted_inference_providers` /
+   `policy.trusted_search_engines` (JSON-list settings, mirroring
+   `allowed_local_hostnames`); no dedicated table.
+
+## References
+
+- Egress package: `src/local_deep_research/security/egress/README.md`
+- SELinux modes — enforcing / permissive / disabled
+- DLP: data-classification (sensitivity labels) + destination-trust / egress
+  controls (information-flow control)

docs/deployment/reverse-proxy.md

@@ -0,0 +1,182 @@
+# Deploying behind a reverse proxy
+
+For anything beyond a single user on `localhost`, run LDR behind a reverse proxy
+(nginx, Caddy, Traefik, Nginx Proxy Manager, …). The proxy terminates TLS and
+handles **HTTPS, response compression, and static-asset caching** for you — so
+LDR itself does none of those (see
+[ADR-0005](../decisions/0005-reject-inapp-response-compression.md) on why
+compression is intentionally the proxy's job, not the app's).
+
+> **Bind LDR to loopback and expose only the proxy.** LDR's default
+> `LDR_WEB_HOST` is `0.0.0.0` (all interfaces) and it always serves **plain
+> HTTP**. Set `LDR_WEB_HOST=127.0.0.1` (or, for Docker, publish to loopback:
+> `-p 127.0.0.1:5000:5000`, or use an internal network). This matters for
+> security — see the next section.
+
+The config below is **illustrative**; directive syntax evolves, so follow the
+linked upstream docs for the current spelling. `5000` is LDR's default port
+(`LDR_WEB_PORT`); substitute yours if you changed it.
+
+## What LDR expects from the proxy
+
+LDR uses Werkzeug's
+[`ProxyFix`](https://werkzeug.palletsprojects.com/en/stable/middleware/proxy_fix/)
+with `x_for=1, x_proto=1` (and `x_host=0, x_port=0`). This makes it read the
+**right-most value** of each forwarded header — the one appended by the single
+proxy directly in front of LDR. `ProxyFix` does not count or validate hops; the
+count just has to equal the number of trusted proxies.
+
+| Forwarded header | Used by LDR? | For |
+|---|---|---|
+| `X-Forwarded-For` | yes | client IP — rate limiting, logging |
+| `X-Forwarded-Proto` | yes | http/https detection → secure cookies, HSTS, the WebSocket same-origin check |
+| `X-Forwarded-Host` | ignored | — |
+| `X-Forwarded-Port` | ignored | — |
+
+This means:
+
+- **Your proxy must set `X-Forwarded-Proto`.** Without it, a TLS-terminating
+  proxy makes LDR think the request is plain HTTP — secure cookies and HSTS are
+  withheld and the same-origin WebSocket check rejects the browser's `https`
+  origin.
+- **Set `X-Forwarded-For`** so client IPs (and rate limiting) are correct;
+  otherwise every request is attributed to the proxy.
+- **`ProxyFix` is always on and trusts these headers unconditionally.** If LDR
+  is reachable directly (not only via the proxy), a client can forge
+  `X-Forwarded-For` (to spoof its IP and evade rate limiting) or
+  `X-Forwarded-Proto: https` (to force secure cookies over plaintext). This is
+  why LDR must be bound to loopback / an internal network. See Flask's
+  [Tell Flask it is behind a proxy](https://flask.palletsprojects.com/en/stable/deploying/proxy_fix/).
+- **Exactly one proxy hop is supported.** The count is fixed at `1` in the app
+  and has no env/setting knob, so a multi-proxy chain or a CDN/Cloudflare Tunnel
+  *in addition* to your proxy is not supported without a code change (LDR would
+  read the inner proxy's address as the client IP).
+- **HSTS and HTTPS redirect:** LDR sends `Strict-Transport-Security`
+  (`max-age=31536000; includeSubDomains`, no `preload`) itself on HTTPS
+  requests, so don't add a duplicate at the proxy. Do add an HTTP→HTTPS redirect
+  at the proxy (shown below).
+
+## nginx
+
+```nginx
+# --- in the http { } context (e.g. conf.d/), shared by all servers ---
+# Map for the WebSocket upgrade; also lets Socket.IO's long-polling fallback
+# (which sends no Upgrade header) keep the connection alive.
+map $http_upgrade $connection_upgrade {
+    default upgrade;
+    ''      close;
+}
+
+# Redirect HTTP to HTTPS
+server {
+    listen 80;
+    server_name ldr.example.com;
+    return 301 https://$host$request_uri;
+}
+
+server {
+    listen 443 ssl;
+    http2 on;                       # `listen ... http2` is deprecated since nginx 1.25.1
+    server_name ldr.example.com;
+
+    # Example paths from certbot/Let's Encrypt; see https://certbot.eff.org/instructions
+    ssl_certificate     /etc/letsencrypt/live/ldr.example.com/fullchain.pem;
+    ssl_certificate_key /etc/letsencrypt/live/ldr.example.com/privkey.pem;
+
+    # nginx's default is 1 MB, which would 413 every research-library upload.
+    # Size this to your upload cap (LDR_SECURITY_UPLOAD_MAX_FILE_SIZE_MB,
+    # default 3072 MB per file). Lower both together to tighten the limit.
+    client_max_body_size 3072m;
+
+    # Compression LDR no longer does in-process (ADR-0005). For Brotli, add the
+    # ngx_brotli module (https://github.com/google/ngx_brotli). nginx ALWAYS
+    # compresses text/html regardless of gzip_types — that's fine here because
+    # LDR's CSRF token is masked per render (see ADR-0005). application/json is
+    # left out so secret-bearing API responses aren't compressed.
+    gzip on;
+    gzip_vary on;
+    gzip_proxied any;
+    gzip_types text/css text/javascript application/javascript image/svg+xml;
+
+    location / {
+        proxy_pass http://127.0.0.1:5000;
+        proxy_http_version 1.1;
+        proxy_set_header Host              $host;
+        proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
+        proxy_set_header X-Forwarded-Proto $scheme;   # required for HTTPS/cookies/HSTS
+
+        # LDR streams live progress as Server-Sent Events on these routes;
+        # don't buffer or time them out during a long research run.
+        proxy_buffering    off;
+        proxy_read_timeout 300s;
+        proxy_send_timeout 300s;
+    }
+
+    # WebSocket (live research progress) needs the HTTP/1.1 upgrade headers.
+    location /socket.io {
+        proxy_pass http://127.0.0.1:5000;
+        proxy_http_version 1.1;
+        proxy_set_header Host              $host;
+        proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
+        proxy_set_header X-Forwarded-Proto $scheme;
+        proxy_set_header Upgrade           $http_upgrade;
+        proxy_set_header Connection        $connection_upgrade;
+        proxy_buffering    off;
+        proxy_read_timeout 3600s;        # generous; keeps idle WebSockets open
+    }
+}
+```
+
+Content-hashed bundles under `/static/dist/` are sent with
+`Cache-Control: public, max-age=31536000, immutable` (the cache key is the
+content hash in the *filename*, e.g. `app.<hash>.js`), so browsers cache them
+without revalidating. Don't add `proxy_cache` to `location /`: LDR is a
+multi-user app with per-user encrypted databases, and caching authenticated
+pages there would leak one user's data to another.
+
+References:
+[proxy module](https://nginx.org/en/docs/http/ngx_http_proxy_module.html) ·
+[`client_max_body_size`](https://nginx.org/en/docs/http/ngx_http_core_module.html#client_max_body_size) ·
+[gzip module](https://nginx.org/en/docs/http/ngx_http_gzip_module.html) ·
+[WebSocket proxying](https://nginx.org/en/docs/http/websocket.html) ·
+[`http2`](https://nginx.org/en/docs/http/ngx_http_v2_module.html) ·
+[Flask-SocketIO deployment](https://flask-socketio.readthedocs.io/en/latest/deployment.html)
+
+## Caddy
+
+Caddy auto-provisions TLS, redirects HTTP→HTTPS, sets `X-Forwarded-For`/`-Proto`
+on [`reverse_proxy`](https://caddyserver.com/docs/caddyfile/directives/reverse_proxy)
+automatically, and upgrades WebSockets transparently — so the whole deployment
+is a few lines:
+
+```caddy
+ldr.example.com {
+    encode gzip               # compression LDR no longer does (ADR-0005)
+    reverse_proxy 127.0.0.1:5000
+}
+```
+
+[Automatic HTTPS](https://caddyserver.com/docs/automatic-https) requires
+`ldr.example.com` to be a real public domain whose DNS points at the host, with
+inbound ports 80 and 443 reachable for the ACME challenge and a writable data
+directory. For an internal/non-public hostname Caddy falls back to its
+locally-trusted internal CA (browsers warn unless you trust its root).
+
+## Notes
+
+- **Single backend only.** LDR doesn't support horizontal scaling — multiple
+  replicas would need Socket.IO sticky sessions and a shared message queue.
+- **Lock down registration** if you expose LDR publicly: it ships its own auth,
+  but `LDR_APP_ALLOW_REGISTRATIONS` defaults to **true** (open self-signup). Set
+  `LDR_APP_ALLOW_REGISTRATIONS=false` after creating your accounts. Proxy-level
+  basic-auth is usually unnecessary given the built-in auth.
+- **Other proxies:** Traefik (set its forwarded-headers trusted IPs),
+  [Nginx Proxy Manager](unraid.md), or a tunnel. Note **Cloudflare Tunnel adds a
+  second hop**, which conflicts with the fixed one-proxy trust model above.
+
+## Related
+
+- [ADR-0005: Reject in-app response compression](../decisions/0005-reject-inapp-response-compression.md)
+- WebSocket / Socket.IO behind a proxy, CSRF/cookie issues: [troubleshooting](../troubleshooting.md)
+- Cross-origin front-ends, allowed WebSocket/CORS origins: [env configuration](../env_configuration.md)
+- [Unraid + Nginx Proxy Manager](unraid.md)

docs/deployment/unraid.md

@@ -0,0 +1,445 @@
+# Local Deep Research on Unraid
+
+This guide covers deploying Local Deep Research (LDR) on Unraid servers.
+
+## 📋 Prerequisites
+
+- **Unraid 6.9 or higher**
+- **Docker enabled** in Unraid settings (default)
+- **Minimum 20GB storage** (more if using local LLMs)
+- **Community Applications plugin** installed (recommended)
+- **Optional: NVIDIA GPU** for local LLM acceleration
+
+## 🚀 Installation Methods
+
+### Method 1: Using Unraid Template (Recommended)
+
+This is the easiest method for Unraid users.
+
+#### Step 1: Add Template Repository
+
+1. Navigate to **Docker** tab in Unraid WebUI
+2. Click on **Docker Repositories** (at the bottom)
+3. Add this URL to **Template repositories**:
+   ```
+   https://github.com/LearningCircuit/local-deep-research
+   ```
+4. Click **Save**
+
+#### Step 2: Install Local Deep Research
+
+1. Go back to **Docker** tab
+2. Click **Add Container**
+3. In the **Template** dropdown, select **LocalDeepResearch**
+4. Review the configuration (see [Configuration](#-configuration) section below)
+5. Click **Apply**
+
+#### Step 3: (Optional) Install Companion Containers
+
+For local LLM and search capabilities, you'll also need:
+
+**Install Ollama (for local LLM):**
+1. Add Container → Search "ollama" in Community Applications
+2. Use `/mnt/user/appdata/local-deep-research/ollama` for config path
+3. Optional: For container communication, create custom network first:
+   ```bash
+   docker network create ldr-network
+   ```
+   Then add to Extra Parameters: `--network=ldr-network`
+4. Apply
+
+**Install SearXNG (for local search):**
+1. Add Container → Search "searxng" in Community Applications
+2. Use `/mnt/user/appdata/local-deep-research/searxng` for config path
+3. Optional: If using custom network from above, add to Extra Parameters: `--network=ldr-network`
+4. Apply
+
+### Method 2: Docker Compose Manager Plugin
+
+If you prefer docker-compose for multi-container setups:
+
+#### Step 1: Install Docker Compose Manager
+
+1. Go to **Apps** tab
+2. Search for "Docker Compose Manager"
+3. Click **Install**
+
+#### Step 2: Create Stack from Repository
+
+1. Navigate to **Docker** tab, scroll to **Compose** section
+2. Click **Add New Stack**
+3. Name it `local-deep-research`
+4. Set **Repository URL**:
+   ```
+   https://github.com/LearningCircuit/local-deep-research.git
+   ```
+5. Set **Compose File**:
+   ```
+   docker-compose.yml:docker-compose.unraid.yml
+   ```
+6. **Branch:** `main`
+7. Click **Save** and **Compose Up**
+
+That's it! The stack will automatically use Unraid-appropriate paths. No manual configuration needed.
+
+**For GPU Support:** Change **Compose File** to:
+```
+docker-compose.yml:docker-compose.unraid.yml:docker-compose.gpu.override.yml
+```
+
+**For Document Collections:** After initial setup, edit `docker-compose.unraid.yml` in the stack to uncomment your document paths (see [Using Local Documents](#-using-local-documents)).
+
+**Important Note:** Containers installed with Docker Compose Manager have limited GUI integration. Updates must be done via the "Update Stack" button in the Compose section, not through the regular Docker UI.
+
+### Method 3: Manual Docker Template
+
+For advanced users who want to customize the template:
+
+1. Download the template:
+   ```bash
+   wget -O /boot/config/plugins/dockerMan/templates-user/local-deep-research.xml \
+     https://raw.githubusercontent.com/LearningCircuit/local-deep-research/main/unraid-templates/local-deep-research.xml
+   ```
+
+2. Go to **Docker** tab → **Add Container**
+3. Select **LocalDeepResearch** from template dropdown
+4. Configure and **Apply**
+
+## ⚙️ Configuration
+
+### Volume Mappings
+
+All volumes should be under `/mnt/user/appdata/local-deep-research/` for best practices:
+
+| Container Path | Unraid Path (Recommended) | Purpose | Required |
+|----------------|---------------------------|---------|----------|
+| `/data` | `/mnt/user/appdata/local-deep-research/data` | User databases, research outputs, cache, logs | Yes |
+| `/scripts` | `/mnt/user/appdata/local-deep-research/scripts` | Startup scripts (for Ollama integration) | Yes |
+| `/root/.ollama` (ollama) | `/mnt/user/appdata/local-deep-research/ollama` | Downloaded LLM models (5-15GB each) | If using Ollama |
+| `/etc/searxng` (searxng) | `/mnt/user/appdata/local-deep-research/searxng` | SearXNG configuration | If using SearXNG |
+
+**Performance Tip:** If your appdata share is set to "cache-only", you can use `/mnt/cache/appdata/local-deep-research/` instead of `/mnt/user/appdata/local-deep-research/` for better performance (bypasses FUSE overhead).
+
+### Port Configuration
+
+**Default Port: 5000**
+
+If port 5000 is already in use on your Unraid server:
+
+1. In the template, change the **Host Port** (left side): `5050:5000`
+2. Do **NOT** change the Container Port (right side) or `LDR_WEB_PORT` variable
+3. Access WebUI at: `http://[unraid-ip]:5050`
+
+### Environment Variables
+
+#### Required Variables (DO NOT CHANGE)
+
+These are pre-configured in the template and **must not** be modified:
+
+| Variable | Value | Purpose |
+|----------|-------|---------|
+| `LDR_WEB_HOST` | `0.0.0.0` | Binds to all interfaces for Docker networking |
+| `LDR_WEB_PORT` | `5000` | Internal container port (change host port instead) |
+| `LDR_DATA_DIR` | `/data` | Internal data directory path |
+
+#### Service Connection Variables
+
+Configure these based on your setup:
+
+| Variable | Default | Description |
+|----------|---------|-------------|
+| `LDR_LLM_OLLAMA_URL` | `http://ollama:11434` | Use this if Ollama is on `ldr-network`<br>Use `http://[IP]:11434` for external Ollama |
+| `LDR_SEARCH_ENGINE_WEB_SEARXNG_DEFAULT_PARAMS_INSTANCE_URL` | `http://searxng:8080` | Use this if SearXNG is on `ldr-network`<br>Configure external search in WebUI otherwise |
+
+#### Optional LLM Configuration
+
+**Leave these EMPTY** unless you want to **LOCK** the configuration (prevents changes via WebUI):
+
+| Variable | Purpose |
+|----------|---------|
+| `LDR_LLM_PROVIDER` | Force LLM provider (ollama, openai, anthropic, google) |
+| `LDR_LLM_MODEL` | Force specific model name |
+| `LDR_LLM_OPENAI_API_KEY` | Lock OpenAI API key |
+| `LDR_LLM_ANTHROPIC_API_KEY` | Lock Anthropic API key |
+| `LDR_LLM_GOOGLE_API_KEY` | Lock Google API key |
+
+> **Important:** Unraid templates create all environment variables even when fields are left blank. Empty values (`""`) are treated as **not set** — they will not override or lock anything. Only non-empty values act as overrides. This means leaving a field blank in the Unraid template is safe and equivalent to not having the variable at all.
+>
+> **Security note:** Setting an API key variable to an empty string does **not** block or clear it. If a key is already stored in the database, it will still be used even when the env var is empty. To explicitly block a key, set it to any non-empty invalid value (e.g., `DISABLED`).
+
+**Recommendation:** Configure these via the WebUI Settings page instead of environment variables for easier management.
+
+### Network Configuration
+
+**Recommended: Bridge Mode with Custom Network**
+
+For multi-container setup (LDR + Ollama + SearXNG):
+- All containers should be on the same network: `ldr-network`
+- Add `--network=ldr-network` to Extra Parameters for each container
+- Containers can communicate using service names (e.g., `http://ollama:11434`)
+
+**Alternative: Individual Containers**
+
+If running LDR alone with external services:
+- Use bridge network (default)
+- Point to external services by IP: `http://192.168.1.100:11434`
+
+## 🎮 Using Local Documents
+
+To search your local documents, use the **Collections** system in the Web UI:
+
+1. Open the LDR Web UI and navigate to the **Collections** page
+2. Create a new collection (e.g., "Research Papers", "Project Docs")
+3. Upload documents directly through the browser — no volume mounts needed
+4. Select your collection as a search engine, or use **"Search All Collections"** to search across everything
+
+## 🎯 GPU Acceleration (NVIDIA)
+
+To use NVIDIA GPU with Ollama for faster local LLM inference:
+
+### Step 1: Install NVIDIA Driver Plugin
+
+1. Go to **Apps** tab
+2. Search for "Nvidia-Driver"
+3. Install the plugin
+4. Select appropriate driver version (start with latest, go older if issues occur)
+5. Reboot Unraid
+
+### Step 2: Configure Docker for NVIDIA Runtime
+
+Edit `/etc/docker/daemon.json`:
+
+```bash
+nano /etc/docker/daemon.json
+```
+
+Add this configuration:
+
+```json
+{
+  "registry-mirrors": [],
+  "insecure-registries": [],
+  "runtimes": {
+    "nvidia": {
+      "path": "nvidia-container-runtime",
+      "runtimeArgs": []
+    }
+  }
+}
+```
+
+Restart Docker:
+```bash
+/etc/rc.d/rc.docker restart
+```
+
+### Step 3: Enable GPU in Ollama Container
+
+**For Template Installation:**
+1. Edit Ollama container
+2. In **Extra Parameters**, add:
+   ```
+   --runtime=nvidia
+   ```
+3. Add environment variables:
+   - `NVIDIA_DRIVER_CAPABILITIES=all`
+   - `NVIDIA_VISIBLE_DEVICES=all`
+4. Apply
+
+**For Docker Compose:**
+Uncomment the GPU sections in the compose file shown above.
+
+### Verify GPU is Working
+
+```bash
+docker exec -it ollama_service nvidia-smi
+```
+
+You should see your GPU listed.
+
+## 💾 Backup and Restore
+
+### Using Unraid's Appdata Backup Plugin
+
+1. Install "CA Appdata Backup / Restore" from Community Applications
+2. Add to backup paths:
+   ```
+   /mnt/user/appdata/local-deep-research/
+   ```
+3. Schedule regular backups
+
+### Manual Backup
+
+```bash
+# Backup data
+tar -czf ldr_backup_$(date +%Y%m%d).tar.gz \
+  /mnt/user/appdata/local-deep-research/data
+
+# Restore data
+tar -xzf ldr_backup_20250120.tar.gz -C /
+```
+
+**What to Backup:**
+- **Critical:** `/mnt/user/appdata/local-deep-research/data` (user databases, research outputs)
+- **Optional:** `/mnt/user/appdata/local-deep-research/ollama` (models can be re-downloaded)
+- **Optional:** `/mnt/user/appdata/local-deep-research/searxng` (minimal config)
+
+## 🔍 Troubleshooting
+
+### Settings Don't Persist
+
+**Symptom:** Settings reset after container restart
+
+**Solution:**
+1. Check volume mapping is correct: `/mnt/user/appdata/local-deep-research/data:/data`
+2. Ensure `LDR_DATA_DIR=/data` is set
+3. Verify `/mnt/user/appdata/local-deep-research/data` exists and has write permissions
+4. Check Unraid logs: **Tools** → **System Log**
+
+### Container Won't Start
+
+**Check dependencies:**
+1. If using `ldr-network`, ensure all containers are on the same network
+2. Verify Ollama and SearXNG are running if referenced
+3. Check logs:
+   ```bash
+   docker logs local-deep-research
+   ```
+
+**Common issues:**
+- Port 5000 conflict → Change host port mapping
+- Network not found → Create network manually: `docker network create ldr-network`
+- Volume permission errors → Ensure paths exist and are writable
+
+### Can't Access WebUI
+
+**Verify network settings:**
+1. Check container is running: **Docker** tab
+2. Verify port mapping: Should show `5000:5000` or your custom mapping
+3. Access via: `http://[unraid-ip]:5000`
+4. Check Unraid firewall settings (if enabled)
+
+**Test container networking:**
+```bash
+docker exec -it local-deep-research wget -O- http://localhost:5000
+```
+
+### GPU Not Detected in Ollama
+
+**Verify driver installation:**
+```bash
+nvidia-smi
+```
+
+**Check container runtime:**
+```bash
+docker inspect ollama_service | grep -i runtime
+```
+
+Should show `"Runtime": "nvidia"`
+
+**Common issues:**
+- Wrong driver version → Try older driver in Nvidia-Driver plugin
+- Runtime not configured → Check `/etc/docker/daemon.json`
+- GPU already in use by VM → Stop VMs using GPU passthrough
+
+### Models Download Slowly or Fail
+
+**For Ollama:**
+1. Check disk space: Models are 5-15GB each
+2. Download manually:
+   ```bash
+   docker exec -it ollama_service ollama pull gemma3:12b
+   ```
+3. Check download progress:
+   ```bash
+   docker logs -f ollama_service
+   ```
+
+### "Update Ready" Always Shows
+
+**This is normal for Docker Compose containers.**
+
+Unraid's native Docker UI doesn't integrate with Docker Compose Manager:
+- Ignore the "Update Ready" label
+- Update via **Docker** tab → **Compose** section → **Update Stack**
+
+## 🔄 Updates
+
+### Manual Updates
+
+**For Template Installation:**
+1. Go to **Docker** tab
+2. Click container's icon → **Force Update**
+3. Container will restart automatically
+
+**For Docker Compose Installation:**
+1. Go to **Docker** tab → **Compose** section
+2. Find your `local-deep-research` stack
+3. Click **Compose Pull** → **Compose Up**
+
+### Automated Updates (Recommended)
+
+**Using Watchtower:**
+
+Watchtower automatically updates your containers when new images are available.
+
+1. Install from Community Applications: Search "Watchtower"
+2. Configure to monitor your containers
+3. Set update schedule (default: daily at midnight)
+
+Watchtower will automatically pull new images and restart containers when updates are available.
+
+**Alternative:** Unraid's built-in Docker Auto Update plugin (if enabled in Settings)
+
+## 🌐 Advanced Configuration
+
+### Reverse Proxy (Nginx Proxy Manager)
+
+To access LDR via custom domain on Unraid:
+
+1. Install "Nginx Proxy Manager" from Community Applications
+2. Add Proxy Host:
+   - **Domain Names:** `ldr.yourdomain.com`
+   - **Forward Hostname/IP:** `local-deep-research` (or IP)
+   - **Forward Port:** `5000`
+   - **Scheme:** `http`
+3. Enable SSL if desired
+
+### External LLM and Search Configuration
+
+For configuring external LLM providers or custom search engines, see the main configuration documentation:
+- [Full Configuration Reference](../CONFIGURATION.md)
+
+All WebUI settings work identically on Unraid as on other platforms.
+
+## 📚 Additional Resources
+
+- **Main Documentation:** [https://github.com/LearningCircuit/local-deep-research](https://github.com/LearningCircuit/local-deep-research)
+- **Configuration Reference:** [CONFIGURATION.md](../CONFIGURATION.md)
+- **API Documentation:** [docs/api-quickstart.md](../api-quickstart.md)
+- **FAQ:** [docs/faq.md](../faq.md)
+- **Discord Support:** [https://discord.gg/ttcqQeFcJ3](https://discord.gg/ttcqQeFcJ3)
+- **Unraid Forums:** [Support Thread](https://forums.unraid.net) *(to be created)*
+
+## ❓ Getting Help
+
+1. **Check logs first:**
+   ```bash
+   docker logs local-deep-research
+   docker logs ollama_service
+   docker logs searxng
+   ```
+
+2. **Search existing issues:** [GitHub Issues](https://github.com/LearningCircuit/local-deep-research/issues)
+
+3. **Ask for help:**
+   - [Discord Server](https://discord.gg/ttcqQeFcJ3)
+   - [GitHub Discussions](https://github.com/LearningCircuit/local-deep-research/discussions)
+   - Unraid Forums (support thread coming soon)
+
+4. **When reporting issues, include:**
+   - Unraid version
+   - Container version (from Docker tab)
+   - Relevant logs
+   - Configuration (without API keys!)

docs/developing.md

@@ -0,0 +1,302 @@
+# Developer Guide
+
+## Architecture Documentation
+
+- **[Architecture Overview](architecture/OVERVIEW.md)** - System components, research flow, and module responsibilities
+- **[Database Schema](architecture/DATABASE_SCHEMA.md)** - Database models and relationships
+- **[Extension Guide](developing/EXTENDING.md)** - How to add custom search engines, strategies, and LLM providers
+- **[Testing and CI](CI_CD_INFRASTRUCTURE.md)** - GitHub Actions workflows, pre-commit hooks, and security scanning
+
+## Development Setup
+
+### Prerequisites
+- **Python**: 3.12+
+- **Node.js**: 24.0.0+
+- **Docker**: Latest version (for production builds)
+- **PDM**: Python package manager
+- **SQLCipher**: Required for encrypted database support. See [SQLCIPHER_INSTALL.md](SQLCIPHER_INSTALL.md) for platform-specific instructions.
+
+### Initial Setup
+
+1. **Clone and Prepare Environment**
+   ```bash
+   git clone git@github.com:LearningCircuit/local-deep-research.git
+   # Or via HTTPS: https://github.com/LearningCircuit/local-deep-research.git
+   cd local-deep-research
+   ```
+
+2. **Backend Setup**
+   ```bash
+   # Create and activate virtual environment
+   python -m venv .venv
+   source .venv/bin/activate
+
+   # Install dependencies
+   pip install pdm
+   pdm install --no-self
+   ```
+
+3. **Frontend Setup**
+   ```bash
+   # Install frontend dependencies
+   npm install
+   ```
+
+4. **Enable Git Hooks**
+   ```bash
+   # Install pre-commit hooks
+   pre-commit install
+   pre-commit install-hooks
+   ```
+   We use the `pre-commit` framework to manage git hooks. This repository includes both standard checks (ruff, eslint, gitleaks) and custom local checks located in `.pre-commit-hooks/`.
+
+## Running the Application
+
+### Development Mode
+
+1. **Start the Backend**
+   ```bash
+   source .venv/bin/activate
+
+   # Option A: Using the installed entry point
+   ldr-web
+
+   # Option B: Using Python module directly
+   python -m local_deep_research.web.app
+   ```
+
+2. **Start the Frontend** (in a new terminal)
+   ```bash
+   npm run dev
+   ```
+   Access the app at `http://localhost:5173`.
+
+### Development Environment Variables
+
+For the full list of all settings and environment variables, see [CONFIGURATION.md](CONFIGURATION.md).
+
+For local development and testing, you may want to configure these environment variables:
+
+| Variable | Default | Description |
+|----------|---------|-------------|
+| `LDR_DATA_DIR` | Platform default | Override data/database storage location |
+| `LDR_BOOTSTRAP_ALLOW_UNENCRYPTED` | `false` | Allow unencrypted database (dev only!) |
+| `CI` or `TESTING` | unset | Enables testing mode (bypasses some security checks) |
+
+> **Warning**: Never set `LDR_BOOTSTRAP_ALLOW_UNENCRYPTED=true` in production. User data will be stored unencrypted.
+
+### Docker (Production-like)
+
+To build and run the entire stack in Docker:
+
+```bash
+docker build -t localdeepresearch/local-deep-research:dev .
+docker run -p 5000:5000 -e LDR_DATA_DIR=/data -v ldr_data:/data localdeepresearch/local-deep-research:dev
+```
+
+### Testing a Release Candidate (Prerelease Image)
+
+When a release is being cut, `.github/workflows/prerelease-docker.yml` publishes
+the RC under two tags on Docker Hub:
+
+- **`prerelease-vX.Y.Z-<sha>`** — immutable, pinned to one specific build. Use
+  this when you want a reproducible test target (e.g. when filing a bug).
+- **`prerelease`** — floating alias that always points at the most recent RC.
+  Use this for quick "try the next release" testing without bumping the tag
+  every cycle.
+
+The simplest way to test an RC alongside your existing stable instance is to
+add a second service to `docker-compose.yml` with a different port and isolated
+volumes, so a broken migration in the RC can't damage your production DB:
+
+```yaml
+  local-deep-research-pre:
+    image: localdeepresearch/local-deep-research:prerelease
+    container_name: local-deep-research-pre
+    networks:
+      - ldr-network
+    extra_hosts:
+      - "host.docker.internal:host-gateway"
+    ports:
+      - "5001:5000"                # production stays on 5000
+    environment:
+      - LDR_WEB_HOST=0.0.0.0
+      - LDR_WEB_PORT=5000
+      - LDR_DATA_DIR=/data
+      - LDR_LLM_OLLAMA_URL=http://ollama:11434
+      - LDR_SEARCH_ENGINE_WEB_SEARXNG_DEFAULT_PARAMS_INSTANCE_URL=http://searxng:8080
+    volumes:
+      - ldr_data_pre:/data        # ← separate from production ldr_data
+      - ldr_scripts_pre:/scripts
+    restart: unless-stopped
+
+volumes:
+  ldr_data_pre:
+  ldr_scripts_pre:
+```
+
+> Copy the `ulimits`, `security_opt`, `cap_drop`, and `cap_add` blocks from the
+> main `local-deep-research` service for the same hardening — they are required
+> for correct startup, not optional.
+
+Then pull and start just the prerelease service:
+
+```bash
+docker compose pull local-deep-research-pre
+docker compose up -d local-deep-research-pre
+```
+
+The UI will be at `http://localhost:5001`. To upgrade to the next RC after it
+ships, re-run `docker compose pull local-deep-research-pre && docker compose up -d local-deep-research-pre`.
+
+## Building
+
+### Building a Package
+
+To build a wheel and source distribution, simply run `pdm build`.
+
+### Building Frontend Assets
+
+If you're developing from source and want to use the Web UI, you need to build the frontend assets:
+
+```bash
+npm install
+npm run build
+```
+
+This builds the Vite frontend into `src/local_deep_research/web/static/dist/`.
+
+> **Note:** pip users don't need this step - pre-built assets are included in the PyPI package.
+
+### Dependency Lockfile Management
+
+This project uses **PDM** with `pdm.lock` to pin exact dependency versions. If you see this warning during Docker builds:
+
+```
+WARNING: Lockfile hash doesn't match pyproject.toml, packages may be outdated
+```
+
+It means `pyproject.toml` has changed but `pdm.lock` hasn't been regenerated.
+
+**How to fix:**
+```bash
+pdm lock
+```
+
+Always commit `pdm.lock` alongside `pyproject.toml` changes to ensure reproducible builds.
+
+## Testing
+
+### Backend Tests (Pytest)
+
+We support two modes of backend testing:
+
+#### 1. Isolated Testing (No Server Required)
+This is the default and recommended way to run unit and integration tests. It uses Flask's `test_client` to mock the server.
+
+**How to run:**
+```bash
+source .venv/bin/activate
+# Run all isolated tests
+pytest tests/
+
+# Run specific API or Auth tests
+pytest tests/api_tests/
+pytest tests/auth_tests/
+```
+
+#### 2. Live System Testing (Requires Running Server)
+These tests make real HTTP requests to a running application instance.
+
+**Prerequisites:**
+1. Start the backend server in one terminal: `ldr-web` (or via Docker).
+2. Run the live test scripts in another terminal.
+
+**How to run:**
+```bash
+python tests/ui_tests/test_simple_research_api.py
+```
+
+### Frontend & E2E Tests (Puppeteer)
+We use `Puppeteer` for UI and End-to-End testing.
+
+**Prerequisites:**
+1. Navigate to the tests directory: `cd tests`
+2. Install test dependencies: `npm install`
+
+**How to run:**
+1. **Ensure the application is running** (locally on port 5000 or via Docker).
+2. Run the test suite:
+   ```bash
+   # Run all UI tests
+   node tests/ui_tests/run_all_ui_tests.js
+
+   # Run specific test
+   node tests/ui_tests/test_simple_auth.js
+   ```
+
+## Database Management
+
+### Backup & Restore
+Before testing changes, you may wish to backup your Local Deep Research data volume.
+
+**Create a Backup**
+```bash
+docker run --rm \
+  -v ldr_data:/from \
+  -v ldr_data-backup:/to \
+  debian:latest \
+  bash -c "cd /from ; tar -cf - . | (cd /to ; tar -xpf -)"
+```
+
+**Restore from Backup**
+
+*Warning: This overwrites existing data*
+```bash
+docker run --rm \
+  -v ldr_data:/target \
+  -v ldr_data-backup:/source \
+  debian:latest \
+  bash -c "rm -rf /target/* /target/.[!.]* ; \
+           cd /source ; tar -cf - . | (cd /target ; tar -xpf -)"
+```
+
+## Troubleshooting
+
+### SQLCipher Issues
+
+See [SQLCIPHER_INSTALL.md](SQLCIPHER_INSTALL.md#troubleshooting) for SQLCipher-related errors.
+
+### Permission Denied on Docker Volume
+
+**Error:** `PermissionError: [Errno 13] Permission denied: '/app/.config/...'`
+
+**Solution:** The volume may have been created with different ownership. Reset it:
+```bash
+docker volume rm ldr_data
+# Re-run the container to create a fresh volume
+```
+
+### Session Lost After Server Restart
+
+**Cause:** The application's secret key is used to sign session cookies.
+
+**Solution:** The secret key is automatically generated on first run and persisted to a `.secret_key` file in your data directory (controlled by `LDR_DATA_DIR`). Sessions are only lost if this file is deleted. If using Docker, ensure your data volume (`ldr_data:/data`) is persistent.
+
+### NoSettingsContextError in Background Threads
+
+**Error:** `NoSettingsContextError: No settings context available`
+
+**Cause:** Background threads don't have access to the Flask request context.
+
+**Solution:** This is handled automatically by the application. If you encounter this during development, ensure your LLM settings are configured via the web UI settings page.
+
+### PDM Lockfile Out of Sync
+
+**Error:** `Lockfile hash doesn't match pyproject.toml`
+
+**Solution:**
+```bash
+pdm lock
+git add pdm.lock
+```

docs/developing/EXTENDING.md

@@ -0,0 +1,680 @@
+# Extension Guide
+
+This guide explains how to extend Local Deep Research with custom components.
+
+## Table of Contents
+
+- [Adding Custom Search Engines](#adding-custom-search-engines)
+- [Adding Custom Search Strategies](#adding-custom-search-strategies)
+- [Using LangChain Retrievers](#using-langchain-retrievers)
+- [Adding Custom LLM Providers](#adding-custom-llm-providers)
+- [Registering Custom LLMs](#registering-custom-llms)
+
+---
+
+## Adding Custom Search Engines
+
+Search engines are responsible for fetching results from external sources. All engines extend `BaseSearchEngine`.
+
+### Basic Search Engine
+
+Create a new file in `src/local_deep_research/web_search_engines/engines/`:
+
+```python
+# search_engine_custom.py
+from typing import Any, Dict, List, Optional
+
+from langchain_core.language_models import BaseLLM
+
+from ...security.secure_logging import logger
+
+from ..search_engine_base import BaseSearchEngine
+
+
+class CustomSearchEngine(BaseSearchEngine):
+    """Custom search engine implementation."""
+
+    # Classification flags - set appropriately for your engine
+    is_public = True       # Searches public internet
+    is_generic = False     # Specialized (vs general web search)
+    is_scientific = False  # Academic/scientific content
+    is_local = False       # Local document search
+    is_news = False        # News content
+    is_code = False        # Code repositories
+    is_lexical = False                  # Uses keyword/lexical search (informational)
+    needs_llm_relevance_filter = False  # Set True to auto-enable LLM relevance filtering
+
+    def __init__(
+        self,
+        max_results: int = 10,
+        credential: Optional[str] = None,
+        llm: Optional[BaseLLM] = None,
+        max_filtered_results: Optional[int] = None,
+        **kwargs,
+    ):
+        """
+        Initialize the search engine.
+
+        Args:
+            max_results: Maximum number of results to return
+            credential: API credential for the service (if required)
+            llm: Language model for relevance filtering
+            max_filtered_results: Max results after filtering
+            **kwargs: Additional parameters
+        """
+        super().__init__(
+            llm=llm,
+            max_filtered_results=max_filtered_results,
+            max_results=max_results,
+        )
+        self.credential = credential
+
+    def _get_previews(self, query: str) -> List[Dict[str, Any]]:
+        """
+        Get preview results (first phase of two-phase retrieval).
+
+        Args:
+            query: Search query
+
+        Returns:
+            List of preview dictionaries with keys:
+            - id: Unique identifier
+            - title: Result title
+            - snippet: Brief description/summary
+            - link: URL to the content
+            - source: Source name (e.g., "CustomEngine")
+        """
+        logger.info(f"Searching custom engine for: {query}")
+
+        # Apply rate limiting before request
+        self._last_wait_time = self.rate_tracker.apply_rate_limit(self.engine_type)
+
+        # Your search implementation here
+        results = self._call_api(query)
+
+        previews = []
+        for item in results:
+            previews.append({
+                "id": item["id"],
+                "title": item["title"],
+                "snippet": item["description"],
+                "link": item["url"],
+                "source": "CustomEngine",
+            })
+
+        return previews
+
+    def _get_full_content(
+        self, relevant_items: List[Dict[str, Any]]
+    ) -> List[Dict[str, Any]]:
+        """
+        Get full content for relevant items (second phase).
+
+        Args:
+            relevant_items: Items that passed relevance filtering
+
+        Returns:
+            Items enriched with full content
+        """
+        results = []
+        for item in relevant_items:
+            # Apply rate limiting
+            self._last_wait_time = self.rate_tracker.apply_rate_limit(self.engine_type)
+
+            # Fetch full content
+            full_content = self._fetch_content(item["link"])
+
+            result = item.copy()
+            result["content"] = full_content
+            result["full_content"] = full_content
+            results.append(result)
+
+        return results
+
+    def _call_api(self, query: str) -> List[Dict]:
+        """Your API implementation."""
+        # Implement your search logic here
+        pass
+
+    def _fetch_content(self, url: str) -> str:
+        """Fetch full content from URL."""
+        # Implement content fetching
+        pass
+```
+
+### Registering the Engine
+
+**Option 1: Register in engine_registry.py (Required)**
+
+Add the engine to `src/local_deep_research/web_search_engines/engine_registry.py` so the system knows how to load it. The registry maps engine names to their Python module and class:
+
+```python
+# In engine_registry.py — ENGINE_REGISTRY dict
+"custom_engine": EngineEntry(
+    module_path=".engines.search_engine_custom",
+    class_name="CustomSearchEngine",
+),
+```
+
+Module paths must be relative (starting with `.`) and listed in the security whitelist (`ALLOWED_MODULE_PATHS` in `module_whitelist.py`).
+
+**Option 1b: Configure user-facing settings (Optional)**
+
+After registering in the engine registry, you can expose user-configurable settings via the settings database:
+
+```python
+# Key: search.engine.web.custom_engine
+config = {
+    "requires_api_key": True,
+    "requires_llm": False,
+    "description": "Custom search engine for specific use case",
+    "strengths": ["Feature 1", "Feature 2"],
+    "weaknesses": ["Limitation 1"],
+    "reliability": 0.8,
+    "default_params": {
+        "max_results": 10
+    }
+}
+```
+
+**Option 2: Modify Factory (For Core Engines)**
+
+Add to `search_engine_factory.py`:
+
+```python
+def create_search_engine(engine_name: str, ...) -> BaseSearchEngine:
+    # ... existing code ...
+
+    if engine_name.lower() == "custom_engine":
+        from .engines.search_engine_custom import CustomSearchEngine
+        return CustomSearchEngine(
+            max_results=max_results,
+            api_key=api_key,
+            llm=llm,
+            **kwargs
+        )
+```
+
+### Search Engine Best Practices
+
+1. **Always apply rate limiting** before API calls:
+   ```python
+   self._last_wait_time = self.rate_tracker.apply_rate_limit(self.engine_type)
+   ```
+
+2. **Set classification flags** accurately - they affect engine selection. For keyword-based engines without ML ranking, set `is_lexical = True` and `needs_llm_relevance_filter = True` — the factory will auto-enable LLM relevance filtering
+
+3. **Handle errors gracefully** - return empty list on failure, don't crash
+
+4. **Use logging** for debugging — engine/provider dirs must import the
+   diagnose-gated `security.secure_logging` wrapper, never raw loguru
+   (a pre-commit hook enforces this), and error messages must be scrubbed
+   before logging because they are production-visible:
+   ```python
+   from ...security.secure_logging import logger
+
+   logger.info(f"Searching for: {query}")
+
+   # In except blocks: never interpolate the raw exception — scrub it first.
+   # (Engines inherit self._scrub_error() from BaseSearchEngine.)
+   except Exception as e:
+       safe_msg = self._scrub_error(e)
+       logger.exception(f"API error: {safe_msg}")
+   ```
+   The hook is syntactic, not data-flow analysis: reviewers should still
+   reject laundering raw exception text through temporary variables such as
+   `msg = str(e); logger.error(msg)` unless the variable came from a scrubber.
+
+5. **Support snippet-only mode** by checking the config:
+   ```python
+   from ...config import search_config
+   if search_config.SEARCH_SNIPPETS_ONLY:
+       return relevant_items  # Skip full content
+   ```
+
+---
+
+## Adding Custom Search Strategies
+
+Strategies define how research is conducted - question generation, iteration, and synthesis.
+
+### Basic Strategy
+
+Create a new file in `src/local_deep_research/advanced_search_system/strategies/`:
+
+```python
+# my_custom_strategy.py
+from typing import Dict, List, Optional
+from loguru import logger
+
+from .base_strategy import BaseSearchStrategy
+
+
+class MyCustomStrategy(BaseSearchStrategy):
+    """Custom search strategy implementation."""
+
+    def __init__(
+        self,
+        search=None,
+        model=None,
+        all_links_of_system=None,
+        settings_snapshot=None,
+        max_iterations: int = 3,
+        **kwargs,
+    ):
+        """
+        Initialize the strategy.
+
+        Args:
+            search: Search engine instance
+            model: LLM for question generation and synthesis
+            all_links_of_system: Shared list for discovered links
+            settings_snapshot: Configuration snapshot
+            max_iterations: Maximum research iterations
+            **kwargs: Additional parameters
+        """
+        super().__init__(
+            all_links_of_system=all_links_of_system,
+            settings_snapshot=settings_snapshot,
+        )
+        self.search = search
+        self.model = model
+        self.max_iterations = max_iterations
+
+    def analyze_topic(self, query: str) -> Dict:
+        """
+        Execute the research strategy.
+
+        Args:
+            query: Research query
+
+        Returns:
+            Dict with:
+            - findings: List of research findings
+            - iterations: Number of iterations completed
+            - questions: Dict of questions by iteration
+            - formatted_findings: Formatted output string
+            - current_knowledge: Accumulated knowledge dict
+            - error: Optional error message
+        """
+        logger.info(f"Starting custom strategy for: {query}")
+
+        findings = []
+        current_knowledge = {}
+
+        try:
+            for iteration in range(1, self.max_iterations + 1):
+                # Update progress
+                self._update_progress(
+                    f"Iteration {iteration}/{self.max_iterations}",
+                    progress_percent=int(iteration / self.max_iterations * 100),
+                    metadata={"iteration": iteration}
+                )
+
+                # Generate questions for this iteration
+                questions = self._generate_questions(query, current_knowledge)
+                self.questions_by_iteration[iteration] = questions
+
+                # Search for each question
+                for question in questions:
+                    results = self._search(question)
+                    findings.extend(results)
+
+                    # Track links
+                    for result in results:
+                        if result.get("link"):
+                            self.all_links_of_system.append(result["link"])
+
+                # Synthesize findings
+                current_knowledge = self._synthesize(findings)
+
+                # Check if we should stop early
+                if self._should_stop(current_knowledge):
+                    logger.info(f"Early stopping at iteration {iteration}")
+                    break
+
+            # Format final output
+            formatted = self._format_findings(findings, current_knowledge)
+
+            return {
+                "findings": findings,
+                "iterations": iteration,
+                "questions": self.questions_by_iteration,
+                "formatted_findings": formatted,
+                "current_knowledge": current_knowledge,
+            }
+
+        except Exception as e:
+            logger.error(f"Strategy error: {e}")
+            return {
+                "findings": findings,
+                "iterations": 0,
+                "questions": self.questions_by_iteration,
+                "formatted_findings": "",
+                "current_knowledge": current_knowledge,
+                "error": str(e),
+            }
+
+    def _generate_questions(self, query: str, knowledge: Dict) -> List[str]:
+        """Generate research questions using the LLM."""
+        prompt = f"""Given the query: {query}
+        And current knowledge: {knowledge}
+        Generate 3 specific research questions."""
+
+        response = self.model.invoke(prompt)
+        # Parse response into questions
+        return self._parse_questions(response.content)
+
+    def _search(self, question: str) -> List[Dict]:
+        """Execute search for a question."""
+        return self.search.run(question)
+
+    def _synthesize(self, findings: List[Dict]) -> Dict:
+        """Synthesize findings into knowledge."""
+        # Implement synthesis logic
+        return {"summary": "...", "key_points": [...]}
+
+    def _should_stop(self, knowledge: Dict) -> bool:
+        """Check if research should stop early."""
+        # Implement stopping criteria
+        return False
+
+    def _format_findings(self, findings: List[Dict], knowledge: Dict) -> str:
+        """Format findings as output string."""
+        # Implement formatting
+        return "Formatted research results..."
+
+    def _parse_questions(self, content: str) -> List[str]:
+        """Parse LLM response into question list."""
+        # Implement parsing
+        return content.strip().split("\n")
+```
+
+### Registering the Strategy
+
+Add to `search_system_factory.py`:
+
+```python
+def create_strategy(strategy_name: str, ...) -> BaseSearchStrategy:
+    strategy_name_lower = strategy_name.lower()
+
+    # ... existing strategies ...
+
+    elif strategy_name_lower in ["my-custom", "mycustom", "custom"]:
+        from .advanced_search_system.strategies.my_custom_strategy import (
+            MyCustomStrategy,
+        )
+        return MyCustomStrategy(
+            search=search,
+            model=model,
+            all_links_of_system=all_links_of_system,
+            settings_snapshot=settings_snapshot,
+            **kwargs
+        )
+```
+
+### Strategy Best Practices
+
+1. **Use progress callbacks** to update the UI:
+   ```python
+   self._update_progress("Searching...", progress_percent=50)
+   ```
+
+2. **Track all discovered links** in `self.all_links_of_system`
+
+3. **Store questions by iteration** in `self.questions_by_iteration`
+
+4. **Access settings** via the snapshot:
+   ```python
+   max_results = self.get_setting("search.max_results", default=10)
+   ```
+
+5. **Handle errors gracefully** - return partial results with error message
+
+---
+
+## Using LangChain Retrievers
+
+The easiest way to add custom search is through LangChain retrievers.
+
+### Registering a Retriever
+
+```python
+from langchain_community.vectorstores import FAISS
+from langchain_openai import OpenAIEmbeddings
+from local_deep_research.web_search_engines.retriever_registry import retriever_registry
+
+# Create your retriever
+embeddings = OpenAIEmbeddings()
+vectorstore = FAISS.from_documents(documents, embeddings)
+retriever = vectorstore.as_retriever(search_kwargs={"k": 10})
+
+# Register globally
+retriever_registry.register("my_documents", retriever)
+
+# Now use in research
+from local_deep_research.api import quick_summary
+
+result = quick_summary(
+    query="What does the documentation say about X?",
+    search_tool="my_documents",  # Use registered retriever
+    programmatic_mode=True
+)
+```
+
+### Passing Retrievers Directly
+
+```python
+from local_deep_research.api import quick_summary
+
+# Create retriever
+retriever = my_vectorstore.as_retriever()
+
+# Pass directly to API
+result = quick_summary(
+    query="Search my documents",
+    retrievers={"private_docs": retriever},
+    search_tool="private_docs",
+    programmatic_mode=True
+)
+```
+
+### Registry Methods
+
+```python
+from local_deep_research.web_search_engines.retriever_registry import retriever_registry
+
+# Register
+retriever_registry.register("name", retriever)
+retriever_registry.register_multiple({"a": ret1, "b": ret2})
+
+# Query
+retriever_registry.get("name")
+retriever_registry.is_registered("name")
+retriever_registry.list_registered()
+
+# Remove
+retriever_registry.unregister("name")
+retriever_registry.clear()
+```
+
+---
+
+## Adding Custom LLM Providers
+
+LLM providers wrap language model APIs for use in LDR.
+
+### Basic Provider
+
+Create in `src/local_deep_research/llm/providers/implementations/`:
+
+```python
+# my_provider.py
+from typing import Dict, Optional
+
+from langchain_core.language_models import BaseChatModel
+from langchain_openai import ChatOpenAI
+
+from ..openai_base import OpenAICompatibleProvider
+
+
+class MyProvider(OpenAICompatibleProvider):
+    """Custom LLM provider."""
+
+    provider_name = "My Provider"
+    api_key_setting = "llm.my_provider.api_key"
+    url_setting = "llm.my_provider.url"
+    default_base_url = "https://api.myprovider.com/v1"
+    default_model = "my-model-v1"
+    # Optional: set to True if missing key should fall back to a placeholder
+    # rather than raising ValueError.
+    api_key_optional = False
+
+    @classmethod
+    def create_llm(
+        cls,
+        model_name: Optional[str] = None,
+        temperature: float = 0.7,
+        settings_snapshot: Optional[Dict] = None,
+        **kwargs
+    ) -> BaseChatModel:
+        """
+        Create LLM instance.
+
+        Args:
+            model_name: Model to use
+            temperature: Sampling temperature
+            settings_snapshot: Configuration
+            **kwargs: Additional parameters
+
+        Returns:
+            LangChain chat model instance
+        """
+        from ....config.thread_settings import get_setting_from_snapshot
+
+        # Resolve API key via the base helper. Raises ValueError when
+        # required and missing, returns the unified placeholder when
+        # api_key_optional=True and the key is unset.
+        api_key = cls.resolve_api_key_or_placeholder(settings_snapshot)
+
+        # Get base URL
+        base_url = get_setting_from_snapshot(
+            cls.url_setting,
+            cls.default_base_url,
+            settings_snapshot=settings_snapshot,
+        )
+
+        return ChatOpenAI(
+            model=model_name or cls.default_model,
+            temperature=temperature,
+            api_key=api_key,
+            base_url=base_url,
+            **kwargs
+        )
+
+    @classmethod
+    def list_models(cls, settings_snapshot: Optional[Dict] = None) -> list[str]:
+        """List available models."""
+        return ["my-model-v1", "my-model-v2", "my-model-large"]
+```
+
+### Register in Auto-Discovery
+
+Drop the provider class file into
+`src/local_deep_research/llm/providers/implementations/`. Auto-discovery
+will scan that directory at import time and register every class whose
+name ends with `Provider`, subclasses `BaseLLMProvider`, and has
+`provider_name` set to a real value (i.e., overridden away from the
+``"unknown"`` default). Setting `provider_name = "unknown"` — or leaving
+it unset on the class — will cause the class to be **silently filtered
+out** of auto-discovery, which is a common gotcha when copying an
+existing provider as a template.
+
+Optional cloud-metadata registration in `auto_discovery.py`:
+
+```python
+PROVIDER_METADATA = {
+    # ... existing providers ...
+    "my_provider": ProviderMetadata(
+        provider_id="my_provider",
+        provider_name="My Provider",
+        company_name="My Company",
+        region="US",
+        country="United States",
+        data_location="US",
+        gdpr_compliant=False,
+        is_cloud=True,
+    ),
+}
+```
+
+---
+
+## Registering Custom LLMs
+
+For programmatic use, register LLMs directly:
+
+```python
+from langchain_openai import ChatOpenAI
+from local_deep_research.llm.llm_registry import register_llm, get_llm_from_registry
+
+# Create custom LLM
+custom_llm = ChatOpenAI(
+    model="gpt-4",
+    temperature=0.5,
+    api_key="...",
+)
+
+# Register it
+register_llm("my_gpt4", custom_llm)
+
+# Use in research
+from local_deep_research.api import quick_summary
+
+result = quick_summary(
+    query="Research topic",
+    llms={"my_gpt4": custom_llm},  # Or use registered name
+    provider_name="my_gpt4",
+    programmatic_mode=True
+)
+```
+
+### Factory Functions
+
+You can also register factory functions:
+
+```python
+def create_my_llm(temperature=0.7):
+    return ChatOpenAI(model="gpt-4", temperature=temperature)
+
+register_llm("my_factory", create_my_llm)
+
+# Will be called when needed
+llm = get_llm_from_registry("my_factory")
+```
+
+### Registry caveat
+
+The built-in providers (ollama, openai, anthropic, ...) live in the same
+registry, auto-registered at import time. `clear_llm_registry()` removes
+them too, and `get_llm()` has no other construction path — every provider
+will raise "was not registered by auto-discovery" until you restore them:
+
+```python
+from local_deep_research.llm.providers import discover_providers
+
+discover_providers(force_refresh=True)
+```
+
+Prefer `unregister_llm("<your name>")` over `clear_llm_registry()` to
+remove only your own registrations.
+
+---
+
+## See Also
+
+- [Architecture Overview](../architecture/OVERVIEW.md) - System architecture
+- [Database Schema](../architecture/DATABASE_SCHEMA.md) - Data models
+- [Full Configuration Reference](../CONFIGURATION.md) - All settings and environment variables
+- [Troubleshooting](../troubleshooting.md) - Common issues
+- [API Quickstart](../api-quickstart.md) - Using the API

docs/developing/credential-scrubbing.md

@@ -0,0 +1,61 @@
+# Credential scrubbing in error/log text
+
+`local_deep_research.security.log_sanitizer` scrubs credentials out of
+exception/error strings **before** they reach a client (HTTP/SSE/JSON
+responses) or the logs. It is a runtime, defense-in-depth *sanitizer* — not a
+git/CI secret *scanner* (use [gitleaks](https://github.com/gitleaks/gitleaks)
+for that, which this repo already runs in pre-commit/CI).
+
+## The two layers
+
+1. **`sanitize_error_message(text)`** — a regex first-pass over `_CREDENTIAL_PATTERNS`
+   for *common credential shapes* (API-key query params, `Authorization:`/
+   `x-api-key:` headers, `user:pass@host`, and well-known token prefixes:
+   `sk-`/`pk-`, Google `AIza`/`ya29.`, GitHub `ghp_`/`github_pat_`, AWS
+   `AKIA`/`ASIA`, Slack `xox*-`, JWTs). `sanitize_error_for_client()` composes
+   it with control-char stripping + length capping.
+2. **`redact_secrets(text, *known_literals)`** — the **backstop**. When you
+   hold the actual secret value (e.g. the configured API key), pass it here so
+   it is scrubbed regardless of shape. This is the real guarantee for
+   *known* secrets; the regex layer is best-effort for *unknown* ones.
+
+Always pair them ("dual-scrub") on any path that surfaces secret-adjacent
+text — via **`scrub_error(error, *known_literals)`** (`security.log_sanitizer`),
+which composes both passes plus the defensive guards (a raising `__str__`
+or a non-string secret cannot crash the `except` handler). Engine
+subclasses use `self._scrub_error(error)`, which resolves the engine's
+`_secret_attrs` and delegates to the same helper. Don't hand-inline the
+two calls: per-site copies are how scrub passes drift apart.
+
+## Design constraints (why it is curated, not exhaustive)
+
+This runs on **arbitrary, possibly attacker-influenced** strings at runtime, so:
+
+- **No ReDoS.** Every pattern must scale linearly — prefer prefix-anchored,
+  single-quantifier regexes; avoid nested quantifiers / ambiguous alternations.
+  Spot-check new patterns on a 200k-char adversarial input.
+- **Over-redaction is the safe failure; under-redaction (a leak) is not** —
+  but gratuitous over-redaction harms log readability, so patterns are
+  anchored/length-floored to avoid eating prose.
+- Keep the set **small and high-signal**. Matching gitleaks' full 100+ rules
+  at runtime is overkill and raises false positives; the `redact_secrets`
+  backstop covers the long tail for known secrets.
+
+## Keeping the patterns current
+
+The prefix regexes mirror the **canonical, actively-maintained gitleaks
+ruleset** — gitleaks' own upstream `config/gitleaks.toml`
+(<https://github.com/gitleaks/gitleaks>), **not** this repo's root
+`.gitleaks.toml` (which only configures the gitleaks *scan* run in
+pre-commit/CI). To refresh when a provider introduces a new token format:
+
+1. Find the rule in gitleaks' upstream `config/gitleaks.toml` (or detect-secrets).
+2. Adapt the regex to a prefix-anchored, ReDoS-safe form here; redact to
+   `[REDACTED_KEY]`.
+3. Add a positive (redacts) **and** a negative (prose not over-redacted) case
+   to `tests/security/test_log_sanitizer.py`.
+4. Run `pytest tests/security/` and a 200k-char ReDoS spot-check.
+
+A periodic (e.g. quarterly) glance at gitleaks' changelog for new
+widely-used token prefixes is enough — this layer only needs the formats that
+plausibly appear in *this app's* error/log text.

docs/developing/resource-cleanup.md

@@ -0,0 +1,971 @@
+# Resource cleanup in LDR
+
+This document captures how LDR manages process-level resources (DB
+connections, HTTP clients, file descriptors, threads) and the reasoning
+trail behind the current model. It exists because file-descriptor
+exhaustion has been a recurring class of bug in LDR, and the *journey*
+of fixing it — what's been tried, what worked, what was ruled out — is
+not reconstructable from `git log` alone.
+
+If you're contributing code that holds a network connection, a database
+session, an LLM client, or a thread, read this before adding `__del__`,
+`weakref.finalize`, or a context manager.
+
+---
+
+## Current model
+
+### Database connections
+
+- **One shared per-user `QueuePool`.** No per-thread engines. Pool
+  sizing: `pool_size=20`, `max_overflow=40`, with periodic `dispose()`
+  every 30 minutes.
+- **SQLCipher is decrypted once per connection-open.** `PRAGMA key`
+  takes ~0.2 ms; pool reuse keeps that off the hot path.
+- Engines are created at login, closed at logout (or process exit via
+  the registered `atexit` shutdown).
+- Background threads (research workers, metric writers, news scheduler
+  jobs) use the same per-user pool — they no longer maintain a separate
+  thread-engine system.
+
+See [ADR-0004](../decisions/0004-nullpool-for-sqlcipher.md) for the
+QueuePool-vs-NullPool decision and PR #3441 for the per-thread-engine
+removal.
+
+### LLM wrappers
+
+LDR wraps every LLM in `ProcessingLLMWrapper` (and optionally
+`RateLimitedLLMWrapper`) so that callers see a uniform interface and
+the project owns the close path:
+
+```
+caller -> ProcessingLLMWrapper.close()
+       -> _close_base_llm(base_llm) in utilities/llm_utils.py
+       -> for ChatOllama:
+            sync httpx client (ollama.Client._client) .close()
+            async httpx client (ollama.AsyncClient._client) .aclose()
+       -> for ChatOpenAI / ChatAnthropic:
+            no close (those use @lru_cache'd shared httpx clients)
+```
+
+Key invariants:
+
+- `ChatOllama` is the *only* provider where `_close_base_llm()` actually
+  closes anything. ChatOpenAI and ChatAnthropic share LRU-cached httpx
+  clients across instances; closing them would break other live LLMs.
+- Both `_client` (sync) **and** `_async_client` (async) are released —
+  the async side is exercised by every `ainvoke()` call (langgraph
+  agents, modular strategies). Closing only the sync side leaks the
+  async transport per call (root cause of #3816).
+- The function is idempotent via an `_ldr_closed` sentinel on the inner
+  httpx clients.
+- The async close uses `asyncio.run(client.aclose())` only when no
+  event loop is currently running. When called from inside async code
+  it skips and leaves the close to the loop's owner.
+
+### Search engines
+
+- `BaseSearchEngine.close()` is the single entry point and **cascades**
+  into `_preview_filters` and `_content_filters`. That cascade is what
+  releases per-engine LLMs (e.g., `JournalReputationFilter.model`),
+  SearXNG sessions, and other filter-held resources.
+- Search-engine cleanup happens at the per-research finally block in
+  `web/services/research_service.py:run_research_process()` and at the
+  programmatic API entry points in `api/research_functions.py`.
+- The `_owns_llm` flag pattern (introduced in #2712) tracks whether a
+  filter or engine constructed its own LLM (and thus owns it) versus
+  borrowed one from a caller (and must not close it).
+
+### Thread lifecycle
+
+- `@thread_cleanup` (decorator on `run_research_process` and similar
+  workers) ensures thread-local DB sessions are released even on
+  abnormal exits.
+- `cleanup_current_thread()` is called from Flask teardown, the queue
+  processor, the auth flow, and the RAG routes — six tier-1 paths in
+  total.
+- Background threads are daemon threads; the process exit handles any
+  thread that did not clean up gracefully.
+
+### Conventions
+
+- **Use `safe_close(resource, "human name")`** from
+  `utilities/resource_utils.py` for every cleanup. Never bare `.close()`
+  in a `finally` (it can mask the original exception).
+- **Prefer `try/finally` over `__del__`**. Python doesn't guarantee
+  finalization order at interpreter exit; `__del__` interacts subtly
+  with reference cycles and `weakref`.
+- **Track ownership explicitly with `_owns_llm` (or analogous flag)**
+  when a class accepts an injected resource that may or may not be its
+  own.
+- **News fragments (`changelog.d/<id>.bugfix.md`) are required for any
+  user-visible cleanup behavior change** — see `changelog.d/README.md`.
+
+---
+
+## How to close X correctly
+
+| You're holding | Do this |
+| --- | --- |
+| A `ChatOllama` (raw or wrapped) | Call `wrapper.close()` in a `finally`, or pass to `safe_close(wrapper, "...")`. The wrapper chain handles both sync and async httpx clients. |
+| A search engine you constructed | `safe_close(engine, "...")` in `finally`. The engine's `close()` cascades into preview/content filters. |
+| A holder class with an LLM | Add a `close()` method, gate the LLM close on `self._owns_llm`, document who calls it. Don't add `__del__`. |
+| A long-lived service holder (news scheduler, etc.) | Wrap construction in `try/finally` at the cycle boundary. Don't store the LLM if you can recreate it cheaply. |
+| A DB session | Use `with get_user_db_session(username) as session:`. Don't bypass via `get_settings_manager(username=...)` without `owns_session=False` (see #3023). |
+| An asyncio event loop | Use the existing one. If you genuinely need a new one (background thread fallback), call `loop.close()` in a `finally` — see `news_strategy.py` for the reference pattern (post-#3018). |
+
+---
+
+## Anti-patterns
+
+These look reasonable but break specific things in this codebase:
+
+- **Adding `__del__` to a class with `close()`.** At interpreter exit
+  the `logger`, `httpx`, and event-loop modules may already be torn
+  down. `__del__` can run after them and raise. Use explicit close in
+  a `finally` instead.
+- **Closing a shared httpx client.** ChatOpenAI / ChatAnthropic share
+  one httpx pool across instances via `@lru_cache`. Closing it kills
+  every other live LLM in the same process. The Ollama check in
+  `_close_base_llm` exists exactly to gate this.
+- **Truthy idempotency sentinels on Mock objects.** `Mock()` without a
+  `spec` auto-generates child Mocks for any attribute access, so
+  `getattr(client, "_ldr_closed", False)` returns a truthy Mock and
+  short-circuits the close. Always use `is True` / `is None` checks
+  for sentinels — see the pattern in `_close_base_llm`.
+- **Skipping `super().close()` in a search-engine subclass.**
+  `BaseSearchEngine.close()` is what cascades into preview/content
+  filters. Override it without calling super and you leak every
+  filter's resources (this was a Copilot finding on #3818).
+- **Treating `asyncio.run()` as safe inside an event loop.** It raises
+  `RuntimeError` if called from a thread that already has a running
+  loop. The pattern in `_close_base_llm` is: detect a running loop
+  with `get_running_loop()`, skip the async close in that branch (the
+  loop owner will close), only call `asyncio.run` in the no-loop case.
+
+---
+
+## History
+
+The FD-leak campaign spans roughly four months of iterative work. Each
+fix narrowed the remaining surface; each subsequent issue was found in
+a corner the previous wave hadn't touched.
+
+### Wave 1 — initial leak inventory (Jan 2026)
+
+- **#1832, #1849, #1856, #1860** — first comprehensive sweep. Identified
+  seven distinct leak sources: `auth_db` engine, `download_management`
+  DB, search cache, subprocess zombies, HTTP sessions in
+  `SemanticScholarSearchEngine` and `BaseDownloader`, Socket.IO threads.
+  Established context-manager + `try/finally` patterns. Added a
+  pre-commit hook to catch missing cleanup at commit time.
+
+### Wave 2 — thread-local engine accumulation (Mar 2026)
+
+- **#2495** — diagnosed that Flask's teardown only cleaned the
+  request-scoped `g.db_session` while a separate `_thread_engines` dict
+  accumulated NullPool engines per thread, leaking ~3 FDs per request.
+  Added `cleanup_current_thread()` across six tier-1 paths.
+- **#2591** — dead-thread engines (when threads crashed they left
+  engines behind) plus `stream=True` socket holds in the generic
+  downloader. Added a throttled dead-thread sweep, removed `stream=True`,
+  raised the Docker ulimit from 1024 to 65536.
+
+### Wave 3 — LLM wrapper lifecycle (Mar 2026)
+
+- **#2708** — diagnosed `ChatOllama` → `httpx.Client` chains with no
+  `__del__`. With the news scheduler triggering 50–300
+  `quick_summary()` calls per hour, a 1024-FD container exhausted in
+  3–4 hours. Wrapped four programmatic API entry points in
+  `try/finally` with explicit close.
+- **#2712** — extracted `close_llm()` to a shared utility. Added
+  `close()` and `_owns_llm` to `NewsAnalyzer`, `HeadlineGenerator`,
+  `TopicGenerator`, `JournalReputationFilter`, `DomainClassifier`,
+  `GitHubSearchEngine`, `IntegratedReportGenerator`,
+  `ElasticsearchSearchEngine`, and the benchmark graders.
+- **#2756** — wrapped bare `.close()` calls in `finally` blocks with
+  `safe_close()` to prevent masking the original exception.
+- **#2732** — moved `close()` into `ProcessingLLMWrapper` and
+  `RateLimitedLLMWrapper` directly; eliminated the standalone
+  `close_llm()` free function.
+
+### Wave 4 — DB session leaks + per-call patterns (late Mar / early Apr 2026)
+
+- **#3018** — `get_settings_manager(username=...)` was bypassing
+  `g.db_session` and creating QueuePool sessions per-thread; live
+  diagnostics showed 321 sockets allocated, only 66 in use.
+  `DownloadService.close()` leaked the inner `SettingsManager` session.
+  Also fixed `TopicBasedRecommender._create_recommendation_card()`
+  (per-call LLM with no cleanup) and an `asyncio.new_event_loop()` in
+  `news_strategy.py` that never closed.
+- **#3204** — test fixtures using `return` instead of `yield` left
+  engines un-disposed. Migrated 8 test files to `yield` +
+  `engine.dispose()`.
+
+### Wave 5 — DB pool architecture (Apr 2026)
+
+- **#3340** — kept QueuePool but minimized FDs (`pool_size=1`,
+  `max_overflow=2`, periodic `dispose()` every 30 min).
+- **#3337** (closed) — proposed switching SQLCipher engines to
+  NullPool for zero persistent FDs. Superseded by #3441.
+- **#3441** — removed per-thread NullPool engines entirely
+  (~2,100 lines of sweep logic deleted) and routed metrics through a
+  single shared per-user QueuePool with bounded sizing
+  (`pool_size=20`, `max_overflow=40`).
+- **#3477** — created [ADR-0004](../decisions/0004-nullpool-for-sqlcipher.md)
+  capturing the final pool model and updated stale FD calculations
+  across docs.
+
+### Wave 6 — async client close (May 2026)
+
+- **#3818** (open, declined for merge) — proposed session-pooling
+  around `safe_get`/`safe_post` to address #3816. The session refactor
+  is reasonable in isolation, but the lsof in #3816 showed ~72% of
+  leaked FDs as `a_inode [eventpoll]` selectors, not HTTP request
+  sockets — pointing at async-client transports rather than `safe_get`
+  callers (whose response bodies were already consumed). See
+  [the PR comment](https://github.com/LearningCircuit/local-deep-research/pull/3818#issuecomment-4402290677)
+  for the full reasoning.
+- **#3855** — extended `_close_base_llm()` to also close
+  `ChatOllama._async_client` (the actual gap the lsof pointed to).
+  Added the `IntegratedReportGenerator` close that was missing from the
+  per-research `finally` block. Idempotency via `_ldr_closed` sentinels
+  on the inner httpx clients.
+
+### Wave 7 — async close inside a running loop (May 2026)
+
+- **#4047** — `_close_base_llm`'s async branch had a documented "skip if
+  a loop is running; loop owner closes" path. **No loop-owner cleanup
+  code existed anywhere in the project**, so when the close was called
+  inside an active asyncio loop the inner `httpx.AsyncClient` (and its
+  `epoll_create` FD) was silently abandoned. Reproduced in production:
+  a v1.6.10 single-host Ollama container reached 1024 FDs with the
+  /proc histogram showing **929 `anon_inode:[eventpoll]` (91%)** — the
+  same FD class as #3816 but in a code path #3855's fix didn't cover.
+  The fix runs the async close in a brief daemon thread that owns its
+  own loop, so `asyncio.run(aclose())` works regardless of the caller's
+  loop state. A bounded 5-second `join` keeps the cleanup from blocking
+  shutdown when the Ollama server is unresponsive; on timeout
+  `_ldr_closed` is left unset so a later call retries, and a WARNING
+  surfaces so the situation is observable instead of silent.
+- **Healthcheck pidfd leak (same PR).** Dockerfile's
+  `HEALTHCHECK CMD python -c "... urllib.request.urlopen(...)"` had no
+  `timeout=` argument; Docker's 10s timeout SIGKILL'd the `sh -c`
+  parent but the python child was reparented to PID 1 and hung
+  forever, each surviving child holding a `pidfd` + TCP socket against
+  the app. Same /proc dump showed **64 `anon_inode:[pidfd]` (6%)** from
+  this. Adding `timeout=8` lets the child return/raise inside Docker's
+  budget so it exits cleanly and gets reaped.
+#### Audit ledger — what the broader sweep checked
+
+The PR included a wide audit (50+ parallel exploration agents across
+seven rounds plus direct `/proc` inspection) to catch any other latent
+FD leak. To save the next contributor from re-running the same checks,
+here is the full ledger:
+
+##### Checked and confirmed clean (no action needed)
+
+- **Non-Ollama LLM providers.** xAI, Google Gemini, OpenRouter, IONOS,
+  LM Studio, llama.cpp HTTP, DeepSeek, OpenAI-compatible endpoint, plus
+  OpenAI and Anthropic themselves. All extend `ChatOpenAI` or
+  `ChatAnthropic`, which use `@lru_cache`'d shared httpx clients.
+  `_close_base_llm`'s short-circuit on these classes is correct by
+  design — closing them would brick every other live LLM in the
+  process.
+- **HTTP session lifecycle.** Six instantiation sites checked
+  (`PricingFetcher` aiohttp, `LDRClient` SafeSession, `BaseDownloader`,
+  `SemanticScholarSearchEngine`, `MCPClient`, `CostCalculator`). All
+  context-managed via `with` or owned by a class with a paired
+  `close()` and `__exit__`.
+- **subprocess / pidfd.** Three call sites, all `subprocess.run()`
+  (blocking). No `subprocess.Popen` paths anywhere in `src/`. No
+  `ProcessPoolExecutor`. No FD leak surface beyond the healthcheck
+  child, already addressed by the Dockerfile `timeout=8` change.
+- **asyncio event loops.** Zero raw `asyncio.new_event_loop()`
+  outside safe `asyncio.run()` patterns. The historical leak in
+  `news_strategy.py` (#3018) is still fixed.
+- **File handles.** All 37 `open()` call sites are inside `with`.
+  Zero bare opens. `tempfile.NamedTemporaryFile` / `TemporaryDirectory`
+  all context-managed.
+- **SocketIO connect/disconnect.** Non-disconnect handlers
+  (`subscribe`, `unsubscribe`, `connect`) do not acquire DB sessions
+  (an early-round agent claim that they did was refuted on re-read).
+  The `__socket_subscriptions` dict is cleaned on disconnect. The
+  PID-1 FD breakdown showed only 3 sockets out of 1024 — socket
+  accumulation is not a contributor.
+
+##### Flagged by audit, then verified NOT a real FD leak
+
+- **OllamaEmbeddings httpx (historical — current state covered in
+  Wave 10 below).** At the time of this Wave-7 audit LDR imported the
+  **deprecated** `langchain_community.embeddings.OllamaEmbeddings`,
+  which used `requests.post()` per call — no persistent httpx client,
+  no `_client` / `_async_client` attribute. Direct introspection:
+  `[a for a in dir(e) if 'client' in a.lower()]` returned `[]`. Zero
+  FDs per call. An audit agent confused this class with `ChatOllama`,
+  which is a different class. The migration to
+  `langchain_ollama.OllamaEmbeddings` predicted in the next subsection
+  has since shipped (#4352/#4353) and the resulting FD-leak regression
+  has been fixed — see Wave 10.
+- **`auth_db` and `journal_quality` engines escaping
+  `shutdown_databases()`.** `auth_db` uses
+  `QueuePool(pool_size=10, max_overflow=20)` and `journal_quality`
+  uses `StaticPool` with `immutable=1`. Both are **bounded** and do
+  not grow at runtime. Live `/proc` on the affected container showed
+  only 21 SQLite-related FDs total on PID 1 — well below the ~91-FD
+  ceiling these unmanaged engines could theoretically reach. The
+  kernel reclaims FDs at process exit regardless of `engine.dispose()`,
+  and SQLite WAL files auto-checkpoint on next open. Missing dispose
+  at exit is hygiene, not a leak.
+- **`LibraryRAGService` in three RAG SSE endpoints.**
+  `rag_routes.py:693, 1054, 1827` do construct the service outside
+  the generator and never close it, **but** `LibraryRAGService.close()`
+  only sets references to `None` — it releases no FDs. FAISS uses
+  `pickle.load()` (not mmap); OllamaEmbeddings holds no FDs per the
+  item above; the SentenceTransformer model+tokenizer mmaps are
+  process-wide singletons. What gets delayed is ~50–200 MB of
+  embedding-model RAM until GC. A memory-pressure question, not the
+  eventpoll FD class this Wave addressed.
+- **Residual `pidfd` accumulation via Playwright fallback** —
+  identified in a Round-8 follow-up after the eventpoll fix landed.
+  Live `/proc` on the prerelease container showed ~29 pidfds steady
+  state, growing ~3.6/hour, all targeting `Pid: -1` (children that
+  had exited). Rate was stable during active benchmark execution,
+  ruling out a per-task source. Eight parallel agents converged on
+  the same chain: `_check_subscription` → `quick_summary` →
+  `FullSearchResults.batch_fetch_and_extract` → `AutoHTMLDownloader`
+  fallback to `PlaywrightHTMLDownloader._fetch_with_playwright`. Each
+  `sync_playwright().start()` invokes
+  `asyncio.create_subprocess_exec()` for the Node.js driver (opens a
+  pidfd via Linux's `PidfdChildWatcher`); the driver then fails
+  because Chromium is not installed in the production `ldr` Dockerfile
+  stage (only `ldr-test` runs `playwright install --with-deps
+  chromium`), and the asyncio child watcher does not promptly close
+  the pidfd on the failed-child exit. CPython 3.14 was confirmed to
+  not use pidfd in `subprocess.py` at all (`subprocess.run`/`Popen`
+  use `waitpid(WNOHANG)` polling), so subprocess-based hypotheses
+  were ruled out. **Fixed by PR #3971** (default
+  `web.enable_javascript_rendering=false`): the fallback short-circuits
+  before any subprocess is spawned, so no pidfd is opened. The PR was
+  motivated by issue #3826 (confusing tracebacks); the FD-leak
+  finding is the second motivation, surfaced here.
+
+##### Minor findings (not steady-state leaks; worth knowing)
+
+- **Daemon threads without explicit shutdown.**
+  `journal_reputation_filter.py` background fetcher, `log_utils.py`
+  queue processor. All daemonized — reaped by the OS at process exit.
+  Not steady-state leaks; no per-request growth.
+- **Abandoned-research thread on socket disconnect.** If a client
+  closes the tab mid-research, the socket subscription is removed but
+  the research thread keeps running until completion;
+  `_active_research[research_id]` is not cleared on disconnect. Not an
+  FD leak; potentially compute/memory waste if the user wanted the
+  research to stop. Out of scope for the FD-leak story.
+
+#### Future-proofing note — `langchain_ollama.OllamaEmbeddings` migration (resolved in Wave 10)
+
+Status: **resolved**. The migration this note predicted shipped in
+#4352/#4353; the FD-leak regression it predicted then surfaced and was
+fixed in Wave 10 (see below). Kept here as the source of the prediction
+that the next contributor's audit can cross-reference.
+
+`langchain_community.embeddings.OllamaEmbeddings` was deprecated ("will
+be removed in langchain 1.0.0", per the import warning). Its replacement,
+`langchain_ollama.OllamaEmbeddings`, **does** carry `_client` and
+`_async_client` attributes — same shape as `ChatOllama`. Verified by
+direct introspection at the time of writing:
+
+```
+langchain_ollama.OllamaEmbeddings client attrs:
+  ['_set_clients', 'async_client_kwargs', 'client_kwargs',
+   'sync_client_kwargs']
+Has _client?       True
+Has _async_client? True
+```
+
+The prediction was: once LDR migrates, the eventpoll FD leak class
+returns for embeddings unless `_close_base_llm` is called on embedding
+instances. The introspection turned out to be slightly different from
+expected — both clients are constructed *eagerly* by a Pydantic
+`@model_validator(mode="after")` in `langchain_ollama.embeddings.py`,
+so the leak fires per-instance regardless of whether the async path is
+exercised. Wave 10 contains the post-mortem and fix.
+
+### Wave 10 — embeddings FD leak after langchain_ollama migration (June 2026)
+
+The migration predicted above shipped without the matching close-path
+generalization, exactly as feared. Verified by four independent agents:
+`langchain_ollama.OllamaEmbeddings(...)` eagerly constructs both a sync
+`ollama.Client` (→ `httpx.Client`) and an async `ollama.AsyncClient`
+(→ `httpx.AsyncClient` → one `epoll_create` FD) inside its
+`@model_validator(mode="after")` at
+`.venv/.../langchain_ollama/embeddings.py:295-315`. No `close()`,
+`aclose()`, `__del__`, or `weakref.finalize` exists on the new class or
+the underlying `ollama` / `httpx` clients, so dropping the Python
+reference does not release the FDs.
+
+`_close_base_llm` already handled the shape — its module-prefix checks
+(`type(...).__module__.startswith("ollama")` at
+`src/local_deep_research/utilities/llm_utils.py:97,114`) match
+`ollama.Client` / `ollama.AsyncClient` regardless of which langchain
+wrapper holds them. The function just wasn't called on embeddings
+instances — `LocalEmbeddingManager.close()` and `LibraryRAGService.close()`
+only nulled their `_embeddings` / `embedding_manager` references,
+relying on GC that would never run the close.
+
+Fix: route the close call through the existing manager lifecycle.
+`LocalEmbeddingManager.close()` now calls `_close_base_llm(self._embeddings)`
+before nulling. `LibraryRAGService.close()` now calls
+`self.embedding_manager.close()` before nulling — guarded by an
+`_owns_embedding_manager` flag so a caller-supplied manager (test
+fixtures, multi-service callers) stays under caller control. The
+`_close_base_llm` docstring is updated to acknowledge it also handles
+`OllamaLLM` and `OllamaEmbeddings`; no behaviour change, only
+documentation. Regression coverage lives next to the existing
+ChatOllama tests in `tests/utilities/test_close_base_llm.py` —
+`TestCloseBaseLLMRealOllamaEmbeddings` is the canary that fires if a
+future migration breaks the close path again.
+
+A follow-up PR (PR-B) hardens the `rag_routes.py` call sites that
+construct `LibraryRAGService` without a `with` block: 4 simple
+synchronous sites get a `with` wrap; 3 SSE-streaming sites have the
+construction moved *inside* the `stream_with_context` generator (a
+`with` at request-handler scope would close the service before the
+stream runs). A safety-net PR (PR-C) registers a `weakref.finalize`
+inside `OllamaEmbeddingsProvider.create_embeddings()` so that callers
+that bypass the manager — for example the programmatic-API examples
+migrated in #4399 — still get eventual cleanup at GC time.
+
+### Round 9 — broader resource audit (May 2026)
+
+Once the FD-leak classes were closed, a follow-up audit looked for
+*other* slow-growth patterns that wouldn't trip the FD counters but
+could still degrade a long-running container: memory and cache growth,
+thread / asyncio Task / lock lifecycle, DB state hygiene beyond
+connections. Three parallel agents per round, two rounds (Round 1
+hypothesis generation, Round 2 fact-check), captured here in
+verified form so the next contributor doesn't re-derive the same
+conclusions.
+
+#### Refuted (false positives from Round 1, verified in Round 2)
+
+- **`@cache` on `get_available_providers`** (was in `config/llm_config.py`;
+  **removed in #4590**, so this no longer exists). Round 1 claimed unbounded
+  cache growth if the function were called with differing `settings_snapshot`
+  dicts. Round 2 verified: dicts are unhashable, so `@cache` would raise
+  `TypeError` on them, not silently grow. In practice the call sites passed
+  `settings_snapshot=None` (hashable, cardinality 1). Not a leak — and the
+  function (a dead duplicate of the provider auto-discovery path) has since
+  been deleted entirely. Kept here for the audit record.
+- **Thread-local Session identity-map growth**
+  (`database/thread_local_session.py`). Round 1 claimed long-running
+  research threads would accumulate ORM objects in the per-thread
+  Session's identity map. Round 2 verified: SQLAlchemy's default
+  `expire_on_commit=True` clears the identity map at every commit;
+  the codebase commits periodically. Bounded by typical query volume,
+  not unbounded by uptime.
+- **`token_usage` table unbounded growth.** Append-only per LLM call
+  with no TTL or retention job. Round 2 verified: **feature by
+  design**. Schema has compound time-series indexes
+  (`idx_token_research_timestamp`, etc.); `/api/context-overflow` and
+  `/metrics/api/metrics` explicitly query historical windows for cost
+  analysis. The table is a permanent audit trail by intent. Adding
+  retention would break the metrics dashboards.
+- **`search_calls` table unbounded growth.** Same shape and same
+  verdict — compound time-series indexes confirm intentional design
+  as a permanent search-analytics record.
+
+#### Fixed in this PR — three per-user lock dicts
+
+- **Three per-user lock dicts** — `_user_init_locks` and `_user_locks`
+  are module-level dicts in `database/library_init.py` and
+  `database/backup/backup_service.py` respectively; `_user_critical_locks`
+  is an instance attribute on the `QueueProcessorV2` singleton in
+  `web/queue/processor_v2.py`. Each stored one `threading.Lock` per
+  username with no removal hook. Bounded ceiling (~296 bytes/entry ×
+  3 dicts at 1000 users = ~900 KB), so not urgent — but easy to fix
+  cleanly. The two module-level dicts now expose
+  `pop_user_init_lock` / `pop_user_lock` functions; the queue
+  processor exposes the equivalent as an instance method
+  `queue_processor.pop_user_critical_lock`. A shared
+  `_pop_per_user_locks(username)` helper in `connection_cleanup.py`
+  calls all three with lazy imports and individual try/except
+  (WARNING-level so dict accumulation is observable, matching the
+  sibling scheduler-unregister error path). The helper is invoked
+  unconditionally — outside the `close_user_database` try/except so
+  it still runs when the DB close itself fails — in both the
+  idle-connection sweeper (`connection_cleanup.py:cleanup_idle_connections`)
+  and the logout / password-change paths (`web/auth/routes.py`).
+  Tests in `tests/web/auth/test_connection_cleanup.py::TestPopPerUserLocks`
+  cover the helper directly and through the idle-close path.
+
+#### Real but small (survives verification)
+
+- **`app_logs` (ResearchLog) table — no automatic retention.** Grows
+  by ~100s-1000s of rows per research. Cleaned only via cascade-delete
+  when the parent `Research` row is deleted manually. Unlike
+  `token_usage` / `search_calls`, this table has no UI dashboard or
+  time-series API consuming it — it's debug context for a specific
+  research session, not an analytics record. For users who keep all
+  research, logs accumulate indefinitely. See "Intentionally not done
+  (deferred)" for the retention design when a symptom report
+  justifies it.
+
+---
+
+## Debugging FD leaks — playbook for the next one
+
+When the next FD leak shows up (and there will be one, eventually), this
+section is the shortcut. It captures the actual diagnostic flow that
+worked across Waves 6 and 7 so a future contributor doesn't have to
+re-derive it from the symptom.
+
+### 0. Symptoms that mean "investigate this as an FD leak"
+
+- Tracebacks like `OSError: [Errno 24] Too many open files`, typically
+  from `selectors.DefaultSelector()` in werkzeug or `send_from_directory`
+  in Flask. These are usually the *first* visible failure.
+- Browser-side MIME-type errors on static assets (`text/html` instead of
+  `text/css` / `application/javascript`). These are downstream of FD
+  exhaustion — Flask can't open the static file, returns an HTML 500,
+  and the browser refuses to apply it because of
+  `X-Content-Type-Options: nosniff`.
+- `High FD count (N) — approaching system limit` warnings from
+  `web/auth/connection_cleanup.py` (fires at FD > 800 every 5-minute
+  cleanup tick).
+- Container health turns `unhealthy` because the healthcheck `urlopen`
+  hangs on a process that no longer has FDs to accept connections.
+
+### 0a. Rule out first — local UI-test "fresh-user churn" false positive
+
+Before treating climbing FDs as a leak, confirm you are measuring the
+**single-CI-user** condition. A very convincing *false* FD leak appears
+when reproducing UI tests locally:
+
+- The Puppeteer harness (`tests/ui_tests/auth_helper.js` →
+  `ensureAuthenticated`) logs in as the shared CI user `test_admin` when
+  `CI=true`. If that login fails, it **falls back to registering a fresh
+  `testuser_<timestamp>` per test**. The usual local trigger is
+  `test_admin` getting *failed-login lockout-locked* after a few
+  iterations.
+- Each fresh user opens its own per-user encrypted DB + engine. Those are
+  disposed only on logout or the ~300s connection-cleanup sweep, so within
+  one sub-300s shard run they accumulate and the server's FD count to
+  `encrypted_databases/*.db(-wal/-shm)` climbs ~linearly (e.g. 0→90 per
+  shard run, 0→533 over six runs). It looks identical to a real per-user
+  connection leak.
+- It is **not** a server bug. In real CI the one working `test_admin` is
+  reused → one engine → FDs bounded by the pool cap (pool_size 20 +
+  max_overflow 40 = 60). Confirm by grepping the server log for many
+  distinct `testuser_<ts>` engine opens, or by checking the username the
+  leaked FDs' DB files belong to.
+
+Concretely: the **chat UI shards** (`chat-core`, `chat-lifecycle`) failing
+in CI were investigated as a per-user DB FD leak and traced *twice* to
+this artifact. Both shards pass locally in faithful CI mode with bounded
+FDs; their CI failures are runner **contention** (60s navigation timeouts
+on a heavily-loaded Docker runner), not a connection leak. Cross-verify
+the user identity before committing to a leak hypothesis.
+
+### 1. Capture diagnostic state BEFORE restarting
+
+The single most important rule: **the snapshot does not survive a
+container restart**. Every minute spent on the live broken container is
+worth an hour of after-the-fact agent guessing. Save the diagnostic
+output to a host-side file first.
+
+#### One-shot host-side snapshot (works even when the container is
+FD-starved enough that `docker exec` can't fork)
+
+```bash
+# Run on the Docker host. No docker exec required.
+P=$(docker inspect -f '{{.State.Pid}}' <container-name>)
+sudo bash -c "
+  echo '=== Total FDs ==='
+  ls /proc/$P/fd | wc -l
+  echo '=== FD-type histogram (digits collapsed) ==='
+  ls -l /proc/$P/fd | awk '{print \$NF}' \
+    | sed -E 's/\[[0-9]+\]/[N]/g; s/[0-9]{4,}/NUM/g' \
+    | sort | uniq -c | sort -rn | head -30
+  echo '=== Counts by category ==='
+  printf 'socket:     %s\n' \$(find /proc/$P/fd -lname 'socket:*'     | wc -l)
+  printf 'pipe:       %s\n' \$(find /proc/$P/fd -lname 'pipe:*'       | wc -l)
+  printf 'eventpoll:  %s\n' \$(find /proc/$P/fd -lname '*eventpoll*'  | wc -l)
+  printf 'pidfd:      %s\n' \$(find /proc/$P/fd -lname '*pidfd*'      | wc -l)
+  printf 'WAL files:  %s\n' \$(find /proc/$P/fd -lname '*-wal'        | wc -l)
+  printf 'SHM files:  %s\n' \$(find /proc/$P/fd -lname '*-shm'        | wc -l)
+  printf '.db files:  %s\n' \$(find /proc/$P/fd -lname '*.db'         | wc -l)
+" | tee /tmp/ldr-fd-snapshot.txt
+```
+
+Why host-side: reading the container's PID 1 FDs from inside the
+container requires the same UID that started PID 1. The Dockerfile
+entrypoint runs as root then `setpriv`s to `ldruser`, so the
+`docker exec` shell (ldruser) cannot `readlink` PID 1's FDs even though
+it can count them. Host root via `sudo` sidesteps the UID check.
+
+#### Inside-container alternative (if the host is locked down)
+
+```bash
+docker exec --user 0 <container-name> sh -c '...same body...'
+```
+
+`--user 0` runs the exec'd shell as root inside the container,
+sidestepping the same UID restriction.
+
+### 2. The lookup table — FD type → likely source
+
+| Dominant FD type            | Likely source                                                                              | Diagnostic deep-dive                                                                                      |
+|-----------------------------|---------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------|
+| `anon_inode:[eventpoll]`    | `asyncio` event loop or `httpx.AsyncClient` selector. Each leaked async client = +1.        | Grep `asyncio.create_subprocess`, `httpx.AsyncClient`, `_async_client`, `ainvoke`. See Wave 6, Wave 7.   |
+| `anon_inode:[pidfd]`        | `asyncio.create_subprocess_*` or `multiprocessing.Process` (uses `pidfd_open` on Linux).    | Read `/proc/PID/fdinfo/N` for each pidfd; the `Pid:` line shows the target (`-1` = child already exited). |
+| `socket:*` (lots)           | HTTP keep-alive, SSE streams, SocketIO connections.                                         | Cross-reference with `/proc/PID/net/tcp` states; check Round 7 R7A8 patterns.                            |
+| `pipe:*` (lots)             | `subprocess.run`/`Popen` with `stdout=PIPE`, multiprocessing IPC, loguru queue.             | Check `subprocess.run` sites and APScheduler executor type.                                              |
+| `REG` `*-wal` / `*-shm`     | SQLCipher in WAL mode. Each pooled connection holds ~3 FDs.                                 | See ADR-0004. If growing without bound, the periodic `engine.dispose()` is silently failing.             |
+| `REG /data/*.db` (lots)     | Plain SQLite connections from an engine without bounded pool.                               | Audit `create_engine` sites (R7A6 caught two unmanaged ones).                                            |
+| `REG /home/...mmap...`      | Memory-mapped model weights or FAISS indexes — usually process-wide singletons (not leaks). | Check whether the count grows per request. If yes → real leak.                                           |
+
+### 3. Pinpointing the source for a specific FD type
+
+#### Eventpoll
+
+`anon_inode:[eventpoll]` always comes from `EpollSelector` — created
+by every asyncio loop and every `httpx.AsyncClient`. Grep:
+
+```
+grep -rn 'asyncio.create_subprocess\|httpx.AsyncClient\|_async_client' src/
+```
+
+Then check whether each site explicitly closes the client. The Wave 7
+fix to `_close_base_llm` is the reference pattern for "close async
+httpx even when called inside a running loop."
+
+#### Pidfd
+
+Pidfds expose their target PID via fdinfo:
+
+```bash
+# Run inside the container (or via docker exec --user 0):
+for fd in $(ls /proc/1/fd 2>/dev/null); do
+  link=$(readlink /proc/1/fd/$fd 2>/dev/null)
+  case "$link" in
+    *pidfd*)
+      tpid=$(awk '/^Pid:/ {print $2}' /proc/1/fdinfo/$fd 2>/dev/null)
+      if [ "$tpid" -gt 0 ] 2>/dev/null; then
+        cmd=$(tr '\0' ' ' < /proc/$tpid/cmdline 2>/dev/null | cut -c1-80)
+        echo "fd=$fd alive pid=$tpid : $cmd"
+      else
+        echo "fd=$fd ORPHAN (child exited; pidfd not closed)"
+      fi
+      ;;
+  esac
+done
+```
+
+A high "ORPHAN" count = something called `asyncio.create_subprocess_*`
+or `multiprocessing.Process`, the child exited, but the pidfd in the
+parent was never closed. Common in Round-8: Playwright's Node.js
+driver subprocess failing because Chromium isn't installed in the
+production image.
+
+**Note:** CPython 3.14's `subprocess.py` does not use pidfd at all
+(`waitpid(WNOHANG)` polling instead). So pidfds in a 3.14 process
+necessarily come from asyncio or multiprocessing, not from
+`subprocess.run` / `Popen`.
+
+#### Syscall-level pinpointing with bpftrace (mysterious cases)
+
+When the source isn't obvious from the FD type, `bpftrace` can record
+the Python stack of every relevant syscall on the live process. This
+would have caught the Playwright leak in seconds instead of two rounds
+of agent exploration. Requires kernel headers and `bpftrace` installed
+on the host (NOT the container — bpftrace runs in host kernel space
+and can target a host PID by number):
+
+```bash
+# Find host-side PID of container's PID 1
+P=$(docker inspect -f '{{.State.Pid}}' <container>)
+
+# Trace every pidfd_open syscall, grouped by user-stack:
+sudo bpftrace -e "tracepoint:syscalls:sys_enter_pidfd_open
+                  /pid == $P/ { @[ustack(perf)] = count(); }"
+
+# Same idea for epoll_create / epoll_create1 (eventpoll FDs):
+sudo bpftrace -e "tracepoint:syscalls:sys_enter_epoll_create1
+                  /pid == $P/ { @[ustack(perf)] = count(); }"
+```
+
+Let it run for a minute, then Ctrl-C; you get a histogram of every
+unique stack that triggered the syscall, ranked by frequency. The hot
+stacks are your culprits. Works for any syscall — useful future
+candidates: `socket`, `inotify_init1`, `timerfd_create`,
+`memfd_create`.
+
+#### WAL/SHM
+
+`engine.dispose()` is expected to release these. If the count climbs
+across the periodic 30-minute dispose cycles, the dispose is silently
+failing. The observability commit (f86c3f7af) elevates dispose
+failures to WARNING — check the logs for `Error disposing engine for
+<user>`.
+
+### 4. Existing instrumentation already in the codebase
+
+- **`_count_open_fds()`** in
+  `src/local_deep_research/web/auth/connection_cleanup.py` —
+  fast `/proc/self/fd`-based counter with macOS fallback. Reusable.
+- **`Resource monitor: open_fds=…`** debug log line in
+  `connection_cleanup.py`, fires every 5-minute cleanup tick.
+- **`High FD count (N)` WARNING** in `connection_cleanup.py`
+  when FDs exceed 800. The single most useful production signal.
+- **`GET /api/v1/health` resource diagnostics** (PR #4915) — for
+  *authenticated* callers the response carries a `resources` block
+  (`fd_count`, `fd_soft_limit`, `fd_hard_limit`, `fd_usage_percent`,
+  `thread_count`) and flips `status` to `"warning"` above 70% FD usage.
+  This is the live, queryable form of the `_count_open_fds()` log
+  signal — `curl` it during a leak hunt instead of grepping container
+  logs. It returns counts only (never fd targets), so no open file
+  paths or socket peers are exposed. Anonymous callers (the Docker
+  healthcheck) get only the basic `status`/`message`/`timestamp`.
+- **In-CI FD-growth canaries** in
+  `tests/utilities/test_close_base_llm.py`. These run on every PR:
+  - `TestCloseBaseLLMRealHttpxAsync::test_no_fd_growth_across_repeated_close_cycles`
+    — guards the eventpoll FD class against Wave-6-shaped regressions.
+  - `TestCloseBaseLLMRealHttpxAsync::test_no_fd_growth_when_closed_inside_running_loop`
+    — guards the Wave-7-shaped in-running-loop skip regression.
+  - `TestAsyncioSubprocessFDBaseline::test_no_fd_growth_across_asyncio_subprocess_cycles`
+    — guards the pidfd FD class against the child-watcher leak shape.
+  - `TestAsyncioSubprocessFDBaseline::test_no_fd_growth_when_subprocess_fails_to_exec`
+    — pins the *exact* Wave-7-pidfd shape (failed exec, child watcher
+    must still clean up). Catches platform-level regressions in
+    Python's asyncio child watcher.
+
+  All four use `_open_fd_count()` (also in that file) which reads
+  `/proc/self/fd` on Linux with an `RLIMIT_NOFILE` fallback on macOS.
+  Slack is +2 FDs across 5–10 iterations. A real per-cycle leak would
+  blow past that.
+
+### 4a. Development-time detection (catch leaks at test time)
+
+Production /proc inspection catches leaks **after** they ship. The
+cheapest catch is to make Python itself complain at test time. Three
+Python features cooperate to surface unclosed resources during a
+normal test run — none of them were on by default during Waves 6 and
+7, which is part of why those leaks made it to production.
+
+**`PYTHONASYNCIODEBUG=1` plus `-W default::ResourceWarning`.** When
+asyncio debug mode is on, unclosed transports/coroutines emit a
+`ResourceWarning` at GC time. The `-W` filter makes Python actually
+display them. Together they would have caught the Wave 7 in-running-loop
+skip: every leaked `httpx.AsyncClient` produces a visible warning the
+first time the GC sweeps after the test fixture exits. From
+[the asyncio dev docs](https://docs.python.org/3/library/asyncio-dev.html):
+
+> When a transport is no longer needed, call its `close()` method to
+> release resources. ... If a transport or an event loop is not closed
+> explicitly, a `ResourceWarning` warning will be emitted in its
+> destructor.
+
+To enable in `pyproject.toml` `[tool.pytest.ini_options]`:
+
+```toml
+filterwarnings = [
+    "default::ResourceWarning",
+]
+env = [
+    "PYTHONASYNCIODEBUG=1",
+]
+```
+
+Or in CI for a one-off check:
+
+```bash
+PYTHONASYNCIODEBUG=1 python -W default::ResourceWarning -m pytest tests/
+```
+
+For a CI gate that **fails** on any leak (more aggressive — use only
+on a targeted subset of tests, not the whole suite, because
+third-party libraries also emit ResourceWarning):
+
+```toml
+filterwarnings = [
+    "error::ResourceWarning",
+]
+```
+
+**`python -X dev`.** Enables Python's dev mode, which turns on a
+bundle of safety checks including ResourceWarning display, asyncio
+debug mode, and warnings as default. Cheap one-flag alternative for
+local development; not recommended in production (overhead).
+
+```bash
+python -X dev -m pytest tests/
+```
+
+**`psutil` for portable FD counting in tests.** Our in-codebase
+`_count_open_fds` uses `/proc/self/fd` (Linux-fast path, macOS
+fallback). `psutil` is the cross-platform alternative many other
+projects use:
+
+- `psutil.Process().num_fds()` — Linux/BSD only; same number as our
+  helper.
+- `psutil.Process().open_files()` — list of named files; gives the
+  paths for `REG`-type FDs (e.g., `/data/*.db-wal`).
+- `psutil.Process().connections(kind='all')` — sockets visible to the
+  process, with state and remote address.
+
+These are useful in unit tests when you want to assert "no new file
+of pattern X is open after the close path runs," and they work on the
+macOS dev environments without `/proc`.
+
+**For tracking which Python object holds a leaked FD: `tracemalloc`
++ `objgraph`.** Not FD tools per se, but when a leak is reproducible,
+take a `tracemalloc` snapshot before and after the suspect operation
+and diff — the new allocation is usually the wrapper holding the FD.
+`objgraph.show_backrefs([leaked_obj])` then renders the reference
+chain keeping it alive. Both are pure-Python and zero-dependency.
+
+### 5. Why we don't have an automated FD-growth test in CI
+
+Several reasons, weighed during Wave 6 and Wave 7:
+
+- **Per-request FD growth is hard to assert.** Many legitimate
+  request paths transiently open and close FDs; a noisy delta is the
+  norm. Distinguishing "leak" from "in-flight" requires a stable
+  quiescent state, which a CI test doesn't naturally provide.
+- **The CI environment spawns its own subprocesses.** pytest,
+  coverage, gunicorn workers (for some test variants), gh-runner
+  cleanups — all add their own FDs that pollute the count.
+- **PID-namespace differences between CI and prod.** Counts you
+  observe in a CI container's /proc are not directly comparable to a
+  production container's /proc; the subprocess sources differ.
+- **The actual leaks have been "slow drip" patterns** that need
+  hours of uptime to surface. Wave 6's eventpoll leak took multiple
+  hours of `ainvoke` calls to reach the 1024 cap. CI can't run for
+  hours per PR.
+
+What works instead:
+1. **Per-leak unit-level regression tests.** Each fix in Waves 1-7
+   landed with a targeted test that exercises the specific close path
+   (e.g. `tests/utilities/test_close_base_llm.py::test_no_fd_growth_when_closed_inside_running_loop`).
+   These are fast, deterministic, and run on every PR.
+2. **Opt-in manual smoke suite** (`RUN_MANUAL_SMOKE=1`) for the
+   end-to-end "run-the-cycle-N-times-and-count" pattern, used during
+   investigation but not on every CI run.
+3. **Production /proc inspection** when a leak is suspected — the
+   playbook above. Faster than CI for the long-drip patterns.
+
+If you want to add a long-run CI job, the right shape would be a
+**nightly** workflow (not per-PR) that:
+
+1. Builds the production Docker image.
+2. Starts it with a synthetic user account and ~5 news subscriptions.
+3. Lets it idle for 20-30 minutes.
+4. Runs the host-side snapshot script above.
+5. Asserts `total FDs < N` and `eventpoll < M` and `pidfd < K`,
+   where the thresholds are tuned for the steady-state ceilings the
+   codebase intentionally permits (auth_db pool, etc.).
+
+That would have caught Waves 6, 7 in a single nightly cycle instead
+of through a user crash report. The reason it doesn't exist yet is
+cost (a half-hour idle job per night per platform) and the lack of a
+clear baseline; the Round-8 finding is the moment to consider adding
+one if you want to invest the maintenance time.
+
+### 6. Lookup: which Wave fixed which leak class
+
+| FD class               | Wave / PR              | Root mechanism                                                            |
+|------------------------|------------------------|---------------------------------------------------------------------------|
+| `eventpoll`            | Wave 6 #3855 + Wave 7 #4047 | ChatOllama `_async_client` not closed (Wave 6) → also not closed when called inside a running loop (Wave 7). |
+| `pidfd` from healthcheck | Wave 7 #4047           | `urlopen` no `timeout=` → child hangs → reparented to PID 1 with pidfd held. |
+| `pidfd` from Playwright fallback | Round 8 / #3971  | Production image lacks Chromium binary; Playwright invocation opens pidfd then fails. |
+| WAL/SHM accumulation   | Wave 5 / ADR-0004      | SQLCipher+WAL leaks handles on out-of-order close; periodic `engine.dispose()` resets the pool.  |
+| Per-thread engine FDs  | Wave 5 #3441           | Removed per-thread `NullPool` engines entirely; shared per-user `QueuePool`. |
+| HTTP session sockets   | Wave 1 / Wave 3        | `SafeSession` / `BaseDownloader` close-in-`finally` discipline.            |
+| `asyncio.new_event_loop` | Wave 4 #3018         | Replaced manual loop creation with `asyncio.run()` in `news_strategy.py`.   |
+
+Use this table to skip the rediscovery step the next time a specific
+FD type dominates a snapshot.
+
+---
+
+## Intentionally not done (deferred)
+
+These showed up during planning and were deliberately *not* done. If
+they get rediscovered as "missing work" by future contributors, please
+reference this section first.
+
+- **`weakref.finalize` defense-in-depth on the LLM wrappers.** Designed
+  and verified safe (no `__del__` conflicts, `__getattr__` doesn't
+  intercept `_finalizer`, no reference cycles). Deferred until a
+  fourth wave of "missed close" leaks justifies adding a new pattern
+  that future contributors must understand. Current explicit-close
+  discipline has held since #2712 / #2732 / #3018.
+- **LLM caching in `get_llm()`.** Bounding total `ChatOllama` instances
+  to N=distinct configs would make leak shapes architecturally
+  impossible. Orthogonal optimization, deferred — adds complexity
+  around settings invalidation and multi-tenant isolation.
+- **Pre-commit hook flagging `get_llm()` callers without `close()`.**
+  Useful in principle, deferred — high false-positive risk
+  (caller-passed LLMs, lazy-init holders, factory-returned LLMs all
+  legitimately don't close). Needs a careful design.
+- **Per-FD-type/inode breakdown on the health endpoint.** The basic
+  version — aggregate FD count, limits, and usage percent on
+  `GET /api/v1/health` — shipped in PR #4915 (see section 4). The two
+  earlier attempts were closed rather than merged: PR #3033 (superseded
+  by #4915, a clean reapplication onto current main) and PR #3036 (a
+  `utilities/fd_monitor.py` FD circuit breaker — closed because its
+  premise, a retry-driven "death spiral," did not match the real
+  WAL/SHM-handle root cause already handled by the periodic pool
+  disposal above; `fd_monitor.py` was never merged and does not exist).
+  A type/inode breakdown (eventpoll vs pidfd vs WAL — the histogram the
+  section-1 `/proc` snapshot produces) is feasible but deferred until an
+  active leak hunt actually needs it.
+- **Automated reproduction of #3816's eventpoll-FD leak in a test
+  suite.** Explored in closed PR #3930 — a single-thread
+  `asyncio.run(ainvoke)` loop against real Ollama does *not* reproduce
+  eventpoll accumulation, because `asyncio.run` deterministically closes
+  its loop's selector each call. Reliable reproduction would need
+  sustained concurrent load (multi-worker harness over a shared loop).
+  In-CI mock + no-network real `ChatOllama` tests in
+  `tests/utilities/test_close_base_llm.py` already cover the close-chain
+  introspection regressions; a load-shape reproduction is deferred
+  until a future leak justifies the maintenance burden.
+- **`app_logs` (ResearchLog) retention setting + scheduled cleanup
+  job.** Identified in Round 9; the only audit finding that wasn't
+  refuted but also isn't impactful enough today. *Trigger to do this
+  work:* a user reports the SQLCipher DB growing >100 MB and
+  complains about query slowdown, OR a self-hosted instance keeping
+  research logs for >1 year sees DB bloat, OR the metrics dashboard
+  starts noting research-detail page load slowdown traced to
+  `app_logs` joins. *Implementation sketch:* add
+  `logs.research_log_retention_days` to
+  `defaults/default_settings.json` (default `0` = disabled, preserves
+  current behavior; e.g. `30` to keep last 30 days). Extend the
+  existing `BackgroundJobScheduler` in `scheduler/background.py`
+  (which already runs `cleanup_inactive_users` hourly and
+  `_reload_config` every 30 min) with a daily `_cleanup_old_research_logs`
+  job that deletes `ResearchLog` rows older than the retention
+  window. Skip rows belonging to favorited / starred researches if a
+  flag exists. ~30 LOC + a regression test that inserts old rows,
+  triggers the job, asserts old rows are deleted and recent ones
+  survive. Add `changelog.d/<id>.feature.md`.
+---
+
+## Glossary
+
+- **`_owns_llm`** — instance flag set in `__init__` to `True` when the
+  class fetched its own LLM via `get_llm()`, `False` when an LLM was
+  injected by the caller. Gates whether `close()` actually closes the
+  LLM.
+- **`safe_close(resource, name)`** — helper in `utilities/resource_utils.py`
+  that calls `resource.close()` inside a try/except, logging on failure.
+  Never raises. Used in every `finally` block.
+- **`_ldr_closed`** — sentinel attribute set on inner httpx clients by
+  `_close_base_llm` to make the function idempotent. Checked with
+  `is True` (not truthy) so Mock objects without a `spec` don't trip
+  the guard.
+- **eventpoll FD** — Linux `a_inode` file descriptor type for
+  `epoll_create`'d kernel objects. Each asyncio event loop registers
+  one. Leaked AsyncClients hold them via the loop's selector.

docs/docker-compose-guide.md

@@ -0,0 +1,198 @@
+# Docker Compose Guide
+
+This guide covers Docker Compose setup for Local Deep Research. For the quickest start, see the [Quick Start](#quick-start) section below.
+
+## Quick Start
+
+### CPU-Only (All Platforms)
+
+Works on macOS (M1/M2/M3/M4 and Intel), Windows, and Linux:
+
+```bash
+curl -O https://raw.githubusercontent.com/LearningCircuit/local-deep-research/main/docker-compose.yml && docker compose up -d
+```
+
+### With NVIDIA GPU (Linux Only)
+
+For hardware-accelerated inference:
+
+```bash
+curl -O https://raw.githubusercontent.com/LearningCircuit/local-deep-research/main/docker-compose.yml && \
+curl -O https://raw.githubusercontent.com/LearningCircuit/local-deep-research/main/docker-compose.gpu.override.yml && \
+docker compose -f docker-compose.yml -f docker-compose.gpu.override.yml up -d
+```
+
+**Prerequisites for GPU:** Install the [NVIDIA Container Toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html) first. For step-by-step Ubuntu/Debian install commands, see the [Installation Guide](installation.md#docker-compose-recommended).
+
+Open http://localhost:5000 after ~30 seconds.
+
+## Using a Different Model
+
+Specify a model with the `LDR_LLM_MODEL` environment variable:
+
+```bash
+LDR_LLM_MODEL=gemma3:4b docker compose up -d
+```
+
+The model will be automatically pulled if not already available.
+
+## Configuration Options
+
+### docker-compose.yml
+
+The base configuration includes:
+
+| Service | Description |
+|---------|-------------|
+| `local-deep-research` | The main web application (port 5000) |
+| `ollama` | Local LLM inference engine |
+| `searxng` | Privacy-focused meta search engine |
+
+### Key Environment Variables
+
+Most settings can be configured through the web UI at http://localhost:5000/settings. Environment variables **override** UI settings and lock them. For the complete list of all environment variables and their defaults, see [CONFIGURATION.md](CONFIGURATION.md).
+
+> **⚠️ Warning:** Setting environment variables causes a hard override—the setting becomes read-only in the UI and cannot be changed until the environment variable is removed. For settings you may want to adjust later, use the web UI instead. Environment variables are best suited for deployment-specific values like `LDR_DATA_DIR` or API keys.
+>
+> **Note:** Empty environment variables (e.g., `LDR_LLM_PROVIDER=` or `LDR_LLM_PROVIDER=""`) are treated as **not set** — they will not override anything and the setting remains editable in the UI. This prevents Docker Compose templates with blank fields from accidentally locking settings. Only non-empty values act as overrides. Setting an API key to an empty string does **not** block or clear it — if a key exists in the database, it will still be used. To explicitly block a key, set it to any non-empty invalid value (e.g., `DISABLED`).
+
+| Variable | Description |
+|----------|-------------|
+| `LDR_WEB_HOST` | Bind address (default: `0.0.0.0` for Docker) |
+| `LDR_WEB_PORT` | Internal port (default: `5000`) |
+| `LDR_DATA_DIR` | Data directory (default: `/data`) |
+| `LDR_APP_ALLOW_REGISTRATIONS` | Allow new user registration (default: `true`). Set to `false` for public deployments after creating your initial account. |
+| `LDR_LLM_PROVIDER` | LLM provider (`ollama`, `openai`, `anthropic`, etc.) |
+| `LDR_LLM_MODEL` | Model name (e.g., `gemma3:12b`) |
+
+### Changing the External Port
+
+Use Docker's port mapping instead of environment variables:
+
+```yaml
+ports:
+  - "8080:5000"  # Expose on port 8080 instead of 5000
+```
+
+### Volume Mounts
+
+| Volume | Purpose |
+|--------|---------|
+| `ldr_data` | Application data |
+| `ldr_scripts` | Startup scripts |
+| `ldr_rag_cache` | RAG index cache |
+| `ollama_data` | Downloaded models |
+| `searxng_data` | Search engine config |
+
+### Resource Limits
+
+> **Warning:** Resource limits in the base `docker-compose.yml` are intentionally minimal:
+> - **`nofile` (file descriptors):** Not set. Docker's daemon default (typically 1M+) is appropriate. Setting a lower value can cause `unable to open database file` errors under load.
+> - **`memlock`:** Set to unlimited so SQLCipher's `mlock()` (a system call that prevents memory from being swapped to disk) can lock encryption keys in RAM when `cipher_memory_security` is enabled (opt-in, off by default).
+> - To customize, use a `docker-compose.override.yml` for your deployment.
+
+### Local Document Collections
+
+Use the **Collections** system in the Web UI to manage your local documents.
+Upload files directly through the Collections page — no volume mounts required.
+
+## Advanced: Cookie Cutter Configuration
+
+For more customization, use Cookie Cutter to generate a tailored docker-compose file:
+
+```bash
+# Install cookiecutter
+pip install --user cookiecutter
+
+# Clone the repository
+git clone https://github.com/LearningCircuit/local-deep-research.git
+cd local-deep-research
+
+# Generate custom configuration
+cookiecutter cookiecutter-docker/
+```
+
+Cookie Cutter will prompt you for:
+
+| Option | Description |
+|--------|-------------|
+| `config_name` | Name for your configuration |
+| `host_port` | Port to expose (default: 5000) |
+| `host_ip` | IP to bind (default: 0.0.0.0) |
+| `host_network` | Use host networking |
+| `enable_gpu` | Enable NVIDIA GPU support |
+| `enable_searxng` | Include SearXNG service |
+
+Then start with:
+
+```bash
+docker compose -f docker-compose.default.yml up -d
+```
+
+## Using External LLM Providers
+
+### OpenRouter (100+ Models)
+
+```yaml
+environment:
+  - LDR_LLM_PROVIDER=openai_endpoint
+  - LDR_LLM_OPENAI_ENDPOINT_URL=https://openrouter.ai/api/v1
+  - LDR_LLM_OPENAI_ENDPOINT_API_KEY=<your-api-key>
+  - LDR_LLM_MODEL=anthropic/claude-3.5-sonnet
+```
+
+### LM Studio (Running on Host)
+
+```yaml
+environment:
+  - LDR_LLM_PROVIDER=lmstudio
+  - LDR_LLM_LMSTUDIO_URL=http://host.docker.internal:1234/v1
+  # - LDR_LLM_LMSTUDIO_API_KEY=<api-key-if-required>   # optional; leave out for unauth instances
+  - LDR_LLM_MODEL=<your-loaded-model>
+```
+
+## Common Commands
+
+```bash
+# Start services
+docker compose up -d
+
+# Start with GPU support
+docker compose -f docker-compose.yml -f docker-compose.gpu.override.yml up -d
+
+# View logs
+docker compose logs -f
+
+# Stop services
+docker compose down
+
+# Update to latest version
+docker compose pull && docker compose up -d
+
+# Remove all data (fresh start)
+docker compose down -v
+```
+
+## Troubleshooting
+
+### Container won't start
+- Check logs: `docker compose logs local-deep-research`
+- Ensure port 5000 is available
+
+### Ollama model not loading
+- Check Ollama logs: `docker compose logs ollama`
+- Verify model name in `LDR_LLM_MODEL` environment variable
+- Ensure sufficient disk space for model download
+
+### GPU not detected
+- Verify NVIDIA drivers: `nvidia-smi`
+- Check container toolkit: `docker run --rm --gpus all nvidia/cuda:11.0-base nvidia-smi`
+
+## Related Documentation
+
+- [Installation Guide](installation.md)
+- [Environment Configuration](env_configuration.md)
+- [Full Configuration Reference](CONFIGURATION.md)
+- [SearXNG Setup](SearXNG-Setup.md)
+- [Unraid Deployment](deployment/unraid.md)
+- [Architecture - Resource Lifecycle](architecture.md#thread--resource-lifecycle)

docs/egress-modes.md

@@ -0,0 +1,149 @@
+# Egress modes — what each one does
+
+> ⚠️ **Experimental.** The egress boundary is new and ships as
+> defense-in-depth, not an absolute guarantee. It blocks the known data-egress
+> paths under the scope you pick, but this is an early version — don't rely on
+> it as your *only* protection for highly sensitive data yet. See the threat
+> model and limitations linked just below.
+
+LDR's **egress scope** (Settings → *Egress Scope*, or the *Privacy & Egress*
+panel on the research form) controls **where your research traffic is allowed
+to go** — which search engines run, whether your LLM/embeddings may be cloud
+services, and which URLs may be fetched. It's the single switch for *"how much
+of this run is allowed to leave my machine?"*
+
+Pick a mode below. The default is **Adaptive**, which just follows your primary
+search engine, so most people never need to think about it.
+
+> This page explains the modes for everyday use. For the threat model,
+> guarantees, and limitations, see
+> [`SECURITY.md`](../SECURITY.md#egress-policy-module) and the technical
+> [egress package README](../src/local_deep_research/security/egress/README.md).
+
+## At a glance
+
+| Mode | Search engines | LLM / embeddings | Best for |
+|---|---|---|---|
+| **Adaptive** *(default)* | follows your primary engine | forced local **only** when the run is private | "just do the sensible thing" |
+| **Unprotected** | any engine (no restriction) | any provider (cloud allowed) | escape hatch — **not recommended**; disables egress protection (hard SSRF / cloud-metadata blocking still applies) |
+| **Public only** | public web/academic engines only | your configured providers | public research; your local collections aren't touched |
+| **Private only** | local engines only (collections, local SearXNG/Ollama) | **forced local** — cloud blocked | sensitive work that must stay on the machine |
+| **Strict** | **only** your one primary engine | your configured providers | a single, exact source with zero expansion |
+
+---
+
+## <a id="adaptive"></a>🔀 Adaptive *(default)*
+
+**Adaptive follows your primary search engine** and resolves to a concrete
+mode for each run:
+
+- Primary is a **public** engine (e.g. SearXNG pointed at a public instance,
+  arXiv, PubMed) → behaves like **Public only**.
+- Primary is a **private** source (a local collection, your library) →
+  behaves like **Private only** (and therefore forces local LLM + embeddings).
+
+Why it's the default: you choose a search engine anyway, and the privacy
+posture "just matches" it. If you make a **private collection** your primary,
+the whole run automatically stays local — nothing leaves the box.
+
+> **Note:** to use a cloud LLM on a private collection, mark the collection
+> **public** (see below) or add the provider to *trusted inference providers*;
+> to drop all restrictions for one run, use **Unprotected**. Adaptive
+> deliberately narrows to match the primary.
+
+## <a id="unprotected"></a>🔓 Unprotected
+
+**Not recommended — an escape hatch.** Egress-scope restrictions are disabled
+for the run: any engine, URL, and LLM/embeddings provider is permitted. The
+hard SSRF and cloud-metadata blocks still apply. A loud, non-dismissible banner
+shows while it is active. Prefer marking a collection **public**, or adding a
+**trusted destination** (`policy.trusted_inference_providers` /
+`policy.trusted_search_engines`) for the specific case, over disabling
+protection wholesale.
+
+> The older **Both** scope (blanket-permit any classified engine) has been
+> **retired**: existing saved `both` configurations are migrated to **Adaptive**
+> (which follows your primary engine and forces local inference for a private
+> primary). If you relied on `both` to use a private collection with a cloud
+> model, mark that collection **public**, or choose **Unprotected** to opt out
+> of protection explicitly.
+
+## <a id="public_only"></a>☁️ Public only
+
+Only **public** web/academic engines run; your **local collections are
+excluded**. URL fetches are allowed to public hosts and blocked for private
+ones. Inference is whatever you configured (cloud allowed). Use it when you
+want public research and don't want your private documents queried at all.
+
+## <a id="private_only"></a>🔒 Private only
+
+The privacy mode. **Only local engines** run (collections, library, a local
+SearXNG/Ollama). Crucially, it **forces local LLM and embeddings** — cloud
+providers (OpenAI, Anthropic, Google, OpenRouter, …) are blocked, so your
+query and your retrieved documents never reach a cloud model. Public URL
+fetches are blocked, and a process-wide socket guard blocks stray outbound
+connections. **Nothing leaves the machine.**
+
+> If you have no local LLM configured, a Private-only run will refuse rather
+> than silently fall back to the cloud — that's intentional (fail-closed).
+
+## <a id="strict"></a>🎯 Strict
+
+The tightest mode: **only your single primary engine** runs — no expansion to
+any other engine at all. At the URL
+layer it behaves like Private-only (private hosts allowed, public blocked), but
+it does **not** force local inference — set the *Require local* toggles if you
+also want local LLM/embeddings.
+
+---
+
+## Per-collection public/private
+
+Each RAG collection has a **public/private** flag (default **private**):
+
+- A **private** collection is excluded under *Public only* / *Adaptive-public*,
+  and when used it forces local inference — its chunks never reach a cloud
+  model.
+- Mark a collection **public** (the *Public collection* checkbox when creating
+  it) only if its contents are non-sensitive and you're happy to process them
+  with cloud inference / use them under public scope.
+
+## The two local-inference toggles
+
+Independent of the scope, you can force local inference any time:
+
+- **Require local LLM endpoint** — refuse cloud LLM providers / non-local URLs.
+- **Require local embeddings** — refuse cloud embedders, and refuse a
+  HuggingFace download for an uncached local model.
+
+Both are **implied automatically** under *Private only* (and Adaptive-private),
+which is why those toggles auto-check and lock when you select Private only.
+
+---
+
+## Per-research overrides
+
+The three primary controls — **Egress Scope**, **Require local LLM endpoint**,
+and **Require local embeddings** — also appear on the research-form page as
+per-run dropdown / checkbox overrides. Values set there apply **only to that
+research run** and do **not** persist to the settings database, so you can do a
+one-off private run without changing your defaults.
+
+## Audit log
+
+Changes to any `policy.*` key, `llm.require_local_endpoint`,
+`llm.allowed_local_hostnames`, or `embeddings.require_local` emit a
+`policy_audit=True` log line so administrators can trace configuration changes.
+Those audit lines are deliberately filtered out of the WebSocket progress
+stream (they never reach browser subscribers).
+
+---
+
+### See also
+
+- [Configuration reference](CONFIGURATION.md#settings-list) — the exact setting
+  keys (`policy.egress_scope`, `llm.require_local_endpoint`,
+  `embeddings.require_local`, `llm.allowed_local_hostnames`) and their
+  auto-generated `LDR_*` environment variables.
+- [`SECURITY.md`](../SECURITY.md#egress-policy-module) — threat model,
+  guarantees, and caveats (including what this does **not** defend against).

docs/elasticsearch_search_engine.md

@@ -0,0 +1,154 @@
+# Elasticsearch Search Engine
+
+This document describes how to use and configure the Elasticsearch search engine within the Local Deep Research project.
+
+## Overview
+
+The Elasticsearch search engine allows you to search for documents within Elasticsearch indices. This is particularly useful for scenarios requiring searches across large volumes of structured or unstructured text data.
+
+## Prerequisites
+
+1.  A running Elasticsearch server (local or remote).
+2.  Elasticsearch Python client library: `elasticsearch>=8.10.0`.
+
+## Installation
+
+Ensure that you have installed the `elasticsearch` Python package. If you are installing Local Deep Research from source, you can run:
+
+```bash
+pip install elasticsearch>=8.10.0
+```
+
+## Basic Usage
+
+### Usage in Code
+
+```python
+from local_deep_research.web_search_engines.engines.search_engine_elasticsearch import ElasticsearchSearchEngine
+
+# Create search engine instance
+es_search = ElasticsearchSearchEngine(
+    hosts=["http://localhost:9200"],  # List of Elasticsearch hosts
+    index_name="my_index",            # Name of the index to search
+    username="user",                  # Optional: Authentication username
+    password="pass",                  # Optional: Authentication password
+    max_results=10                    # Maximum number of results to return
+)
+
+# Execute search
+results = es_search.run("Your search query")
+
+# Process results
+for result in results:
+    print(f"Title: {result.get('title')}")
+    print(f"Snippet: {result.get('snippet')}")
+    print(f"Content: {result.get('content')}")
+```
+
+### Advanced Search
+
+The ES search engine supports various advanced search methods:
+
+```python
+# Use Elasticsearch Query String syntax
+results = es_search.search_by_query_string("title:keyword AND content:search_text")
+
+# Use Elasticsearch DSL (Domain Specific Language)
+results = es_search.search_by_dsl({
+    "query": {
+        "bool": {
+            "must": {"match": {"content": "search term"}},
+            "filter": {"term": {"category": "technology"}}
+        }
+    }
+})
+```
+
+## Configuration
+
+### Search Engine Parameters
+
+| Parameter | Type | Default Value | Description |
+| :--- | :--- | :--- | :--- |
+| hosts | List[str] | ["http://localhost:9200"] | List of Elasticsearch server addresses |
+| index_name | str | "documents" | Name of the index to search |
+| username | Optional[str] | None | Authentication username |
+| password | Optional[str] | None | Authentication password |
+| api_key | Optional[str] | None | API Key authentication |
+| cloud_id | Optional[str] | None | Elastic Cloud ID |
+| max_results | int | 10 | Maximum number of results |
+| highlight_fields | List[str] | ["content", "title"] | Fields to highlight |
+| search_fields | List[str] | ["content", "title"] | Fields to search |
+| filter_query | Optional[Dict] | None | Optional filter query |
+| llm | Optional[BaseLLM] | None | LLM for relevance filtering |
+| max_filtered_results | Optional[int] | None | Maximum number of filtered results |
+
+## Indexing Data
+
+For ease of use, we provide the `ElasticsearchManager` utility class to help index data:
+
+```python
+from local_deep_research.utilities.es_utils import ElasticsearchManager
+
+# Create ES Manager
+es_manager = ElasticsearchManager(
+    hosts=["http://localhost:9200"]
+)
+
+# Create index
+es_manager.create_index("my_index")
+
+# Index a single document
+es_manager.index_document(
+    index_name="my_index",
+    document={
+        "title": "Document Title",
+        "content": "Document content...",
+    }
+)
+
+# Bulk index documents
+documents = [
+    {"title": "Document 1", "content": "Content 1"},
+    {"title": "Document 2", "content": "Content 2"}
+]
+es_manager.bulk_index_documents("my_index", documents)
+
+# Index file (automatically extract content)
+es_manager.index_file("my_index", "path/to/document.pdf")
+
+# Index all files in a directory
+es_manager.index_directory(
+    "my_index",
+    "path/to/docs",
+    file_patterns=["*.pdf", "*.docx", "*.txt"]
+)
+```
+
+## Examples
+
+Refer to `examples/elasticsearch_search_example.py` for a complete usage example.
+
+## Running Examples
+
+Ensure Elasticsearch is running, then execute:
+
+```bash
+python examples/elasticsearch_search_example.py
+```
+
+## Troubleshooting
+
+### Cannot connect to Elasticsearch
+
+-   Ensure the Elasticsearch server is running.
+-   Verify the host address and port.
+-   Verify authentication credentials (if required).
+-   Check network connection and firewall settings.
+
+### Empty Search Results
+
+-   Ensure the index exists and contains data.
+-   Check if the search fields are correct.
+-   Try using a simpler query.
+-   Check Elasticsearch logs for more information.

docs/elasticsearch_search_engine.zh.md

@@ -0,0 +1,154 @@
+# Elasticsearch 搜索引擎
+
+这个文档介绍如何在 Local Deep Research 项目中使用和配置 Elasticsearch 搜索引擎。
+
+## 概述
+
+Elasticsearch 搜索引擎允许你搜索 Elasticsearch 索引中的文档。这对于需要搜索大量结构化或非结构化文本数据的场景非常有用。
+
+## 前提条件
+
+1. 运行中的 Elasticsearch 服务器（本地或远程）
+2. Elasticsearch Python 客户端库：`elasticsearch>=8.10.0`
+
+## 安装
+
+确保你已经安装了 `elasticsearch` Python 包。如果你是从源代码安装 Local Deep Research，可以运行：
+
+```bash
+pip install elasticsearch>=8.10.0
+```
+
+## 基本用法
+
+### 在代码中使用
+
+```python
+from local_deep_research.web_search_engines.engines.search_engine_elasticsearch import ElasticsearchSearchEngine
+
+# 创建搜索引擎实例
+es_search = ElasticsearchSearchEngine(
+    hosts=["http://localhost:9200"],  # Elasticsearch 服务器地址
+    index_name="my_index",            # 要搜索的索引名称
+    username="user",                  # 可选：认证用户名
+    password="pass",                  # 可选：认证密码
+    max_results=10                    # 返回结果数量
+)
+
+# 执行搜索
+results = es_search.run("你的搜索查询")
+
+# 处理结果
+for result in results:
+    print(f"标题: {result.get('title')}")
+    print(f"片段: {result.get('snippet')}")
+    print(f"内容: {result.get('content')}")
+```
+
+### 高级搜索
+
+ES搜索引擎支持多种高级搜索方式：
+
+```python
+# 使用 Elasticsearch 查询字符串语法
+results = es_search.search_by_query_string("title:关键词 AND content:内容")
+
+# 使用 Elasticsearch DSL（领域特定语言）
+results = es_search.search_by_dsl({
+    "query": {
+        "bool": {
+            "must": {"match": {"content": "搜索词"}},
+            "filter": {"term": {"category": "技术"}}
+        }
+    }
+})
+```
+
+## 配置说明
+
+### 搜索引擎参数
+
+| 参数 | 类型 | 默认值 | 说明 |
+|------|------|--------|------|
+| hosts | List[str] | ["http://localhost:9200"] | Elasticsearch 服务器地址列表 |
+| index_name | str | "documents" | 要搜索的索引名称 |
+| username | Optional[str] | None | 认证用户名 |
+| password | Optional[str] | None | 认证密码 |
+| api_key | Optional[str] | None | API 密钥认证 |
+| cloud_id | Optional[str] | None | Elastic Cloud ID |
+| max_results | int | 10 | 最大结果数 |
+| highlight_fields | List[str] | ["content", "title"] | 要高亮显示的字段 |
+| search_fields | List[str] | ["content", "title"] | 要搜索的字段 |
+| filter_query | Optional[Dict] | None | 可选的过滤查询 |
+| llm | Optional[BaseLLM] | None | 用于相关性过滤的语言模型 |
+| max_filtered_results | Optional[int] | None | 过滤后的最大结果数 |
+
+## 索引数据
+
+为了便于使用，我们提供了 `ElasticsearchManager` 工具类来帮助索引数据：
+
+```python
+from local_deep_research.utilities.es_utils import ElasticsearchManager
+
+# 创建 ES 管理器
+es_manager = ElasticsearchManager(
+    hosts=["http://localhost:9200"]
+)
+
+# 创建索引
+es_manager.create_index("my_index")
+
+# 索引单个文档
+es_manager.index_document(
+    index_name="my_index",
+    document={
+        "title": "文档标题",
+        "content": "文档内容...",
+    }
+)
+
+# 批量索引文档
+documents = [
+    {"title": "文档1", "content": "内容1"},
+    {"title": "文档2", "content": "内容2"}
+]
+es_manager.bulk_index_documents("my_index", documents)
+
+# 索引文件(自动提取内容)
+es_manager.index_file("my_index", "path/to/document.pdf")
+
+# 索引整个目录中的文件
+es_manager.index_directory(
+    "my_index",
+    "path/to/docs",
+    file_patterns=["*.pdf", "*.docx", "*.txt"]
+)
+```
+
+## 示例
+
+查看 `examples/elasticsearch_search_example.py` 获取完整的使用示例。
+
+## 运行示例
+
+确保 Elasticsearch 正在运行，然后执行：
+
+```bash
+python examples/elasticsearch_search_example.py
+```
+
+## 故障排除
+
+### 无法连接到 Elasticsearch
+
+- 确保 Elasticsearch 服务器正在运行
+- 检查主机地址和端口是否正确
+- 验证认证凭据（如果需要）
+- 检查网络连接和防火墙设置
+
+### 搜索结果为空
+
+- 确保索引存在并且包含数据
+- 检查搜索字段是否正确
+- 尝试使用更简单的查询
+- 检查 Elasticsearch 日志获取更多信息

docs/enhancements/metrics/dashboard-improvements.md

@@ -0,0 +1,249 @@
+# Metrics Dashboard Enhancement Status
+
+## Current State (Implemented Features)
+
+The metrics dashboard now provides comprehensive analytics with:
+
+### ✅ Core Analytics
+- **Summary cards**: Total tokens, total researches, avg response time, success rate
+- **Expandable token breakdown**: Click "Total Tokens Used" to view detailed input/output splits
+- **Dual filtering system**: Time range (7D, 30D, 3M, 1Y, All) + Research mode (Quick Summary, Detailed, All)
+- **Response time tracking**: Converted from milliseconds to seconds for better readability
+- **Success rate monitoring**: Percentage of successful LLM calls
+
+### ✅ Advanced Token Analytics
+- **Per-research averages**: Input, output, and total tokens per research session
+- **Total usage breakdown**: Complete token consumption across all sessions
+- **Model usage analysis**: Detailed breakdown by model with token/call counts
+- **Time-series visualization**: Token consumption trends over time with interactive charts
+
+### ✅ Enhanced Data Collection
+- **Call stack tracking**: Function-level LLM usage tracking with file/function context
+- **Research mode filtering**: Separate analytics for quick vs detailed research types
+- **Phase-based tracking**: Token usage by research phase and search iteration
+- **Search engine tracking**: Monitor which search engines are being used
+
+### ✅ User Experience Improvements
+- **Interactive tooltips**: Contextual help throughout the dashboard
+- **Responsive filtering**: Real-time dashboard updates when changing filters
+- **Clean visual design**: Elegant expandable cards with smooth animations
+- **Empty state handling**: Graceful handling when no data is available
+
+## Pending Enhancement Proposals
+
+### 1. Time-Series Analytics 📈 *(Partially Implemented)*
+
+**✅ Completed:**
+- Basic time-series token usage visualization
+- Date range selectors (7D, 30D, 3M, 1Y, All)
+- Timestamp-based queries in token counter
+
+**🔄 Remaining:**
+- **Research frequency patterns** - Identify peak usage times and patterns
+- **Performance trends** - Monitor efficiency improvements/degradation over time
+- **Seasonal analysis** - Understand usage patterns across different time periods
+- **Hourly/daily heatmaps** - Visual patterns of usage intensity
+
+### 2. Cost Analysis 💰
+
+Transform token counts into actionable financial insights:
+- **Token costs per model** - Calculate actual spending based on model pricing
+- **Cost breakdown by research** - Show real $ spent per query
+- **Budget tracking** - Set and monitor spending limits with alerts
+- **Cost efficiency metrics** - ROI analysis for research investments
+- **Projected costs** - Estimate future spending based on usage trends
+
+**Implementation:**
+- Add pricing configuration for different models
+- Create cost calculation utilities
+- Build budget management interface
+- Add cost-based alerts and notifications
+
+### 3. Quality & Performance Metrics ⚡ *(Partially Implemented)*
+
+**✅ Completed:**
+- **Research success rates** - Monitor completion vs failure rates
+- **Average response time tracking** - LLM call response times in seconds
+- **Error rate tracking** - Monitor and analyze failure patterns
+
+**🔄 Remaining:**
+- **Average research duration** - Time-to-completion analytics for entire research sessions
+- **Token efficiency** - Quality of results per token spent
+- **Query complexity analysis** - Resource usage for simple vs complex queries
+- **Quality scoring mechanisms** - Automated research quality assessment
+
+### 4. Individual Research Deep Dive 🔍 *(Partially Implemented)*
+
+**✅ Completed:**
+- **Recent researches list** - Clickable list with token counts and timestamps
+- **Phase-based token tracking** - Usage by search, analysis, synthesis phases
+- **Model utilization tracking** - Which models used for what tasks
+
+**🔄 Remaining:**
+- **Per-Research Page** (`/metrics/research/{id}`):
+  - **Token usage timeline** - Visual flow of token consumption during research
+  - **Search engine performance** - Effectiveness of different engines for the query
+  - **Cost analysis** - Complete financial breakdown
+  - **Efficiency scoring** - Performance rating vs similar researches
+
+### 5. Comparative Analytics 📊
+
+Enable data-driven optimization decisions:
+- **Model performance comparison** - Efficiency metrics across different models
+- **Research type analysis** - Scientific vs general queries patterns
+- **A/B testing framework** - Compare different research strategies
+- **Benchmarking** - Performance against historical averages
+- **Best practices identification** - Highlight most efficient approaches
+
+**Implementation:**
+- Build comparison visualization tools
+- Add research categorization system
+- Create A/B testing infrastructure
+- Design benchmarking algorithms
+
+### 6. Resource Optimization Insights 🎯
+
+Automated recommendations for cost optimization:
+- **Waste detection** - Identify failed high-cost queries
+- **Model recommendations** - Suggest cheaper alternatives for similar tasks
+- **Usage anomalies** - Detect unusual patterns requiring attention
+- **Optimization suggestions** - Actionable insights for cost reduction
+- **Efficiency alerts** - Notify when performance degrades
+
+**Implementation:**
+- Create anomaly detection algorithms
+- Build recommendation engines
+- Add automated alert systems
+- Design optimization suggestion framework
+
+### 7. Advanced Visualizations 📉
+
+Enhanced visual analytics:
+- **Heatmaps** - Usage patterns by time of day/week
+- **Flow diagrams** - Token flow through research stages
+- **Comparative charts** - Side-by-side efficiency comparisons
+- **Interactive dashboards** - Drill-down capabilities
+- **Custom visualization builder** - User-configurable charts
+
+**Implementation:**
+- Integrate advanced charting libraries
+- Create interactive visualization components
+- Build custom chart configuration tools
+- Add dashboard personalization features
+
+### 8. Export and Reporting 📋
+
+Data portability and external integration:
+- **CSV/PDF exports** - Download metrics data
+- **Scheduled reports** - Automated weekly/monthly summaries
+- **Custom date range analysis** - Flexible reporting periods
+- **API endpoints** - Integration with external analytics tools
+- **Email notifications** - Automated report delivery
+
+**Implementation:**
+- Add export functionality to existing endpoints
+- Create report generation system
+- Build scheduling infrastructure
+- Design external API interface
+
+### 9. Real-time Monitoring 🔴
+
+Live system monitoring:
+- **Live token usage** - Real-time consumption tracking
+- **System health metrics** - API response times, error rates
+- **Resource alerts** - Threshold-based notifications
+- **Active research monitoring** - Track ongoing research progress
+- **Performance dashboards** - Real-time system status
+
+**Implementation:**
+- Add WebSocket-based real-time updates
+- Create system health monitoring
+- Build alert management system
+- Design real-time dashboard components
+
+### 10. Enhanced Model Analytics 📄
+
+Deep-dive model performance analysis:
+
+**Per-Model Page** (`/metrics/model/{model_name}`):
+- **Historical usage trends** - Model adoption over time
+- **Cost efficiency analysis** - Price/performance comparisons
+- **Task-specific performance** - Effectiveness for different research types
+- **Usage recommendations** - When to use this model
+- **Comparative benchmarks** - Performance vs other models
+
+**Implementation:**
+- Create model-specific analytics pages
+- Add task categorization system
+- Build performance comparison tools
+- Design recommendation algorithms
+
+## Technical Considerations
+
+### Database Enhancements
+- Add cost tracking columns to metrics tables
+- Implement time-series optimized storage
+- Create indexes for performance queries
+
+### Frontend Architecture
+- Modular component design for reusability
+- Responsive design for mobile accessibility
+- Progressive loading for large datasets
+
+### API Design
+- RESTful endpoints for all metrics data
+- Pagination for large result sets
+- Caching for frequently accessed data
+
+### Performance Optimization
+- Database query optimization
+- Client-side caching strategies
+- Lazy loading for complex visualizations
+
+## Recent Achievements Summary
+
+### December 2024 Major Release
+The metrics dashboard received a comprehensive overhaul with the following key features:
+
+1. **Dual Filtering System** 🎯
+   - Time range filtering: 7D, 30D, 3M, 1Y, All
+   - Research mode filtering: Quick Summary, Detailed, All
+   - Real-time dashboard updates when filters change
+
+2. **Expandable Token Analytics** 📊
+   - Interactive token breakdown card with click-to-expand functionality
+   - Detailed input/output token splits
+   - Average per research vs total usage comparison
+   - Smooth animations and elegant design
+
+3. **Enhanced Data Collection** 📈
+   - Comprehensive call stack tracking with file/function context
+   - Research phase and iteration tracking
+   - Response time monitoring (converted to human-readable seconds)
+   - Success rate calculation and display
+
+4. **Improved User Experience** ✨
+   - Interactive tooltips throughout the dashboard
+   - Clean, responsive design with smooth animations
+   - Graceful empty state handling
+   - Contextual help and explanations
+
+## Next Priority Features
+
+Based on current gaps, the highest value remaining enhancements are:
+
+1. **Cost Analysis** 💰 - Transform token usage into financial insights
+2. **Real-time Monitoring** 🔴 - Live system monitoring and alerts
+3. **Individual Research Pages** 🔍 - Detailed per-research analytics
+4. **Export and Reporting** 📋 - Data portability and automated reports
+
+## Success Metrics
+
+- **User Engagement**: Time spent on metrics pages
+- **Decision Impact**: Changes in model usage patterns
+- **Cost Savings**: Reduction in unnecessary token spending
+- **Efficiency Gains**: Improvement in tokens-per-successful-research ratio
+
+---
+
+*This document serves as a living specification for metrics dashboard enhancements. Each proposal should be evaluated for feasibility, user value, and implementation complexity before development.*

docs/enhancements/metrics/enhanced-token-tracking.md

@@ -0,0 +1,306 @@
+# Enhanced Token Tracking Data Collection
+
+## Current Implementation Status
+
+### ✅ **Implemented Features**
+- **Token Usage Tracking** - Basic token metrics fully operational
+- **Model Usage Tracking** - Model performance and usage patterns
+- **Star Reviews Analytics** - Comprehensive rating analytics with visualizations
+- **Metrics Dashboard Integration** - Full web UI with charts and data tables
+- **Database Schema** - Stable schema with fixed foreign key constraints
+
+### Current Token Tracking
+
+The existing system tracks basic token usage:
+- **research_id** - Links to specific research sessions
+- **model_name** - LLM model used
+- **provider** - Model provider (OpenAI, Anthropic, Ollama, etc.)
+- **prompt_tokens, completion_tokens, total_tokens** - Token usage counts
+- **timestamp** - When the LLM call was made
+
+**Note**: Database foreign key constraints were recently fixed to ensure all token tracking data is properly saved without constraint errors.
+
+## Proposed Enhanced Data Collection
+
+### ✅ **Already Available (Can Add Immediately)**
+
+#### Research Context
+- **research_query** - The original research question/query
+- **research_mode** - `"quick"` or `"detailed"` research mode
+- **research_phase** - Current stage of research:
+  - `init`, `setup`, `iteration_1`, `iteration_2`, etc.
+  - `question_generation`, `parallel_search`, `search_complete`
+  - `output_generation`, `report_generation`, `complete`, `error`
+
+#### Performance Metrics
+- **response_time_ms** - Duration of LLM call (wrap with timing)
+- **success_status** - `"success"`, `"error"`, `"timeout"`
+- **error_type** - Specific error if call failed
+
+#### Search Engine Context
+- **search_engines_planned** - Which engines were planned (from progress messages)
+- **search_engine_selected** - Which engine was actually used
+- **search_iteration** - Which search iteration this relates to
+
+### ⚠️ **Requires Minor Implementation Work**
+
+#### Cost Analysis
+- **estimated_cost_usd** - Calculated cost based on model pricing tables
+- **model_tier** - Classification: `"free"`, `"paid"`, `"premium"`
+
+#### Enhanced Performance
+- **query_length** - Character count of original research query
+- **tokens_per_second** - Processing speed metric (calculated from response_time_ms)
+
+#### Session Context
+- **session_id** - Unique session identifier for grouping related requests
+- **request_source** - Origin of request: `"web_ui"`, `"api"`, `"cli"`, `"benchmark"`
+
+### ❌ **Not Feasible with Current Architecture**
+
+#### Complex Quality Metrics
+- **response_quality_score** - Would require LLM evaluation system
+- **search_results_used** - Not directly tracked in current system
+- **result_relevance** - Would need sophisticated analysis
+
+#### Advanced Context
+- **retry_count** - Would require modifying LLM retry logic
+- **cache_hit** - No caching system currently implemented
+- **concurrent_requests** - Complex to track accurately
+
+## Implementation Details
+
+### Current Database Schema (Implemented)
+
+```sql
+-- TokenUsage table (current implementation)
+CREATE TABLE token_usage (
+    id INTEGER PRIMARY KEY,
+    research_id INTEGER,  -- Foreign key constraint removed to fix tracking
+    model_name TEXT,
+    provider TEXT,
+    prompt_tokens INTEGER,
+    completion_tokens INTEGER,
+    total_tokens INTEGER,
+    timestamp DATETIME DEFAULT CURRENT_TIMESTAMP
+);
+
+-- ModelUsage table (current implementation)
+CREATE TABLE model_usage (
+    id INTEGER PRIMARY KEY,
+    research_id INTEGER,  -- Foreign key constraint removed to fix tracking
+    model_name TEXT,
+    provider TEXT,
+    timestamp DATETIME DEFAULT CURRENT_TIMESTAMP
+);
+
+-- ResearchRating table (fully implemented)
+CREATE TABLE research_rating (
+    id INTEGER PRIMARY KEY,
+    research_id INTEGER NOT NULL,
+    rating INTEGER NOT NULL CHECK (rating BETWEEN 1 AND 5),
+    feedback TEXT,
+    timestamp DATETIME DEFAULT CURRENT_TIMESTAMP
+);
+```
+
+### Enhanced Database Schema (Proposed for Future)
+
+```sql
+-- Enhanced TokenUsage table
+CREATE TABLE enhanced_token_usage (
+    id INTEGER PRIMARY KEY,
+    research_id INTEGER,
+    model_name TEXT,
+    provider TEXT,
+
+    -- Existing token metrics
+    prompt_tokens INTEGER,
+    completion_tokens INTEGER,
+    total_tokens INTEGER,
+
+    -- NEW: Research context (available immediately)
+    research_query TEXT,
+    research_mode TEXT,
+    research_phase TEXT,
+    search_iteration INTEGER,
+
+    -- NEW: Performance metrics (available immediately)
+    response_time_ms INTEGER,
+    success_status TEXT DEFAULT 'success',
+    error_type TEXT,
+
+    -- NEW: Search engine context (available immediately)
+    search_engines_planned TEXT, -- JSON array
+    search_engine_selected TEXT,
+
+    -- NEW: Cost analysis (minor work required)
+    estimated_cost_usd REAL,
+    model_tier TEXT,
+
+    -- NEW: Enhanced metrics (minor work required)
+    query_length INTEGER,
+    tokens_per_second REAL,
+    session_id TEXT,
+    request_source TEXT,
+
+    -- Timing
+    timestamp DATETIME DEFAULT CURRENT_TIMESTAMP
+);
+```
+
+### Integration Points
+
+#### TokenCountingCallback Enhancement
+The callback can be enhanced to capture additional context:
+
+```python
+class TokenCountingCallback:
+    def __init__(self, research_id=None, research_context=None):
+        self.research_id = research_id
+        self.research_context = research_context or {}
+
+    def on_llm_start(self, serialized, prompts, **kwargs):
+        self.start_time = time.time()
+        # Extract research phase from context
+        # Extract search engine info from context
+
+    def on_llm_end(self, response, **kwargs):
+        self.response_time = (time.time() - self.start_time) * 1000
+        # Calculate additional metrics
+        # Save enhanced data to database
+```
+
+#### Progress Message Integration
+Extract search engine information from existing progress messages:
+
+```python
+# From research progress system
+if "SEARCH_PLAN:" in message:
+    engines = message.split("SEARCH_PLAN:")[1].strip()
+    research_context["planned_engines"] = engines
+
+if "ENGINE_SELECTED:" in message:
+    engine = message.split("ENGINE_SELECTED:")[1].strip()
+    research_context["selected_engine"] = engine
+```
+
+#### Cost Calculation System
+Add model pricing configuration:
+
+```python
+MODEL_PRICING = {
+    "gpt-4": {"prompt": 0.03, "completion": 0.06},  # per 1K tokens
+    "gpt-3.5-turbo": {"prompt": 0.001, "completion": 0.002},
+    "claude-3-opus": {"prompt": 0.015, "completion": 0.075},
+    # ... other models
+}
+
+def calculate_cost(model_name, prompt_tokens, completion_tokens):
+    if model_name in MODEL_PRICING:
+        pricing = MODEL_PRICING[model_name]
+        prompt_cost = (prompt_tokens / 1000) * pricing["prompt"]
+        completion_cost = (completion_tokens / 1000) * pricing["completion"]
+        return prompt_cost + completion_cost
+    return 0.0
+```
+
+## Benefits of Enhanced Tracking
+
+### Immediate Value
+- **Cost visibility** - See actual spending per research and model
+- **Performance monitoring** - Track response times and success rates
+- **Phase analysis** - Understand where tokens are consumed in research process
+- **Search optimization** - Identify most effective search engines
+
+### Analytics Possibilities
+- **Cost optimization** - Identify expensive queries and suggest alternatives
+- **Performance trends** - Monitor system performance over time
+- **Usage patterns** - Understand how different research modes perform
+- **Model comparison** - Compare efficiency across different models
+
+### Future Dashboard Features
+- **Cost breakdown charts** - Visual cost analysis per model/research
+- **Performance timeline** - Response time trends
+- **Phase efficiency** - Token usage by research stage
+- **Search engine effectiveness** - Success rates by engine
+
+## Implementation Priority
+
+### ✅ Phase 1: Core Infrastructure (COMPLETED)
+1. ✅ Fixed database foreign key constraints to enable token tracking
+2. ✅ Implemented star reviews analytics system
+3. ✅ Created comprehensive metrics dashboard with visualizations
+4. ✅ Added SQLAlchemy ORM-based data access patterns
+5. ✅ Integrated Chart.js for data visualization
+6. ✅ Added responsive design with dark theme integration
+
+### Phase 2: Enhanced Token Tracking (NEXT)
+1. Add research context (query, mode, phase)
+2. Add response timing
+3. Add success/error tracking
+4. Add search engine context from progress messages
+
+### Phase 3: Cost Analysis
+1. Implement model pricing configuration
+2. Add cost calculation to token tracking
+3. Add model tier classification
+
+### Phase 4: Advanced Features
+1. Add session tracking
+2. Add request source detection
+3. Implement calculated metrics (tokens_per_second, etc.)
+
+## Data Sources
+
+### Research Service Context
+- Research query, mode, and ID available in research service
+- Phase information from progress callback system
+- Session context from Flask request
+
+### Progress Message System
+- Search engine planning and selection
+- Research iteration tracking
+- Error states and completion status
+
+### LLM Integration Points
+- Token counts from LLM responses
+- Model and provider information
+- Response timing from callback wrapper
+
+## Recent Implementation Work
+
+### Star Reviews Analytics (December 2024)
+- **Complete implementation** of dedicated star reviews page at `/metrics/star-reviews`
+- **API endpoints** for real-time data aggregation using SQLAlchemy ORM
+- **Interactive visualizations** including:
+  - LLM model performance bar charts
+  - Search engine effectiveness charts
+  - Rating distribution over time
+  - Research analytics with clickable links to research details
+- **Dark theme integration** with responsive design
+- **Comprehensive testing** with Puppeteer UI automation
+
+### Database Fixes (December 2024)
+- **Critical fix**: Removed foreign key constraints from TokenUsage and ModelUsage tables
+- **Problem**: Foreign keys referenced non-existent "research_history" table causing silent failures
+- **Solution**: Modified `src/local_deep_research/metrics/db_models.py` to use simple integer fields
+- **Result**: Token tracking now works correctly without constraint errors
+
+### Files Modified
+- `src/local_deep_research/web/routes/metrics_routes.py` - Star reviews routes and API
+- `src/local_deep_research/web/templates/pages/star_reviews.html` - Complete analytics UI
+- `src/local_deep_research/web/templates/pages/metrics.html` - Navigation integration
+- `src/local_deep_research/metrics/db_models.py` - Database schema fixes
+- `tests/ui_tests/test_star_reviews_debug.js` - Comprehensive UI testing
+
+### Current Status
+- ✅ Token tracking fully operational
+- ✅ Star reviews analytics complete and deployed
+- ✅ Database schema stable and constraint-free
+- ✅ Web UI responsive and functional
+- ✅ Testing infrastructure in place
+
+---
+
+*This enhanced tracking system provides significantly more insight into research performance, costs, and patterns while remaining feasible to implement with the current architecture. The core infrastructure is now complete and ready for Phase 2 enhancements.*

docs/env_configuration.md

@@ -0,0 +1,262 @@
+# Configuring Local Deep Research with Environment Variables
+
+> **Note:** For most users, the **Web UI Settings** is the recommended way to configure Local Deep Research. Environment variables are primarily useful for Docker deployments, CI/CD pipelines, and server configurations where the web UI is not accessible during startup.
+>
+> For a complete auto-generated reference of **all** settings, defaults, and environment variables, see [CONFIGURATION.md](CONFIGURATION.md).
+
+You can override any configuration setting in Local Deep Research using environment variables. This is useful for:
+
+- Setting up multiple environments (development, production)
+- Changing settings without modifying configuration files
+- Providing sensitive information like API keys securely
+- Setting server ports for Docker or cloud deployments
+
+## Environment Variable Format
+
+To override a setting, convert its key to uppercase, replace dots with underscores, and prefix with `LDR_`:
+
+```
+setting.key.name  →  LDR_SETTING_KEY_NAME
+```
+
+For example:
+- `app.debug` → `LDR_APP_DEBUG`
+- `llm.model` → `LDR_LLM_MODEL`
+- `search.tool` → `LDR_SEARCH_TOOL`
+
+> **Important:** Environment variables **override** UI settings and lock them — the setting becomes read-only in the UI until the environment variable is removed. For settings you may want to adjust later, use the Web UI instead.
+>
+> **Note on empty values:** Empty environment variables (e.g., `LDR_LLM_PROVIDER=""`) are treated as **not set** — they will not override anything and the setting remains editable in the UI. This is by design: deployment tools like Unraid and Docker Compose templates often create all environment variables even when fields are left blank. Only non-empty values act as overrides. To remove an override, delete the environment variable entirely rather than setting it to an empty string. To explicitly block a setting, set it to any non-empty invalid value (e.g., `DISABLED`).
+
+For the complete list of all settings, their environment variable names, and default values, see [CONFIGURATION.md](CONFIGURATION.md).
+
+## API Keys
+
+API keys are best set using environment variables for security. Only the `LDR_` prefixed version is needed.
+
+> **Security note:** Setting an API key variable to an empty string (e.g., `LDR_LLM_OPENAI_API_KEY=""`) does **not** block or clear the key — it is treated as unset, and the key remains editable in the UI. If a key is already stored in the database, it will still be used. To explicitly block a key, set it to any non-empty invalid value (e.g., `DISABLED`).
+
+```bash
+# LLM API keys
+export LDR_LLM_OPENAI_API_KEY=your-openai-key-here
+export LDR_LLM_ANTHROPIC_API_KEY=your-anthropic-key-here
+export LDR_LLM_OPENROUTER_API_KEY=your-openrouter-key-here
+
+# Search engine API keys
+export LDR_SEARCH_ENGINE_WEB_BRAVE_API_KEY=your-brave-key-here
+export LDR_SEARCH_ENGINE_WEB_SERPAPI_API_KEY=your-serpapi-key-here
+export LDR_SEARCH_ENGINE_WEB_TAVILY_API_KEY=your-tavily-key-here
+```
+
+For the full list of API key environment variables, see [CONFIGURATION.md](CONFIGURATION.md).
+
+## LLM Provider Configuration
+
+### OpenRouter
+
+[OpenRouter](https://openrouter.ai/) provides access to 100+ models through an OpenAI-compatible API. To use OpenRouter:
+
+1. Get an API key from [openrouter.ai](https://openrouter.ai/)
+2. Configure using one of these methods:
+
+**Method 1: Via Web UI (Recommended)**
+- Navigate to Settings → LLM Provider
+- Select "OpenAI-Compatible Endpoint"
+- Set Endpoint URL to: `https://openrouter.ai/api/v1`
+- Enter your OpenRouter API key
+- Select your desired model
+
+**Method 2: Via Environment Variables**
+
+```bash
+# Required environment variables for OpenRouter
+export LDR_LLM_PROVIDER=openai_endpoint
+export LDR_LLM_OPENAI_ENDPOINT_URL=https://openrouter.ai/api/v1
+export LDR_LLM_OPENAI_ENDPOINT_API_KEY="<your-api-key>"
+export LDR_LLM_MODEL=anthropic/claude-3.5-sonnet  # or any OpenRouter model
+```
+
+**Method 3: Docker Compose**
+
+Add to your `docker-compose.yml` environment section:
+
+```yaml
+services:
+  local-deep-research:
+    environment:
+      - LDR_LLM_PROVIDER=openai_endpoint
+      - LDR_LLM_OPENAI_ENDPOINT_URL=https://openrouter.ai/api/v1
+      - LDR_LLM_OPENAI_ENDPOINT_API_KEY=<your-api-key>
+      - LDR_LLM_MODEL=anthropic/claude-3.5-sonnet
+```
+
+**Available Models**: Browse models at [openrouter.ai/models](https://openrouter.ai/models)
+
+**Note**: OpenRouter uses the OpenAI-compatible API, so you select "OpenAI-Compatible Endpoint" as the provider and change the endpoint URL to OpenRouter's API.
+
+### Other OpenAI-Compatible Providers
+
+The same configuration pattern works for any OpenAI-compatible API service:
+
+```bash
+# Generic pattern for OpenAI-compatible APIs
+export LDR_LLM_PROVIDER=openai_endpoint
+export LDR_LLM_OPENAI_ENDPOINT_URL=https://your-provider.com/v1
+export LDR_LLM_OPENAI_ENDPOINT_API_KEY="<your-api-key>"
+export LDR_LLM_MODEL="<your-model-name>"
+```
+
+## Docker Usage
+
+For Docker deployments, you can pass environment variables when starting containers:
+
+```bash
+docker run -p 5000:5000 \
+  -e LDR_LLM_OPENAI_API_KEY=your-api-key-here \
+  -e LDR_SEARCH_TOOL=wikipedia \
+  local-deep-research
+```
+
+## Migrating from server_config.json
+
+The `server_config.json` file is deprecated and will be removed in a future release. Migrate your settings to environment variables using this mapping:
+
+| server_config.json key | Environment Variable |
+|------------------------|---------------------|
+| `host` | `LDR_WEB_HOST` |
+| `port` | `LDR_WEB_PORT` |
+| `debug` | `LDR_APP_DEBUG` |
+| `use_https` | `LDR_WEB_USE_HTTPS` |
+| `allow_registrations` | `LDR_APP_ALLOW_REGISTRATIONS` |
+| `rate_limit_default` | `LDR_SECURITY_RATE_LIMIT_DEFAULT` |
+| `rate_limit_login` | `LDR_SECURITY_RATE_LIMIT_LOGIN` |
+| `rate_limit_registration` | `LDR_SECURITY_RATE_LIMIT_REGISTRATION` |
+| `rate_limit_settings` | `LDR_SECURITY_RATE_LIMIT_SETTINGS` |
+
+After setting the environment variables, delete `server_config.json` from your data directory. The web UI will show a warning banner while the file still exists.
+
+## Common Operations
+
+### Changing the Web Port
+
+```bash
+export LDR_WEB_PORT=8080  # Linux/Mac
+set LDR_WEB_PORT=8080     # Windows
+```
+
+> **Note:** `LDR_APP_PORT` is a separate setting for notification URL generation. To change the port the server listens on, use `LDR_WEB_PORT`. Requires server restart.
+
+### Setting API Keys
+
+```bash
+# Linux/Mac
+export LDR_LLM_ANTHROPIC_API_KEY=your-api-key-here
+
+# Windows
+set LDR_LLM_ANTHROPIC_API_KEY=your-api-key-here
+```
+
+### Changing Search Engine
+
+```bash
+export LDR_SEARCH_TOOL=wikipedia  # Linux/Mac
+set LDR_SEARCH_TOOL=wikipedia     # Windows
+```
+
+### Data Directory Location
+
+By default, Local Deep Research stores all data (database, research outputs, cache, logs) in platform-specific user directories. You can override this location using the `LDR_DATA_DIR` environment variable:
+
+```bash
+# Linux/Mac
+export LDR_DATA_DIR=/path/to/your/data/directory
+
+# Windows
+set LDR_DATA_DIR=C:\path\to\your\data\directory
+```
+
+All application data will be organized under this directory:
+- `$LDR_DATA_DIR/ldr.db` - Application database
+- `$LDR_DATA_DIR/research_outputs/` - Research reports
+- `$LDR_DATA_DIR/cache/` - Cached data
+- `$LDR_DATA_DIR/logs/` - Application logs
+
+### Debug Logging (`LDR_APP_DEBUG` and `LDR_LOGURU_DIAGNOSE`)
+
+`LDR_APP_DEBUG=true` raises the log level to `DEBUG` for more informative output. It does **not**, on its own, enable Loguru's `diagnose` mode, which renders the `repr()` of every local variable in every traceback frame on exceptions and can materialise credentials (API keys, the SQLCipher master password, `Authorization` headers) into your logs.
+
+To opt into that variable-level detail, set `LDR_LOGURU_DIAGNOSE=true` **in addition to** `LDR_APP_DEBUG=true`. Keep it disabled in production; a warning is emitted whenever it is active.
+
+```bash
+# Verbose logs, no local-variable dumps (safe default for debugging)
+export LDR_APP_DEBUG=true
+
+# Full exception diagnostics including frame locals (NEVER in production)
+export LDR_APP_DEBUG=true
+export LDR_LOGURU_DIAGNOSE=true
+```
+
+### Database Configuration (SQLCipher)
+
+Database encryption settings are configured exclusively via environment variables (they cannot be changed through the Web UI). These settings are applied at database creation time and must remain consistent for the database to be accessible.
+
+For the full list of database configuration variables, defaults, constraints, and deprecated aliases, see the **Pre-Database (Env-Only) Settings** section in [CONFIGURATION.md](CONFIGURATION.md#pre-database-env-only-settings).
+
+### Upload Size Limit
+
+The per-file upload cap is read from `LDR_SECURITY_UPLOAD_MAX_FILE_SIZE_MB` at startup. The value is in **megabytes**; the default is **3072 MB (3 GB)**. Values must be positive integers; zero, negative, or non-integer values fall back to the default with a warning.
+
+```bash
+# Allow up to 5 GB per file
+export LDR_SECURITY_UPLOAD_MAX_FILE_SIZE_MB=5120
+
+# Tighten to 100 MB
+export LDR_SECURITY_UPLOAD_MAX_FILE_SIZE_MB=100
+```
+
+This cap applies to both the research-upload and RAG-collection upload endpoints. A separate library-side setting (`research_library.max_pdf_size_mb`) controls whether a PDF that passes the upload cap can also be *stored* in the library — both default to 3 GB and should typically be raised or lowered together.
+
+Memory usage stays bounded regardless of the cap: multipart uploads are spooled to disk above a 5 MB threshold (`DiskSpoolingRequest.max_form_memory_size`), so the per-file cap does not directly affect RAM consumption.
+
+Requires a server restart to take effect.
+
+### CORS / WebSocket Security
+
+These settings control Cross-Origin Resource Sharing (CORS) for API routes and WebSocket connections. For the full list of security environment variables and their defaults, see [CONFIGURATION.md](CONFIGURATION.md#pre-database-env-only-settings).
+
+WebSocket connections also require an authenticated session in addition to passing the CORS check below. The CORS setting controls *which origins* may attempt a handshake; the auth requirement ensures only logged-in users can complete one.
+
+**Values:**
+- `*` — Allow all origins (most permissive)
+- Empty string or unset — Same-origin only (most restrictive)
+- Comma-separated list — Allow specific origins only
+
+**Examples:**
+
+```bash
+# Same-origin only (the default — leave both unset, shown here for clarity)
+export LDR_SECURITY_WEBSOCKET_ALLOWED_ORIGINS=""
+
+# Allow specific cross-origin front-ends (recommended when you need cross-origin)
+export LDR_SECURITY_CORS_ALLOWED_ORIGINS="https://example.com,https://app.example.com"
+export LDR_SECURITY_WEBSOCKET_ALLOWED_ORIGINS="https://example.com,https://app.example.com"
+
+# Allow ALL origins — disables CORS/WebSocket origin checks.
+# Trusted local/dev networks only; do not use in production.
+export LDR_SECURITY_CORS_ALLOWED_ORIGINS="*"
+export LDR_SECURITY_WEBSOCKET_ALLOWED_ORIGINS="*"
+```
+
+**Docker Compose example:**
+
+```yaml
+services:
+  local-deep-research:
+    environment:
+      # Same-origin is the secure default — omit these unless you serve the UI
+      # from a different origin, then list it explicitly (avoid "*").
+      - LDR_SECURITY_CORS_ALLOWED_ORIGINS=https://app.example.com
+      - LDR_SECURITY_WEBSOCKET_ALLOWED_ORIGINS=https://app.example.com
+```
+
+**Note:** If WebSocket connections fail after upgrading to the same-origin default, the usual cause is a TLS-terminating reverse proxy that doesn't forward `X-Forwarded-Proto` (see the Socket.IO proxy snippet in [troubleshooting](troubleshooting.md) — the same header is needed for secure cookies/HSTS). Otherwise, set `LDR_SECURITY_WEBSOCKET_ALLOWED_ORIGINS` to your front-end origin (e.g. `https://app.example.com`); a fully cross-origin front-end must also set `LDR_SECURITY_CORS_ALLOWED_ORIGINS`. Using `*` re-enables allow-all and is not recommended outside trusted local/dev networks.

docs/faq.md

@@ -0,0 +1,577 @@
+# Frequently Asked Questions (FAQ)
+
+> **Note**: This documentation is maintained by the community and may contain inaccuracies. While we strive to keep it up-to-date, please verify critical information and report any errors via [GitHub Issues](https://github.com/LearningCircuit/local-deep-research/issues).
+
+## Table of Contents
+
+1. [General Questions](#general-questions)
+2. [Installation & Setup](#installation--setup)
+3. [Configuration](#configuration)
+4. [Common Errors](#common-errors)
+5. [Search Engines](#search-engines)
+6. [LLM Configuration](#llm-configuration)
+7. [Local Document Search](#local-document-search)
+8. [Performance & Optimization](#performance--optimization)
+9. [Docker Issues](#docker-issues)
+10. [Platform-Specific Issues](#platform-specific-issues)
+
+## General Questions
+
+### What is Local Deep Research (LDR)?
+
+LDR is an open-source AI research assistant that performs systematic research by breaking down complex questions, searching multiple sources in parallel, and creating comprehensive reports with proper citations. It can run entirely locally for complete privacy.
+
+### How is LDR different from ChatGPT or other AI assistants?
+
+LDR focuses specifically on research with real-time information retrieval. Key differences:
+- Provides citations and sources for claims
+- Searches multiple databases including academic papers
+- Can run completely offline with local models
+- Open source and customizable
+- Searches your own documents
+
+### Is LDR really free?
+
+Yes! LDR is open source (MIT license). Costs only apply if you:
+- Use cloud LLM providers (OpenAI, Anthropic)
+- Use premium search APIs (Tavily, SerpAPI)
+- Need cloud hosting infrastructure
+
+Local models (Ollama) and free search engines have no costs.
+
+### Can I use LDR completely offline?
+
+Partially. You can:
+- Use local LLMs (Ollama) offline
+- Search local documents offline
+- But web search requires internet
+
+For intranet/offline environments, configure LDR to use only local documents and disable web search.
+
+## Chat Mode
+
+> **Experimental** — interface and behavior may change before GA.
+
+### What's the difference between Chat Mode and submitting a research query on the home page?
+
+Chat Mode is for multi-turn conversations where each question builds on previous answers — the session accumulates entities, topics, and sources across the whole conversation. A single query on the home page starts fresh each time. Use Chat Mode if you want to explore a topic progressively; use single queries for one-off lookups.
+
+For details, see [Chat Mode in features.md](features.md#chat-mode).
+
+### Does Chat Mode use the same settings as regular research mode?
+
+Yes — same LLM and search engines. But chat always runs in "quick" mode (1 iteration). Three chat-specific settings tune context depth and title generation: in Settings, on the **All Settings** tab, click the **Chat** section header to expand it. There you'll find `chat.max_findings_to_include`, `chat.llm_title_generation`, and `chat.title_llm_timeout_seconds` (hard wall-clock timeout for the title-generation LLM call so a slow endpoint can't block title generation).
+
+### Can I save or export conversations from Chat Mode?
+
+Yes. Sessions persist across logouts in your per-user database — you can archive, reactivate, or permanently delete them via the UI. To export a conversation, click the **Export** button in the chat header to download the session as a Markdown file.
+
+## Installation & Setup
+
+### What are the system requirements?
+
+- **Python**: 3.10 or newer
+- **RAM**: 8GB minimum (16GB recommended for larger models)
+- **GPU VRAM** (for Ollama):
+  - 7B models: 4GB VRAM minimum
+  - 13B models: 8GB VRAM minimum
+  - 30B models: 16GB VRAM minimum
+  - 70B models: 48GB VRAM minimum
+- **Disk Space**:
+  - 100MB for LDR
+  - 1-2GB for SearXNG
+  - 5-15GB per Ollama model
+- **OS**: Windows, macOS, Linux
+
+### Do I need Docker?
+
+Docker is recommended but not required. You can:
+- Use Docker Compose (easiest)
+- Use Docker containers individually
+- Install via pip without Docker
+
+### Which installation method should I use?
+
+- **Docker Compose**: Best for production use
+- **Docker**: Good for quick testing
+- **Pip package**: Best for development or Python integration
+
+### How do I set up SearXNG?
+
+SearXNG is a privacy-respecting metasearch engine. Learn more at the [SearXNG repository](https://github.com/searxng/searxng).
+
+```bash
+docker pull searxng/searxng
+docker run -d -p 8080:8080 --name searxng searxng/searxng
+```
+
+Then set the URL to `http://localhost:8080` in LDR settings.
+
+### The cookiecutter command fails on Windows
+
+For Windows users, you can use the generated docker-compose file directly instead of running cookiecutter:
+```yaml
+services:
+  local-deep-research:
+    build:
+      context: .
+      dockerfile: Dockerfile
+    ports:
+      - "5000:5000"
+    environment:
+      - SEARXNG_URL=http://searxng:8080
+    depends_on:
+      - searxng
+
+  searxng:
+    image: searxng/searxng:latest
+    ports:
+      - "8080:8080"
+```
+
+## Configuration
+
+### How do I change the LLM model?
+
+1. **Via Web UI**: Settings → LLM Provider → Select model
+2. **Via Environment**: Set `LDR_LLM_MODEL` and `LDR_LLM_PROVIDER`
+3. **Via API**: Pass model parameters in requests
+
+### Where should I configure settings?
+
+**Important**: The `.env` file method is deprecated. Use the web UI settings instead:
+1. Run the web app: `python -m local_deep_research.web.app`
+2. Navigate to Settings
+3. Configure your preferences
+4. Settings are saved to the database
+
+For a complete reference of all settings, defaults, and environment variables, see [CONFIGURATION.md](CONFIGURATION.md).
+
+### How do I download Ollama models in Docker?
+
+**Note**: If you use cookiecutter with Ollama, it will automatically download an initial model that you specify during setup.
+
+To manually download additional models:
+```bash
+# Connect to the Ollama container
+docker exec -it ollama ollama pull llama3:8b
+
+# Or if using docker-compose
+docker-compose exec ollama ollama pull llama3:8b
+```
+
+### Which Ollama model should I use?
+
+Recommended models:
+- **Best quality**: `llama3:70b` (requires 48GB+ VRAM)
+- **Balanced**: `gemma3:12b` (good quality/speed trade-off)
+- **Fastest**: `llama3:8b`, `mistral:7b`, or `gemma:7b`
+
+For data-driven picks, see the community-maintained **[LDR Benchmarks dataset on Hugging Face](https://huggingface.co/datasets/local-deep-research/ldr-benchmarks)** — accuracy results submitted by other LDR users across local and cloud models, sortable by model, search engine, and strategy. Useful before downloading multi-GB weights.
+
+## Common Errors
+
+### "Error: max_workers must be greater than 0"
+
+This means LDR cannot connect to your LLM. Check:
+1. Ollama is running: `ollama list`
+2. You have models downloaded: `ollama pull llama3:8b`
+3. Correct model name in settings
+4. For Docker: Ensure containers can communicate
+
+### "No module named 'local_deep_research'"
+
+Reinstall the package:
+```bash
+pip uninstall local-deep-research
+pip install local-deep-research
+```
+
+### "404 Error" when viewing results
+
+This issue should be resolved in versions 0.5.2 and later. If you're still experiencing it:
+1. Refresh the page
+2. Check if research actually completed in logs
+3. Update to the latest version
+
+### Research gets stuck or shows empty headings
+
+Common causes:
+- "Search snippets only" disabled (must be enabled for SearXNG)
+- Rate limiting from search engines
+- LLM connection issues
+
+Solutions:
+1. Reset settings to defaults
+2. Use fewer iterations (2-3)
+3. Limit questions per iteration (3-4)
+
+### "'str' object has no attribute 'items'"
+
+This issue should be fixed in recent versions. If you encounter it, ensure you're using the correct environment variable format. Remove deprecated variables:
+- `LDR_SEARCH_ENGINE_WEB`
+- `LDR_SEARCH_ENGINE_AUTO`
+- `LDR_SEARCH_ENGINE_DEFAULT`
+
+Use `LDR_SEARCH_TOOL` instead if needed.
+
+### Chinese / Japanese / Korean text is missing from exported PDFs
+
+PDF export uses WeasyPrint, which resolves glyphs through the host's installed fonts. If your system has no CJK font installed, those characters disappear silently from the PDF even though they render fine in the browser. Install a CJK font package:
+
+- **Debian/Ubuntu:** `sudo apt install fonts-noto-cjk && fc-cache -fv`
+- **Fedora/RHEL:** `sudo dnf install google-noto-sans-cjk-fonts && fc-cache -fv`
+- **Alpine:** `apk add font-noto-cjk`
+- **macOS / Windows:** CJK fonts ship with the OS — no install needed.
+- **Docker (official image):** `fonts-noto-cjk` is bundled, no action needed.
+
+After installing, restart LDR and re-export the PDF.
+
+### Emojis show up as empty boxes in exported PDFs
+
+Emojis in the markdown are routed through the host's emoji font by WeasyPrint. If the system has no emoji font installed, each emoji codepoint renders as an empty box ("tofu") in the PDF. Install an emoji font package:
+
+- **Debian/Ubuntu:** `sudo apt install fonts-noto-color-emoji && fc-cache -fv`
+- **Fedora/RHEL 10+:** `sudo dnf install google-noto-color-emoji-fonts && fc-cache -fv`
+- **RHEL 9 / CentOS Stream 9:** `sudo dnf install google-noto-emoji-color-fonts && fc-cache -fv` (package was renamed to `google-noto-color-emoji-fonts` in EL10)
+- **Alpine:** `apk add font-noto-emoji` (the package name omits "color" but ships the color `NotoColorEmoji.ttf`)
+- **macOS / Windows:** emoji fonts ship with the OS — no install needed.
+- **Docker (official image):** `fonts-noto-color-emoji` is bundled, no action needed.
+
+After installing, restart LDR and re-export the PDF.
+
+## Search Engines
+
+### SearXNG connection errors
+
+1. **Verify SearXNG is running**:
+   ```bash
+   docker ps | grep searxng
+   curl http://localhost:8080
+   ```
+
+2. **For Docker networking issues**:
+   - Use `http://searxng:8080` (container name) not `localhost`
+   - Or use `--network host` mode
+
+3. **Check browser access**: Navigate to `http://localhost:8080`
+
+### Rate limit errors
+
+Solutions:
+1. Check status: `python -m local_deep_research.web_search_engines.rate_limiting status`
+2. Reset limits: `python -m local_deep_research.web_search_engines.rate_limiting reset`
+3. Use the langgraph-agent strategy (the default), which can route around rate-limited engines
+4. Add premium search engines
+
+### "Invalid value" errors from SearXNG
+
+Ensure "Search snippets only" is enabled in settings. This is required for SearXNG.
+
+### Captcha errors
+
+Some search engines detect bot activity. Solutions:
+- Use SearXNG instead of direct search engines
+- Add delays between searches
+- Use premium APIs (Tavily, SerpAPI)
+
+## LLM Configuration
+
+### Cannot connect to Ollama
+
+1. **Verify Ollama installation**:
+   ```bash
+   ollama --version
+   ollama list
+   ```
+
+2. **For Docker**: Use correct URL
+   - From host: `http://localhost:11434`
+   - From container: `http://ollama:11434` or `http://host.docker.internal:11434`
+
+### LM Studio connection issues
+
+LM Studio runs on your host machine, but Docker containers can't reach `localhost` (it refers to the container itself). If you see "Model 1" / "Model 2" instead of actual models, this is why.
+
+**Mac/Windows (Docker Desktop):**
+- Use `http://host.docker.internal:1234` instead of `localhost:1234`
+
+**Linux (#1358):**
+
+Option A - Use your host's actual IP address:
+1. Find your IP: `hostname -I | awk '{print $1}'` (gives something like `192.168.1.xxx`)
+2. Set LM Studio URL to: `http://192.168.1.xxx:1234`
+3. Ensure LM Studio is listening on `0.0.0.0` (not just localhost)
+
+Option B - Enable `host.docker.internal` on Linux:
+Add to your docker-compose.yml:
+```yaml
+services:
+  local-deep-research:
+    extra_hosts:
+      - "host.docker.internal:host-gateway"
+```
+Then use `http://host.docker.internal:1234`
+
+### No models appear after I set my LM Studio API key
+
+If you paste the API key into Settings → LLM → LM Studio and immediately click the model-refresh button, the key may not have been saved yet — the field saves on blur (when you click or tab away from it), not on keypress. Click outside the API key field, or press Tab, to save the value first, then click the refresh button. The model list should populate normally.
+
+### LM Studio API key — what value should I use?
+
+LM Studio does not validate API keys by default. You can leave the API key field **blank**, or set it to any non-empty string (e.g. `lm-studio` or `not-needed`). Either will work.
+
+### Should I use the LM Studio provider or the generic OpenAI-compatible provider?
+
+Use the dedicated **LM Studio** provider (Settings → LLM → Provider → LM Studio) rather than the generic *OpenAI-compatible* option. The dedicated provider is pre-configured with the correct defaults for LM Studio and avoids common compatibility issues.
+
+### Context length not respected
+
+Known issue with Ollama (#500). Workaround:
+- Set context length when pulling model: `ollama pull llama3:8b --context-length 8192`
+
+### Model not in dropdown list
+
+Current limitation (#179). Workarounds:
+1. Type the exact model name in the dropdown field
+2. Edit database directly
+3. Use environment variables
+
+### How do I use OpenRouter?
+
+[OpenRouter](https://openrouter.ai/) provides access to 100+ models through a single API. It uses an OpenAI-compatible API format.
+
+**Quick Setup:**
+
+1. **Get API Key**: Sign up at [openrouter.ai](https://openrouter.ai/) and generate an API key
+
+2. **Configure via Web UI** (Recommended):
+   - Navigate to Settings → LLM Provider
+   - Select "OpenAI-Compatible Endpoint" (not "OpenRouter" - use the generic OpenAI endpoint option)
+   - Set Endpoint URL: `https://openrouter.ai/api/v1`
+   - Enter your OpenRouter API key
+   - Select a model from the dropdown (it will auto-populate with OpenRouter models)
+
+3. **Configure via Environment Variables** (for Docker/CI/CD):
+   ```bash
+   export LDR_LLM_PROVIDER=openai_endpoint
+   export LDR_LLM_OPENAI_ENDPOINT_URL=https://openrouter.ai/api/v1
+   export LDR_LLM_OPENAI_ENDPOINT_API_KEY="<your-api-key>"
+   export LDR_LLM_MODEL=anthropic/claude-3.5-sonnet
+   ```
+
+4. **Docker Compose Example**:
+   ```yaml
+   services:
+     local-deep-research:
+       environment:
+         - LDR_LLM_PROVIDER=openai_endpoint
+         - LDR_LLM_OPENAI_ENDPOINT_URL=https://openrouter.ai/api/v1
+         - LDR_LLM_OPENAI_ENDPOINT_API_KEY=<your-api-key>
+         - LDR_LLM_MODEL=anthropic/claude-3.5-sonnet
+   ```
+
+**Available Models**: Browse at [openrouter.ai/models](https://openrouter.ai/models)
+
+**Common Issues**:
+- **"Model not found"**: Ensure the model name exactly matches OpenRouter's format (e.g., `anthropic/claude-3.5-sonnet`)
+- **Authentication errors**: Verify your API key is correct and has credits
+- **Can't find OpenRouter in provider list**: Select "OpenAI-Compatible Endpoint" instead
+
+See also: [Environment Variables Documentation](env_configuration.md#openrouter) | [Full Configuration Reference](CONFIGURATION.md)
+
+## Local Document Search
+
+### How do I search my local documents?
+
+Use the **Collections** system in the Web UI:
+
+1. **Navigate** to the Collections page in the sidebar
+2. **Create a collection** (e.g., "Research Papers", "Project Docs")
+3. **Upload documents** directly through the UI — supported formats include PDF, TXT, MD, DOCX, and many more
+4. **Search** your collections by selecting them as a search engine, or use **"Search All Collections"** (Library RAG) to search across everything
+
+### Local search not finding documents
+
+Common issues:
+1. **First search is slow** — initial indexing takes time
+2. **File types** — ensure supported formats (PDF, TXT, MD, DOCX)
+3. **Collection not indexed** — re-upload or re-index via the Collections UI
+
+## Performance & Optimization
+
+### Research is too slow
+
+1. **Reduce complexity**:
+   - In the Web UI: Use Settings to reduce iterations and questions per iteration
+   - Via API:
+   ```python
+   quick_summary(
+       query="your query",
+       iterations=1,  # Start with 1
+       questions_per_iteration=2  # Limit sub-questions
+   )
+   ```
+
+2. **Use faster models**:
+   - Local: `mistral:7b`
+   - Cloud: `gpt-3.5-turbo`
+
+3. **Enable "Search snippets only"** (required for SearXNG)
+
+### High memory usage
+
+- Use smaller models (7B instead of 70B)
+- Limit document collection size
+- Use quantized models (GGUF format)
+
+## Docker Issues
+
+### Containers can't communicate
+
+1. **Use Docker Compose** (recommended)
+2. **Or use host networking**:
+   ```bash
+   docker run --network host ...
+   ```
+3. **Check container names** in URLs
+
+### Port 5000 not accessible on Windows
+
+This usually means you copied a `docker run … --network host …` recipe from the Linux quick-start. `--network host` is a Linux-only feature: on Docker Desktop (Mac, Windows, WSL2) it silently drops the `-p 5000:5000` publish, so the WebUI looks unreachable. As a side effect, `localhost` inside the container also stops resolving to your host's Ollama / SearXNG ports, so once you remove `--network host` the container can no longer reach them on `localhost`.
+
+**Easiest fix:** use Docker Compose instead. The bundled `docker-compose.yml` wires SearXNG and Ollama via service names (`http://searxng:8080`, `http://ollama:11434`) so nothing on the host needs `localhost`/`host.docker.internal` swapping:
+
+```bash
+curl -O https://raw.githubusercontent.com/LearningCircuit/local-deep-research/main/docker-compose.yml
+docker compose up -d
+```
+
+**If you want to stay on `docker run`:** drop `--network host`, keep `-p 5000:5000`, and point Ollama/SearXNG at `host.docker.internal` instead of `localhost`. Either set the URLs in **Settings → LLM** and **Settings → Search → SearXNG** after first login, or pass them as env vars on launch:
+
+```bash
+docker run -d -p 5000:5000 \
+  --name local-deep-research \
+  --add-host=host.docker.internal:host-gateway \
+  --volume 'deep-research:/data' \
+  -e LDR_DATA_DIR=/data \
+  -e LDR_LLM_OLLAMA_URL=http://host.docker.internal:11434 \
+  -e LDR_SEARCH_ENGINE_WEB_SEARXNG_DEFAULT_PARAMS_INSTANCE_URL=http://host.docker.internal:8080 \
+  localdeepresearch/local-deep-research
+```
+
+Note: env vars passed on `docker run` always win over values you later change in the Settings UI (the env-var override is checked on every read), so if you plan to manage URLs from the UI, leave the `-e LDR_...` lines off.
+
+### "Database is locked" errors
+
+Stop all containers and restart:
+```bash
+docker-compose down
+docker-compose up -d
+```
+
+## Unraid Deployment
+
+### How do I install LDR on Unraid?
+
+See the [complete Unraid deployment guide](deployment/unraid.md) for detailed instructions on:
+- Template repository installation (recommended)
+- Docker Compose Manager setup
+- GPU acceleration
+- Volume configuration
+
+### Common Unraid Issues
+
+**Settings don't persist?** Check that `/data` is mapped to `/mnt/user/appdata/local-deep-research/data`
+
+**Port 5000 in use?** Change host port mapping to `5050:5000` (don't change container port)
+
+**GPU not detected?** Install Nvidia-Driver plugin and configure Docker runtime - see [GPU setup guide](deployment/unraid.md#-gpu-acceleration-nvidia)
+
+**"Update Ready" always showing?** Normal for Docker Compose Manager - update via Compose section, not Docker tab
+
+For complete troubleshooting, see the [Unraid deployment guide](deployment/unraid.md#-troubleshooting)
+
+### Can't access LDR WebUI from other devices
+
+Check network configuration:
+1. Verify container is running in **Docker** tab
+2. Ensure port mapping is correct (`5000:5000`)
+3. Test locally first: `http://[unraid-ip]:5000`
+4. Check Unraid firewall if enabled
+5. For multi-container setup, ensure all on same network (`ldr-network`)
+
+### Docker in Proxmox LXC: Permission Errors
+
+If you see either of these errors in container logs:
+```
+chown: changing ownership of '/data/logs': Operation not permitted
+```
+or:
+```
+error: failed switching to "ldruser": operation not permitted
+```
+
+This means Linux capabilities needed by the entrypoint are blocked by the LXC container. The `chown` error indicates missing `CAP_CHOWN`/`CAP_FOWNER`, while the `setpriv` error indicates missing `CAP_SETUID`/`CAP_SETGID`.
+
+**Solutions (try in order):**
+
+1. **Ensure nesting is enabled** in your Proxmox LXC container:
+   - Proxmox UI → Container → Options → Features → check "Nesting"
+   - Or in config: `features: nesting=1,keyctl=1`
+
+2. **If nesting is already enabled**, your LXC may have a restrictive AppArmor profile. Try:
+   ```bash
+   # In /etc/pve/lxc/<CTID>.conf
+   lxc.apparmor.profile: unconfined
+   ```
+   Then restart the LXC container.
+
+3. **Use a privileged LXC container** (trades security for compatibility):
+   - When creating the LXC with Proxmox community scripts, select "Privileged" in advanced settings
+
+**Background:** The LDR container runs its entrypoint as root to fix volume permissions, then uses `setpriv` to drop to a non-root user (`ldruser`) for security. `setpriv` calls `setuid()`/`setgid()` which require these capabilities. In standard Docker this works out of the box, but Docker-inside-LXC inherits the outer container's capability restrictions.
+
+## Platform-Specific Issues
+
+### Windows filename errors (#339)
+
+LDR may generate invalid filenames. Fixed in recent versions, update to latest.
+
+### macOS: Port 5000 in use (AirPlay conflict)
+
+macOS Monterey (12.0+) uses port 5000 for AirPlay Receiver, which conflicts with LDR's default port. See [Port Conflict Troubleshooting](troubleshooting.md#port-conflicts) for solutions.
+
+### macOS M1/M2/M3 issues
+
+- Build your own Docker image for ARM
+- Use native Ollama installation
+- Some models may not be optimized for Apple Silicon
+
+### WSL2 networking problems
+
+Common on Windows. Solutions:
+1. Use `127.0.0.1` instead of `0.0.0.0`
+2. Check WSL2 firewall settings
+3. Restart WSL: `wsl --shutdown`
+
+## Getting Help
+
+- **Discord**: [Join our community](https://discord.gg/ttcqQeFcJ3)
+- **GitHub Issues**: [Report bugs](https://github.com/LearningCircuit/local-deep-research/issues)
+- **Reddit**: [r/LocalDeepResearch](https://www.reddit.com/r/LocalDeepResearch/)
+
+When reporting issues, include:
+- Error messages and logs
+- Your configuration (OS, Docker/pip, models)
+- Steps to reproduce
+- What you've already tried
+
+## Related Documentation
+
+- [Installation Guide](https://github.com/LearningCircuit/local-deep-research/wiki/Installation)
+- [Search Engines Guide](search-engines.md)
+- [Features Documentation](features.md)
+- [API Documentation](api-quickstart.md)
+- [Configuration Guide](env_configuration.md)
+- [Full Configuration Reference](CONFIGURATION.md)

docs/features.md

@@ -0,0 +1,367 @@
+# Features Documentation
+
+This comprehensive guide covers all features available in Local Deep Research (LDR).
+
+> **Note**: This documentation is maintained by the community and may contain inaccuracies. While we strive to keep it up-to-date, please verify critical information and report any errors via [GitHub Issues](https://github.com/LearningCircuit/local-deep-research/issues).
+
+## Table of Contents
+
+1. [Research Modes](#research-modes)
+2. [Search Capabilities](#search-capabilities)
+3. [LLM Integration](#llm-integration)
+4. [User Interface Features](#user-interface-features)
+5. [Advanced Features](#advanced-features)
+6. [Developer Features](#developer-features)
+7. [Performance Features](#performance-features)
+
+## Research Modes
+
+### Quick Summary Mode
+
+Fast research mode that provides concise answers with citations.
+
+**Features:**
+- Automatic query decomposition
+- Parallel search execution
+- Smart result synthesis
+- Citation tracking
+- Structured output with tables when relevant
+
+**Usage:**
+```python
+from local_deep_research.api import quick_summary
+
+result = quick_summary(
+    query="Your research question",
+    iterations=2,  # Number of research iterations
+    questions_per_iteration=3  # Sub-questions per iteration
+)
+```
+
+### Detailed Research Mode
+
+Comprehensive analysis mode for in-depth exploration of topics.
+
+**Features:**
+- Section-based research organization
+- Multiple research cycles
+- Cross-reference validation
+- Extended context windows
+- Detailed citation management
+
+### Report Generation Mode
+
+Creates professional research reports with proper structure.
+
+**Features:**
+- Automatic table of contents
+- Section headers and organization
+- Executive summary generation
+- Bibliography management
+- Export to PDF/Markdown
+
+### Document Analysis Mode
+
+Searches and analyzes your private document collections.
+
+**Features:**
+- Multiple document formats supported
+- Vector-based semantic search
+- Collection management
+- Incremental indexing
+- Privacy-preserved processing
+
+### Chat Mode
+
+> **Experimental** — interface and behavior may change before GA.
+
+Interactive multi-turn research conversations. Each session accumulates context across turns and supports streaming progress and follow-up refinement via the sidebar **Chat** link or `/chat/`. Designed for exploring a topic progressively rather than one-off lookups; for single queries, use a research mode directly from the home page.
+
+**Features:**
+- Multi-turn conversation with accumulated context (entities, topics, source count)
+- Live streaming of research steps and citations as the answer is built
+- Persistent sessions in your per-user database (encrypted by default; survive logout)
+- Session lifecycle: archive, reactivate, permanently delete
+- Optional LLM-generated session titles (toggle via `chat.llm_title_generation`)
+- Export a session as Markdown
+- Always uses "quick" research mode (v1); one in-flight research per session
+
+## Search Capabilities
+
+### Multi-Engine Search
+
+Simultaneously query multiple search engines for comprehensive results.
+
+**Supported Engines:**
+- Academic: arXiv, PubMed, Semantic Scholar
+- General: Wikipedia, SearXNG, DuckDuckGo
+- Technical: GitHub, Elasticsearch
+- Custom: Local documents, LangChain retrievers
+
+### Intelligent Query Routing
+
+The default langgraph-agent strategy selects appropriate search engines dynamically based on query type:
+- Scientific queries → Academic engines
+- Code questions → GitHub + technical sources
+- General knowledge → Wikipedia + web search
+
+### Adaptive Rate Limiting
+
+**Features:**
+- Learns optimal wait times per engine
+- Automatic retry with exponential backoff
+- Fallback engine selection
+- Rate limit status monitoring
+
+### Search Strategies
+
+Search strategies:
+- `source-based`: Comprehensive research with detailed source tracking
+- `focused-iteration`: Iterative refinement, quick Q&A (highest factual accuracy)
+- `focused-iteration-standard`: Comprehensive variant with broader exploration
+- `topic-organization`: Clusters sources into topics for structured output
+- `mcp`: Agentic ReAct-pattern research using MCP tools
+- `langgraph-agent`: Autonomous agentic research
+
+See [Architecture Overview](architecture/OVERVIEW.md) for details.
+
+## LLM Integration
+
+### Local Models (via Ollama)
+
+**Supported Models:**
+- Llama 3 (8B, 70B)
+- Mistral (7B, 8x7B)
+- Gemma (7B, 12B)
+- DeepSeek Coder
+- Custom GGUF models
+
+**Features:**
+- Complete privacy
+- No API costs
+- Model hot-swapping
+- GPU acceleration support
+
+### Cloud Models
+
+**Providers:**
+- OpenAI (GPT-3.5, GPT-4)
+- Anthropic (Claude 3 family)
+- Google (Gemini models)
+- OpenRouter (100+ models)
+
+**Features:**
+- Automatic fallback
+- Cost tracking per model
+- Token usage monitoring
+- Model comparison tools
+
+## User Interface Features
+
+### Web Interface
+
+**Core Features:**
+- Real-time research progress
+- Interactive result exploration
+- Settings management
+- Research history
+- Export capabilities
+
+### Keyboard Shortcuts
+
+- `ESC`: Cancel current operation
+- `Ctrl+Shift+1`: Quick Summary mode
+- `Ctrl+Shift+2`: Detailed Research mode
+- `Ctrl+Shift+3`: Report Generation
+- `Ctrl+Shift+4`: Settings
+- `Ctrl+Shift+5`: Analytics
+
+### Real-time Updates
+
+**WebSocket Features:**
+- Live research progress
+- Streaming results
+- Status notifications
+- Error handling
+- Connection management
+
+### Export Options
+
+**Formats:**
+- PDF with formatting
+- Markdown with citations
+- JSON for programmatic use
+- Plain text
+- HTML with styling
+
+## Advanced Features
+
+### LangChain Integration
+
+Connect any LangChain-compatible retriever:
+
+```python
+from local_deep_research.api import quick_summary
+
+result = quick_summary(
+    query="Internal documentation query",
+    retrievers={"company_docs": your_retriever},
+    search_tool="company_docs"
+)
+```
+
+**Supported Vector Stores:**
+- FAISS
+- Chroma
+- Pinecone
+- Weaviate
+- Elasticsearch
+- Custom implementations
+
+### MCP Server (Claude Integration)
+
+Use LDR as a research tool directly from Claude Desktop or other MCP-compatible AI assistants.
+
+**Features:**
+- 8 tools (5 research, 3 discovery) accessible via Model Context Protocol
+- STDIO transport for secure local operation
+- Per-call settings overrides
+- Autonomous agentic research (langgraph-agent strategy) with dynamic tool selection
+- Document analysis with RAG pipeline
+
+See [MCP Server Guide](mcp-server.md) for setup and usage.
+
+### REST API
+
+Full HTTP API for language-agnostic access:
+
+```bash
+# Quick summary
+POST /api/v1/quick_summary
+
+# Detailed research
+POST /api/v1/detailed_research
+
+# Report generation
+POST /api/v1/generate_report
+```
+
+**Features:**
+- OpenAPI specification
+- Authentication support
+- Rate limiting
+- Webhook callbacks
+- Batch processing
+
+### Analytics Dashboard
+
+**Metrics Tracked:**
+- Cost per research/model
+- Token usage patterns
+- Response times
+- Success rates
+- Search engine health
+- User ratings
+
+**Time Ranges:**
+- Last 7 days
+- Last 30 days
+- Last 90 days
+- All time
+
+### Research History
+
+**Features:**
+- Full research archive
+- Search within results
+- Tagging system
+- Sharing capabilities
+- Version tracking
+
+## Developer Features
+
+### Python SDK
+
+```python
+from local_deep_research import ResearchClient
+
+client = ResearchClient(
+    llm_provider="ollama",
+    llm_model="llama3:8b",
+    search_engines=["searxng", "arxiv"]
+)
+
+result = client.research(
+    query="Your question",
+    strategy="focused_iteration"
+)
+```
+
+### Benchmarking System
+
+**Features:**
+- SimpleQA dataset support
+- Custom dataset creation
+- Performance metrics
+- A/B testing framework
+- Configuration optimization
+
+**Usage:**
+```bash
+python -m local_deep_research.benchmarks.cli.benchmark_commands simpleqa \
+    --examples 100 \
+    --search-tool searxng
+```
+
+
+### Command Line Tools
+
+```bash
+# Run benchmarks from CLI
+python -m local_deep_research.benchmarks.cli.benchmark_commands simpleqa --examples 50
+
+# Manage rate limiting
+python -m local_deep_research.web_search_engines.rate_limiting status
+python -m local_deep_research.web_search_engines.rate_limiting reset --engine SearXNGSearchEngine
+```
+
+## Performance Features
+
+### Caching System
+
+**Document Embedding Cache:**
+- Caches document embeddings for faster subsequent searches
+
+### Parallel Processing
+
+**Optimization:**
+- Concurrent search queries
+- Parallel LLM calls
+- Async result processing
+- Thread pool management
+
+### Resource Management
+
+**Features:**
+- Token budget enforcement
+- Request queuing
+- Graceful degradation
+
+
+## Security Features
+
+### Privacy Protection
+
+- Local processing options
+- No telemetry by default
+- Secure credential storage
+
+
+## Related Documentation
+
+- [Search Engines Guide](search-engines.md)
+- [API Quickstart](api-quickstart.md)
+- [Configuration Guide](env_configuration.md)
+- [Full Configuration Reference](CONFIGURATION.md)
+- [Troubleshooting](troubleshooting.md)
+- [Analytics Dashboard](analytics-dashboard.md)

docs/install-pip.md

@@ -0,0 +1,75 @@
+# Python Package (pip) Installation Guide
+
+> **Note:** For most users, **Docker is preferred** as it handles all dependencies automatically. pip install is best suited for **developers** or users who want to integrate LDR into existing Python projects.
+
+## Quick Install
+
+```bash
+# Step 1: Install the package
+pip install local-deep-research
+
+# Step 2: Setup SearXNG for best results
+docker pull searxng/searxng
+docker run -d -p 8080:8080 --name searxng searxng/searxng
+
+# Step 3: Install Ollama from https://ollama.ai
+
+# Step 4: Download a model
+ollama pull gemma3:12b
+
+# Step 5: Start the web interface
+ldr-web
+```
+
+Open http://localhost:5000 after a few seconds.
+
+
+## SQLCipher (Database Encryption)
+
+LDR uses SQLCipher for AES-256 encrypted databases. Pre-built `sqlcipher3` wheels are available for Windows, macOS, and Linux — most users won't need to compile anything.
+
+- **Full setup instructions:** [SQLCipher Install Guide](SQLCIPHER_INSTALL.md)
+- **Skip encryption:** If you don't need database encryption, set `export LDR_BOOTSTRAP_ALLOW_UNENCRYPTED=true` to use standard SQLite instead. API keys and data will be stored unencrypted.
+- **Docker:** Includes SQLCipher out of the box — no extra setup needed.
+
+## Optional Dependencies
+
+### MCP Server
+
+For integration with Claude Desktop or Claude Code:
+
+```bash
+pip install "local-deep-research[mcp]"
+```
+
+## Platform Notes
+
+> **Windows PDF Export:** PDF export requires Pango/Cairo system libraries. See the [WeasyPrint installation guide](https://doc.courtbouillon.org/weasyprint/stable/first_steps.html) for setup instructions.
+
+> **CJK characters in PDF exports:** WeasyPrint resolves glyphs through the host's installed fonts. If your research results contain Chinese, Japanese, or Korean characters and they disappear from the downloaded PDF, install a CJK font package:
+>
+> - **Debian/Ubuntu:** `sudo apt install fonts-noto-cjk && fc-cache -fv`
+> - **Fedora/RHEL:** `sudo dnf install google-noto-sans-cjk-fonts && fc-cache -fv`
+> - **Alpine:** `apk add font-noto-cjk`
+> - **macOS:** ships with PingFang / Hiragino — no install needed.
+> - **Windows:** ships with Microsoft YaHei / SimSun — no install needed.
+>
+> Docker users on the official image do not need to do anything; `fonts-noto-cjk` is bundled.
+
+> **Emoji in PDF exports:** Emojis in the markdown are rendered through the host's emoji font. If they appear as empty boxes ("tofu") in the PDF, install an emoji font package:
+>
+> - **Debian/Ubuntu:** `sudo apt install fonts-noto-color-emoji && fc-cache -fv`
+> - **Fedora/RHEL 10+:** `sudo dnf install google-noto-color-emoji-fonts && fc-cache -fv`
+> - **RHEL 9 / CentOS Stream 9:** `sudo dnf install google-noto-emoji-color-fonts && fc-cache -fv` (package was renamed to `google-noto-color-emoji-fonts` in EL10)
+> - **Alpine:** `apk add font-noto-emoji` (the package name omits "color" but ships the color `NotoColorEmoji.ttf`)
+> - **macOS / Windows:** ships with the OS — no install needed.
+>
+> Docker users on the official image do not need to do anything; `fonts-noto-color-emoji` is bundled.
+
+## Development from Source
+
+For contributing or running from the latest code, see the [Development Guide](developing.md).
+
+---
+
+[Back to Installation Overview](installation.md)

docs/installation.md

@@ -0,0 +1,134 @@
+# Installation
+
+This page gives you copy-pasteable commands for every install method. For deeper configuration, follow the links to the dedicated guides.
+
+**Prerequisites:** [Docker](https://docs.docker.com/engine/install/) and [Docker Compose](https://docs.docker.com/compose/install/) are required for Options 1–3 below (pip users only need Docker for the optional SearXNG container).
+
+
+## Docker Compose (Recommended)
+
+The easiest way to get started. Bundles LDR, Ollama, and SearXNG in one command.
+
+**CPU-only (all platforms):**
+
+```bash
+curl -O https://raw.githubusercontent.com/LearningCircuit/local-deep-research/main/docker-compose.yml && docker compose up -d
+```
+
+**With NVIDIA GPU (Linux only):**
+
+Prerequisites — install the NVIDIA Container Toolkit (Ubuntu/Debian):
+
+```bash
+curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg \
+  && curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
+    sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
+    sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
+
+sudo apt-get update
+sudo apt-get install nvidia-container-toolkit -y
+sudo systemctl restart docker
+
+# Verify installation
+nvidia-smi
+```
+
+> **Note:** For RHEL/CentOS/Fedora, Arch, or other distributions, see the [NVIDIA Container Toolkit installation guide](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html).
+
+Then start the stack:
+
+```bash
+curl -O https://raw.githubusercontent.com/LearningCircuit/local-deep-research/main/docker-compose.yml && \
+curl -O https://raw.githubusercontent.com/LearningCircuit/local-deep-research/main/docker-compose.gpu.override.yml && \
+docker compose -f docker-compose.yml -f docker-compose.gpu.override.yml up -d
+```
+
+**Optional alias for convenience:**
+
+```bash
+alias docker-compose-gpu='docker compose -f docker-compose.yml -f docker-compose.gpu.override.yml'
+# Then simply use: docker-compose-gpu up -d
+```
+
+**Windows (PowerShell):**
+
+```powershell
+curl.exe -O https://raw.githubusercontent.com/LearningCircuit/local-deep-research/main/docker-compose.yml
+if ($?) { docker compose up -d }
+```
+
+**Use a different model:**
+
+```bash
+curl -O https://raw.githubusercontent.com/LearningCircuit/local-deep-research/main/docker-compose.yml && MODEL=gpt-oss:20b docker compose up -d
+```
+
+Open http://localhost:5000 after ~30 seconds.
+
+> **Note:** `curl -O` will overwrite existing docker-compose.yml files in the current directory.
+
+**DIY docker-compose:** See [docker-compose.yml](../docker-compose.yml) for a compose file with reasonable defaults. Things you may want to configure: Ollama GPU driver, context length, keep alive duration, and model selection.
+
+For Cookiecutter setup, environment variables, and troubleshooting, see the [Docker Compose Guide](docker-compose-guide.md).
+
+## Docker
+
+Run each container individually for a minimal setup.
+
+**Linux (native Docker Engine):**
+
+```bash
+# Step 1: Pull and run SearXNG for optimal search results
+docker run -d -p 8080:8080 --name searxng searxng/searxng
+
+# Step 2: Pull and run Ollama
+docker run -d -p 11434:11434 --name ollama ollama/ollama
+docker exec ollama ollama pull gpt-oss:20b
+
+# Step 3: Pull and run Local Deep Research
+docker run -d -p 5000:5000 --network host \
+  --name local-deep-research \
+  --volume 'deep-research:/data' \
+  -e LDR_DATA_DIR=/data \
+  localdeepresearch/local-deep-research
+```
+
+**Mac / Windows / WSL2 (Docker Desktop):**
+
+`--network host` doesn't work on Docker Desktop — it silently drops the port publish, and `localhost` inside the LDR container no longer reaches the Ollama/SearXNG containers. Drop `--network host`, keep `-p 5000:5000`, and point Ollama and SearXNG at `host.docker.internal` via env vars:
+
+```bash
+# Steps 1 and 2 (SearXNG + Ollama) are the same as above.
+
+# Step 3: Pull and run Local Deep Research
+docker run -d -p 5000:5000 \
+  --name local-deep-research \
+  --add-host=host.docker.internal:host-gateway \
+  --volume 'deep-research:/data' \
+  -e LDR_DATA_DIR=/data \
+  -e LDR_LLM_OLLAMA_URL=http://host.docker.internal:11434 \
+  -e LDR_SEARCH_ENGINE_WEB_SEARXNG_DEFAULT_PARAMS_INSTANCE_URL=http://host.docker.internal:8080 \
+  localdeepresearch/local-deep-research
+```
+
+(`--add-host` is a no-op on Mac/Windows where `host.docker.internal` is already auto-injected, but makes the same recipe work on Linux Docker Desktop.)
+
+If you'd rather not pass env vars, you can launch without them and then change the URLs after first login under **Settings → LLM → Ollama URL** and **Settings → Search → SearXNG → Instance URL**. For most users on these platforms, [Docker Compose](#docker-compose-recommended) is simpler — it wires the URLs up automatically via service names.
+
+Open http://localhost:5000 after ~30 seconds.
+
+## Python Package (pip)
+
+Best for developers or users integrating LDR into existing Python projects.
+
+```bash
+pip install local-deep-research
+```
+
+For full setup (SearXNG, Ollama, SQLCipher), see the [pip Guide](install-pip.md).
+
+## Unraid
+
+Local Deep Research is available as a pre-configured template for Unraid servers with sensible defaults, automatic SearXNG/Ollama integration, and NVIDIA GPU passthrough support.
+
+For full setup instructions, see the [Unraid Guide](deployment/unraid.md).

docs/journal-quality.md

@@ -0,0 +1,173 @@
+# Journal Quality System
+
+The journal quality system automatically scores academic journals encountered during research, filters out predatory publications, and provides a dashboard for exploring journal metrics.
+
+## Overview
+
+When you search using academic engines (ArXiv, OpenAlex, Semantic Scholar, NASA ADS), every journal is automatically scored on a 1–10 scale using real bibliometric data. The emitted scores are non-contiguous — the system only produces `{1, 4, 5, 6, 7, 8, 10}`; values 2, 3, and 9 are reserved but never assigned. Predatory journals are auto-removed from results. Scores are cached so subsequent lookups are instant.
+
+## Quality Scale
+
+| Score | Tier | Description | Example |
+|-------|------|-------------|---------|
+| 10 | Elite | Top-tier journals with h-index > 150 | Nature, Science, NEJM |
+| 7–8 | Strong | Strong Q1 journals, h-index 41–150 | PLOS ONE, IEEE Trans. |
+| 5–6 | Moderate | Solid journals, DOAJ-listed OA journals | Many field-specific journals |
+| 4 | Default | Low h-index or unknown venue (no data in any bundled source) | Newer, niche, or unindexed journals |
+| 1 | Predatory | Flagged by Stop Predatory Journals — auto-removed | SPJ list entries |
+
+## Threshold Semantics
+
+The threshold setting controls how aggressively the filter drops results (the filter rejects any journal whose score is below the threshold):
+
+| Threshold | Effect |
+|-----------|--------|
+| **2 (default)** | Drop only predatory (score 1). Keep everything else — including default/unknown venues. |
+| 3 | Same as 2 — no scores fall in the 2–3 gap. |
+| 4 | Same as 2 — no scores fall in the 2–3 gap. |
+| 5 | Also drop default/unknown venues (score 4). |
+| 6 | Also drop the long tail of moderate journals (score 5). |
+| 7+ | Keep only strong/elite journals. Aggressive — use only when you specifically want high-quality filtering. |
+
+The default of **2** is intentionally conservative: it removes flagged predatory venues (we have positive evidence of fraud) but doesn't silently delete sources just because we don't have bibliometric data on them.
+
+## How Scoring Works
+
+The system uses a tiered approach — the first tier that finds the journal wins:
+
+1. **Tier 1 — Predatory Check**: Checks against the Stop Predatory Journals lists (~2,500 entries). If flagged AND not whitelisted (DOAJ/high h-index), the result is auto-removed.
+
+<!-- TODO(post-merge): reconcile 212K vs 280K wording — 212K is
+     journal-type entries; 280K includes all OpenAlex source types
+     (journals, conferences, repositories, book series). Docs should
+     settle on one number with a clear qualifier. -->
+2. **Tier 2 — OpenAlex Snapshot**: Looks up the journal in the bundled OpenAlex dataset (~280K sources, downloaded fresh from the OpenAlex S3 bulk dump). Scores based on h-index thresholds (>150 → 10, >75 → 8, >40 → 7, >20 → 6, >10 → 5, else → 4) with DOAJ cross-referencing. Quartile (Q1–Q4) is derived from cited_by_count percentiles per source type.
+
+3. **Tier 3 — DOAJ Check**: For journals with ISSN not in OpenAlex, checks DOAJ status. Listed = score 5. (DOAJ retired its "Seal" in April 2025, so the former Seal = score 8 tier no longer exists.)
+
+4. **Tier 3.5 — Institution Affiliation Salvage**: For preprints (arXiv, bioRxiv) without a journal, fall back to scoring the author's institution using ~120K OpenAlex Institutions records. Capped at 6 — never beats a real venue match.
+
+5. **Tier 0 cache — LLM-only** (checked before Tiers 3.6/4): Returns a previously-computed LLM score if the journal has been analyzed by the same LLM within the re-analysis window (default 365 days). Tiers 1–3.5 are not cached — reference-DB lookups are already instant (100–300µs) and re-checking every time keeps scores current as the bundled data is rebuilt.
+
+6. **Tier 3.6 — LLM Name Cleanup (off by default)**: Asks the LLM to canonicalize a dirty journal name (e.g. strip a volume reference), then retries the Tier 2 OpenAlex lookup under the cleaned name. Enable via `search.journal_reputation.enable_llm_scoring`.
+
+7. **Tier 4 — LLM Analysis (off by default)**: Last resort for unknown journals. Uses SearXNG web search + LLM to assess reputation. Enable via `search.journal_reputation.enable_llm_scoring`. Only Tier 4 results are persisted to the per-user cache.
+
+7. **No-signal pass-through**: When *no* tier produces a signal, `derive_quality_score` returns `None` and the source is kept without a quality tag. Distinct from predatory (score 1, auto-removed) or unknown-venue default (score 4, rendered as `[Unranked ★]`).
+
+## Predatory-List Overrides
+
+Tier 1 does **not** auto-remove a journal that Stop Predatory Journals flagged if **either** of the following is true:
+
+- the journal is listed in DOAJ, **or**
+- the journal's h-index exceeds `PREDATORY_WHITELIST_HINDEX` (default `10`, defined in `constants.py`).
+
+This deliberate whitelist protects against false positives. Community predatory lists occasionally flag mainstream high-volume open-access publishers, typically because of rapid-publication or fee-structure concerns rather than actual fraud. A journal with real citation impact (h-index > 10) or a DOAJ listing is treated as legitimate enough to keep, even if a blacklist disagrees. The system prioritizes evidence of real scholarly impact over reputation heuristics, at the cost of letting a small number of borderline venues through.
+
+If your workflow needs stricter filtering — for example in a systematic review where any blacklist hit should be a hard stop — lower `PREDATORY_WHITELIST_HINDEX` toward `0` (in `constants.py` or via a local override) so fewer flagged journals are rescued. Raising the threshold has the opposite effect, making the predatory filter more forgiving.
+
+## Data Sources
+
+This system is made possible by the following open academic data initiatives. All bundled data has MIT or CC0-compatible licenses:
+
+| Source | Entries | License | What It Provides | Website |
+|--------|---------|---------|-----------------|---------|
+| [OpenAlex](https://openalex.org) | ~212K | CC0 Public Domain | h-index, impact factor, DOAJ status | openalex.org |
+| [DOAJ](https://doaj.org) | ~22K | CC0 (metadata) | Open access verification | doaj.org |
+| [Stop Predatory Journals](https://predatoryjournals.org) | ~2.5K | MIT License | Predatory journal/publisher blacklist | predatoryjournals.org |
+
+We are grateful to these projects for making academic quality data freely available to the research community.
+
+## Dashboard
+
+Access the journal quality dashboard at **Analytics → Journals** or `/metrics/journals`.
+
+Features:
+- **Summary stats**: Total journals, average quality, predatory count, DOAJ listed
+- **Quality distribution chart**: Bar chart showing score distribution (1-10)
+- **Score source chart**: Doughnut showing OpenAlex vs DOAJ vs LLM breakdown
+- **Searchable table**: 212K journals with pagination, filtering by tier/source, sortable columns
+- **Trust indicators**: DOAJ checkmark, predatory warning per journal
+
+## Source Quality Tags
+
+Research reports include quality tags next to each source:
+
+```
+[1] Physical Review Letters [Q1 ★★★★★] (source nr: 1)
+    URL: https://arxiv.org/abs/...
+
+[2] Some Niche Journal [Q3 ★★] (source nr: 2)
+    URL: https://arxiv.org/abs/...
+```
+
+## Configuration
+
+| Setting | Default | Description |
+|---------|---------|-------------|
+| `search.journal_reputation.threshold` | `2` | Minimum quality score to keep a result (1-10). Default 2 = predatory only. |
+| `search.journal_reputation.enable_llm_scoring` | `false` | Enable Tier 4 LLM analysis for journals not in any bundled dataset |
+| `search.journal_reputation.exclude_non_published` | `false` | Drop results without journal reference |
+| `search.journal_reputation.reanalysis_period` | `365` | Days before re-scoring cached journals |
+| `search.journal_reputation.max_context` | `3000` | Max chars for LLM analysis context |
+
+Per-engine enable/disable:
+- `search.engine.web.arxiv.journal_reputation.enabled`
+- `search.engine.web.openalex.journal_reputation.enabled`
+- `search.engine.web.semantic_scholar.journal_reputation.enabled`
+- `search.engine.web.nasa_ads.journal_reputation.enabled`
+
+### Tier 4 Cost & Latency
+
+Enabling `search.journal_reputation.enable_llm_scoring` activates Tier 4 (LLM + web-search analysis for journals that miss every bundled tier). Each unknown-journal scoring costs:
+
+- **Latency**: roughly 3–10 s per unique journal (one SearXNG query + one LLM call; results are deduped per batch).
+- **Tokens**: ~300–500 tokens per analysis depending on snippets and model.
+
+The result is cached in the per-user `journals` table for `search.journal_reputation.reanalysis_period` days (default 365), so the cost is amortized across repeated searches of the same rare journal. Leave it off unless you regularly encounter unindexed venues and can tolerate the per-source overhead.
+
+## Troubleshooting
+
+### Journal score looks too low, or the journal was filtered out
+
+- Check `search.journal_reputation.threshold`. The default of `2` only removes predatory journals — raise to `5`+ to also drop unknown / low-impact venues.
+- Look up the journal in the `/metrics/journals` dashboard. If it's missing, none of OpenAlex, DOAJ, or the predatory list matched it.
+- If the journal is in the predatory list but you believe it's legitimate, verify the whitelist: journals with h-index > `JOURNAL_PREDATORY_WHITELIST_HINDEX` (see `src/local_deep_research/constants.py`) or a DOAJ listing override the predatory flag automatically.
+
+### Journal not appearing in results
+
+- Predatory matches are auto-removed (score 1) unless whitelisted. If you need them back, raise the threshold above 1 — you'll keep the row but see the `[Predatory]` quality tag.
+- If Tier 4 is disabled and the journal is unindexed, no score is generated and the source is returned without a quality tag. Enable Tier 4 (`enable_llm_scoring: true`) if you want a score for rare journals.
+
+### Performance feels slow
+
+- The first access after upgrade or a fresh install downloads the bundled data (~1–2 min on a normal connection). Subsequent requests use the cached local DB.
+- If Tier 4 is on, each unknown journal adds 3–10 s (see the cost section above). Raise `reanalysis_period` to cache longer, or disable Tier 4 if you don't need it.
+
+## Database Management
+
+The journal-quality reference database (`journal_quality.db`) is built automatically on first access and lives in the user data directory:
+
+- **Linux / macOS**: `~/.local/share/local-deep-research/journal_quality.db` (or `$XDG_DATA_HOME/local-deep-research/journal_quality.db` if set)
+- **Windows**: `%APPDATA%\local-deep-research\journal_quality.db`
+
+The file is stored `chmod 0o444` read-only and opened with SQLite `mode=ro&immutable=1` on every connection — the application never writes to it after the build step. To force a fresh rebuild (after corruption or to pick up newer upstream data), delete the file and restart. The next request will re-download and rebuild it in 1–2 minutes.
+
+Do not edit the DB in place: the pre-commit hook at `.pre-commit-hooks/check-journal-quality-readonly.py` rejects any code that opens it writable.
+
+## API Endpoints
+
+| Endpoint | Method | Description |
+|----------|--------|-------------|
+| `/metrics/journals` | GET | Journal quality dashboard page |
+| `/metrics/api/journals` | GET | Paginated journal data with filtering/sorting |
+
+Query parameters for `/metrics/api/journals`:
+- `page` (int): Page number (default 1)
+- `per_page` (int): Results per page (default 50, max 200)
+- `search` (string): Filter by journal name
+- `tier` (string): elite, strong, moderate, low, predatory
+- `score_source` (string): openalex, doaj, llm, predatory
+- `sort` (string): quality, h_index, impact_factor, name, publisher
+- `order` (string): asc, desc
+- `include_summary` (bool): Include summary stats and chart data

docs/library-and-rag.md

@@ -0,0 +1,315 @@
+# Research Library & RAG Guide
+
+This guide covers the Research Library for document management and the RAG (Retrieval-Augmented Generation) system for semantic search.
+
+## Table of Contents
+
+- [Overview](#overview)
+- [Managing Documents](#managing-documents)
+- [Collections](#collections)
+- [RAG Indexing](#rag-indexing)
+- [Semantic Search](#semantic-search)
+- [Embedding Models](#embedding-models)
+- [Configuration](#configuration)
+
+---
+
+## Overview
+
+The Research Library allows you to:
+- **Upload documents** (PDFs, text files, markdown)
+- **Organize into collections** for different projects or topics
+- **Index for semantic search** using RAG (vector embeddings)
+- **Search your documents** using natural language queries
+
+Access the library at: `http://localhost:5000/library`
+
+---
+
+## Managing Documents
+
+### Supported File Types
+
+| Format | Extension | Notes |
+|--------|-----------|-------|
+| PDF | `.pdf` | Text extracted automatically |
+| Plain Text | `.txt` | Direct text storage |
+| Markdown | `.md`, `.markdown` | Rendered as text |
+| HTML | `.html`, `.htm` | Tags stripped, text extracted |
+| Word | `.docx` (and `.doc`*) | Text extracted via `unstructured` |
+| OpenDocument Text | `.odt` | Text extracted via `unstructured` |
+| PowerPoint | `.pptx` (and `.ppt`*) | Slide text extracted |
+| Excel | `.xlsx`, `.xls` | Cell text extracted |
+| Rich Text | `.rtf` | Text extracted |
+| EPUB | `.epub` | Text extracted |
+| Email | `.eml` | Body text extracted |
+| Data | `.csv`, `.tsv`, `.json`, `.yaml`, `.yml`, `.xml`, `.toml` | Parsed to text |
+| Notebooks | `.ipynb` | Cell sources and outputs |
+| Web archives | `.mhtml`, `.mht` | Saved web pages |
+
+The upload dialog's file picker is populated from the live list of formats the
+server can actually parse (`GET /library/api/config/supported-formats`), so it
+only offers formats whose parser dependencies are installed.
+
+\* The legacy binary formats `.doc` and `.ppt` are offered **only** when
+LibreOffice (`soffice`) is installed, because `unstructured` converts them to
+the modern format with it. Image formats (`.png`, `.jpg`, …) are offered
+**only** when the optional OCR extras (`pytesseract` plus the `tesseract`
+system binary) are installed. The default Docker image ships neither, so those
+formats are not offered there.
+
+### Uploading Documents
+
+1. Navigate to **Library** in the sidebar
+2. Click **Upload** or drag files into the upload area
+3. Select a collection (or use the default "Library")
+4. Documents are processed and text is extracted
+
+### Storage Modes
+
+| Mode | Description | Use Case |
+|------|-------------|----------|
+| **Database** | PDFs stored encrypted in SQLCipher | Default, most secure |
+| **Text-only** | Only extracted text stored | Save space |
+
+### Document Actions
+
+- **View** - Open document details and extracted text
+- **Download PDF** - Get original file (if stored)
+- **Download Text** - Export extracted text
+- **Delete** - Remove from library
+
+---
+
+## Collections
+
+Collections organize your documents into groups.
+
+### Creating a Collection
+
+1. Go to **Library** → **Collections**
+2. Click **Create Collection**
+3. Enter a name and optional description
+4. Click **Create**
+
+### Managing Collections
+
+- **Add documents** - Upload directly to collection or move existing docs
+- **Remove documents** - Documents can exist in multiple collections
+- **Delete collection** - Choose to keep or delete orphaned documents
+- **Index collection** - Build RAG index for semantic search
+
+### Default Collection
+
+The "Library" collection is created automatically and serves as the default destination for uploads.
+
+---
+
+## RAG Indexing
+
+RAG (Retrieval-Augmented Generation) enables semantic search over your documents.
+
+### How It Works
+
+```
+Document → Split into Chunks → Generate Embeddings → Store in Vector Index
+```
+
+1. **Chunking** - Documents split into overlapping segments
+2. **Embedding** - Each chunk converted to a vector using AI model
+3. **Indexing** - Vectors stored in FAISS for fast similarity search
+
+### Indexing a Collection
+
+1. Go to **Library** → **Collections**
+2. Select a collection
+3. Click **Index for Search** (or **Rebuild Index**)
+4. Wait for indexing to complete (progress shown)
+
+### Index Status
+
+| Status | Meaning |
+|--------|---------|
+| **Not Indexed** | Documents not searchable |
+| **Indexing** | Currently processing |
+| **Indexed** | Ready for semantic search |
+| **Needs Reindex** | New documents added since last index |
+
+---
+
+## Semantic Search
+
+Once indexed, search your documents using natural language.
+
+### Using Collection Search
+
+1. Select a collection with indexed documents
+2. Enter a natural language query
+3. Results ranked by semantic similarity
+
+### Using in Research
+
+When conducting research, you can:
+1. Set search tool to your collection name
+2. LDR will search your documents instead of the web
+3. Combine with web search via the default langgraph-agent strategy, which can query your collections and web engines in the same run
+
+Example with Python API:
+```python
+from local_deep_research.api import quick_summary
+
+result = quick_summary(
+    query="What does the documentation say about authentication?",
+    search_tool="my_collection",  # Use your collection name
+    programmatic_mode=True
+)
+```
+
+---
+
+## Embedding Models
+
+Choose the embedding model based on your needs.
+
+### Available Providers
+
+#### Sentence Transformers (Local - Default)
+
+Runs locally, no API key required.
+
+| Model | Dimensions | Best For |
+|-------|------------|----------|
+| `all-MiniLM-L6-v2` | 384 | General use (fast) |
+| `all-mpnet-base-v2` | 768 | Higher quality |
+| `multi-qa-MiniLM-L6-cos-v1` | 384 | Q&A tasks |
+| `paraphrase-multilingual-MiniLM-L12-v2` | 384 | Multi-language |
+
+#### Ollama (Local)
+
+Uses your local Ollama installation.
+
+- Default model: `nomic-embed-text`
+- Requires Ollama running locally
+- Configure URL in Settings → LLM → Ollama
+
+#### OpenAI (Cloud)
+
+Uses OpenAI's embedding API.
+
+- Default model: `text-embedding-3-small`
+- Requires OpenAI API key
+- Higher quality, requires internet
+
+### Changing Embedding Model
+
+1. Go to **Library** → **Embedding Settings**
+2. Select provider and model
+3. Click **Save**
+
+> **Note:** Changing models requires reindexing existing collections.
+
+---
+
+## Configuration
+
+### Chunking Settings
+
+| Setting | Default | Description |
+|---------|---------|-------------|
+| Chunk Size | 1000 | Characters per chunk |
+| Chunk Overlap | 200 | Overlap between chunks |
+| Splitter Type | recursive | How text is split |
+
+**Splitter Types:**
+- `recursive` - Split by paragraphs, then sentences (recommended)
+- `token` - Split by token count
+- `sentence` - Split by sentences
+- `semantic` - Split by semantic similarity
+
+### Index Settings
+
+| Setting | Default | Description |
+|---------|---------|-------------|
+| Distance Metric | cosine | Similarity calculation |
+| Index Type | flat | Exact search (most accurate) |
+
+**Distance Metrics:**
+- `cosine` - Angle-based similarity (recommended)
+- `l2` - Euclidean distance
+- `dot_product` - Dot product similarity
+
+### File Locations
+
+| Data | Location |
+|------|----------|
+| Document database | `~/.local-deep-research/` |
+| FAISS indices | `~/.cache/local_deep_research/rag_indices/` |
+
+---
+
+## API Reference
+
+### Collection Endpoints
+
+| Endpoint | Method | Description |
+|----------|--------|-------------|
+| `/library/api/collections` | GET | List all collections |
+| `/library/api/collections` | POST | Create collection |
+| `/library/api/collections/<id>` | PUT | Update collection |
+| `/library/api/collections/<id>` | DELETE | Delete collection |
+
+### Document Endpoints
+
+| Endpoint | Method | Description |
+|----------|--------|-------------|
+| `/library/api/documents` | GET | List documents |
+| `/library/api/document/<id>` | GET | Get document details |
+| `/library/api/document/<id>` | DELETE | Delete document |
+| `/library/api/document/<id>/text` | GET | Get extracted text |
+| `/library/api/document/<id>/pdf` | GET | Download PDF |
+
+### RAG Endpoints
+
+| Endpoint | Method | Description |
+|----------|--------|-------------|
+| `/library/api/rag/settings` | GET | Get RAG configuration |
+| `/library/api/rag/configure` | POST | Update RAG settings |
+| `/library/api/rag/info` | GET | Get index statistics |
+| `/library/api/collections/<id>/index` | GET | Start indexing (SSE) |
+
+---
+
+## Troubleshooting
+
+### Documents Not Appearing
+
+- Check file format is supported
+- Verify upload completed successfully
+- Refresh the library page
+
+### Search Not Working
+
+- Ensure collection is indexed (check status)
+- Try rebuilding the index
+- Check embedding model is configured
+
+### Slow Indexing
+
+- Large documents take longer
+- Consider using smaller chunk sizes
+- Local embedding models are slower than cloud
+
+### Memory Issues
+
+- Reduce chunk size
+- Index fewer documents at once
+- Use a lighter embedding model
+
+---
+
+## See Also
+
+- [Architecture Overview](architecture/OVERVIEW.md) - System architecture
+- [Extension Guide](developing/EXTENDING.md) - Adding custom retrievers
+- [Full Configuration Reference](CONFIGURATION.md) - All settings and environment variables
+- [API Quickstart](api-quickstart.md) - Using the API

docs/mcp-server.md

@@ -0,0 +1,467 @@
+# MCP Server Guide
+
+LDR provides an MCP (Model Context Protocol) server that exposes its research capabilities to AI assistants like Claude Desktop, Claude Code, and OpenClaw. MCP is an open protocol by Anthropic that lets AI applications call external tools over a standardized interface.
+
+The MCP server exposes 8 research tools — 5 research tools and 3 discovery tools — over **STDIO transport only** (local use, no network exposure). MCP support is an optional dependency that must be installed separately.
+
+## Quick Start
+
+### Installation
+
+```bash
+pip install "local-deep-research[mcp]"
+```
+
+### Claude Desktop Configuration
+
+Add the following to your `claude_desktop_config.json`:
+
+```json
+{
+  "mcpServers": {
+    "local-deep-research": {
+      "command": "ldr-mcp",
+      "env": {
+        "LDR_LLM_PROVIDER": "openai",
+        "LDR_LLM_OPENAI_API_KEY": "sk-..."
+      }
+    }
+  }
+}
+```
+
+### Claude Code Configuration
+
+Add to your `.mcp.json` (project-level) or `~/.claude/mcp.json` (global):
+
+```json
+{
+  "mcpServers": {
+    "local-deep-research": {
+      "command": "ldr-mcp",
+      "env": {
+        "LDR_LLM_PROVIDER": "ollama",
+        "LDR_LLM_OLLAMA_URL": "http://localhost:11434"
+      }
+    }
+  }
+}
+```
+
+### OpenClaw Configuration
+
+Add LDR as a skill in your `openclaw.json`:
+
+```json
+{
+  "mcpServers": {
+    "local-deep-research": {
+      "command": "ldr-mcp",
+      "env": {
+        "LDR_LLM_PROVIDER": "openai",
+        "LDR_LLM_OPENAI_API_KEY": "sk-..."
+      }
+    }
+  }
+}
+```
+
+The configuration format is the same as Claude Desktop. See the [OpenClaw MCP documentation](https://docs.openclaw.com/skills/mcp) for details on skill registration.
+
+### Verify Installation
+
+Run `ldr-mcp` in a terminal. It should start and wait for STDIO input (no output means it's working). Press `Ctrl+C` to stop.
+
+### First Research
+
+Open Claude Desktop and try:
+
+```
+Use quick_research to find information about quantum computing applications
+```
+
+## Configuration
+
+The MCP server uses the same settings system as the main LDR application. There are no dedicated MCP-specific environment variables. All configuration is done through standard `LDR_*` environment variables, set in the Claude Desktop config's `env` block.
+
+### Environment Variables
+
+**LLM Settings:**
+
+| Variable | Description | Example |
+|----------|-------------|---------|
+| `LDR_LLM_PROVIDER` | LLM provider | `openai`, `ollama`, `anthropic` |
+| `LDR_LLM_MODEL` | Model name | `gpt-4`, `llama3:8b`, `claude-sonnet-4-20250514` |
+| `LDR_LLM_OPENAI_API_KEY` | OpenAI API key | `sk-...` |
+| `LDR_LLM_TEMPERATURE` | Generation temperature | `0.7` |
+
+**Search Settings:**
+
+| Variable | Description | Example |
+|----------|-------------|---------|
+| `LDR_SEARCH_TOOL` | Default search engine | `searxng`, `arxiv`, `wikipedia` |
+| `LDR_SEARCH_SEARCH_STRATEGY` | Default strategy | `source-based`, `focused-iteration` |
+| `LDR_SEARCH_ITERATIONS` | Default iteration count | `2` |
+| `LDR_SEARCH_QUESTIONS_PER_ITERATION` | Questions per iteration | `3` |
+
+> **Note:** The strategy variable uses a double underscore (`SEARCH_SEARCH_STRATEGY`) because the settings key is `search.search_strategy`.
+
+**Optional Search API Keys:**
+
+| Variable | Description |
+|----------|-------------|
+| `LDR_TAVILY_API_KEY` | Tavily search API key |
+| `LDR_BRAVE_SEARCH_API_KEY` | Brave Search API key |
+| `LDR_SERPAPI_API_KEY` | SerpAPI key |
+
+### Example Configurations
+
+**OpenAI (default):**
+
+```json
+{
+  "mcpServers": {
+    "local-deep-research": {
+      "command": "ldr-mcp",
+      "env": {
+        "LDR_LLM_PROVIDER": "openai",
+        "LDR_LLM_MODEL": "gpt-4",
+        "LDR_LLM_OPENAI_API_KEY": "sk-..."
+      }
+    }
+  }
+}
+```
+
+**Ollama (fully local, no API key needed):**
+
+```json
+{
+  "mcpServers": {
+    "local-deep-research": {
+      "command": "ldr-mcp",
+      "env": {
+        "LDR_LLM_PROVIDER": "ollama",
+        "LDR_LLM_MODEL": "llama3:8b"
+      }
+    }
+  }
+}
+```
+
+**Anthropic:**
+
+```json
+{
+  "mcpServers": {
+    "local-deep-research": {
+      "command": "ldr-mcp",
+      "env": {
+        "LDR_LLM_PROVIDER": "anthropic",
+        "LDR_LLM_MODEL": "claude-sonnet-4-20250514",
+        "LDR_LLM_ANTHROPIC_API_KEY": "sk-ant-..."
+      }
+    }
+  }
+}
+```
+
+### Logging
+
+All log output goes to **stderr** (stdout is reserved for the JSON-RPC protocol). The log level is hardcoded to `INFO` — there is no environment variable to change it.
+
+## Available Tools
+
+### Research Tools
+
+#### `quick_research`
+
+Fast research summary. Typically takes **1-5 minutes**.
+
+**Parameters:**
+
+| Parameter | Type | Required | Description |
+|-----------|------|----------|-------------|
+| `query` | string | Yes | Research question (max 10,000 chars) |
+| `search_engine` | string | No | Search engine to use (e.g., `"searxng"`, `"arxiv"`, `"wikipedia"`) |
+| `strategy` | string | No | Research strategy (e.g., `"source-based"`, `"focused-iteration"`) |
+| `iterations` | integer | No | Number of search iterations (1-10) |
+| `questions_per_iteration` | integer | No | Questions per iteration (1-10) |
+
+**Returns:**
+
+```json
+{
+  "status": "success",
+  "summary": "Research summary text...",
+  "findings": ["finding1", "finding2"],
+  "sources": ["https://example.com/source1"],
+  "iterations": 3,
+  "formatted_findings": "Formatted markdown findings..."
+}
+```
+
+Best for: fast fact-checking, simple queries, getting a quick overview.
+
+---
+
+#### `detailed_research`
+
+Comprehensive research analysis. Typically takes **5-15 minutes**.
+
+**Parameters:**
+
+| Parameter | Type | Required | Description |
+|-----------|------|----------|-------------|
+| `query` | string | Yes | Research question (max 10,000 chars) |
+| `search_engine` | string | No | Search engine to use |
+| `strategy` | string | No | Research strategy |
+| `iterations` | integer | No | Number of search iterations (1-20) |
+| `questions_per_iteration` | integer | No | Questions per iteration (1-10) |
+
+**Returns:**
+
+```json
+{
+  "status": "success",
+  "query": "original query",
+  "research_id": "unique-id",
+  "summary": "Detailed summary...",
+  "findings": ["finding1", "finding2"],
+  "sources": ["https://example.com/source1"],
+  "iterations": 5,
+  "formatted_findings": "Formatted markdown findings...",
+  "metadata": {"timestamp": "...", "search_tool": "...", "strategy": "..."}
+}
+```
+
+Best for: in-depth analysis, nuanced topics, comprehensive coverage.
+
+---
+
+#### `generate_report`
+
+Full structured markdown report with sections, citations, and bibliography. Typically takes **10-30 minutes**.
+
+**Parameters:**
+
+| Parameter | Type | Required | Description |
+|-----------|------|----------|-------------|
+| `query` | string | Yes | Research topic (max 10,000 chars) |
+| `search_engine` | string | No | Search engine to use |
+| `searches_per_section` | integer | No | Searches per report section (1-10, default 2) |
+
+**Returns:**
+
+```json
+{
+  "status": "success",
+  "content": "# Report Title\n\n## Section 1\n...",
+  "metadata": {"timestamp": "...", "query": "..."}
+}
+```
+
+Best for: publication-quality structured reports with proper citations.
+
+---
+
+#### `analyze_documents`
+
+Search and analyze documents in a local collection using RAG. Typically takes **30 seconds - 2 minutes**.
+
+**Parameters:**
+
+| Parameter | Type | Required | Description |
+|-----------|------|----------|-------------|
+| `query` | string | Yes | Search query (max 10,000 chars) |
+| `collection_name` | string | Yes | Engine ID of the collection (e.g., `"collection_3"`). Use `list_search_engines()` to find available collections. |
+| `max_results` | integer | No | Maximum documents to retrieve (1-100, default 10) |
+
+**Returns:**
+
+```json
+{
+  "status": "success",
+  "summary": "Summary of findings from documents...",
+  "documents": [{"content": "...", "metadata": {"source": "file.pdf", "page": 1}}],
+  "collection": "my-papers",
+  "document_count": 5
+}
+```
+
+The data flow is: `collection_name` → FAISS index lookup → semantic similarity search → LLM summarization.
+
+Best for: searching uploaded PDFs and documents in local collections. Use `list_search_engines()` to discover available collection IDs.
+
+---
+
+#### `search`
+
+Raw search results without LLM processing. Typically takes **5-30 seconds**. No LLM cost.
+
+**Parameters:**
+
+| Parameter | Type | Required | Description |
+|-----------|------|----------|-------------|
+| `query` | string | Yes | Search query (max 10,000 chars) |
+| `engine` | string | Yes | Search engine to use (e.g., `"arxiv"`, `"wikipedia"`, `"searxng"`). Use `list_search_engines()` to see options. |
+| `max_results` | integer | No | Maximum results to return (1-100, default 10) |
+
+**Returns:**
+
+```json
+{
+  "status": "success",
+  "query": "quantum computing",
+  "engine": "arxiv",
+  "result_count": 10,
+  "results": [
+    {
+      "title": "Quantum Computing: An Overview",
+      "link": "https://arxiv.org/abs/2301.12345",
+      "snippet": "We present a comprehensive overview..."
+    }
+  ]
+}
+```
+
+Common engines: `arxiv` (academic papers), `pubmed` (medical literature), `wikipedia` (encyclopedic), `searxng` (meta-search), `github` (code/repos), `semantic_scholar` (citations). Use `list_search_engines()` for the full list.
+
+Best for: raw search results for your own analysis, quick lookups, checking what sources are available before running full research. Especially useful for **monitoring and subscriptions** — check for new content regularly without LLM cost.
+
+### Discovery Tools
+
+These tools return instantly and are useful for exploring available options.
+
+#### `list_search_engines`
+
+Returns available search engines with their descriptions, strengths, weaknesses, and whether they require an API key or run locally.
+
+#### `list_strategies`
+
+Returns available research strategies with their names and descriptions.
+
+#### `get_configuration`
+
+Returns the current server configuration including LLM provider, model, temperature, and search defaults. **API keys are intentionally excluded** from the response.
+
+## Research Strategies Guide
+
+LDR supports the following research strategies via MCP:
+
+| Strategy | Speed | Accuracy | Best For |
+|----------|-------|----------|----------|
+| `source-based` | Medium | High | Topics needing authoritative citations |
+| `focused-iteration` | Medium | Highest (~95%) | Complex factual / technical topics |
+| `focused-iteration-standard` | Medium | High | Comprehensive long-form answers with citations |
+| `topic-organization` | Medium | High | Structured output clustered by theme |
+| `langgraph-agent` | Varies | High | Autonomous agentic research across engines |
+
+Use `list_strategies()` to see all available strategies and their descriptions.
+
+## Error Handling
+
+All tool calls return structured error responses when something goes wrong. Errors are classified into these categories:
+
+| Error Type | Cause | User-Facing Message |
+|------------|-------|---------------------|
+| `validation_error` | Bad parameters (empty query, out-of-range values) | Specific message (e.g., "Query cannot be empty") |
+| `auth_error` | Invalid or missing API key (401) | "...failed (auth_error). Check server logs." |
+| `service_unavailable` | Provider or search engine down (503) | "...failed (service_unavailable). Check server logs." |
+| `timeout` | Operation took too long | "...failed (timeout). Check server logs." |
+| `rate_limit` | API quota exceeded (429) | "...failed (rate_limit). Check server logs." |
+| `connection_error` | Network connectivity issue | "...failed (connection_error). Check server logs." |
+| `model_not_found` | Model doesn't exist (404) | "...failed (model_not_found). Check server logs." |
+| `unknown` | Unclassified error | "...failed (unknown). Check server logs." |
+
+All errors are logged to stderr with full detail. User-facing messages are sanitized — no stack traces or API keys are exposed.
+
+## Security Model
+
+- **STDIO-only transport** — The server runs `mcp.run(transport="stdio")`, which cannot be accessed over a network. Only the parent process (e.g., Claude Desktop, Claude Code, OpenClaw) can communicate with it.
+- **No authentication needed** — OS-level process isolation provides security. The parent process controls access.
+- **API keys never returned** — `get_configuration` intentionally excludes API keys from its response.
+- **Error messages sanitized** — No internal details, stack traces, or credentials in error responses.
+- **Input validation** — All parameters have strict bounds and type checking.
+- **Per-call settings overrides** — Settings overrides from tool parameters are in-memory only and not persisted.
+- **`@no_db_settings`** — Prevents database settings access from MCP calls.
+
+> **Security Note:** This MCP server is designed for **local use only**. Do not expose it over a network without implementing proper security controls (OAuth, rate limiting). See the [MCP Security Guide](https://modelcontextprotocol.io/docs/tutorials/security/security_best_practices) for network deployment requirements.
+
+## Docker Deployment
+
+The MCP server uses STDIO transport, which requires direct process communication with the host AI assistant (e.g., Claude Desktop). This means it **must run on the host machine**, not inside a Docker container.
+
+- **MCP server** (`ldr-mcp`) — install and run on the **host** machine only
+- **Web service** (`ldr-web`) — can run in Docker
+
+The Docker image does not include MCP extras, and adding them would not help — the STDIO transport cannot bridge the container boundary to reach Claude Desktop.
+
+## Usage Examples
+
+### Prompt Patterns
+
+**Quick fact-checking:**
+```
+Use quick_research to find: What is the current population of Tokyo?
+```
+
+**Deep analysis:**
+```
+Use detailed_research with focused-iteration strategy to analyze:
+What are the latest advances in solid-state battery technology?
+```
+
+**Full report:**
+```
+Generate a report on the impact of AI on drug discovery using source-based strategy
+```
+
+**Document search:**
+```
+Search collection 'research-papers' for: machine learning optimization techniques
+```
+
+**Agentic research:**
+```
+Use the langgraph-agent strategy to research the environmental impact of cryptocurrency mining,
+considering both proof-of-work and proof-of-stake systems
+```
+
+**Individual search engines (no LLM cost, fast):**
+```
+Search arxiv for recent papers on diffusion models
+Search pubmed for CRISPR clinical trials 2024
+Search wikipedia for quantum error correction
+Search github for agentic research frameworks
+```
+
+The `search` tool is especially useful for **monitoring and subscriptions** — check for new content on a topic regularly without burning LLM tokens. An AI agent can call `search` to get raw results, then decide whether to run a full `detailed_research` only when something interesting appears.
+
+### Tips
+
+- Use `search` for fast, free lookups before committing to a full research run
+- Start with `quick_research` to test your setup, then upgrade to `detailed_research` for depth
+- Use `focused-iteration` strategy for highest accuracy on technical topics
+- Lower temperature (0.3-0.5) for factual research, higher (0.8-1.2) for creative exploration (valid range: 0.0-2.0)
+- Call `list_search_engines` and `list_strategies` first to see what's available in your configuration
+- Use `get_configuration` to verify your LLM and search settings are correct
+
+## Troubleshooting
+
+| Problem | Solution |
+|---------|----------|
+| Server won't start | Verify MCP extras are installed: `pip install "local-deep-research[mcp]"` |
+| "API key" errors | Check env vars in your Claude Desktop config (e.g., `LDR_LLM_OPENAI_API_KEY`) |
+| "Invalid strategy" error | Run `list_strategies()` to see valid strategy names |
+| "Unknown search engine" error | Run `list_search_engines()` to see available engines |
+| No results returned | Try a different `search_engine` or make your query more specific |
+| Server logs | Check stderr output. Log level is hardcoded to INFO. |
+
+## Related Documentation
+
+- [API Quickstart](api-quickstart.md) — HTTP and Python API access
+- [Search Engines Guide](search-engines.md) — Available search engines and configuration
+- [Configuration Reference](CONFIGURATION.md) — All LDR settings
+- [Features Overview](features.md) — All LDR features
+- [CLI Tools](cli-tools.md) — Command-line tools including `ldr-mcp`

docs/metrics-dashboard.md

@@ -0,0 +1,290 @@
+# Metrics & Analytics Dashboard
+
+This guide covers the Metrics Dashboard for monitoring research performance, costs, and usage analytics.
+
+## Table of Contents
+
+- [Overview](#overview)
+- [Research Metrics](#research-metrics)
+- [Cost Analytics](#cost-analytics)
+- [Rate Limiting](#rate-limiting)
+- [Link Analytics](#link-analytics)
+- [Star Reviews](#star-reviews)
+- [API Reference](#api-reference)
+
+---
+
+## Overview
+
+The Metrics Dashboard provides insights into your research activity:
+- **Token usage** and LLM costs
+- **Search performance** and rate limiting
+- **Research quality** ratings
+- **Link analytics** and domain classification
+
+Access the dashboard at: `http://localhost:5000/metrics`
+
+---
+
+## Research Metrics
+
+### Overview Page
+
+The main metrics page shows:
+
+| Metric | Description |
+|--------|-------------|
+| Total Researches | Number of research sessions |
+| Total Tokens | Tokens consumed across all research |
+| Total Cost | Estimated LLM API costs |
+| Average Duration | Mean research completion time |
+
+### Per-Research Metrics
+
+Click on any research to see detailed metrics:
+
+#### Timeline
+
+View the research execution timeline:
+- Start and end times
+- Iteration progress
+- Question generation timing
+- Search execution timing
+
+#### Search Metrics
+
+Details about search operations:
+- Queries executed
+- Results retrieved
+- Search engine usage
+- Response times
+
+#### Links
+
+All links discovered during research:
+- Source URLs
+- Domain classification
+- Relevance scores
+
+---
+
+## Cost Analytics
+
+### Token Tracking
+
+The system tracks token usage for cost estimation:
+
+| Metric | Description |
+|--------|-------------|
+| Input Tokens | Tokens sent to LLM |
+| Output Tokens | Tokens received from LLM |
+| Total Tokens | Combined usage |
+| Estimated Cost | Based on model pricing |
+
+### Cost Breakdown
+
+View costs broken down by:
+- **Model** - Which LLM models were used
+- **Provider** - OpenAI, Anthropic, etc.
+- **Time Period** - Daily, weekly, monthly
+- **Research** - Per-research costs
+
+### Accessing Cost Analytics
+
+1. Navigate to **Metrics** → **Costs**
+2. View summary statistics
+3. Filter by date range
+4. Export data as needed
+
+### Pricing Information
+
+The system includes pricing data for common models:
+
+```
+GET /metrics/api/pricing
+GET /metrics/api/pricing/<model_name>
+```
+
+---
+
+## Rate Limiting
+
+### Monitoring Rate Limits
+
+View current rate limiting status for search engines:
+
+1. Navigate to **Metrics** → **Rate Limiting**
+2. See per-engine statistics
+
+| Column | Description |
+|--------|-------------|
+| Engine | Search engine name |
+| Current Wait | Active wait time |
+| Success Rate | Percentage of successful requests |
+| Total Requests | Request count |
+| Last Updated | Most recent activity |
+
+### Rate Limit Management
+
+From the dashboard you can:
+- View current rate limit estimates
+- See learned optimal wait times
+- Monitor success rates
+
+For CLI management, see [CLI Tools](cli-tools.md#rate-limiting-cli).
+
+---
+
+## Link Analytics
+
+### Domain Classification
+
+The system classifies domains found in research:
+
+| Category | Examples |
+|----------|----------|
+| Academic | arxiv.org, pubmed.gov |
+| News | bbc.com, reuters.com |
+| Government | .gov, .edu |
+| Commercial | company websites |
+| Reference | wikipedia.org |
+
+### Link Statistics
+
+View aggregate statistics:
+- Total links discovered
+- Unique domains
+- Category distribution
+- Top domains by frequency
+
+### Accessing Link Analytics
+
+1. Navigate to **Metrics** → **Links**
+2. View domain breakdown
+3. Filter by category
+4. See classification summary
+
+### Classification API
+
+Classify domains programmatically:
+
+```
+POST /metrics/api/domain-classifications/classify
+{
+  "domains": ["example.com", "arxiv.org"]
+}
+```
+
+---
+
+## Star Reviews
+
+### Rating Research Quality
+
+Rate your research results to track quality over time:
+
+1. Open a completed research
+2. Click the star rating (1-5)
+3. Optionally add feedback text
+
+### Viewing Ratings
+
+Navigate to **Metrics** → **Star Reviews** to see:
+- Average rating across all research
+- Rating distribution
+- Recent ratings with feedback
+- Trends over time
+
+### Rating API
+
+```
+# Get ratings for a research
+GET /metrics/api/ratings/<research_id>
+
+# Submit a rating
+POST /metrics/api/ratings/<research_id>
+{
+  "rating": 5,
+  "feedback": "Excellent results"
+}
+```
+
+---
+
+## API Reference
+
+### Metrics Endpoints
+
+| Endpoint | Method | Description |
+|----------|--------|-------------|
+| `/metrics/api/metrics` | GET | Overview metrics |
+| `/metrics/api/metrics/enhanced` | GET | Detailed metrics |
+| `/metrics/api/metrics/research/<id>` | GET | Per-research metrics |
+| `/metrics/api/metrics/research/<id>/timeline` | GET | Research timeline |
+| `/metrics/api/metrics/research/<id>/search` | GET | Search metrics |
+| `/metrics/api/metrics/research/<id>/links` | GET | Research links |
+
+### Cost Endpoints
+
+| Endpoint | Method | Description |
+|----------|--------|-------------|
+| `/metrics/api/pricing` | GET | Model pricing data |
+| `/metrics/api/pricing/<model>` | GET | Specific model pricing |
+| `/metrics/api/cost-calculation` | POST | Calculate costs |
+| `/metrics/api/research-costs/<id>` | GET | Research cost breakdown |
+| `/metrics/api/cost-analytics` | GET | Cost analytics summary |
+
+### Rate Limiting Endpoints
+
+| Endpoint | Method | Description |
+|----------|--------|-------------|
+| `/metrics/api/rate-limiting` | GET | Rate limit statistics |
+| `/metrics/api/rate-limiting/current` | GET | Current rate limits |
+
+### Rating Endpoints
+
+| Endpoint | Method | Description |
+|----------|--------|-------------|
+| `/metrics/api/ratings/<id>` | GET | Get research rating |
+| `/metrics/api/ratings/<id>` | POST | Submit rating |
+| `/metrics/api/star-reviews` | GET | All ratings summary |
+
+### Domain Classification Endpoints
+
+| Endpoint | Method | Description |
+|----------|--------|-------------|
+| `/metrics/api/domain-classifications` | GET | All classifications |
+| `/metrics/api/domain-classifications/summary` | GET | Classification summary |
+| `/metrics/api/domain-classifications/classify` | POST | Classify domains |
+| `/metrics/api/domain-classifications/progress` | GET | Classification progress |
+| `/metrics/api/link-analytics` | GET | Link analytics data |
+
+---
+
+## Troubleshooting
+
+### No Metrics Showing
+
+- Ensure you have completed at least one research
+- Check that token tracking is enabled
+- Refresh the page
+
+### Cost Estimates Incorrect
+
+- Verify model pricing is up to date
+- Check provider is correctly identified
+- Token counts may vary by model
+
+### Rate Limiting Data Missing
+
+- Run some searches to generate data
+- Check the rate limiting CLI for status
+
+---
+
+## See Also
+
+- [CLI Tools](cli-tools.md) - Rate limiting CLI
+- [Full Configuration Reference](CONFIGURATION.md) - All settings and environment variables
+- [Troubleshooting](troubleshooting.md) - Common issues
+- [Architecture Overview](architecture/OVERVIEW.md) - System design

docs/news-subscriptions.md

@@ -0,0 +1,298 @@
+# News Subscriptions Guide
+
+This guide covers the News Subscription system for automated research monitoring and news tracking.
+
+## Table of Contents
+
+- [Overview](#overview)
+- [Creating Subscriptions](#creating-subscriptions)
+- [Managing Subscriptions](#managing-subscriptions)
+- [Update Frequency](#update-frequency)
+- [Subscription History](#subscription-history)
+- [Scheduler](#scheduler)
+- [Notifications](#notifications)
+- [API Reference](#api-reference)
+
+---
+
+## Overview
+
+News Subscriptions allow you to:
+- **Monitor topics** with automatic periodic research
+- **Track search queries** as living news feeds
+- **Get notifications** when new content is found
+- **Organize subscriptions** into folders
+
+Access subscriptions at: `http://localhost:5000/news/subscriptions`
+
+### Subscription Types
+
+| Type | Description | Use Case |
+|------|-------------|----------|
+| **Search** | Recurring search query | "AI regulations news" |
+| **Topic** | Topic monitoring with evolution tracking | "Machine Learning" |
+
+---
+
+## Creating Subscriptions
+
+### From the Web UI
+
+1. Navigate to **News** → **Subscriptions**
+2. Click **New Subscription**
+3. Configure your subscription:
+
+#### Basic Settings
+
+| Field | Description |
+|-------|-------------|
+| **Query/Topic** | What to search for |
+| **Name** | Friendly name (optional) |
+| **Frequency** | How often to update |
+| **Folder** | Organization folder (optional) |
+
+#### Research Settings
+
+| Field | Default | Description |
+|-------|---------|-------------|
+| Model Provider | OLLAMA | LLM provider to use |
+| Model | (varies) | Specific model |
+| Search Engine | (your default search tool) | Search engine to use |
+| Iterations | 3 | Research depth |
+| Questions per Iteration | 5 | Breadth per cycle |
+| Strategy | source-based | Research approach |
+
+#### Submit Options
+
+- **Create & Run Now** - Save and immediately execute
+- **Test Run** - Test configuration without saving
+- **Create Subscription** - Save for scheduled execution
+
+### From Research Results
+
+When viewing research results, click **Subscribe** to create a subscription based on that query.
+
+---
+
+## Managing Subscriptions
+
+### Dashboard Overview
+
+The subscriptions dashboard shows:
+
+| Metric | Description |
+|--------|-------------|
+| Total | All subscriptions |
+| Active | Currently running |
+| Paused | Temporarily stopped |
+| Updates Today | Refreshes in last 24h |
+
+### Subscription Actions
+
+| Action | Description |
+|--------|-------------|
+| **Run Now** | Execute immediately |
+| **Pause** | Stop scheduled updates |
+| **Resume** | Restart paused subscription |
+| **Edit** | Modify settings |
+| **View History** | See past research runs |
+| **Delete** | Remove subscription |
+
+### Filtering & Search
+
+- **Search** - Filter by name or query text
+- **Status** - Show active, paused, or all
+- **Frequency** - Filter by update interval
+- **Folder** - Filter by organization folder
+
+### Folders
+
+Organize subscriptions into folders:
+
+1. Click **Manage Folders**
+2. Create folders with names
+3. Assign subscriptions to folders
+4. Filter view by folder
+
+---
+
+## Update Frequency
+
+### Preset Intervals
+
+| Frequency | Interval |
+|-----------|----------|
+| Hourly | 60 minutes |
+| Every 3 hours | 180 minutes |
+| Every 6 hours | 360 minutes |
+| Every 12 hours | 720 minutes |
+| Daily | 1440 minutes |
+| Weekly | 10080 minutes |
+
+### Custom Intervals
+
+Set any interval between 1 hour and 30 days (in minutes).
+
+### Jitter
+
+A small random delay (up to 5 minutes) is added to prevent all subscriptions from running simultaneously.
+
+---
+
+## Subscription History
+
+View the research history for any subscription:
+
+1. Click on a subscription
+2. Select **View History**
+
+### History Details
+
+| Field | Description |
+|-------|-------------|
+| **Research Title** | Query that was executed |
+| **Timestamp** | When it ran |
+| **Status** | Completed or failed |
+| **Duration** | How long it took |
+| **Topics** | Extracted topics |
+
+Click on any research run to view full results.
+
+---
+
+## Scheduler
+
+The scheduler runs subscriptions automatically in the background.
+
+### Scheduler Status
+
+The status bar shows:
+- Whether scheduler is running
+- Number of active jobs
+- Whether your subscriptions are tracked
+
+### Starting the Scheduler
+
+The scheduler starts automatically when you log in. Your session remains active for 48 hours.
+
+### Manual Control
+
+- **Run Now** - Execute any subscription immediately
+- **Pause** - Stop a subscription temporarily
+- **Resume** - Restart a paused subscription
+
+### Error Handling
+
+If a subscription fails:
+1. It retries with exponential backoff
+2. After 10 consecutive failures, it's automatically disabled
+3. You receive a notification about the error
+4. You can manually resume it anytime
+
+---
+
+## Notifications
+
+Get notified when subscriptions update or encounter errors.
+
+### Notification Events
+
+| Event | Description |
+|-------|-------------|
+| **subscription_update** | New content found |
+| **subscription_error** | Update failed |
+
+### Setup
+
+Configure notifications in **Settings** → **Notifications**:
+
+1. Add notification service URLs (Discord, Slack, Email, etc.)
+2. Enable subscription notifications
+3. Test your configuration
+
+See [NOTIFICATIONS.md](NOTIFICATIONS.md) for supported services.
+
+---
+
+## API Reference
+
+### Subscription Endpoints
+
+| Endpoint | Method | Description |
+|----------|--------|-------------|
+| `/news/api/subscriptions` | GET | List all subscriptions |
+| `/news/api/subscriptions` | POST | Create subscription |
+| `/news/api/subscriptions/<id>` | GET | Get subscription details |
+| `/news/api/subscriptions/<id>` | PUT | Update subscription |
+| `/news/api/subscriptions/<id>` | DELETE | Delete subscription |
+| `/news/api/subscriptions/<id>/history` | GET | Get subscription history |
+
+### Folder Endpoints
+
+| Endpoint | Method | Description |
+|----------|--------|-------------|
+| `/news/api/subscription/folders` | GET | List folders |
+| `/news/api/subscription/folders` | POST | Create folder |
+
+### Scheduler Endpoints
+
+| Endpoint | Method | Description |
+|----------|--------|-------------|
+| `/news/api/scheduler/status` | GET | Get scheduler status |
+| `/news/api/scheduler/start` | POST | Start scheduler |
+
+### Example: Create Subscription
+
+```python
+import requests
+
+response = requests.post(
+    "http://localhost:5000/news/api/subscriptions",
+    json={
+        "query": "AI safety news",
+        "name": "AI Safety Updates",
+        "refresh_minutes": 1440,  # Daily
+        "model_provider": "ollama",
+        "search_engine": "searxng",
+        "iterations": 3
+    },
+    cookies={"session": your_session_cookie}
+)
+```
+
+---
+
+## Troubleshooting
+
+### Subscription Not Updating
+
+1. Check scheduler status (should be "Running")
+2. Verify subscription is not paused
+3. Check for errors in subscription history
+4. Try "Run Now" to test manually
+
+### Too Many Updates
+
+- Increase the update frequency (e.g., daily instead of hourly)
+- Pause subscriptions you don't need actively
+
+### Missing Notifications
+
+1. Verify notification service is configured
+2. Test notification URL in settings
+3. Check subscription notification events are enabled
+
+### Scheduler Not Starting
+
+- Log out and log back in
+- Check browser console for errors
+- Verify server is running
+
+---
+
+## See Also
+
+- [NOTIFICATIONS.md](NOTIFICATIONS.md) - Notification service setup
+- [Full Configuration Reference](CONFIGURATION.md) - All settings, defaults, and environment variables
+- [API Quickstart](api-quickstart.md) - API authentication
+- [Features](features.md) - Feature overview

docs/news/EXCEPTION_HANDLING.md

@@ -0,0 +1,221 @@
+# News API Exception Handling
+
+## Overview
+
+The news API module now uses a structured exception handling approach instead of returning error dictionaries. This provides better error handling, cleaner code, and consistent API responses.
+
+## Exception Hierarchy
+
+All news API exceptions inherit from `NewsAPIException`, which provides:
+- Human-readable error messages
+- HTTP status codes
+- Machine-readable error codes
+- Optional additional details
+
+```python
+from local_deep_research.news.exceptions import NewsAPIException
+
+class NewsAPIException(Exception):
+    def __init__(self, message: str, status_code: int = 500,
+                 error_code: Optional[str] = None,
+                 details: Optional[Dict[str, Any]] = None)
+```
+
+## Available Exceptions
+
+### InvalidLimitException
+- **Status Code**: 400
+- **Error Code**: `INVALID_LIMIT`
+- **Usage**: When an invalid limit parameter is provided
+- **Example**:
+```python
+if limit < 1:
+    raise InvalidLimitException(limit)
+```
+
+### SubscriptionNotFoundException
+- **Status Code**: 404
+- **Error Code**: `SUBSCRIPTION_NOT_FOUND`
+- **Usage**: When a requested subscription doesn't exist
+- **Example**:
+```python
+if not subscription:
+    raise SubscriptionNotFoundException(subscription_id)
+```
+
+### SubscriptionCreationException
+- **Status Code**: 500
+- **Error Code**: `SUBSCRIPTION_CREATE_FAILED`
+- **Usage**: When subscription creation fails
+- **Example**:
+```python
+except Exception as e:
+    raise SubscriptionCreationException(str(e), {"query": query})
+```
+
+### SubscriptionUpdateException
+- **Status Code**: 500
+- **Error Code**: `SUBSCRIPTION_UPDATE_FAILED`
+- **Usage**: When subscription update fails
+
+### SubscriptionDeletionException
+- **Status Code**: 500
+- **Error Code**: `SUBSCRIPTION_DELETE_FAILED`
+- **Usage**: When subscription deletion fails
+
+### DatabaseAccessException
+- **Status Code**: 500
+- **Error Code**: `DATABASE_ERROR`
+- **Usage**: When database operations fail
+- **Example**:
+```python
+except Exception as e:
+    raise DatabaseAccessException("operation_name", str(e))
+```
+
+### NewsFeedGenerationException
+- **Status Code**: 500
+- **Error Code**: `FEED_GENERATION_FAILED`
+- **Usage**: When news feed generation fails
+
+### ResearchProcessingException
+- **Status Code**: 500
+- **Error Code**: `RESEARCH_PROCESSING_FAILED`
+- **Usage**: When processing research items fails
+
+### NotImplementedException
+- **Status Code**: 501
+- **Error Code**: `NOT_IMPLEMENTED`
+- **Usage**: For features not yet implemented
+- **Example**:
+```python
+raise NotImplementedException("feature_name")
+```
+
+### InvalidParameterException
+- **Status Code**: 400
+- **Error Code**: `INVALID_PARAMETER`
+- **Usage**: When invalid parameters are provided
+
+### SchedulerNotificationException
+- **Status Code**: 500
+- **Error Code**: `SCHEDULER_NOTIFICATION_FAILED`
+- **Usage**: When scheduler notification fails (non-critical)
+
+## Flask Integration
+
+### Error Handlers
+
+The Flask application automatically handles `NewsAPIException` and its subclasses:
+
+```python
+@app.errorhandler(NewsAPIException)
+def handle_news_api_exception(error):
+    return jsonify(error.to_dict()), error.status_code
+```
+
+### Response Format
+
+All exceptions are converted to consistent JSON responses:
+
+```json
+{
+    "error": "Human-readable error message",
+    "error_code": "MACHINE_READABLE_CODE",
+    "status_code": 400,
+    "details": {
+        "additional": "context",
+        "if": "available"
+    }
+}
+```
+
+## Migration Guide
+
+### Before (Error Dictionaries)
+
+```python
+def get_news_feed(limit):
+    if limit < 1:
+        return {
+            "error": "Limit must be at least 1",
+            "news_items": []
+        }
+
+    try:
+        # ... code ...
+    except Exception as e:
+        logger.exception("Error getting news feed")
+        return {"error": str(e), "news_items": []}
+```
+
+### After (Exceptions)
+
+```python
+def get_news_feed(limit):
+    if limit < 1:
+        raise InvalidLimitException(limit)
+
+    try:
+        # ... code ...
+    except NewsAPIException:
+        raise  # Re-raise our custom exceptions
+    except Exception as e:
+        logger.exception("Error getting news feed")
+        raise NewsFeedGenerationException(str(e))
+```
+
+## Benefits
+
+1. **Cleaner Code**: Functions focus on their primary logic without error response formatting
+2. **Consistent Error Handling**: All errors follow the same format
+3. **Better Testing**: Easier to test exception cases with pytest.raises
+4. **Type Safety**: IDEs can better understand return types
+5. **Centralized Logging**: Error logging can be handled in one place
+6. **HTTP Status Codes**: Proper HTTP status codes are automatically set
+
+## Testing
+
+Test exception handling using pytest:
+
+```python
+import pytest
+from local_deep_research.news.exceptions import InvalidLimitException
+
+def test_invalid_limit():
+    with pytest.raises(InvalidLimitException) as exc_info:
+        news_api.get_news_feed(limit=-1)
+
+    assert exc_info.value.status_code == 400
+    assert exc_info.value.details["provided_limit"] == -1
+```
+
+## Best Practices
+
+1. **Always catch and re-raise NewsAPIException subclasses**:
+```python
+except NewsAPIException:
+    raise  # Let Flask handle it
+except Exception as e:
+    # Convert to appropriate NewsAPIException
+    raise DatabaseAccessException("operation", str(e))
+```
+
+2. **Include relevant details**:
+```python
+raise SubscriptionCreationException(
+    "Database constraint violated",
+    details={"query": query, "type": subscription_type}
+)
+```
+
+3. **Use appropriate exception types**: Choose the most specific exception that matches the error condition
+
+4. **Log before raising**: For unexpected errors, log the full exception before converting to NewsAPIException
+
+## Future Improvements
+
+- Add retry logic for transient errors
+- Implement exception metrics/monitoring
+- Add internationalization support for error messages
+- Create exception middleware for advanced error handling

docs/processes/review-process/README.md

@@ -0,0 +1,144 @@
+# PR Review Process
+
+This document describes how PRs flow through review at LDR. It's the maintainer-facing companion to [CONTRIBUTING.md](../../../CONTRIBUTING.md).
+
+> **Folder convention:** this is the first document under `docs/processes/`. Each folder under `docs/processes/<process-name>/` documents one organizational process, with `README.md` as the entry point so visiting the folder URL on GitHub renders it directly. Future processes (release, security review, etc.) should follow the same pattern.
+
+## Overview
+
+PRs come from a mix of sources: core maintainers, the extended reviewer team, one-off external contributors, AI bots, and Dependabot. Triage labels are auto-applied at PR open and toggled by review events so reviewers can find the right work to focus on without manually scanning every PR.
+
+## Quick reference
+
+| Label | Meaning | Who acts next |
+|-------|---------|---------------|
+| `external-contributor` | PR author is outside the maintainer team | First codeowner to triage |
+| `first-time-contributor` | Author's first PR to this repo | A maintainer should welcome and review |
+| `bot` | PR opened by an automated account (`*[bot]`, moltenbot, etc.) | A codeowner — apply higher scrutiny |
+| `needs-codeowner-review` | Awaiting first review from a global codeowner | Global codeowners |
+| `awaiting-author` | Codeowner requested changes; ball is with author | The PR author |
+| `awaiting-codeowner` | Author has responded; needs codeowner re-review | Global codeowners |
+| `needs-rework` | PR shows low engagement — broken tests, mechanical churn, ignored feedback, scope violation | The PR author (substantive rework) |
+
+Labels are managed by:
+- `.github/labels.yml` (declarative definitions)
+- `.github/workflows/labels-sync.yml` (creates/updates labels)
+- `.github/workflows/pr-triage.yml` (toggles them per PR)
+
+## Who's a maintainer
+
+**Global codeowners** (review required for any path; merge approval):
+- @LearningCircuit
+- @hashedviking
+- @djpetti
+
+**Extended reviewer team** (codeowners for tests, CI, docs, and templates):
+- @scottvr
+- @tombii
+- @prashant-sharma-cmd
+- @elpikola
+- @shreydekate
+
+See [`.github/CODEOWNERS`](../../../.github/CODEOWNERS) for the full path-to-owner mapping. The global-owners list is mirrored in `.github/workflows/pr-triage.yml`; both must stay in sync.
+
+## PR triage queue
+
+The canonical search filters maintainers and reviewers should run regularly. Each one returns the PRs that are most useful to act on next.
+
+**External PRs awaiting first codeowner look** — the most important queue:
+
+```
+is:open is:pr -author:dependabot[bot] -author:LearningCircuit -author:hashedviking -author:djpetti label:needs-codeowner-review
+```
+
+**Likely-stale PRs** (author hasn't responded; candidates to nudge or close — replace the date with ~30 days ago):
+
+```
+is:open is:pr label:awaiting-author updated:<2026-04-08
+```
+
+**Bot PRs needing higher-scrutiny review** — treat these as proposals, not contributions:
+
+```
+is:open is:pr label:bot label:needs-codeowner-review
+```
+
+**First-time contributors** — extra welcoming and coaching:
+
+```
+is:open is:pr label:first-time-contributor
+```
+
+**Author has responded; needs re-review** — triage these regularly to avoid ping-pong delays:
+
+```
+is:open is:pr label:awaiting-codeowner
+```
+
+## Lifecycle of a PR
+
+```
+PR opened (external)  →  needs-codeowner-review
+                         + external-contributor
+                         + first-time-contributor (if first PR)
+                         + bot (if automated account)
+
+needs-codeowner-review  ──[codeowner: changes_requested]──>  awaiting-author
+needs-codeowner-review  ──[codeowner: approved]───────────>  (no lifecycle label)
+
+awaiting-author         ──[author pushes commit]──────────>  awaiting-codeowner
+awaiting-author         ──[codeowner dismisses review]───>  needs-codeowner-review
+
+awaiting-codeowner      ──[codeowner: approved]───────────>  (no lifecycle label)
+awaiting-codeowner      ──[codeowner: changes_requested]─>  awaiting-author
+```
+
+`commented` reviews are a no-op — they don't move the lifecycle. Use approve / request-changes / dismiss to move state.
+
+## Reviewer responsibilities
+
+When you pick up a PR with `needs-codeowner-review`:
+- Read the PR description for the contributor's verification narrative — what they tested by hand. Missing or generic descriptions are a signal — see "Spotting low-engagement PRs" below.
+- Submit one of: approve, request changes, or comment-with-questions. Don't leave the PR in limbo.
+- For external/bot PRs, double-check tests actually pass on the branch (CI green != tests pass meaningfully).
+
+When you see `awaiting-codeowner`:
+- The author has already responded. Don't make them wait — pick it up promptly or hand off.
+
+When you see `needs-rework`:
+- Don't review line-by-line. The PR is in a state that needs the author to take a substantive next step.
+
+## Spotting low-engagement PRs
+
+Reviewers should look for these heuristics — they're stronger signals than "was AI used":
+
+- **Broken tests on the branch.** The author didn't run them locally before submitting.
+- **Mechanical churn across many files.** 30+ files of same-shape edits suggests a one-shot generation rather than considered changes.
+- **Author doesn't respond to specific review questions**, or responds with another regenerated diff rather than addressing the question directly.
+- **Generic description with no concrete verification narrative** — nothing in the PR body suggests the author exercised the change beyond pushing it.
+- **Atomic-scope violations.** Multiple unrelated changes bundled. CONTRIBUTING.md is explicit: one logical change per PR.
+
+### Recommended response
+
+Apply `needs-rework`, post a comment listing the concrete issues, ask for either a split or a specific revision. **Do not do the fix work for the contributor unless they go silent for an extended period.** Doing the work for them rewards low-engagement submissions.
+
+## Bot PRs specifically
+
+Bot-author PRs (`moltenbot000`, similar AI agents) have no human in the loop on the PR side. Iterating with the bot via PR comments rarely produces meaningful revisions — the bot typically regenerates rather than addressing specific feedback.
+
+Treat bot PRs as **proposals, not contributions**. If substantive issues exist:
+
+- Take the good parts forward as a maintainer-authored PR (cherry-pick + fix).
+- Close the bot PR with a clear rationale comment.
+
+PR #3847 was the working example for this pattern: a moltenbot PR with a sound architectural idea (loguru sink-level redaction) wrapped in mechanical call-site churn and broken tests. The right path was to land the patcher idea cleanly via a maintainer-authored PR and close the bot's submission.
+
+## When to escalate
+
+- **Security findings during review** (exposed secrets, auth bypass, injection): apply `security-review-needed` and ping `@LearningCircuit`. See the [security review process](../security-review-process/README.md) for the security-specific flow.
+- **Legal or licensing concerns**: core team only (`@LearningCircuit @hashedviking @djpetti`). Do not merge.
+- **Persistent disagreement on approach**: pause, take it to Discord or an issue for broader input rather than escalating in PR comments.
+
+## Out of scope here
+
+This document is about the **review** process. Adjacent processes (release, security review, deprecation) are documented separately under `docs/processes/<process-name>/` (when migrated) or in their existing locations.

docs/processes/security-review-process/README.md

@@ -0,0 +1,109 @@
+# Security Review Process
+
+## Overview
+
+This repository uses an automated alert system to highlight when PRs modify security-critical files. Instead of blocking development, we provide clear warnings and checklists to ensure dangerous changes get proper scrutiny.
+
+## How It Works
+
+### 🚨 Automatic Detection
+When a PR modifies any of these files:
+- Database encryption (`encrypted_db.py`, `sqlcipher_*.py`)
+- Authentication (`/auth/` directory, `password*.py`)
+- Security utilities (`/security/` directory)
+- Encryption/decryption code (`*encrypt*.py`, `*decrypt*.py`, `*crypto*.py`)
+
+The CI will:
+1. **Post a prominent warning comment** with security-specific review checklist
+2. **Add labels** to the PR (`security-review-needed`, `touches-encryption`, etc.)
+3. **Create a status check** showing "Security Review Required" (informational, not blocking)
+
+### 📋 Review Checklists
+
+The bot creates specific checklists based on what was modified:
+
+#### For Encryption Changes:
+- SQLCipher pragma order (key must come first)
+- No hardcoded keys
+- Backward compatibility
+- Migration paths
+
+#### For Authentication Changes:
+- No auth bypasses
+- Secure session handling
+- Proper password hashing
+- No privilege escalation
+
+#### For All Security Changes:
+- No exposed secrets
+- No debug bypasses
+- Safe error messages
+- Secure logging
+
+## For Developers
+
+### When You Modify Security Files:
+
+1. **Expect the warning** - It's not a failure, just a heads-up
+2. **Self-review first** - Go through the checklist yourself
+3. **Document your changes** - Explain WHY the security code needed to change
+4. **Test thoroughly** - Especially with existing encrypted databases
+5. **Be patient** - Security reviews take time
+
+### Red Flags to Avoid:
+
+❌ Changing pragma order in SQLCipher code
+❌ Adding "temporary" auth bypasses
+❌ Logging passwords or encryption keys
+❌ Hardcoding credentials
+❌ Disabling security checks "for testing"
+
+## For Reviewers
+
+### When You See the Warning:
+
+1. **Take it seriously** - These files can break security if done wrong
+2. **Go through the checklist** - Don't just check boxes, verify each item
+3. **Test locally** - Especially encryption changes with real databases
+4. **Ask questions** - If something seems off, dig deeper
+5. **Get a second opinion** - For CRITICAL changes, ask another reviewer
+
+### What Caused Our Previous Issue:
+
+The SQLCipher pragma order bug that corrupted databases happened because:
+- Pragmas were applied BEFORE setting the encryption key
+- This wasn't caught in review despite being in "hotfix" branch
+- The change seemed logical but was actually breaking
+
+**Lesson:** Even "obvious" changes to encryption code need careful review!
+
+## The Philosophy
+
+- **Warnings, not blocks** - We trust developers but want awareness
+- **Specific guidance** - Checklists for what to look for
+- **Risk-based** - More critical files get scarier warnings
+- **Educational** - Help reviewers know what to check
+
+## Labels Added Automatically
+
+- `security-review-needed` - Always added for security files
+- `touches-encryption` - Database encryption modified
+- `touches-authentication` - Auth system modified
+- `critical-changes` - Multiple security systems affected
+
+## It's Not Perfect
+
+This system won't catch everything:
+- Logic bugs in security code
+- Subtle vulnerabilities
+- Dependencies with security issues
+
+It's meant to make reviewers **pause and think** when touching dangerous code.
+
+## Questions?
+
+If you're unsure about a security change:
+1. Ask in the PR for additional reviewers
+2. Test with production-like data
+3. Consider doing a security-focused code review session
+4. When in doubt, be more cautious

docs/processes/test-review/README.md

@@ -0,0 +1,185 @@
+# Test review (PUNCHLIST) — completion report
+
+This document records the outcome of the systematic test-suite review
+tracked in `/home/coder1/.claude/plans/test-review/PUNCHLIST.md` (the
+canonical punchlist lives outside the repo).
+
+Of **1,023 unique punchlist entries**, every entry has been either
+**addressed by a PR** or **categorized as no-action-needed** with
+documented rationale. This file enumerates each category so a future
+audit can verify completeness without rebuilding the audit from
+scratch.
+
+## Category A — Addressed by a PR (734 entries)
+
+Each entry in this category was actioned by exactly one PR. The PRs
+fall into the buckets below.
+
+### A1. Mass shadow-file cleanup (~3,100 tests removed)
+
+| PR | Files removed | Tests removed |
+|---|---|---|
+| #4241 | 52 (import-existence tautologies — surgical per-test removal) | 140 |
+| #4242 | 45 `tests/news/test_*_behavior.py` (0 SUT imports) | 2,428 |
+| #4243 | 26 non-news shadow files (0 SUT imports) | 683 |
+
+### A2. Tier 1 placeholders + OVERMOCKED strengthening
+
+| PR | Description |
+|---|---|
+| #4244 | Script-style `test_custom_context.py` deleted |
+| #4245 | **Production bug fix** in `SQLCardStorage.create()` + 2 OVERMOCKED tests strengthened |
+| #4246 | Rating-storage OVERMOCKED test strengthened |
+| #4247 | 7 no-assertion placeholders across 7 files |
+| #4273 | 2 broad-status-list tautologies in `test_context_overflow_api.py` — exposed and fixed a wrong test URL |
+| #4274 | 5 `ui_tests/test_*_uuid*` diagnostic scripts (REAL_NETWORK, NO_ASSERT) |
+| #4275 | `test_google_pse.py` H7_MOCK_IDENTITY tautology |
+| #4276 | Mock-roundtrip test in `test_llm_provider_integration.py` |
+| #4290 | 4 final `assert True` placeholders found by exhaustive AST scan |
+
+### A3. Tier 2 tautology + NO_ASSERT cleanup
+
+| PR | Description |
+|---|---|
+| #4248 | 3 tautology asserts in `test_domain_classifier.py` |
+| #4249 | 4 tautology asserts across bytes_loader, json_utils, headline_generator, content_fetcher |
+| #4250 | 2 no-assert placeholders in `test_llm_config.py` |
+| #4251 | 2 explicit DELETE NO_FAILURE_PATH theme tests |
+| #4252 | 17 more NO_FAILURE_PATH theme tests |
+| #4277 | 10 hasattr-only tests in `test_research_metrics_extended.py` |
+| #4278 | 4 weak tests in `test_token_counter_extended.py` |
+| #4279 | LRU `or True` tautology in `test_search_cache_stampede.py` |
+| #4280 | 7 no-assertion tests in `test_search_integration.py` |
+| #4281 | 10 no-assertion tests across two `test_search_integration_*` files |
+| #4282 | 13 no-assertion downloader + rag-route-coverage tests |
+| #4286 | 8 no-assert / dead-code security + rate-limit tests |
+| #4287 | 7 no-assert "should not raise" metrics tests |
+
+### A4. Tier 3 SHADOW / TESTS_STDLIB / H4_ASSERTS_MOCK
+
+| PR | Description |
+|---|---|
+| #4269 | Tier 3 dead-code + stdlib shadow + redundant API tests |
+| #4270 | 3 shadow / script-style test files (text_cleaner, duplicate_links_fix) |
+| #4271 | 1 status-or-tautology fix + 3 hasattr-only tests in `test_diversity_explorer.py` |
+| #4272 | Whole-file `test_evidence_evaluator.py` (27 tests, 0 SUT imports) |
+| #4283 | 7 H4_ASSERTS_MOCK tests in `test_adaptive_explorer.py` |
+| #4284 | 2 H4_ASSERTS_MOCK tests in `test_parallel_explorer.py` |
+| #4285 | 3 H4_ASSERTS_MOCK / H6_only_isinstance tests in `test_diversity_explorer_coverage.py` |
+
+### A5. Tier 4 FLAKY / freezegun / requires_llm
+
+| PR | Description |
+|---|---|
+| #4288 | Gated `test_research_creation.py` with `@pytest.mark.requires_llm` (6 tests) |
+| #4289 | Migrated 3 `test_search_cache.py` TTL tests to freezegun |
+| #4291 | **Implemented unicode + URL-encoded path-traversal detection in PathValidator** — flipped 3 xfail tests to passing |
+
+### A6. Tier 5 REDUNDANT dedupes (16 PRs)
+
+| PR | Description |
+|---|---|
+| #4253 | Shadow `domain_classifier/test_models.py` deleted |
+| #4254 | 6 duplicates in `test_search_engine_factory_coverage` files |
+| #4255 | 10 duplicates in `test_dual_confidence_checker.py` (batch 1) |
+| #4256 | 4 shadow weighted_score tests in `test_base_constraint_checker.py` |
+| #4257 | 6 more duplicates in `test_dual_confidence_checker.py` (batch 2) |
+| #4258 | Whole-file `test_evidence_analyzer_coverage.py` (8 duplicates) |
+| #4259 | 17 duplicate EvidenceType tests in `test_base_evidence.py` |
+| #4260 | 8 duplicates in `test_rejection_engine_extended.py` |
+| #4261 | 2 cross-file duplicates (MCP + evidence analyzer) |
+| #4263 | 19 duplicates in `test_evaluator_integration.py` |
+| #4264 | 24 duplicates in `test_evaluator.py` (kept 2 unique survivors) |
+| #4265 | 18 duplicates in `test_findings_repository.py` |
+| #4266 | 16 duplicates across 2 filter test files |
+| #4267 | 22 cross-file duplicates between evaluator pure_logic + high_value |
+| #4268 | 7 misc duplicates across 4 files |
+
+### A7. False-positive documentation
+
+| PR | Description |
+|---|---|
+| #4292 | Documented `test_handles_errors_gracefully` as intentional startup-resilience behavior (PR #2118 walked back PR #2235's contract) |
+
+## Category B — Marked KEEP by the punchlist itself (294 entries)
+
+These entries appear in PUNCHLIST.md with `recommended_action: KEEP`
+or `(see Tier guidance)` indicating the test is **intentionally** a
+smoke / no-raise / one-liner enum-value check. The PUNCHLIST does not
+request a PR for these.
+
+Representative examples:
+- `test_app_coverage.py::test_https_branch_does_not_raise` — KEEP (smoke test)
+- `test_app_factory.py::test_response_has_security_headers` — KEEP
+- `test_rate_limiting_tracker_coverage.py::test_programmatic_mode_no_op` — KEEP
+- `test_search_cache.py::test_stampede_protection_single_fetch` — KEEP
+
+## Category C — Stale line-number references (~250 entries)
+
+These entries reference test methods that **still exist** in code but
+**no longer have the flagged issue** — the assertion has already been
+narrowed or the surrounding logic fixed by prior PRs.
+
+Verified examples (by `grep -cE "status_code in \[[^]]*200[^]]*(401|403|404|500)"`):
+
+| File | Punchlist entries | Actual current tautologies |
+|---|---|---|
+| `tests/research_library/routes/test_rag_routes.py` | 49 | **0** (all narrowed) |
+| `tests/research_library/routes/test_library_routes.py` | 48 | **0** |
+| `tests/web/routes/test_context_overflow_api.py` | 28 | 0 (last 2 fixed in #4273) |
+
+These entries cannot be addressed by a new PR — there is nothing left
+to change in code. A future PUNCHLIST regeneration would not produce
+these entries against the current state of the repo.
+
+## Category D — Legitimate config-smoke tests (5 entries)
+
+`tests/database/test_alembic_migrations.py` has 5 tests with
+`try/finally + no explicit assertion`:
+
+- `test_migration_with_read_only_database`
+- `test_migration_with_busy_timeout`
+- `test_migration_with_echo_enabled`
+- `test_migration_with_pool_pre_ping`
+- `test_migration_with_static_pool`
+
+Each test exercises a different SQLAlchemy engine configuration with
+`run_migrations(engine)`. Per the inline comment "raises on failure",
+the implicit assertion IS the contract — verifying the migration
+framework works against each engine config. Adding an explicit
+`assert get_current_revision(engine) == get_head_revision()` would
+only restate what `run_migrations(engine)` already enforces internally.
+
+These are legitimate engine-config smoke tests, not anti-patterns.
+
+## Cumulative impact
+
+- **50 PRs created**
+- **~3,720 weak/shadow tests removed or strengthened**
+- **~46,500 lines of dead/duplicate test code removed**
+- **1 production bug fixed** (`SQLCardStorage.create()` — silent
+  dropping of flat `source_type`)
+- **1 test bug fixed** (wrong URL in `test_context_overflow_api.py`
+  was passing on 404 silently)
+- **1 SUT feature implemented** (PathValidator now detects URL-encoded,
+  double-URL-encoded, and unicode-look-alike path-traversal attacks)
+- **6 Ollama-dependent integration tests** properly gated with
+  `@pytest.mark.requires_llm`
+- **3 TTL tests** migrated from `time.sleep` to `freezegun`
+
+## Verification
+
+To verify no more mechanically-addressable items remain, the following
+exhaustive AST scans were run across `tests/**/test_*.py`:
+
+| Pattern | Tests matching as of session end |
+|---|---|
+| Body is pure `pass` or empty (no skip marker) | 0 |
+| All assertions are literal `assert True` | 0 |
+| `import X; assert X is not None` (only assertion) | 0 |
+| `result is None or result is not None` tautology | 0 (in non-pending-PR code) |
+| `or True` literal-tautology asserts | 0 |
+| Shadow file (0 SUT imports, ≥5 tests, no `client.*` usage) | 0 deletable; remaining 4 have legitimate non-import-based SUT testing |
+
+Every concrete pattern flagged by the heuristics in PUNCHLIST.md
+methodology section H1–H12 has been swept.

docs/release_notes/0.2.0.md

@@ -0,0 +1,74 @@
+# Local Deep Research v0.2.0 Release Notes
+
+We're excited to announce Local Deep Research v0.2.0, a major update that brings significant improvements to research capabilities, performance, and user experience.
+
+## Major Enhancements
+
+### New Search Strategies
+- **Parallel Search**: Lightning-fast research that processes multiple questions simultaneously
+- **Iterative Deep Search**: Enhanced exploration of complex topics with improved follow-up questions
+- **Cross-Engine Filtering**: Smart result ranking across multiple search engines for higher quality information
+
+### Improved Search Integrations
+- **Enhanced SearxNG Support**: Better integration with self-hosted SearxNG instances
+- **Improved GitHub Integration**: More effective search and analysis of code repositories
+- **Better Source Selection**: Refined logic for choosing the most appropriate search engines per query
+
+### Technical Improvements
+- **Unified Database**: All settings and history now in a single `ldr.db` database
+- **Improved Ollama Integration**: Better reliability and error handling with local models
+- **Enhanced Error Recovery**: More graceful handling of connectivity issues and API errors
+
+### User Experience
+- **Enhanced Logging Panel**: Improved visibility with duplicate detection and better filtering
+- **Streamlined Settings UI**: Reorganized settings interface with better organization
+- **Research Progress Tracking**: More detailed real-time updates during research
+
+### Development Improvements
+- **PDM Support**: Switched to PDM for dependency management
+- **Pre-commit Hooks**: Added linting and code quality checks
+- **Code Security**: Added CodeQL integration with analysis scripts
+- **Improved Documentation**: Better development guides and setup instructions
+
+## API Changes
+
+- Renamed and consolidated some API functions for consistency
+- Added support for additional parameters in research configuration
+- Improved error handling and response formatting
+
+## Migration Notes
+
+- The application now uses a unified database (`ldr.db`) that will automatically migrate data from older databases
+- If upgrading from v0.1.x, your settings and research history will be automatically migrated on first run
+- The `llm_config.py` file has been deprecated in favor of direct environment variable configuration
+
+## Bug Fixes
+
+- Fixed issues with settings persistence across sessions
+- Resolved UI rendering problems in the history and results pages
+- Fixed socket.io event handling and client disconnection issues
+- Improved handling of large document collections
+- Fixed API endpoint URL inconsistencies
+
+## Contributors
+This release represents the combined efforts of multiple contributors :
+- @djpetti, @HashedViking, @LearningCircuit (core contributors to this release; sorted alphabetically)
+- @dim-tsoukalas, @scottvr (sorted alphabetically)
+
+## Get Involved
+
+- Join our [Discord](https://discord.gg/ttcqQeFcJ3) for support and discussions
+- Follow our [Subreddit](https://www.reddit.com/r/LocalDeepResearch/) for announcements and updates
+- Report bugs and request features on our [GitHub Issues](https://github.com/LearningCircuit/local-deep-research/issues)
+
+---
+
+## Installation
+
+Download the [Windows Installer](https://github.com/LearningCircuit/local-deep-research/releases/download/v0.2.0/LocalDeepResearch_Setup.exe) or install via pip:
+
+```bash
+pip install local-deep-research
+```
+
+Requires Ollama or other LLM provider. See the [README](https://github.com/LearningCircuit/local-deep-research/blob/main/README.md) for complete setup instructions.

docs/release_notes/0.4.0.md

@@ -0,0 +1,55 @@
+# Local Deep Research v0.4.0 Release Notes
+
+We're excited to announce Local Deep Research v0.4.0, bringing significant improvements to search capabilities, model integrations, and overall system performance.
+
+## Major Enhancements
+
+### LLM Improvements
+- **Custom OpenAI Endpoint Support**: Added support for custom OpenAI-compatible endpoints
+- **Dynamic Model Fetching**: Improved model discovery for both OpenAI and Anthropic using their official packages
+- **Increased Context Window**: Enhanced default context window size and maximum limits
+
+### Search Enhancements
+- **Journal Quality Assessment**: Added capability to estimate journal reputation and quality for academic sources
+- **Enhanced SearXNG Integration**: Fixed API key handling and prioritized SearXNG in auto search
+- **Elasticsearch Improvements**: Added English translations to Chinese content in Elasticsearch files
+
+### User Experience
+- **Search Engine Visibility**: Added display of selected search engine during research
+- **Better API Key Management**: Improved handling of search engine API keys from database settings
+- **Custom Context Windows**: Added user-configurable context window size for LLMs
+
+### System Improvements
+- **Logging System Upgrade**: Migrated to `loguru` for improved logging capabilities
+- **Memory Optimization**: Fixed high memory usage when journal quality filtering is enabled
+- **Resumable Benchmarks**: Added support for resuming interrupted benchmark runs
+
+## Bug Fixes
+- Fixed broken SearXNG API key setting
+- Memory usage optimizations for journal quality filtering
+- Cleanup of OpenAI endpoint model loading features
+- Various fixes for evaluation scripts
+- Improved settings manager reliability
+
+## Development Improvements
+- Added test coverage for settings manager
+- Cleaner code organization for LLM integration
+- Enhanced API key handling from database settings
+
+## New Contributors
+- @JayLiu7319 contributed support for Custom OpenAI Endpoint models
+
+## Full Changelog
+For complete details of all changes, see the [full changelog](https://github.com/LearningCircuit/local-deep-research/compare/v0.3.12...v0.4.0).
+
+---
+
+## Installation
+
+Download the [Windows Installer](https://github.com/LearningCircuit/local-deep-research/releases/download/v0.4.0/LocalDeepResearch_Setup.exe) or install via pip:
+
+```bash
+pip install local-deep-research
+```
+
+Requires Ollama or other LLM provider. See the [README](https://github.com/LearningCircuit/local-deep-research/blob/main/README.md) for complete setup instructions.

docs/release_notes/1.6.0.md

@@ -0,0 +1,62 @@
+# Journal Quality Redesign — release notes (pending)
+
+Staging notes for the journal-quality redesign shipped by [#3081](https://github.com/LearningCircuit/local-deep-research/pull/3081). Fold into the next tagged version's release-notes file when cutting the release.
+
+## Major Features
+
+### Tiered journal quality scoring
+
+The journal-reputation filter now uses a five-tier pipeline (predatory check → OpenAlex snapshot → DOAJ → institution affiliation salvage → optional LLM) instead of calling an LLM per source. A bundled read-only reference database of ~280K academic venues + ~120K institutions powers Tiers 1–3 in 100–300 µs per lookup, eliminating the multi-second per-source latency of the previous LLM-only path.
+
+See `docs/journal-quality.md` for the full scoring table, data-source licenses, and the Tier-1 predatory-whitelist override.
+
+### Journal dashboard
+
+New analytics view at **Analytics → Journals** lists every source in the reference DB with server-side filtering, pagination, and sortable columns (h-index, quartile, impact factor, quality score, DOAJ status, predatory flag). Works on ~200K rows without loading them all into memory thanks to the shared read-only SQLite reference DB (`mode=ro&immutable=1`, `chmod 0o444` after build).
+
+### Quality tags in research output
+
+Each source in the research report now carries a compact quality tag:
+
+```
+[1] Physical Review Letters [Q1 ★★★★★]
+[2] Some Niche Journal [Q3 ★★]
+```
+
+Predatory-flagged sources that are not whitelisted are auto-removed before the report is assembled.
+
+## BREAKING — `journals` table columns removed
+
+The per-user encrypted `journals` table no longer stores the following columns:
+
+`issn`, `issn_list`, `publisher`, `openalex_source_id`, `source_type`, `h_index`, `impact_factor`, `works_count`, `cited_by_count`, `is_in_doaj`, `has_doaj_seal`, `is_predatory`, `predatory_source`, `is_indexed_in_scopus`, `data_version`, `sjr_quartile`.
+
+These metrics are now served exclusively by the bundled read-only reference DB (`journal_quality.db`). The per-user `journals` table is now a Tier-4 LLM cache only and retains: `id`, `name`, `name_lower`, `quality`, `score_source`, `quality_model`, `quality_analysis_time`.
+
+### Impact
+
+- If you have custom SQL tooling that queries the removed columns on the per-user DB, point it at the reference DB via the new accessor (`journal_quality/db.py`) or the `/api/journals` dashboard endpoint instead.
+- The upgrade itself preserves every row in the `journals` table — only the dropped columns disappear. `quality`, `name`, `name_lower`, and the LLM-cache columns stay.
+- No action is needed for users who do not write their own SQL.
+
+## New data downloads on first use
+
+The journal-quality reference DB builds itself on first access from five third-party snapshots (OpenAlex, DOAJ, Stop Predatory Journals, JabRef abbreviations, OpenAlex Institutions — all CC0 or MIT licensed). The build streams several hundred MB from the OpenAlex S3 bulk dump plus smaller snapshots from the other sources, unpacks to ~1 GB of intermediate working set, and typically takes 1–2 minutes on a normal connection. All five sources fetch **in parallel** via a ThreadPoolExecutor — wall-clock is bound by the slowest source (the OpenAlex sources snapshot at ~30–60 s) rather than the sum. On fresh installs the first research request returns immediately with `[preprint — not in journal catalog]` / `[journal quality data is downloading…]` placeholder tags; a second search a minute later picks up the real Q-tier scores. The dashboard's Journals page (`/metrics/journals`) shows per-source progress bars during the build.
+
+## Settings
+
+One new opt-in toggle and four opt-out per-engine toggles:
+
+- `search.journal_reputation.enable_llm_scoring` (default `false`, **opt-in**) — if enabled, Tier 3.6 and Tier 4 use SearXNG + the LLM for unknown journals.
+- `search.engine.web.arxiv.journal_reputation.enabled` (default `true`, opt-out)
+- `search.engine.web.openalex.journal_reputation.enabled` (default `true`, opt-out)
+- `search.engine.web.semantic_scholar.journal_reputation.enabled` (default `true`, opt-out)
+- `search.engine.web.nasa_ads.journal_reputation.enabled` (default `true`, opt-out) — per-engine toggles for running the reputation filter over each academic-search engine's results; disable individually to skip filtering for a given engine.
+
+## Operational notes
+
+- The first user request after upgrade pays a one-time migration cost on the per-user `journals` table (SQLite batch rebuild). Typical libraries complete in under a second; very large libraries (100k+ journals) may stall 2–5 seconds.
+- The first visit to **Analytics → Journals** before the reference DB has finished building shows the page frame with a per-source download banner rather than a data grid; `/api/journals` returns HTTP 503 with `{"status": "error", "message": "Journal reference database not available."}` until the build completes. A research request run in parallel kicks off the build in a background daemon thread, so warming the filter path first is the fastest way to make the dashboard live on a fresh install.
+- Windows installs now enforce read-only on the reference DB via `SetFileAttributesW`, matching the POSIX `chmod 0o444` behavior.
+- The bulk data download fails fast with a clear error if free disk space falls below 2 GB.
+- Transient network failures during the bulk download retry up to three times with exponential backoff (1/2/4 s), respecting `Retry-After` on HTTP 429.

docs/release_notes/1.6.10.md

@@ -0,0 +1,85 @@
+
+### 💥 Breaking Changes
+
+- **`llm.model` no longer auto-fills `gemma3:12b`.** Pre-1.7 installs silently downloaded a multi-GB Ollama binary on first use. The field is now empty by default — pick a model in **Settings → LLM**, or research will fail loudly with a clear `ValueError`. Existing users with a saved `llm.model` setting are unaffected; fresh installs see an inline warning in Settings (`aria-live="polite"`) and a clear error on research start. Affects all built-in providers (`openai`, `anthropic`, `ollama`, `lmstudio`, `llamacpp`, `google`, `openrouter`, `ionos`, `xai`). ([#3670](https://github.com/LearningCircuit/local-deep-research/pull/3670))
+- **`llamacpp` provider migrated from in-process to HTTP.** No longer loads `.gguf` files in-process via `llama-cpp-python` — connects to `llama-server`'s OpenAI-compatible HTTP endpoint instead. Removed settings: `llm.llamacpp_model_path`, `llm.llamacpp_n_batch`, `llm.llamacpp_n_gpu_layers`, `llm.llamacpp_f16_kv`. New settings: `llm.llamacpp.url` (default `http://localhost:8080/v1`), `llm.llamacpp.api_key` (optional, for auth proxies). Migration: run `llama-server -m <your-model.gguf>` and the default URL picks it up; alternatively use the `openai_endpoint` provider pointed at the same URL. ([#3670](https://github.com/LearningCircuit/local-deep-research/pull/3670))
+
+### 🔒 Security
+
+- Fix LM Studio model detection (#3800) — the settings dropdown was silently dropping models for every auto-discovered provider except Ollama / OpenAI / Anthropic / OpenAI-Compatible-Endpoint, including LM Studio, Llama.cpp, Google Gemini, xAI, OpenRouter, and IONOS. While investigating, also fixed a separate credential-leak class where the route's auto-discovery loop sent every other provider's configured API key in the `Authorization` header to the local LM Studio / llama-server endpoint on every settings-page load. ([#3800](https://github.com/LearningCircuit/local-deep-research/pull/3800))
+- Fix SSRF parser-differential bypass (GHSA-g23j-2vwm-5c25). URLs containing
+  backslash, whitespace, or ASCII control bytes are now rejected upfront by the
+  SSRF validator and notification-URL validator; hostname extraction switched
+  from `urllib.parse.urlparse` to `urllib3.util.parse_url` so the validator and
+  the HTTP client agree on destination by construction. Credit: @Fushuling,
+  @RacerZ-fighting.
+- Hardened SSRF defenses against AWS ECS task metadata
+  (`169.254.170.2`, `169.254.170.23`), Tencent Cloud (`169.254.0.23`),
+  and AlibabaCloud (`100.100.100.200`) metadata endpoints — these are
+  now always blocked alongside the existing AWS IMDS / Azure / OCI /
+  DigitalOcean entry (`169.254.169.254`). Redacted credentials, path,
+  and query from URL-rejection logs (operators with grep/regex tooling
+  on the rejection log lines will see authority-only `scheme://host:port`
+  instead of full URLs going forward).
+- SSRF defense-in-depth: block IPv6 transition prefixes that can wrap
+  private IPv4 destinations on hosts with kernel sit0/NAT64 routes.
+
+  - `2002::/16` (6to4, RFC 3056 — deprecated by RFC 7526)
+  - `64:ff9b::/96` (NAT64 well-known, RFC 6052)
+  - `64:ff9b:1::/48` (NAT64 local-use, RFC 8215 — same SSRF threat class
+    as the WKP; missing it earned a HackerOne bounty against
+    ssrf_filter)
+  - `2001::/32` (Teredo, RFC 4380)
+  - `100::/64` (IPv6 discard, RFC 6666)
+  - `::/96` (IPv4-Compatible IPv6, RFC 4291 §2.5.5.1 — DEPRECATED 2006;
+    same SSRF threat class as the transition prefixes)
+
+  The metadata-IP block is hardened against IPv6-wrapped IMDS access:
+  when an IPv6 destination falls in a NAT64 prefix, the embedded IPv4 is
+  extracted and matched against `ALWAYS_BLOCKED_METADATA_IPS`, so
+  `[64:ff9b::a9fe:a9fe]` cannot reach 169.254.169.254 even on a NAT64
+  host.
+
+  Operators on IPv6-only deployments using DNS64+NAT64 (where outbound
+  IPv4 traffic is synthesized through `64:ff9b::/96`) can opt back in via
+  the env-only setting `security.allow_nat64`
+  (`LDR_SECURITY_ALLOW_NAT64=true`). The opt-in is scoped strictly to
+  the two NAT64 prefixes — 6to4, Teredo, and discard remain blocked
+  unconditionally, and the IMDS embedded-IPv4 carve-out still applies.
+
+### ✨ New Features
+
+- **LM Studio optional API key.** New `llm.lmstudio.api_key` setting for LM Studio instances with authentication enabled. ([#3670](https://github.com/LearningCircuit/local-deep-research/pull/3670))
+- **Release notes now use [towncrier](https://towncrier.readthedocs.io) news fragments.** Each PR drops one tiny `changelog.d/<id>.<category>.md` file (categories: `breaking`, `security`, `feature`, `bugfix`, `removal`, `misc`). At release prep, the maintainer runs `pdm run towncrier build --version <X.Y.Z> --yes`, which renders the fragments into `docs/release_notes/<X.Y.Z>.md` and removes them. Replaces the previous shared-file workflow, which scaled poorly at LDR's PR throughput. See `changelog.d/README.md` for the convention. ([#3773](https://github.com/LearningCircuit/local-deep-research/pull/3773))
+- **New citation format: source-tagged with global numbering.** Reports can now render citations as `[arxiv-1]`, `[openai.com-2]`, `[arxiv-3]` — the source tag identifies what kind of source each citation is (short URLClassifier tag for known academic sources, cleaned domain for generic web URLs, or the collection name for local RAG/library hits, e.g. `[my-papers-4]`), while the number stays the original global counter so labels never collide and inline citations match the bibliography order. Opt in via `report.citation_format → "Source-tagged with global numbering"`; the existing default remains `Numbers with hyperlinks [1]`. ([#4012](https://github.com/LearningCircuit/local-deep-research/pull/4012))
+- Context-overflow warnings (front-page banner, result-page banner, completion toast) now show a clickable action that takes you straight to the relevant diagnostics — historical warnings link to the context-overflow metrics page, while per-research warnings link to that research's overflow section on its details page.
+- Context-overflow warnings now include structured fields (`research_id`, `model`, `provider`) and detect both input-only and total-context (input+output) overflow. Four distinct tags (`[provider-confirmed]`, `[total-context]`, `[estimated]`, `[estimated-total-context]`) make different overflow causes distinguishable in logs and alerts. Hosted providers (OpenAI, Anthropic, OpenRouter) get estimation-based detection so their truncation events are no longer invisible. The high-context VRAM warning now links directly to the context-overflow metrics page.
+- OpenAI-compatible runtime errors (LM Studio / vLLM / llama.cpp server / OpenRouter / custom endpoint) now surface a message that names the provider, configured base URL, and model. The existing `Error type: <code>` token convention is extended with seven `openai_*` codes (`openai_connection_refused`, `openai_timeout`, `openai_auth`, `openai_permission_denied`, `openai_model_not_found`, `openai_bad_request`, `openai_unknown`), and `ErrorReporter` maps each to the right `ErrorCategory`. The original exception text is preserved in a `Details:` suffix; userinfo embedded in a base URL (`https://u:key@host/v1`) is stripped before display.
+- The `/metrics/context-overflow` page is reorganised around per-research truncation diagnosis: redundant aggregate sections (Token Usage Overview, Token Usage Over Time, Model Usage Breakdown, Context Limit Distribution doughnut, Token Usage by Phase) are dropped in favour of empty-state messages and a recoloured and re-shaped scatter chart that buckets points by context-utilisation ratio (green circle / amber triangle / red diamond — shape gives a redundant signal for colourblind users) rather than per-limit dashed reference lines. Tooltips now explicitly label requests with unknown context limits or unknown token counts. The main `/metrics` dashboard gains a small Context Overflow summary panel linking to the full diagnostics page; on aggregation failure the panel shows `—` rather than a misleading `0%`.
+- Upload rate limits are now configurable via `LDR_SECURITY_RATE_LIMIT_UPLOAD_USER` and `LDR_SECURITY_RATE_LIMIT_UPLOAD_IP`. The default cap is raised from `10 per minute;100 per hour` to `60 per minute;1000 per hour`, unblocking bulk RAG library uploads.
+
+### 🐛 Bug Fixes
+
+- Fix progress page log panel sizing so it uses CSS layout instead of JavaScript height calculations. ([#2606](https://github.com/LearningCircuit/local-deep-research/pull/2606))
+- Progress page current-task text now preserves line breaks and wraps long generated task messages instead of collapsing them into a hard-to-read wall of text. ([#2609](https://github.com/LearningCircuit/local-deep-research/pull/2609))
+- Live research log entries now keep the same newest-first visual order as loaded log history instead of appearing at the bottom of the log panel. ([#2610](https://github.com/LearningCircuit/local-deep-research/pull/2610))
+- **Journal-quality dashboard ("Your Research") no longer shows orphan papers from deleted research sessions.** Deleting a research session previously left behind `Paper` rows whose `PaperAppearance` had cascade-deleted, so the dashboard kept counting them. The aggregate top-200 query and the predatory-blocked count now exclude papers that have no remaining appearances. Per-research views were already correct. ([#3544](https://github.com/LearningCircuit/local-deep-research/pull/3544))
+- Fix the model dropdown in Settings to populate for every auto-discovered LLM provider, not just Ollama / OpenAI / Anthropic / OpenAI-Compatible-Endpoint. LM Studio, Llama.cpp, Google Gemini, xAI, OpenRouter, and IONOS were silently dropped by the frontend (`processModelData`) even when the backend returned their models correctly. ([#3800](https://github.com/LearningCircuit/local-deep-research/pull/3800))
+- Fix file descriptor exhaustion (#3816) caused by `ChatOllama._async_client` not being released alongside `_client`. `_close_base_llm()` now closes both the sync and async httpx clients owned by `ChatOllama`, addressing the eventpoll-FD growth seen in long-running deployments. ([#3816](https://github.com/LearningCircuit/local-deep-research/pull/3816))
+- Fix the research details "View Journals" button so it opens the scoped journal-quality dashboard instead of a missing page. ([#3825](https://github.com/LearningCircuit/local-deep-research/pull/3825))
+- Fix Download Manager failure after 20–30 PDFs with `UNIQUE constraint failed: documents.document_hash` and the cascading PendingRollbackError that prevented further downloads. ([#3827](https://github.com/LearningCircuit/local-deep-research/pull/3827))
+- Fix login/registration over `http://localhost:5001` on Docker Desktop.
+  The session cookie's `Secure` flag is now driven purely by the request
+  protocol (HTTPS) rather than by source-IP heuristics, so HTTP requests
+  from any IP — including Docker Desktop's NAT gateway — now keep their
+  cookies and CSRF tokens. ([#3849](https://github.com/LearningCircuit/local-deep-research/pull/3849))
+- Fix Ollama embedding indexing aborting with `"input length exceeds the context length"` (HTTP 500) by exposing a configurable `embeddings.ollama.num_ctx` setting. The new field appears on the embedding settings page when the Ollama provider is selected and ships with a default of 8192 tokens, which covers chunk sizes up to ~5000 characters for `nomic-embed-text`, `mxbai-embed-large`, `bge-m3`, and `qwen3-embedding`. ([#3870](https://github.com/LearningCircuit/local-deep-research/pull/3870))
+- LangGraph agent strategy error messages now include the exception type, and the `'str' object has no attribute 'model_dump'` failure (issue #3897) is translated into a rich on-result-page recommendation listing the three possible causes — proxy/shim, serving stack, or model — with concrete remediation steps for each. ([#3926](https://github.com/LearningCircuit/local-deep-research/pull/3926))
+- The Context Overflow summary panel on `/metrics` now honors the dashboard's research-mode filter (Quick / Detailed / All) the same way the rest of the panels do.
+
+### 📝 Other Changes
+
+- Added `LDR_DISABLE_RATE_LIMITING` as the canonical name for disabling HTTP rate limiting, matching the `LDR_` env-var convention. The legacy `DISABLE_RATE_LIMITING` still works but emits a deprecation warning. Note: this is distinct from `LDR_RATE_LIMITING_ENABLED`, which controls the adaptive search-engine rate limiter — a different subsystem (#3905).
+- Migrated CI workflows to the canonical `LDR_DISABLE_RATE_LIMITING` env var (added in #3936) and closed a test-isolation gap in `test_enabled_by_default` that did not clear the canonical var.
+- Removed unused chart-rendering JavaScript on the context-overflow analytics page (token-usage-over-time stacked bar, model-token-stats table, context-limits-distribution doughnut, and the configured-limits list). These functions stopped being called when the page was reorganised to focus on truncation diagnosis (#3792); deleting them now drops ~200 lines of dead code without any user-visible change.
+- The prerelease Docker workflow now also re-points the floating `prerelease` tag at each new RC manifest. Testers can pin `image: localdeepresearch/local-deep-research:prerelease` in their compose file and run `docker compose pull && docker compose up -d` to fetch the latest RC without editing the tag each cycle. Versioned tags (`prerelease-vX.Y.Z-<sha>`) are still published for reproducible testing.

docs/release_notes/1.6.11.md

@@ -0,0 +1,28 @@
+
+### 🔒 Security
+
+- Extended the cloud-metadata IP absolute-block in `NotificationURLValidator` to Apprise plugin schemes (signal, gotify, ntfy/ntfys, mattermost, rocketchat, matrix, json, xml, form, mailto). Previously only the http/https branch ran the IMDS guard, so a notification URL like `signal://169.254.169.254/...` would round-trip to Apprise even though Apprise translates it into a plain HTTP request against that host. LAN/loopback reach for self-hosted plugin endpoints (the #4006 use case) is unchanged.
+- Fix Google Gemini API key leak in `list_models_for_api` error log. The
+  URL constructed at request time embeds the key as a `?key=...` query
+  parameter (per Google's documented API), so when the underlying request
+  failed, the `requests` exception message — and therefore the
+  `logger.exception(...)` traceback — contained the full URL with the key.
+  The except handler now redacts the key from the message and uses
+  `logger.warning` (no exception chain).
+
+### 🐛 Bug Fixes
+
+- **JavaScript rendering disabled by default in the production Docker image (#3826).** The headless-browser fallback in the content fetcher (Crawl4AI/Playwright) was previously attempted on every fetch even though the default Docker production image ships without a Chromium binary - each attempt failed loudly and contributed to the memory growth reported in issue #3826. A new `web.enable_javascript_rendering` setting (default `false`) gates the fallback. In limited (mostly accidental) internal benchmark comparisons - between dev instances that happened to have Chromium installed and routine Docker runs that did not - JS rendering did not measurably improve research quality, and most regular benchmark runs are on Docker without Chromium anyway, so the change does not regress observed quality. To enable: install Chromium in the runtime environment (`playwright install --with-deps chromium`) and toggle the setting in the UI. ([#3826](https://github.com/LearningCircuit/local-deep-research/pull/3826))
+- **Embeddings now work against OpenAI-compatible local servers (LM Studio, vLLM, llama.cpp).** Previously the OpenAI embeddings provider only listed itself as available when an API key was configured, and the relevant `embeddings.openai.*` settings were not registered for the UI to surface — so users who ran an OpenAI-compatible local server had no way to point the embeddings tab at it. The provider now also reports available when only `embeddings.openai.base_url` is set, substitutes a placeholder API key for keyless endpoints (matching the LM Studio LLM provider's behavior), and ships a new `settings_openai_embeddings.json` defaults file that registers `embeddings.openai.api_key`, `embeddings.openai.base_url`, `embeddings.openai.model`, and `embeddings.openai.dimensions`. The model-list lookup now routes through the configured `base_url` so model discovery also targets the local server. ([#3883](https://github.com/LearningCircuit/local-deep-research/pull/3883))
+- **"Start Research" button no longer silently does nothing for users with a non-multiple-of-512 context window value.** The hidden `context_window` input on the research form had a `step="512"` constraint, so any stored value not aligned to that grid (e.g. `25000`) failed HTML5 form validation. Because the input lives inside a `display:none` container that is only shown for local providers, the browser could not focus the invalid field to report the error, so submission was aborted with no visible message and no log line — the click appeared to do nothing. Relaxed the step to `1` so any in-range integer is accepted; the value is still bounded by `min`/`max` and persisted unchanged. ([#3909](https://github.com/LearningCircuit/local-deep-research/pull/3909))
+- **Chinese/Japanese/Korean text now renders in exported PDFs.** The default PDF stylesheet hard-coded a Latin-only font stack, so any CJK characters in the research result were dropped silently from the download even though they displayed correctly in the browser. The minimal CSS now includes a broad CJK fallback chain (Noto Sans CJK, PingFang, Hiragino, Apple SD Gothic Neo, Microsoft YaHei/JhengHei, Yu Gothic, Malgun Gothic, SimSun) covering Windows, macOS, and Linux desktops out of the box, and the Docker image now installs `fonts-noto-cjk` so the slim base image has glyph coverage. Linux pip/server installs still need a CJK font package on the host — see [install-pip.md](../install-pip.md) and the [FAQ](../faq.md#chinese--japanese--korean-text-is-missing-from-exported-pdfs) for the per-distro commands. ([#4055](https://github.com/LearningCircuit/local-deep-research/pull/4055))
+- Cross-engine filtering now keeps LLM ranking fallbacks within the subset of results the model actually evaluated, preventing unevaluated search results from leaking into downstream synthesis.
+- Fix file-descriptor leak (`anon_inode:[eventpoll]`) in `_close_base_llm` when invoked while an asyncio loop is running. The previous code path skipped the async-client close in that case and documented that the "loop owner" would close instead — but no loop-owner cleanup actually exists in the project, so the inner `httpx.AsyncClient` (and its `epoll_create` FD) was silently abandoned. The same skip-path fires under the default `langgraph-agent` strategy because LangGraph dispatches some tool steps via asyncio internally, so close calls reached from a sync `finally` can still land inside a live loop. Cleanup now runs in a brief daemon thread that owns its own loop, so `asyncio.run(aclose())` works regardless of the caller's loop state; a bounded 5-second `join` keeps it from holding up shutdown if the Ollama server is unresponsive. This is the gap left by #3855's coverage of #3816 — same leak class, different code path.
+
+  Also fixes a smaller secondary leak in the Docker `HEALTHCHECK`: `urllib.request.urlopen('http://localhost:5000/api/v1/health')` had no `timeout=` argument, so when the app slowed down (FD exhaustion or otherwise) Docker's 10s healthcheck timeout SIGKILL'd the `sh -c` wrapper but the python child got reparented to PID 1 and hung forever, each contributing a `pidfd` and a TCP socket against the app. Adding `timeout=8` lets the child return/raise before Docker's wall so it exits cleanly and gets reaped.
+
+### 📝 Other Changes
+
+- Added a "Testing a Release Candidate (Prerelease Image)" subsection to the developer guide (`docs/developing.md`) covering both the floating `:prerelease` and immutable `prerelease-vX.Y.Z-<sha>` Docker Hub tags, with a side-by-side compose snippet that runs the RC on port 5001 with isolated volumes so a broken migration cannot affect a production instance.
+- Pool-dispose failures in the periodic WAL/SHM handle-release workaround (ADR-0004) now log at ``WARNING`` instead of ``DEBUG`` so silent drift becomes visible. Documented in ``docs/CONFIGURATION.md`` that ``LDR_APP_DEBUG=true`` also enables Loguru ``diagnose=True``, which can materialise sensitive locals into exception traces — do not enable in production.
+- Refactored environment settings to use specialized exception classes, improving error observability and alignment with `TRY003` standards.

docs/release_notes/1.6.12.md

@@ -0,0 +1,22 @@
+
+### 🔒 Security
+
+- RAG collection upload endpoint now validates per-file size (50MB limit) and per-request file count (200 limit), matching the research upload endpoint. Previously only the request-level `MAX_CONTENT_LENGTH` (10GB) was enforced, allowing a single oversized file or a request with thousands of zero-byte files to reach the per-file processing loop. Pre-flight `Content-Length` check rejects oversized files before reading bytes into memory.
+
+### ✨ New Features
+
+- **New citation format: source-tagged with global numbering.** Reports can now render citations as `[arxiv-1]`, `[openai.com-2]`, `[arxiv-3]` — the source tag identifies what kind of source each citation is (short URLClassifier tag for known academic sources, cleaned domain for generic web URLs, or the collection name for local RAG/library hits, e.g. `[my-papers-4]`), while the number stays the original global counter so labels never collide and inline citations match the bibliography order. Opt in via `report.citation_format → "Source-tagged with global numbering"`; the existing default remains `Numbers with hyperlinks [1]`. ([#4012](https://github.com/LearningCircuit/local-deep-research/pull/4012))
+- **Add LaTeX math rendering support.** Mathematical formulas written with `$...$` (inline) and `$$...$$` (display) notation are now rendered using KaTeX in research reports.
+- Per-PDF library storage cap (`research_library.max_pdf_size_mb`) default raised from 100 MB to 3 GB so it no longer silently truncates large academic PDFs after the recent upload-validator bump. The setting's UI ceiling is also raised from 500 MB to 10 GB, and the `PDFStorageManager` / `download_service` fallback defaults are updated to match. Deployments that want a tighter bound can still lower the setting via the UI.
+- Per-file upload cap (`FileUploadValidator.MAX_FILE_SIZE`) is now configurable and defaults to 3 GB (was 50 MB) so large academic datasets and PDFs fit out of the box. Deployments that want a tighter bound can lower it via the `LDR_SECURITY_UPLOAD_MAX_FILE_SIZE_MB` environment variable or the `security.upload_max_file_size_mb` setting. Memory usage stays bounded by the existing 5 MB spool-to-disk threshold on multipart requests.
+- The library RAG indexer now prunes old quarantined FAISS index files (`<hash>.faiss.corrupt-*` and matching `.pkl.corrupt-*`) at quarantine time, keeping only the 5 most recent per base path. Prevents the `rag_indices/` cache directory from filling up on systems that experience recurring corruption, while preserving recent diagnostic artefacts. Follow-up to #4197 / #4200.
+
+### 🐛 Bug Fixes
+
+- Fixed `UnicodeDecodeError` on Windows when loading settings, security config, benchmark results, and Vite manifest files. All text-mode `open()` calls now use explicit UTF-8 encoding, and JSON config files use `utf-8-sig` to handle BOM-prefixed files from Windows editors. Fixes #3743. ([#3797](https://github.com/LearningCircuit/local-deep-research/pull/3797))
+- Fixed embedding-model dropdown showing "No models available" with LM Studio (and other OpenAI-compatible local servers) when an embedding model whose name didn't include the literal `embedding` token was loaded (e.g. `nomic-embed-text-v1.5`). The OpenAI and Ollama embedding providers no longer guess from the model name — every model the endpoint reports is shown so the user can pick the one they actually loaded. Ollama still tags models when its `/api/show` capabilities response is available. Fixes #4195. ([#4195](https://github.com/LearningCircuit/local-deep-research/pull/4195))
+- Closed a follow-up race condition in the library RAG indexer where two concurrent workers indexing different documents into the same collection could lose each other's embeddings from the FAISS file — last writer's `save_local` overwrote the earlier writer's chunks (the chunks survived in the DB, but the index file was missing them until a force-reindex rebuilt it). The save path now reloads from disk under the per-`(user, index_path)` lock before adding, so concurrent writers' chunks are merged instead of overwriting each other. Follow-up to #4197/#4200.
+- Fixed a data-loss bug where the library RAG index file (`.faiss`) was silently deleted when integrity verification failed during concurrent auto-indexing — destroying hundreds of previously-indexed documents in one go. Corrupted index files are now quarantined to `<path>.corrupt-<ns>` so they remain recoverable, and concurrent indexers are serialised by a per-`(user, index_path)` lock so the race that produced the checksum mismatch no longer occurs. Also drops the per-document `PRAGMA wal_checkpoint(FULL)` that contributed to `database is locked` errors under bulk-download concurrency. Fixes #4197.
+- Paginate the ``/history/logs/<id>`` endpoint (default 500, clamped to 5000) so loading a long research's history no longer materialises every ResearchLog row server-side or pushes a 50+ MB JSON response that the browser ultimately prunes to 500 entries anyway. Complements the live-socket truncation in PR #4004.
+- Three module-level per-user lock dicts (`_user_init_locks` in `database/library_init.py`, `_user_locks` in `database/backup/backup_service.py`, `_user_critical_locks` on `QueueProcessorV2`) previously accumulated one `threading.Lock` entry per username over the process lifetime with no removal hook on user-close. Bounded by total user count (~900 bytes/user across all three), so not visible on typical self-hosted instances — but long-lived multi-user deployments with user-account churn would see slow memory creep. The three modules now expose `pop_user_*_lock(username)` helpers and a shared `_pop_per_user_locks` call in the connection-cleanup module fires them from both the idle-cleanup sweeper and the logout / password-change paths, matching the cleanup already done for scheduler-job registrations and session-password store.
+- Tighten the `model_dump` error-pattern regex introduced in #3926 so the rich proxy/shim hint only fires for the canonical `object has no attribute 'model_dump'` AttributeError, not for unrelated traces that happen to contain the substring `model_dump`.

docs/release_notes/1.6.13.md

@@ -0,0 +1,7 @@
+
+### 🐛 Bug Fixes
+
+- Citation handlers and the report structure generator now route LLM responses through `get_llm_response_text`, stripping `<think>` reasoning blocks from synthesized answers and reports (previously leaked verbatim with reasoning models) and normalizing string/message responses in one place.
+- Cross-engine filtering now deduplicates repeated LLM-ranked result indices so the same source does not appear multiple times with different citation numbers.
+- Fix `AttributeError` in citation handlers during follow-up research when LLMs return string responses instead of message objects.
+- Restore `str()` coercion in the precision citation handler's key-fact extraction so it always returns a string (fixes a mypy `no-any-return` failure introduced in #3884).

docs/release_notes/1.6.3.md

@@ -0,0 +1,60 @@
+# 1.6.3 — release notes
+
+LLM provider rework shipped by [#3670](https://github.com/LearningCircuit/local-deep-research/pull/3670). 1.6.3 was tagged without release notes; this file documents its breaking changes retroactively.
+
+## BREAKING — `llm.model` no longer auto-fills `gemma3:12b`
+
+Before this release, leaving the **Language Model** field empty in Settings caused fresh installs to silently download a 7–12 GB Ollama binary (`gemma3:12b`) the user had not asked for. The default is now an empty string (`""`), and `get_llm()` raises `ValueError("LLM model not configured...")` when the field is unset for any provider that requires a model name.
+
+### Impact
+
+- **Fresh installs**: open Settings → LLM, pick a provider, and choose a model name (e.g. `llama3.1:8b` for Ollama, `gpt-4o-mini` for OpenAI, `claude-3-5-sonnet-20241022` for Anthropic). The model field shows a yellow inline warning while empty so the requirement is discoverable.
+- **Existing installs**: any user who already had `llm.model` saved (including `gemma3:12b`) keeps their value — `import_settings(overwrite=False)` at `settings/manager.py:1129` preserves any non-`None` DB value, so only fresh DBs and never-written rows pick up the new empty default.
+- The corresponding HTTP endpoint `/api/check/ollama_model` now returns HTTP 400 with `error_type: "model_not_configured"` when neither the query parameter nor `llm.model` is set, instead of silently substituting `gemma3:12b`.
+
+## BREAKING — `llamacpp` provider switched to HTTP
+
+The `llamacpp` provider no longer loads `.gguf` files in-process via `langchain_community.llms.LlamaCpp`. It now talks HTTP to llama.cpp's OpenAI-compatible `llama-server`, mirroring the LM Studio integration.
+
+### Migration steps for existing llamacpp users
+
+1. Install `llama.cpp` (https://github.com/ggerganov/llama.cpp) if you do not already have it.
+2. Start the server:
+   ```
+   llama-server -m /path/to/your/model.gguf
+   ```
+   Default port is `8080`.
+3. Open Settings → LLM and confirm `llm.llamacpp.url` is `http://localhost:8080/v1` (or wherever your server runs). Set `llm.llamacpp.api_key` only if your server is behind an auth proxy.
+4. Set `llm.model` to the model name `llama-server` is hosting (e.g. `llama-3.1-8b-instruct`).
+
+### Removed settings
+
+The four old in-process settings are gone:
+
+- `llm.llamacpp_model_path`
+- `llm.llamacpp_n_gpu_layers`
+- `llm.llamacpp_n_batch`
+- `llm.llamacpp_f16_kv`
+
+Existing rows for these keys remain in the per-user DB but are no longer read by any code path; they will be silently ignored. Tuning that previously lived in these settings now belongs to the `llama-server` command line.
+
+### New settings
+
+- `llm.llamacpp.url` — HTTP endpoint URL (default `http://localhost:8080/v1`).
+- `llm.llamacpp.api_key` — optional, only needed if you put an auth proxy in front of `llama-server`.
+
+## Cross-provider default-model cleanup
+
+All built-in providers (`anthropic`, `google`, `openrouter`, `xai`, `ionos`, `openai`, `ollama`, `lmstudio`, `llamacpp`, `openai_endpoint`) now have `default_model = ""` and refuse to silently substitute. Where a user-specified model is missing, the central check in `get_llm()` raises a `ValueError` with an actionable message naming the offending setting key. 73 tests were updated to pass `model_name=...` explicitly rather than relying on the old defaults.
+
+## Better error message when local LLM servers are not running
+
+When a user selects `llamacpp`, `lmstudio`, `openai_endpoint` (or another OpenAI-compatible provider) and the configured server is not reachable, the raw `openai`/`httpx` connection error is rewritten — by the friendly-error layer added in #4027 — into a message that names the provider and base URL, e.g. `Cannot reach <provider> at <url>. Check that the server is running and the URL is correct.` so the user sees something actionable instead of a stack trace.
+
+## Optional API keys for `lmstudio` and `llamacpp`
+
+Both providers now support an optional `api_key` setting (`llm.lmstudio.api_key`, `llm.llamacpp.api_key`) for users who put their local server behind an auth proxy. If left blank, the providers use a placeholder key that bare `llama-server` / LM Studio ignore. No action required for default setups.
+
+## Settings UI
+
+The **LLM Provider** dropdown now lists `llama.cpp (Local)` as a selectable option (it was reachable only by manual config-edit in 1.6.2). The empty-`llm.model` warning live-updates on both keystroke and dropdown selection, mirroring the API-key-not-configured pattern in `openai_base.py`.

docs/release_notes/1.6.8.md

@@ -0,0 +1,33 @@
+# 1.6.8 — release notes
+
+## Bug fixes
+
+- **(#3747) Restored login for databases created before v1.4.0.**
+  Users whose encrypted user database was created before 2026-03-25
+  (v1.4.0, when Alembic migrations were introduced) could not log in
+  after upgrading: their databases lacked the `alembic_version` row, and
+  the migration runner attempted to apply migrations from scratch against
+  a legacy column shape. Migration `0007`'s index backfill then failed on
+  missing columns (e.g. `settings.category`), leaving the database in a
+  corrupted intermediate state.
+
+  This release detects pre-Alembic databases on first launch, stamps them
+  at the correct baseline (revision `0001`), and lets the remaining
+  migrations apply cleanly. Look for the `BUG-3747:` log line at startup
+  to confirm the recovery path engaged.
+
+  Affected users just need to update to `1.6.8` (or `:latest` / `:1.6`)
+  and restart — the recovery is automatic on the next launch.
+
+  ### Hardening
+
+  - `stamp_database()` is now race-tolerant: concurrent stampers (e.g. two
+    same-user logins arriving simultaneously) no longer trigger
+    `OperationalError` / `IntegrityError` on the duplicate
+    `alembic_version` insert. The race-tolerance is narrowly scoped to
+    `alembic_version`-related errors, so disk-full / corruption / unrelated
+    `SQLITE_BUSY` errors continue to propagate.
+  - `run_migrations()` refuses to operate on what looks like an auth-DB
+    shape (only `users` table, optionally with `alembic_version`) — defense
+    in depth against accidentally routing the auth engine through the
+    user-DB migration runner.

docs/release_notes/1.7.0.md

@@ -0,0 +1,81 @@
+
+### 🔒 Security
+
+- Extended `redact_secrets()` coverage from the Google provider (#4070) to every credential-bearing exception-logging call site: the OpenAI-compatible provider base (`list_models`), the custom OpenAI endpoint provider, the OpenAI embedding provider, and the web search-engine subsystem (`BaseSearchEngine.run()` plus the main HTTP-call error paths in Tavily, Exa, Serper, Google PSE, Mojeek, PubMed, Semantic Scholar, Guardian, ScaleSerp, SerpAPI, and GitHub). API keys embedded in upstream SDK error messages or echoed-back URLs/headers no longer leak into logs. Guardian and ScaleSerp chain explicit-value redaction with their existing regex sanitizers for defense in depth. Consolidates #4168, #4175, and #4181. See #4131. ([#4426](https://github.com/LearningCircuit/local-deep-research/pull/4426))
+- Extended `redact_secrets()` coverage to the NASA ADS search engine, which was missed by the original #4131 sweep. Its `_get_previews` catch-all used `logger.exception`, whose traceback frames hold `self.headers` (the `Authorization: Bearer <key>` value) and would render the API key under loguru diagnose mode. The except block now uses a redacted `logger.warning`. Follow-up to #4131.
+- Loguru ``diagnose=True`` exception rendering (which dumps the ``repr()`` of every local variable in every traceback frame) is now gated behind a separate explicit ``LDR_LOGURU_DIAGNOSE`` opt-in instead of riding on ``LDR_APP_DEBUG``. Previously, enabling ``LDR_APP_DEBUG`` for general debug output also turned on localvar dumps, so any exception could write frame-local credentials — API keys, the SQLCipher master password, ``Authorization`` headers — into every log sink. ``LDR_APP_DEBUG`` now controls log *level* only; ``diagnose`` stays off unless the operator additionally sets ``LDR_LOGURU_DIAGNOSE=true``, and an explicit warning is emitted when they do.
+- The MCP server subprocess (``ldr-mcp``) now pins ``diagnose=False`` on its stderr sink. loguru defaults ``diagnose=True``, which renders ``repr()`` of every traceback frame's locals on exception, so the many ``logger.exception(...)`` paths in the MCP request handlers were writing frame-local credentials (``api_key``, ``Authorization`` headers, search-engine secrets) into the MCP client's stderr log on any failure. Companion to the ``LDR_LOGURU_DIAGNOSE`` lockdown for the web/CLI logger; the MCP subprocess has no debug mode, so the gate is unconditionally off.
+- The benchmark CLI (``ldr-benchmark`` / ``benchmarks/cli/benchmark_commands.py``) now pins ``diagnose=False`` on both of its ``logger.add(sys.stderr, ...)`` calls. loguru defaults ``diagnose=True``, which renders ``repr()`` of every traceback frame's locals on exception, so the many ``logger.exception(...)`` paths in the benchmarks package (LLM grader calls, search-engine runners, dataset loaders) were writing frame-local credentials — LLM ``api_key``, ``Authorization`` headers, search-engine secrets — to stderr on any failure. Companion to the ``LDR_LOGURU_DIAGNOSE`` lockdown for the web/CLI logger (#4384) and the MCP subprocess (#4394); the benchmark CLI also bypasses ``config_logger``, so the env-var gate did not reach it.
+- The chat `PATCH` (rename/archive) and `DELETE` session routes now carry the same per-user rate limit as the other state-changing chat endpoints. They were previously bounded only by the global limiter, leaving an uneven abuse surface across the session API.
+- The example scripts under ``examples/`` (``example_browsecomp.py`` and the three ``api_usage/programmatic/`` snippets) now pass ``diagnose=False`` to their ``logger.add(sys.stderr, ...)`` calls. Loguru defaults ``diagnose=True``, which renders ``repr()`` of every local in every traceback frame on exception. These files are templates that users copy into their own scripts, so the previous default propagated the same credential-in-traceback leak fixed for the application logger and MCP subprocess (#4185, #4384) into anyone's downstream code.
+
+### ✨ New Features
+
+- **Chat Mode** — Interactive multi-turn research conversations. Each session accumulates context across turns (entities, topics, source count), streams research steps and citations live as the answer is built, and persists in your per-user database (encrypted by default). Sessions can be archived, reactivated, deleted, or exported as Markdown. Reach it from the **Chat** sidebar link or `/chat/`. See [features.md#chat-mode](../features.md#chat-mode) for the full feature description. ([#2953](https://github.com/LearningCircuit/local-deep-research/pull/2953))
+- **"Summarizing…" indicator for Chat follow-ups** — When you send a follow-up question, the thinking indicator now reads "Summarizing previous conversation…" while the earlier turns are condensed into context for the new research, instead of showing blank dots. It clears the moment research starts streaming its progress.
+- **Configurable follow-up context in Chat Mode** — Follow-up questions now build on the earlier conversation instead of starting cold. By default, the prior conversation is condensed into a short summary focused on your new question (using your configured LLM) and passed to the follow-up research as its "previous findings", keeping multi-turn research on-topic and within context limits. You can change this with the new **Follow-up Context Mode** setting (`chat.followup_context_mode`):
+
+  - `summary` (default) — an LLM summary of the conversation, focused on your new question
+  - `raw` — the most recent research findings, truncated
+  - `full` — the entire conversation transcript
+  - `none` — no prior context (just your new question and the original topic)
+
+  Only `summary` makes an extra LLM call per follow-up; the other modes add no model cost.
+- Chat Mode: the thinking bubble now shows what the agent is currently reasoning about between tool calls — the LLM's intermediate prose (e.g. "I should compare the published benchmark results next…") appears above the bouncing dots and overwrites with each new step, instead of leaving a static indicator for the entire research duration.
+- Chat mode now shows an elapsed-seconds counter ("Stopping research… (Ns)") in the progress task line after the user clicks Stop, so it's clear the click registered even when the worker thread is blocked inside an LLM HTTP call (thinking-mode models like Qwen 3 and DeepSeek-R1 can take 30+ seconds to surface a stopping checkpoint because no chunks yield during ``<think>`` blocks). The counter only appears after 3 seconds elapsed — quick stops show the plain "Stopping research…" label without a number. The counter is cleared automatically whenever research stops, completes, or errors out. This is a UX-only mitigation; reducing the underlying termination latency would require deeper work such as per-token streaming or signal-based HTTP interruption.
+- Chat mode now streams inline citation hyperlinks *as the answer is written* rather than only after the final save. Each chunk sent to the client is run through the same citation formatter the final answer uses, with a small carry buffer that holds incomplete `[N` tokens straddling chunk boundaries until the closing `]` arrives — so the user sees `[[arxiv.org-1]](url)` (or whichever format their ``report.citation_format`` setting selects) appearing live, and there is no visible format-flip when the saved answer replaces the streaming bubble on completion. The server-side partial-content buffer still stores the raw chunk so terminate-mid-stream saves aren't double-formatted on resume.
+- Chat-mode live milestones now use friendlier wording for the agent's tool calls and observations. ``Tool: search_pubmed — "covid"`` becomes ``🔍 Searching PubMed: "covid"``, ``Tool: fetch_url — "https://…"`` becomes ``📖 Reading the page: "https://…"``, and tool results show as ``📄 From {engine}: …`` so the thinking-text reads like a narration of what the agent is doing rather than a dump of internal tool names. Engines without an explicit display name fall back to a cleaned ``Title Case`` form of the raw tool name, so newly added search engines work without a code change.
+
+### 🐛 Bug Fixes
+
+- Added a `weakref.finalize`-based safety net inside the Ollama
+  embeddings factory so that programmatic API callers and example
+  scripts that construct `OllamaEmbeddings` directly — bypassing
+  the managed RAG service lifecycle — still release their underlying
+  httpx clients when the instance is garbage-collected.
+- Chat Mode correctness, settings, and accessibility fixes. (1) The streaming citation path in ``base_citation_handler._invoke_with_streaming`` now routes its joined chunks through ``get_llm_response_text`` before returning, so a reasoning model's ``<think>…</think>`` block no longer leaks into the persisted chat answer — ``.stream()`` bypasses ``ProcessingLLMWrapper.invoke`` (the only place tags were stripped), and the streamed result now matches the non-streaming ``invoke()`` contract. (2) ``SettingsManager.set_setting``'s self-heal block only re-points a row's ``type`` when the key matches a known prefix; keys outside the dispatch map (``focused_iteration.*``, ``langgraph_agent.*``, which ship as ``type=SEARCH``) are no longer silently demoted to ``APP`` on every edit. (3) ``report_assembly_service.assemble_full_report`` no longer wraps ``_build_sources_markdown`` in a blanket ``except`` that dropped the entire Sources block on error — failures propagate to the caller's existing 500 path instead of rendering a report that looks complete but is silently missing all sources. (4) The chat ``PATCH /sessions`` route now checks the boolean returns of ``update_session_title`` / ``reactivate_session`` / ``archive_session`` and returns 500 when a DB write failed but the session still exists, instead of reporting ``success: true`` with the stale row. (5) ``ChatService.delete_session`` sets the in-memory termination flags only **after** the delete commits, so a failed commit can no longer kill the in-flight research of a session that still exists. (6) ``build_research_context`` now populates ``original_query`` (the session's first user message) so the contextual follow-up strategy keeps its topic anchor instead of reading an empty string. (7) The mid-stream termination paths set ``streaming_state["_termination_handled"]`` so the ``ResearchTerminatedException`` handler no longer calls ``handle_termination`` a second time — eliminating the duplicate SUSPENDED status update, duplicate final socket emit, and doubled test-mode sleep. (8) The chat page route uses ``render_template_with_defaults`` so ``?v={{ version }}`` asset cache-busting is no longer empty. (9) The direct-mode capacity-reject path in ``processor_v2._start_research_directly`` now increments ``QueueStatus.queued_tasks`` (via ``add_task_metadata``) for the re-queued research — previously the counter stayed 0, so ``_process_user_queue`` could treat the queue as empty and leave the row undispatched until later user activity re-pumped the counter. (10) The streaming chat bubble no longer sets a nested ``role="status"`` inside the ``role="log"`` message list, which had made screen readers announce every streamed chunk twice. A regression test asserts the streaming path strips ``<think>`` from the returned synthesis.
+- Chat Mode data-integrity fixes — message ordering now respects ``sequence_number`` for same-millisecond rows, the "Load older" cursor uses a composite ``(created_at, id)`` key so boundary rows are no longer silently dropped, Markdown export pages through the full conversation (previously truncated to 50 messages), and the partial-save idempotency flag is set only after the database write succeeds so a transient failure no longer permanently loses the assistant response.
+- Chat Mode fixes for error/stopped-state button visibility, streaming citation carry-over, and accessibility. ``handleResearchError`` and ``handleResearchSuspended`` in ``chat.js`` now call ``showSessionButtons()`` so the edit-title / export controls reappear after a Stop or a failed research — previously only the successful-completion path restored them, leaving the buttons hidden until page reload after a stop or error. The streaming completion finalizer now flushes the citation carry buffer (``_flush_carry`` exposed via ``streaming_state``) before emitting the ``is_final`` sentinel, so an LLM stream that ends mid-token like ``"[12"`` no longer silently drops the leading bracket from the client's accumulated text. ``_PARTIAL_BRACKET_RE`` accepts the lenticular opener ``【`` in addition to ASCII ``[`` — some LLMs emit Chinese-style citation brackets and the citation formatter already recognises them, so the carry buffer needed to hold those back the same way. The error-path ``add_message`` in ``research_service.py`` now passes ``allow_archived=True`` matching the completion and stop-and-partial paths, so a session archived mid-flight no longer silently swallows the "research failed" message. ``ChatService.get_in_progress_research_id`` now re-raises ``DB_EXCEPTIONS`` instead of returning ``None``: a transient DB error during session load no longer looks identical to "no research running" and the route handler returns a 500 the client can surface as an error banner. Chat-css gains visible ``:focus-visible`` rings on send / stop / edit-title / ``#chat-page .ldr-btn``, a real focus ring on ``#chat-input`` (replacing the invisible 15%-opacity wrapper shadow), ``prefers-reduced-motion`` gates on the streaming caret pulse and thinking-dot bounce animations, ``:focus-within`` reveal for the per-message timestamp meta (previously hover-only and therefore invisible to keyboard users), a 44×44 minimum touch target for the mobile send button (WCAG 2.5.5), and the responsive breakpoint aligned to ``767px`` so the chat container's top-bar height assumption no longer disagrees with the rest of the project at exactly 768px. Minor cleanup: ``timedelta`` is now imported at module scope in ``chat/routes.py`` instead of inside ``send_message``, ``ChatContextManager`` is no longer re-exported from ``chat/__init__.py`` (every caller imports from ``chat.context`` directly), and the docs/features.md chat-mode section now mentions Markdown export.
+- Chat Mode input-validation and resource-bound hardening. ``send_message`` now parses the numeric search settings (``search.iterations``, ``search.questions_per_iteration``) **before** the atomic write that commits the user message + IN_PROGRESS research row — a malformed (non-numeric) settings value now returns a clean 400 instead of raising an unhandled 500 after the rows are committed, which previously orphaned them and left the session unusable via the per-session 409 guard. The inline-citation carry buffer in ``research_service._make_chat_stream_callback`` is now capped at 64 bytes: a never-closing ``[`` followed by an endless digit run from a misbehaving LLM is flushed raw past the cap instead of growing the buffer without bound. The client-side ``streamedContent`` accumulator in ``chat.js`` is likewise capped at 256 KB (mirroring the server's ``_MAX_PARTIAL_BUFFER_BYTES``) with a one-time "(Response truncated — exceeded display limit.)" notice, so a model with no ``max_tokens`` can't OOM the browser tab. Two secondary citation regexes that the earlier lenticular-bracket work had missed now also accept ``【N】``: the source-list parser in ``citation_formatter.py`` (RIS export) and the citation-strip in ``benchmarks/graders.py``. Regression tests were added for the carry-buffer flush + overflow contract and the LLM-title newline strip.
+- Chat Mode reliability fixes. The log panel now loads historical logs when expanded on a chat page (the toggle previously held a stale null research id). Delete / clear-history failures now surface an error notification instead of silently doing nothing. When a completed research's answer can't be loaded into the chat, the message now links to the full report (which is still saved) rather than a dead-end "no report available". The header "New Chat" / "Export" buttons are now styled (their base CSS wasn't loaded on the chat page). Stopping a research while it is still starting up now reliably terminates it instead of being silently ignored, and a late error event can no longer overwrite an already-rendered completed answer.
+- Chat Mode rendering and log-safety fixes. ``chat.css``'s `.ldr-chat-container` height now uses `calc(100dvh - …)` (with `100vh` retained as the previous-line fallback, matching the pattern already used in ``mobile-navigation.css``) so iOS Safari no longer buries the chat input below the collapsible URL bar. ``chat.js``'s ``handleResearchComplete`` else-branch (the one that fires when streaming didn't reach its `is_final` sentinel — most commonly after a flush-then-disconnect race introduced by the `_flush_carry` change) now reuses the in-place-swap pattern from the streaming-complete branch when a partial bubble is already on screen: it removes the streaming class and renders the formatted message into the same `.ldr-chat-message-text` element instead of `.remove()` + `addMessageToUI`. Eliminates the 5-8.5 s vanish-then-refetch flicker. The original detach-and-reinsert path is preserved for the no-partial-bubble case where no flicker would occur anyway. ``ChatService.regenerate_title_with_llm`` now strips `\n` and `\r` from the LLM-returned title before storage; the title is interpolated into loguru f-strings (the "title already set" log line one above this fix), so an embedded newline would otherwise forge what looked like a second log entry in aggregators. Also keeps ``document.title`` and ``chatTitle.textContent`` visually clean.
+- Chat Mode report-assembly, layout, and accessibility fixes. ``format_links_to_markdown`` (``utilities/search_utilities.py``) now coerces citation indices to ``str`` before ``sorted()`` so mixed ``int``/``str`` values arriving from different strategies (e.g. ``recursive_decomposition_strategy.py`` emits ``int``, ``_build_sources_markdown``'s fallback emits ``str``) no longer crash report assembly with ``TypeError: '<' not supported between instances of 'str' and 'int'``; the sort still produces numeric order. ``chat.css`` desktop and mobile chat-container heights now subtract the real ``.ldr-top-bar`` heights (60px / 50px) instead of the previous over-reserved 80px / 60px, ending the viewport overflow that clipped the chat input area. ``research_service.py`` ``add_progress_step`` failure path now zeroes ``event_data`` inside its ``except`` block so a DB-lock that swallows the persistence call also suppresses the live socket emit — preserving the documented live/reload symmetry invariant (a chat step the client sees live is the same set the persisted log returns on reload). Two icon-only delete buttons in ``history.js`` gained ``aria-label`` / ``title`` for WCAG 2.1 Level A compliance.
+- Chat Mode: when research completes but the streaming-bubble swap path doesn't end up with a visible assistant response (transient socket drop, race during session switch, missed ``is_final`` chunk, etc.), the chat now silently re-renders from the DB-authoritative ``/messages`` endpoint instead of leaving the user staring at an empty page until they manually refresh.
+- Citation formatter: ``apply_inline_hyperlinks`` (the fallback path that chat-mode answers always hit because the langgraph-agent synthesis doesn't emit a ``## Sources`` block in its prose) now dispatches on ``CitationFormatter.mode`` instead of hard-coding ``NUMBER_HYPERLINKS``. The user's ``report.citation_format`` setting is now honored for chat-mode citations — picking "Domain ID Hyperlinks" or "Source-Tagged Hyperlinks" actually produces ``[[arxiv.org-1]](url)`` / ``[[arxiv-1]](url)`` instead of every chat answer coming out as ``[[1]](url)``.
+- Citation formatter: ``apply_inline_hyperlinks`` now accepts either ``"url"`` or ``"link"`` as the destination key on each source dict. Searxng-sourced results carry the destination under ``"link"`` (``search_engine_searxng.py``), which the fallback hyperlink path silently dropped — so chat-mode answers that did not emit a ``## Sources`` block in their synthesis shipped with plain ``[N]`` brackets instead of clickable citations even though the Sources section beneath the answer was fully populated.
+- Fix report structure parsing crashing with `IndexError` on a numbered section line without a period (e.g. `1 Introduction`), and stop truncating section names that contain periods (`1. U.S. Policy` is now kept whole instead of becoming `U`). The section number is now split on the first period only, with a guard for malformed lines.
+- Fixes for bugs that affect the **non-chat** research path as well as chat. (1) ``LangGraphAgentStrategy.analyze_topic`` no longer overwrites its ``query`` parameter with a truncated tool-call argument inside the tool-call display loop — because ``langgraph-agent`` is the default strategy, after the first ``web_search`` the original research question was silently replaced by a ≤80-char search arg, which then fed the citation re-synthesis, the fallback synthesis, and the recorded ``question`` field, steering the final answer at the wrong question. (2) The queue dispatcher (``processor_v2._start_queued_researches``) now has a dedicated ``except SystemAtCapacityError`` clause: a transient at-capacity rejection is left QUEUED for the next tick instead of being counted toward ``SPAWN_RETRY_LIMIT`` and wrongly marked FAILED after a few ticks under load. (3) ``cleanup_research_resources`` now reports the real terminal status (passed in by the caller — SUSPENDED on user termination, FAILED on error) instead of a hard-coded ``COMPLETED``, so stopping a research no longer emits a spurious "completed" socket signal (which produced a stray "[Stopped]" bubble in chat and a misleading 100%/Completed flip on the standard progress page); chat.js's completion/suspension handlers also cross-guard each other. (4) Chat-triggered research now inserts a ``UserActiveResearch`` row, so it counts toward the per-user ``app.max_concurrent_researches`` cap the same way UI-launched research does — previously chat research was invisible to the cap (queried but never recorded), letting multiple chat tabs bypass it. The row is committed atomically with the research row, cleaned up on spawn failure, and removed on normal completion by the existing completion-sweep middleware.
+
+  Regression tests added for all four.
+- GitHub search relevance ranking now rejects negative, out-of-range, and non-integer result indices and deduplicates repeated ones, so a malformed LLM response can no longer select the wrong result, list the same repository twice, or discard all results.
+- Library RAG service is now closed at the end of every HTTP request
+  that uses it — including streaming endpoints. Together with the
+  embedding-manager close path, this stops file-descriptor accumulation
+  under sustained library indexing/search traffic.
+- News subscriptions run via "Run now" or the overdue-subscription check no longer force the LLM to ``ollama``/``llama3`` when the subscription has no explicit model configured. The run paths in ``news/flask_api.py`` hardcoded those values, which (being truthy) overrode the user's configured ``llm.provider``/``llm.model`` — so a user on OpenAI/Anthropic whose subscription carried no model would silently get an Ollama run, typically failing. The hardcoded defaults are removed; an unset model now passes through as ``None`` so ``start_research`` falls back to the user's settings.
+- Stopped a per-RAG-request file-descriptor leak introduced when the
+  embeddings provider migrated to `langchain_ollama.OllamaEmbeddings`.
+  The library RAG service now closes the underlying httpx clients on
+  teardown, preventing eventpoll FD accumulation under sustained
+  indexing/search load.
+- Strip `<think>` reasoning blocks from the main synthesis path (`synthesize_findings` and the standard knowledge generator), not just the citation handlers — reasoning-model output no longer leaks `<think>…</think>` into final answers. Also fixes precision/forced answer extractors emitting a stray `". <content>"` when the model returns an empty (or think-only) response.
+- The WebSocket subscribe ownership check now recognizes benchmark runs in addition to normal research. The new per-user ownership gate (and the removal of the cross-user broadcast fallback) only matched ``ResearchHistory`` UUIDs, so the benchmark page — which subscribes with an integer ``BenchmarkRun`` id — was rejected and its live progress (current-task detail and the SearXNG rate-limit warning) was silently dropped. ``_user_owns_research`` now also accepts the user's own ``BenchmarkRun`` rows from their per-user encrypted database, restoring benchmark live progress without widening the authorization boundary.
+- The central LLM wrapper now normalizes string-returning providers into a message object and applies `<think>`-tag stripping to async (`ainvoke`) calls too, so any LLM obtained from `get_llm` yields a consistent, think-free `.content` — eliminating `'str' object has no attribute 'content'` crashes at the source. Message objects keep their `tool_calls`/`reasoning_content` (only `.content` is rewritten).
+- The chat ``send_message`` endpoint now uses the shared ``@require_json_body`` decorator like the other state-changing chat POSTs. It was the only one validating the body inline, so a non-JSON content type slipped past the consistent 400 contract; requiring an ``application/json`` body also hardens CSRF on the heaviest chat endpoint (it launches a research run). Behavior for valid requests is unchanged.
+- The queued-research dispatch loop now reverts its queue-counter claim when a global-capacity reject re-queues a research. Previously, ``_start_queued_researches`` marked the task ``processing`` (decrementing ``queued_tasks``, incrementing ``active_tasks``) and, on ``SystemAtCapacityError``, only reset the row's ``is_processing`` flag — leaking a slot into ``active_tasks`` on every capacity-rejected retry. Under sustained capacity pressure ``queued_tasks`` would drift to 0, at which point the per-user queue processor treated the queue as empty and stopped dispatching the still-present rows. ``update_task_status`` now supports a ``processing`` → ``queued`` transition that restores the counts, and the capacity-reject path uses it.
+- `GET /api/report/<id>` now returns the `sources` field from the structured `research_resources` table instead of the dead `all_links_of_system` metadata key. After the chat-mode-v2 report refactor that key is never written, so the field returned an empty list for every research created since — even though the assembled `content` and the news feed already read sources from the table. (#3665)
+
+### 📝 Other Changes
+
+- **`research_history.report_content` shape — answer-only at persistence time.** Internal/library change tied to the Chat Mode work: `report_content` now stores the synthesized answer (LLM prose + inline citations) rather than the full `format_findings` blob (which previously embedded `## Sources` and `## Research Metrics` sections). User-facing views are unchanged — `assemble_full_report()` in `web/services/report_assembly_service.py` reconstructs the legacy display shape on demand for the history page, exports, and the chat "View full research" link, and legacy rows that still contain the inline sections are detected and not double-appended. **Direct callers of `storage.get_report()` / `storage.get_report_with_metadata()` should switch to `assemble_full_report()` if they need the legacy shape.**
+- CI: fix the freshly-merged ``check-shadow-tests`` pre-commit hook using ``os.path.basename()``, which the repo's own ``check-pathlib-usage`` hook forbids. Because PR pre-commit runs ``--all-files`` against the PR-merged-into-main tree, this one line on main turned every open PR's pre-commit red. Switched to ``pathlib.Path(path).name`` (behaviour verified identical to ``os.path.basename`` across separator/edge cases).
+- CI: split ``tests/web/routes/test_settings_routes_coverage.py`` out of the parallel pytest run into a dedicated serial step (same pattern as the existing ``fd_canary`` step). The 94 tests in that file all use the ``authenticated_client`` fixture (register + login + SQLCipher KDF + 10 Alembic migrations per test), and under ``-n auto`` they accumulate enough connection / FD pressure inside the assigned xdist worker to deadlock it silently late in the run — the worker dies with ``[gw0] node down: Not properly terminated`` (no timeout printed), its coverage data is dropped, and the 50% fail-under gate then fails the job even though every test that completed actually passed. Skipping individual tests in this file just relocates the death (confirmed empirically on this PR's earlier iteration). The serial step keeps all 94 tests running, appends into the main run's ``.coverage`` data via ``--cov-append``, and runs the final coverage report + 50% gate after both contributions have merged. Defence-in-depth: pairs with #4393's 180s/thread timeout.
+- CI: switch ``pytest-timeout`` from ``signal`` to ``thread`` method and raise the global per-test timeout from 60s to 180s. On Python 3.14 the SIGALRM-based interrupt was firing inside ``weakref.py`` cleanup, corrupting xdist workers and dropping their coverage data — which then put total coverage below the 50% gate even when every test that completed actually passed. The thread method raises a Python-level exception in the main thread instead of interrupting at an arbitrary safe-point, and 180s gives the heavy ``authenticated_client`` register+login fixture (SQLCipher KDF + Alembic migrations) headroom under ``-n auto`` + coverage contention.
+- Chat Mode polish and test-quality improvements. ``chat/routes.py`` adopts the shared ``@require_json_body(error_format="success")`` decorator on the three POST/PATCH endpoints (``create_session``, ``generate_session_title``, ``update_session``), replacing 4-block inline ``isinstance(data, dict)`` guards. ``UserActiveResearch`` stale-row reclaim is extracted from both ``chat.routes.send_message`` and ``research_routes.start_research`` into a shared ``reclaim_stale_user_active_research(db, username, *, grace_cutoff_dt=None, logger=None)`` helper in ``web/routes/globals.py`` — chat passes a 30s grace cutoff (because chat send can race with its own concurrent sibling), research_routes passes None (matching its pre-existing behaviour). ``ChatService.delete_session`` now imports ``set_termination_flag`` at module top-level instead of inside the function body (no circular import requires the deferred import). The dummy ``"I understand. What else would you like to know?"`` assistant bubble that ``chat.js`` injected when the server returned no ``research_id`` is gone — the thinking indicator is now cleared without injecting a placeholder reply. ``chat.js::loadSession`` now focuses the input on completion, matching ``startNewChat``. Three brittle test patterns are replaced: the seven copy-pasted ``captured_username`` + ``@contextmanager`` blocks in ``test_chat_user_isolation.py`` are consolidated under a single ``username_capturing_db`` fixture; the 16 inline ``SocketIOService._instance = None`` reset lines in ``test_chat_socket_events.py`` become a single autouse fixture that runs in setup AND teardown regardless of test outcome; ``test_chat_settings_integration.py``'s three tautological dict-assertion tests are replaced with four real tests that exercise ``_load_settings`` against patched ``SettingsManager`` and validate every snapshot-extraction key still exists in ``default_settings.json``. ``test_chat_send_message_reclaim.py``'s three source-grep tests against ``chat/routes.py`` are replaced with eight behaviour tests that drive the new shared reclaim helper against a real SQLite session (covering grace-window respect, live-thread skip, username scoping, and the without-cutoff immediate-reclaim mode used by research_routes). ``test_chat_service.py`` drops three ``if hasattr(...)`` assertion guards that would have silently passed when a real model regressed; the ``get_in_progress_research_id`` DB-error test inverts to assert propagation now that the service re-raises instead of returning None. ``test_chat_api.py``'s ordering assertion no longer hides behind ``if len(sessions) >= 2:``.
+- Chat Mode polish fixes. The agent-thinking milestone for ``research_subtopic`` now reads ``🔬 Investigating subtopic: "topic1, topic2"`` instead of empty quotes — the display code now picks up the tool's actual ``subtopics: list[str]`` argument and stringifies the list. ``send_message``'s ``UserActiveResearch`` reclaim block now emits a ``logger.warning`` mirroring the ``ResearchHistory`` reclaim above it, so operators can trace why a per-user concurrency cap was released. ``_validate_title`` strips Unicode format / line-separator characters (``Cf``/``Zl``/``Zp`` — zero-width spaces ``U+200B-U+200D`` and BOM ``U+FEFF``) before the emptiness check, so a title of 500 zero-width chars is rejected instead of saving a session that looks blank in the UI. ``regenerate_title_with_llm`` is now idempotent: it skips the LLM call if the session title no longer matches the non-LLM fallback, so a concurrent tab's edit (or the user's manual edit completing first) is not overwritten by a stale LLM round-trip. The dead ``build_research_context_for_session()`` wrapper in ``chat/context.py`` is removed along with its dedicated tests — production code uses ``ChatContextManager`` directly.
+- Chat follow-up research now logs which prior-context mode ran and how many characters of context it built. Previously the context-building step (especially the LLM summary) produced no log output, so a follow-up's preparation looked like an unexplained pause.
+- Clarified the auto-generated `app.debug` / `LDR_APP_DEBUG` description in `default_settings.json` (and therefore `docs/CONFIGURATION.md`) to reflect the split introduced when `LDR_LOGURU_DIAGNOSE` was added: `LDR_APP_DEBUG` now raises log level only and explicitly does NOT enable Loguru's local-variable dumps on its own. Companion to the `LDR_LOGURU_DIAGNOSE` security fix.
+- Document a known limitation: LangGraph `create_agent`/`bind_tools` (in the `langgraph-agent` and `mcp` strategies) resolve to the base LLM via `ProcessingLLMWrapper.__getattr__`, bypassing the wrapper's `<think>`-tag stripping. Added in-code notes at the three call sites. This is a cosmetic leak only (reasoning-model `<think>` blocks may appear in agent output); it does not affect direct `invoke()` calls, which still go through the wrapper.
+- Documented that the News API `findings` field is the answer-only report content after the chat-mode-v2 refactor (intentional): structured top-N source links live in the separate `links` array rather than an embedded `## Sources` blob. No behavior change. (#3665)
+- Test fixtures and dev scripts that re-configure loguru (``tests/conftest.py`` loguru-caplog fixture, the two ``tests/api_tests/`` ``__main__`` runners, ``tests/test_openai_api_key_e2e.py``, ``tests/settings/test_manager_behavior.py``, ``tests/journal_quality/test_db.py``, ``tests/performance/mcp/echo_server.py``) now pass ``diagnose=False`` to their ``logger.add(...)`` calls, matching the production policy locked in by #4384 / #4394. loguru defaults ``diagnose=True``, which renders ``repr()`` of every traceback frame's local on exception; pinning it off keeps frame-local credentials (api keys, SQLCipher passwords, ``Authorization`` headers) out of pytest output and CI logs even when a test crashes mid-fixture. Hygiene change only — no behavior change for green tests.
+- The news-feed "time ago" formatter no longer swallows unparseable timestamps behind a silent "Recently" label. Since `created_at` is always written as a valid ISO timestamp, a value that won't parse means a corrupt row — that row is now logged and skipped by the feed builder instead of rendering with a misleading time.

docs/release_notes/1.8.1.md

@@ -0,0 +1,278 @@
+
+### 💥 Breaking Changes
+
+- Removed the `mcp` / `agentic` (ReAct) search strategy. Existing selections are automatically migrated to `langgraph-agent` (a near functional superset); connecting to external MCP servers as research tool sources is no longer supported. (LDR still ships its own MCP **server** for exposing research to assistants like Claude — that is unaffected.) ([#4548](https://github.com/LearningCircuit/local-deep-research/pull/4548))
+- **Breaking (programmatic API):** `get_llm(provider=…)` and `get_embeddings(provider=…)` now **fail closed** when called with no settings snapshot (`settings_snapshot=None`) for any provider that is not a known local-default. Snapshot-less callers may instantiate only the localhost-default providers — LLM: `ollama`, `lmstudio`, `llamacpp`; embeddings: `sentence_transformers`, `ollama` — plus LLMs you registered in-process via the programmatic API (`quick_summary(llms={…})`). Any other provider (incl. `openai`, `anthropic`, `google`, `openrouter`, and any future cloud provider) raises `PolicyDeniedError` instead of silently constructing a cloud client. This closes a snapshot-less egress hole; programmatic callers that relied on building a cloud provider without a snapshot must now pass a `settings_snapshot` (or register the LLM in-process). The allowlists are intentionally tight so unknown/new providers fail closed by default.
+- **The `auto` and `parallel` meta search engines have been removed** (including the `meta` and `parallel_scientific` aliases). The default langgraph-agent strategy replaces them — it selects search engines dynamically per query, so a separate LLM-based engine picker is redundant. Stored values (`search.tool` setting, news subscriptions, queued researches, saved benchmark configs) are migrated automatically on upgrade (migration 0013; `search.tool` values pointing at a removed engine become `searxng`). **What to do:** API callers passing `search_tool="auto"` or `"parallel"` must pick a concrete engine (e.g. `searxng`), and `LDR_SEARCH_TOOL=auto` environment overrides must be updated likewise.
+- **Upgrade behaviour change:** the default egress scope is now `adaptive` (was effectively `both`). Adaptive follows your primary search engine, so on upgrade a config with a *concrete* primary may narrow: a public primary (e.g. SearXNG/arXiv) now excludes local collections (public-only behaviour), and a *private collection* primary forces local LLM + embeddings. To keep the previous "any engine, cloud inference" behaviour, set Egress Scope to **Both**. See docs/egress-modes.md.
+
+### 🔒 Security
+
+- News recommendation logging now sanitizes externally-derived topic strings (and search error text) before interpolating them into log records, closing a log-injection vector where crafted news content could forge log lines or inject terminal escape sequences. ([#3767](https://github.com/LearningCircuit/local-deep-research/pull/3767))
+- **`/settings/api` no longer leaks env-overridden API keys to authenticated users.** The bulk settings JSON endpoint now redacts password-typed values (`llm.openai.api_key`, search-engine API keys, etc.) so they come back as `[REDACTED]` instead of plaintext. Metadata is preserved so the UI still renders correctly. Note: the settings form template still pre-fills password inputs server-side — that's a separate UX issue and not addressed here. ([#3947](https://github.com/LearningCircuit/local-deep-research/pull/3947))
+- **Settings form no longer pre-fills password inputs with the stored value**, eliminating an authenticated View-Source disclosure of API keys / OAuth tokens. Password fields render empty with a placeholder indicating configuration state and `autocomplete="new-password"` to prevent browser caching. ([#3954](https://github.com/LearningCircuit/local-deep-research/pull/3954))
+- Extended `redact_secrets()` coverage to credential-bearing exception handlers in the encrypted-database, web queue, and scheduler subsystems — sites where the user's SQLCipher master password is in lexical scope. Unlike API keys (rotatable), the SQLCipher master password is unrecoverable (TRUST.md §5), so a leak via a rendered traceback or upstream exception message is a permanent compromise. Covers `database/encrypted_db.py` (6 sites in `create_user_database` / `open_user_database` / `change_password`), `web/queue/processor_v2.py:_start_research_directly`, and 11 sites across `scheduler/background.py` (subscription scheduling, document processing, RAG indexing, overdue-subscription handling, subscription research trigger). See #4182. ([#4182](https://github.com/LearningCircuit/local-deep-research/pull/4182))
+- Egress policy hardening (PR #4300, Round 6 review): 24 fixes across DNS classification (process-global socket timeout removed, threading.RLock around cache mutations), PDP coverage (nested-dict settings unwrap, NAT64 metadata reclassification, fetch_content tool gate), engine/RAG/data coverage (MetaSearchEngine Wikipedia fallback routed through factory, all 5 direct LibraryRAGService sites pre-flight-checked, SearchCache hash includes scope to close cache-bypass), and UI/cross-cutting work (per-research policy overrides on the research form, settings dashboard renders the policy keys, benchmark snapshot threading, policy_audit WebSocket filter, audit log on policy-key changes, dismiss-flag rename to follow the dismiss_* convention). 23 regression tests added.
+
+  A second adversarial review round added 13 more fixes: closed two egress bypasses (Elasticsearch `cloud_id` is now refused under private scope; OpenAI/discovered-provider model-list probes are scope-gated in both the settings and RAG model endpoints); closed a metadata-SSRF gap (literal cloud-metadata IPs are now classified PUBLIC on the literal-IP branch of `_classify_host`, matching the DNS branch); ADAPTIVE now resolves a registered local-retriever primary to PRIVATE_ONLY (was BOTH, which leaked the private corpus to cloud inference); the PEP-578 audit-hook backstop is now armed on non-web entry points (CLI / news scheduler / programmatic API) and re-armed on ThreadPoolExecutor pool workers; fixed the SearXNG `url_setting` key so a local SearXNG is no longer over-blocked under PRIVATE_ONLY; `allowed_local_hostnames` no longer rejects unresolvable intranet hosts on save; the denied-fetch quota no longer counts benign parse failures; and `DownloadService` fails closed when the settings backend errors. 10 regression tests added.
+
+  Follow-up regression fix: user-registered in-process LLMs (the programmatic API's `llms={...}` and plugins) are no longer denied with `provider_url_unset` when a run resolves to require-local inference (e.g. ADAPTIVE with a registered-retriever primary). The PDP now distinguishes user registrations from auto-discovered built-in providers — a custom name shadowing a built-in cloud provider stays blocked, and snapshot-less calls keep failing closed for everything else. 3 regression tests added.
+
+  Third adversarial review round (5×20 agents) confirmed and fixed 6 more issues (4 of 11 verified findings were declined as not-exploitable or design-intent, 1 was a false positive): the `evaluate_url` cloud-metadata block now normalizes alternate IPv4 encodings (octal `0251.0376.0251.0376`, hex, integer) so an IMDS literal can't read as an allowed public host under PUBLIC_ONLY/BOTH and always denies with the explicit `blocked_metadata_ip` reason; `dangerous_scheme` (javascript:/data:/file: hrefs) no longer counts toward the per-run denied-fetch quota (matching `unsupported_scheme`, so a document full of data: URIs can't starve later legitimate fetches); `evaluate_llm_endpoint` and `evaluate_embeddings` now percent-decode the host before classification (consistency with `evaluate_url`, so a legitimate percent-encoded local endpoint isn't wrongly denied under require-local); `context_from_snapshot` fails closed with `ValueError` on a non-dict snapshot instead of a swallowable `AttributeError`; and the run-scoped DNS classification cache is now first-writer-wins so concurrent disagreeing lookups (round-robin DNS) can't flip a hostname's classification mid-run. 11 regression tests added.
+
+  Round 2 of the 5×20 review (audit-hook + thread-context propagation) confirmed and fixed 4 issues (of 21 findings: 6 false positives, 1 already-fixed, several lower-severity backstop-completeness gaps deferred with the primary PEPs as the live gate): the PEP-578 audit hook now decodes bytes-encoded socket addresses before classifying (CPython fires `socket.connect` with a bytes host, which previously passed straight through the PRIVATE_ONLY/STRICT backstop); `set_active_context` rejects an unresolved ADAPTIVE-scope context (it would have silently no-op'd the hook); `active_egress_context` saves and restores the previous context so a nested activation no longer wipes the parent's; and the default `langgraph-agent` strategy now re-arms the audit-hook backstop inside each subagent ThreadPoolExecutor worker (threading.local isn't inherited by pool workers). 4 regression tests added.
+
+  Round 3 of the 5×20 review (call-site PEPs / data-egress paths) confirmed the empty-snapshot fail-open as the one genuinely exploitable issue and fixed it (most of the 18 findings were lower-severity than first rated — UI-display consistency, config-time validations already backstopped by execution-time PEPs, documented accepted-risk redirects, or backward-compatible empty-snapshot defaults): the `quick_summary` REST endpoint now fails CLOSED (HTTP 503) when the user's settings snapshot can't be loaded, instead of continuing with an empty `{}` snapshot that silently downgraded a configured PRIVATE_ONLY/require-local user to the permissive BOTH scope. News-subscription policy validation now also surfaces an incoherent egress config (STRICT scope + meta-picker primary, which raises ValueError) as a fail-fast misconfiguration error at create/update time rather than silently skipping the check. 1 regression test added.
+
+  Round 4 of the 5×20 review (adversarial bypass hunting) found no new exploitable bypass — every candidate was a known/documented accepted-risk (DNS-rebinding and redirect TOCTOU, per-context fetch quota, connect-only audit-hook scope), a fail-safe over-restriction, or an operator-trust-boundary case (a programmatic caller controls its own snapshot). One cheap defense-in-depth completion was applied: `evaluate_url` now also blocks cloud-metadata endpoints reachable by hostname (GCP `metadata.google.internal` / `metadata.goog`, any case) and strips an insignificant trailing dot before the metadata checks, so `metadata.google.internal.` and `169.254.169.254.` can no longer read as an allowed public host (closing the hostname/trailing-dot gaps the round-1 octal/hex/integer fix didn't cover). 2 regression tests added.
+
+  Round 5 of the 5×20 review (quality / docs / UI / completeness) fixed 5 issues: policy-audit denial logs now redact URLs (scheme://host:port only) via the existing `redact_url_for_log` at all four sites (ContentFetcher, DownloadService, and both full_search paths), so a denied URL carrying userinfo credentials or an API-key query param is no longer written verbatim to the audit log; the ADAPTIVE warning-banner render path (`/api/warnings`) now resolves scope with `allow_dns=False` so a settings-page load can't block up to the DNS timeout on a synchronous getaddrinfo for a URL-engine primary; `CONFIGURATION.md` was regenerated so its Settings List documents the egress keys and their `LDR_*` env vars (the hand-written prose, which the on-merge auto-regeneration would have wiped, moved into the hand-maintained egress-modes.md with the cross-link fixed); plus a missing `_retriever_is_local` return annotation and an added BOTH-scope allow test. 3 regression tests added. ([#4300](https://github.com/LearningCircuit/local-deep-research/pull/4300))
+- FAISS RAG indexes are now loaded through a restricted unpickler that only permits the two classes a legitimate docstore contains, instead of `FAISS.load_local(..., allow_dangerous_deserialization=True)`. A tampered `index.pkl` (the docstore pickle that is actually deserialized) can no longer execute arbitrary code on load — closing the pickle-deserialization RCE for every case, including a first-encounter index with no integrity record and a `.pkl`-only swap that leaves the `.faiss` checksum unchanged. The fix needs no integrity record, schema change, or re-indexing. The loader is fail-closed: if a future LangChain/Pydantic upgrade changes the docstore pickle format to use other classes, affected indexes raise `UnpicklingError` on load (and are quarantined and rebuilt) rather than silently deserializing — recover by re-indexing the affected collection, or by widening the loader's allow-list to the new class. ([#4632](https://github.com/LearningCircuit/local-deep-research/pull/4632))
+- **WebSocket/Socket.IO connections now default to same-origin only.** When `LDR_SECURITY_WEBSOCKET_ALLOWED_ORIGINS` is unset or empty, cross-origin WebSocket connections are rejected instead of being allowed from any origin — closing a Cross-Site WebSocket Hijacking gap as defense-in-depth (the session cookie's `SameSite=Lax` already blocked the classic cross-site case) and matching the same-origin default already used for HTTP CORS. To run a cross-origin front-end, set `LDR_SECURITY_WEBSOCKET_ALLOWED_ORIGINS` (and `LDR_SECURITY_CORS_ALLOWED_ORIGINS`) to its origin. If you terminate TLS at a reverse proxy, forward `X-Forwarded-Proto` so the same-origin check sees `https` — otherwise the WebSocket handshake is rejected (see the troubleshooting guide). Set `*` to restore the previous allow-all behavior on trusted local/dev networks. A rejected WebSocket handshake is now logged with a warning naming the origin and how to allow it, so a misconfigured origin is diagnosable instead of a silently frozen progress UI. ([#4807](https://github.com/LearningCircuit/local-deep-research/pull/4807))
+- **Local file/RAG path validation no longer 500s or blocks legitimate paths.** Browsing to a restricted path on a non-root deployment returned a 500 (an uncaught permission error from a symlink pre-check) instead of a clean "invalid path" response, and legitimately symlinked index folders (Docker/Kubernetes bind mounts, macOS `/tmp`) were wrongly rejected; both are fixed, and the system-directory block now also holds on macOS (where `/etc` is itself a symlink to `/private/etc`). Separately, a `LDR_DATA_DIR` path containing an apostrophe (e.g. `/home/O'Brien/ldr`) is now fully supported: it no longer crashes startup, and encrypted-database backups handle it too (the apostrophe is safely escaped in the backup's `ATTACH DATABASE` statement). ([#4808](https://github.com/LearningCircuit/local-deep-research/pull/4808))
+- **Restricted-directory access blocks no longer log the user's path.** When local file/RAG path validation blocks access to a system directory, the error log now names only which restricted directory was hit (e.g. `/etc`) instead of the user's full resolved path, which could contain a username. ([#4820](https://github.com/LearningCircuit/local-deep-research/pull/4820))
+- **Local-folder RAG indexing no longer logs the path you submitted when it's rejected.** A failed path validation on the local-folder indexing endpoint logged the raw submitted path (which can contain a username); it now logs a generic "Path validation failed" message instead. The `Invalid path` response is unchanged. ([#4825](https://github.com/LearningCircuit/local-deep-research/pull/4825))
+- **Local-folder RAG indexing logs the folder/file name instead of the full path.** The "indexing complete" log and the per-file indexing-error log now record only the folder/file basename rather than the full filesystem path (which can contain a username), while keeping the same diagnostic detail. ([#4831](https://github.com/LearningCircuit/local-deep-research/pull/4831))
+- **API error responses no longer leak raw exception text or server config (CWE-209).** Several response paths embedded internal exception detail into messages returned to the client: the `/api/news/*` endpoints (database/SQLAlchemy errors), the start-research and news-subscription egress prechecks (raw `ValueError`s), the research failure handler (unrecognized exceptions *and* LLM/search/provider "configuration error" detail — including server-level endpoints and file paths, since settings can come from `LDR_*` environment overrides) via `/api/research/<id>/status` and the persisted error report, and the benchmark status endpoint (`GET /benchmark/api/status/<id>`, which returned a failed run's raw `str(e)`). All now return generic, category-appropriate messages with an actionable hint where one applies, while the full cause is retained server-side in the logs — hardening shared/multi-user deployments where a non-admin user must not see another tenant's or the server's internal configuration. ([#4843](https://github.com/LearningCircuit/local-deep-research/pull/4843))
+- Local-folder RAG indexing now validates the supplied file glob patterns against an allowlist, rejecting any pattern that could escape the indexed folder (e.g. `../../etc/*` or an absolute `/etc/*`). Previously only the base folder was validated, so on multi-tenant/shared deployments a crafted pattern could read server-side files into the requesting user's library. ([#4846](https://github.com/LearningCircuit/local-deep-research/pull/4846))
+- Local-folder RAG indexing now confines globbed matches to the indexed folder: a symlink inside the folder that points outside it (a linked file, or a linked directory recursive globbing descends into) is skipped rather than read. Previously such a symlink could expose files outside the folder, since `glob` follows symlinks and only the base folder was validated. Complements the glob-pattern validation in #4846; together they close the folder-escape surface on this endpoint. ([#4848](https://github.com/LearningCircuit/local-deep-research/pull/4848))
+- **`GET /history/report/<id>` no longer leaks the settings snapshot (API keys).** The report route returned the persisted `research_meta` wholesale in its response metadata, and `research_meta` includes `settings_snapshot` — which holds all application settings, including API keys, tokens, and base URLs. It now strips `settings_snapshot` via the same `strip_settings_snapshot()` helper its four sibling routes already use, while preserving every other (non-sensitive) metadata field the report view needs. ([#4853](https://github.com/LearningCircuit/local-deep-research/pull/4853))
+- Symlink-loop confinement for RAG local indexing now works on Python 3.13+. The check relied on `Path.resolve()` raising on a symlink loop, which newer Python no longer does; it now resolves with `strict=True` so a planted in-base symlink loop is correctly skipped rather than silently kept. ([#4864](https://github.com/LearningCircuit/local-deep-research/pull/4864))
+- **Benchmark YAML downloads no longer leak the evaluation `endpoint_url` in the default export.** The default (summary) download is meant to be safe to share, but it still emitted the evaluator's `endpoint_url`, which can be a private/internal host. It's now gated behind the existing **Export → "Include settings snapshot"** opt-in — matching the settings-snapshot gating — so the default summary stays shareable and reproducibility-minded users still get it on demand.
+- **Closed three remaining password-leak paths in the settings module.** `GET /settings/api/<key>` and `GET /settings/api/bulk` now redact password-typed values to `[REDACTED]` (matching the bulk `/settings/api` endpoint redacted in the previous release). `POST /save_settings` (the JS-disabled form-encoded fallback) now treats an empty value for a password setting as a no-op, so no submission path can wipe a stored API key with empty string.
+- Add a process-wide PEP 578 `sys.audit` hook (`security/egress_audit_hook.py`) that gates `socket.connect` against the active EgressContext. This is the secondary line of defense — every explicit PEP we wired (ContentFetcher, evaluate_llm_endpoint, MCP download_content, journal_reputation_filter, …) remains the primary line and still fires first. The hook catches what those PEPs cannot: a third-party library that opens its own connection, a new code path added without policy awareness, a langchain tool registered by an MCP server, prompt-injection steering a tool into raw `requests.get`, or anything else that ultimately calls `socket.connect`. The hook is installed once on first import of `local_deep_research.security` (idempotent — PEP 578 hooks cannot be removed) and is INACTIVE by default: with no `EgressContext` registered on the current thread, every connect passes through unmodified, so test runners, import-time helpers, and scripts that touch a socket are unaffected. Workers opt in by calling `set_active_context(ctx)` (or using the `active_egress_context` context manager); the research worker in `web/services/research_service.py` now does this after building the snapshot, and the centralized `database/thread_local_session._ThreadCleanup` exit handler clears it on worker shutdown so a pooled thread cannot leak one run's scope into the next task. The hook gates only AF_INET / AF_INET6 — AF_UNIX, AF_NETLINK and friends pass through. It does NOT defend against an adversary with code execution in the LDR process (they can clear the active context or add a passthrough hook themselves); for that, layer OS-level controls per SECURITY.md. 18 unit tests in `tests/security/test_egress_audit_hook.py` pin the contract: install idempotency, get/set/clear, context-manager cleanup on exception, per-scope behaviour against real raw sockets (PRIVATE_ONLY blocks 8.8.8.8 and permits 127.0.0.1; PUBLIC_ONLY mirrors; STRICT blocks public hosts; BOTH allows either), IPv6 loopback bracket handling, AF_UNIX pass-through, and per-thread isolation under concurrency.
+- Bumped the Docker base image from python:3.14.5-slim to python:3.14.6-slim, picking up the upstream CPython fixes for CVE-2026-9669 (bz2 decompressor reuse out-of-bounds write), CVE-2026-7774 (tarfile data_filter path-traversal bypass), and CVE-2026-3276 (unicodedata.normalize CPU exhaustion). Also suppressed three base-image CVEs with no fix available in Debian trixie (graphite2 CVE-2026-50593, expat CVE-2026-50219, perl HTTP::Tiny CVE-2026-7010 — none reachable in this container) in .grype.yaml, and removed five suppressions for python CVEs that the 3.14.5/3.14.6 releases fixed.
+- Closed a residual SQLCipher-password leak left after the #4182 sweep: `open_user_database` redacted the migration-failure log line but still re-raised `DatabaseInitializationError` with the unredacted original error embedded in its message and chained via `from init_err`. The caller (`thread_local_session`) logs that typed error with `logger.exception`, which re-rendered the chain — and the password its frame locals carry under `diagnose=True` — defeating the redaction. The typed error now carries the redacted message and breaks the chain with `from None` (ADR-0003). See #4182.
+- Closed another `diagnose=True` frame-locals leak of the SQLCipher master password from the #4182 sweep: `ThreadSafeMetricsWriter.get_session` opened a per-thread metrics session with `password` live in the frame and logged failures with `logger.exception`, so a rendered traceback could persist the plaintext password (unrecoverable — TRUST.md §5). It now logs with `logger.warning` (no traceback); the exception still propagates to the caller via `raise`, so nothing is swallowed. The module is added to the `test_password_redaction_invariant` allow-list so CI permanently blocks any regression. See #4182.
+- Closed the last `diagnose=True` frame-locals leak of the SQLCipher master password from the #4182 sweep: the two consumers of `DatabaseInitializationError` — `thread_local_session.get_session` and `ResourceStatusTracker.__init__` — caught the error and logged it with `logger.exception` while `password` was a live local in the frame, so a rendered traceback could persist the plaintext password (unrecoverable — TRUST.md §5). Both now log with `logger.warning` (no traceback); the redacted detail is already logged at the raise site. The two modules are added to the `test_password_redaction_invariant` allow-list so CI permanently blocks any regression. See #4182.
+- Completed the #4182 logging chokepoint: the optional file sink (`LDR_ENABLE_FILE_LOGGING`) now also runs with `diagnose=False`, matching the database and frontend sinks. Previously it still honored `LDR_LOGURU_DIAGNOSE`, so an operator with debug + diagnose + file logging all enabled could persist frame-local credentials (including the unrecoverable SQLCipher master password — TRUST.md §5) into an unencrypted log file. Frame-local exception dumps now render only to the ephemeral stderr sink; no persisted or shipped sink (DB, browser, file) ever renders them. See #4182.
+- Completed the analytics-page innerHTML hardening pass: coerced the `frequency_rank` value (`#N` domain rank badge) in `link_analytics.html` with `Number()` — the one numeric sibling left un-coerced next to `usageCount`/`usagePercentage`/`researchDiversity` — and escaped `research_id` in `cost_analytics.html`'s "most expensive research" list (`encodeURIComponent()` in the `href`, `window.escapeHtml()` in the link text, mirroring `star_reviews.html`). Both are defense-in-depth on raw `${...}` interpolation into `innerHTML` with no DOMPurify barrier; neither is currently exploitable (rank is a server-side int, `research_id` is a `uuid4` and the cost-analytics render path is presently disabled), but together they make the PR's "across analytics pages" scope complete.
+- Egress policy hardening (Round 7 review): close four PEP gaps surfaced by a 120-agent re-audit and apply the same fix to four sibling sites. (1) LLM gate at config/llm_config.py now uses a known-local allow-list (`ollama`/`lmstudio`/`llamacpp`) when no snapshot is supplied — previously the gate silently no-op'd, letting cloud LLMs instantiate from snapshot-less callers under `llm.require_local_endpoint=true`. (2) `JournalReputationFilter` forwards `settings_snapshot` to `get_llm()` and `create_default` lets `PolicyDeniedError` propagate instead of silently returning None. (3) MCP `_discover_mcp_tools` / `_execute_mcp_tool` block under STRICT / PRIVATE_ONLY (and on absent snapshot), with a policy_audit log and a UI progress signal. (4) `_execute_download_content` narrows its bare `except Exception` so corrupted scope values fail closed instead of dropping to SSRF-only. The same bare-except fail-open pattern is fixed consistently in `notifications/manager.py`, `research_library/services/download_service.py` (paired with a `_policy_locked` short-circuit in the URL check), `web_search_engines/search_engine_base.py` (disables `include_full_content` when policy can't be evaluated), and `web/routes/research_routes.py` (STRICT + meta-picker now surfaces as 400 at run-start). Adds a Flask `PolicyDeniedError` handler so denials escaping a synchronous PEP return a clean 400 with the decision reason. Regression tests: parametrized no-snapshot fail-closed for every cloud provider plus ambiguous (`openai_endpoint`) and hypothetical (`groq`, `mistral`, `cohere`) entries; allow-list passes for the three known-local providers; MCP scope gate (STRICT/PRIVATE_ONLY/no-snapshot/corrupted policy) verifies the connection manager is never touched and the empty list is cached; download_content refuses without constructing ContentFetcher; journal_reputation_filter forwards snapshot and propagates PolicyDeniedError.
+- Egress-policy follow-ups after the 5×20 review: (1) ContentFetcher now relaxes a downloader's `SafeSession` to allow private IPs under `PRIVATE_ONLY`, mirroring `policy_aware_validate_url`, so a private/lab URL the policy already approved is no longer rejected by the downloader's own strict SSRF re-validation; (2) the PEP-578 audit-hook backstop is now re-armed in the `NewsAggregationStrategy` analysis ThreadPoolExecutor worker (threading.local isn't inherited by pool workers) and in the document scheduler's worker thread (built from the user's saved settings), so scheduled downloads and news-analysis LLM calls keep defense-in-depth parity under `PRIVATE_ONLY`/`STRICT`. The primary snapshot-based PEPs already gated these paths; this restores the secondary net. (MCP search runs synchronously on the already-armed strategy thread, so it needed no change.)
+- Extracted `updateEnhancedDomainList` from the `link_analytics.html` inline script into `static/js/pages/link_analytics_render.js` (surgical extraction mirroring PR #4584) so it can be tested in isolation. Added 7 Vitest XSS regression tests at `tests/js/pages/link-analytics-xss.test.js` covering the escape sites fixed in #3095 (Number()-coercion of `research_diversity`/`usage_count`/`usage_percentage`, `escapeHtml` of research-link queries and classification fields, `encodeURI`/`encodeURIComponent` of domain names and research IDs, and the inline-script wrapper pattern at `link_analytics.html:797-802`). These tests run in PR CI via the existing Vitest job. Added a new `link-analytics` Puppeteer shard for full-page verification of `/metrics/links` in the release pipeline (strict-mode only), with runtime assertions that no `<script>` element is injected and that the "Recent Researches (N total)" header shows a numeric `N`.
+- Hardened the SQLCipher-master-password frame-locals leak class (#4182) at two levels. (1) Sink level: `log_utils` now forces `diagnose=False` on the database sink (which persists into the user's own encrypted DB) and the frontend sink (which ships to the browser), so loguru can never render frame-local credentials into a durable or remote sink even when an operator enables `LDR_LOGURU_DIAGNOSE` for local debugging — a single chokepoint covering every credential-bearing exception handler app-wide. (2) Targeted sweep: the credential-centric session helpers `metrics/search_tracker`, `metrics/token_counter`, `database/library_init`, `database/backup/backup_executor`, `web_search_engines/rate_limiting/tracker`, and `research_library/.../research_history_indexer` now log failures with `logger.warning` instead of `logger.exception`, and are added to the `test_password_redaction_invariant` allow-list so CI permanently blocks regressions. Big multi-purpose route/service handlers are intentionally left logging full tracebacks for unrelated errors — they are covered by the sink-level guard. See #4182.
+- Operator-configured LLM provider URLs (`llm.ollama.url`, `llm.lmstudio.url`,
+  `llm.llamacpp.url`, `llm.openai_endpoint.url`) are now validated against the
+  same SSRF rules as outbound HTTP before the LangChain SDK constructor runs.
+  Closes an auth-gated SSRF gap where the SDK's internal `httpx` client would
+  otherwise bypass the existing `safe_requests` guard.
+
+  Follow-up review closed two more sites in the same vuln class: the
+  OpenAI-compatible base class's model-listing (`list_models_for_api`, used by
+  the custom-endpoint / LM Studio / llama.cpp providers) now validates
+  `base_url` before constructing the OpenAI client and returns an empty list on
+  a blocked URL, and `OpenAIProvider` now validates `llm.openai.api_base` before
+  handing it to `ChatOpenAI`.
+- RAG auto-indexing now applies backpressure to its background worker pool. The `ThreadPoolExecutor` previously had an unbounded internal queue; under sustained upload bursts (possible under the configurable upload rate cap from #3935), this could queue thousands of indexing jobs and exhaust memory. The pool is now capped at 100 in-flight + queued jobs; submissions over the cap are dropped with a clear log warning ("Auto-index queue saturated… trigger a manual reindex if needed"). Documents still upload successfully — only the *automatic* indexing is skipped, and users can manually reindex via the UI.
+- Routed every search-engine site that scrubbed errors via `redact_secrets` through a single `BaseSearchEngine._scrub_error()` helper (regex sanitize + literal redaction), replacing ~60 hand-copied call sites. This closes the per-site drift class that could drop a credential, and upgrades several engines whose logged errors previously skipped the regex sanitization pass.
+- Search, fetch, and agent tool error messages are now scrubbed of credentials before they reach the LLM and the user-visible research output — a search-engine or LLM exception can embed a request URL carrying an API key, so these are now routed through the same `sanitize_error_for_client` helper used for download errors.
+- The MCP search strategy now filters its specialized search-engine tool list by the active egress scope (parity with the LangGraph strategy): forbidden engines never appear in the tool schema the LLM sees, and a policy denial at execution surfaces as an audit log line instead of a generic tool error.
+- The ``custom_endpoint`` URL supplied to ``/api/start_research``, ``/api/news/subscriptions`` (POST and PUT/PATCH), and ``/api/followup/start`` is now validated for SSRF at the request boundary (rejecting cloud-metadata / link-local targets, always blocked by ``ssrf_validator``) before any research is queued. This is fail-fast defense-in-depth — the OpenAI-compatible provider re-validates the same URL via ``assert_base_url_safe`` before the LangChain client is built — but the route-layer check rejects early, before a DB row is written or a thread is spawned, and keeps the endpoint out of the logs. The endpoint is normalized exactly as the provider normalizes it, so scheme-less local endpoints (``localhost:11434``, ``192.168.1.10:8000``) are accepted; private IPs and localhost still pass, so local LLM providers (Ollama, LM Studio, vLLM) are unaffected.
+- The error-message credential scrubber now also redacts GitHub (`ghp_`/`github_pat_`), AWS (`AKIA`/`ASIA`/…), Slack (`xox*-`/`xapp-`), Google OAuth (`ya29.`) tokens and JWTs — extending coverage to more common credential formats before error text reaches clients or logs.
+- The per-run denied-fetch quota (`MAX_DENIED_FETCHES_PER_RUN`) is now aggregated per *run* instead of per `EgressContext`. Each call site (content fetcher, full-content search, download service, the audit hook) builds its own `EgressContext`, so the counter previously reset whenever a new engine/fetcher was constructed — letting a malicious indexed document evade the exhaustion guard by spreading denied fetches across contexts. The counter is now anchored to the run's armed audit-hook context (shared across the whole run, re-armed identically on pool workers), with a safe fallback to the local context for snapshot-less / programmatic callers.
+- The server now logs a loud warning at startup when the SQLCipher KDF is weakened below the production floor (reached only in test mode, e.g. ``LDR_TEST_MODE``) while user databases already exist. New databases would be created with this weak at-rest work factor, and — separately — any existing database created at a higher KDF can no longer be opened: its on-disk key is unchanged, but the server now derives a different, weaker key, so decryption fails and login returns a generic "Invalid username or password" 401 for every affected user (the KDF-mismatch symptom class behind PR #4775). Silent on fresh deployments and when the effective KDF is at/above the floor.
+- The shared error-message credential scrubber now also redacts OAuth `access_token`/`refresh_token` and other sensitive URL query parameters, Azure `subscription-key`, `Authorization:`/`x-api-key:` headers, and Google (`AIza…`) API keys — strengthening credential redaction for every code path that surfaces error text to clients or logs.
+- Under the PRIVATE_ONLY egress scope (and Adaptive resolving to private), notification dispatch via non-http Apprise vendor schemes (`slack://`, `discord://`, `telegram://`, …) is now refused — those send to external vendor APIs and can't be verified local. Address a self-hosted notifier by its `http(s)://` URL instead, which is allowed as a private host. Other scopes are unchanged.
+- Wrapped the `research_diversity` interpolation in the link-analytics "Recent Researches (N total)" header in `Number()`, matching the same coercion already applied to the sibling `🔍 N researches` badge in the same template. Closes a defense-in-depth gap (raw `${researchDiversity}` into `innerHTML` with no DOMPurify) left inconsistent by PR #3095. Not currently exploitable — the value is a Python `len(...)` int server-side — but aligns with the PR's stated scope of coercing numeric fields used in template-literal text positions.
+
+### ✨ New Features
+
+- **Star Reviews dashboard gains six new charts.** The `/metrics/star-reviews` page now shows a doughnut distribution, rating-volume overlay on the trends chart, research-mode comparison, LLM satisfaction breakdown (stacked bars), quality-dimensions radar, and a recent-feedback section. Two bugs were also fixed: recent ratings displayed "Invalid Date" and showed generic fallbacks instead of actual query text and mode. ([#3804](https://github.com/LearningCircuit/local-deep-research/pull/3804))
+- **Benchmark YAML downloads now show the LDR version and `date_tested` from when the benchmark *ran*, not at download time.** Previously a v1.6.5 run downloaded on v1.6.7 incorrectly stamped v1.6.7 — making cross-run comparison impossible. The full per-key settings snapshot active at the run (API keys redacted) is now captured at run time and is available **opt-in** via the new **Export → "Include settings snapshot"** checkbox; by default the YAML contains only the summary configuration, so it's safe to share without exposing internal URLs or local paths. Benchmarks created before this release show `ldr_version: unknown (pre-0014 run)` and, when settings are requested, `settings: null` — no backfill, but no breakage either. ([#3844](https://github.com/LearningCircuit/local-deep-research/pull/3844))
+- Add TinyFish as an optional hosted search engine with structured search results and browser-rendered content extraction. ([#4057](https://github.com/LearningCircuit/local-deep-research/pull/4057))
+- Added a "Sampling Seed" field to the benchmark configuration (default 42): the same seed and example counts now select the same benchmark questions on every run, so accuracy numbers are comparable across runs and configurations. The seed is stored with each benchmark run. Leave the field empty to draw a fresh random sample each run (the previous behavior). ([#4516](https://github.com/LearningCircuit/local-deep-research/pull/4516))
+- Library searches now warn in the research log when results are incomplete because one or more collections failed, naming the affected collections; failed-collection error messages use collection names instead of internal ids. ([#4520](https://github.com/LearningCircuit/local-deep-research/pull/4520))
+- **New "Anthropic-Compatible Endpoint" LLM provider** for self-hosted services that speak the Anthropic Messages API (`/v1/messages`) rather than the OpenAI chat-completions format. Select it under Settings → LLM, set `llm.anthropic_endpoint.url` (and an optional `llm.anthropic_endpoint.api_key` — keyless gateways are supported), and LDR builds a `ChatAnthropic` client pointed at your endpoint. The base URL is SSRF-validated before any request, and a real `ANTHROPIC_API_KEY` in the environment is never sent to a self-hosted endpoint. The official cloud Anthropic provider is unchanged. ([#4581](https://github.com/LearningCircuit/local-deep-research/pull/4581))
+- Add per-message **Copy**, **Retry**, and **Delete** actions to chat — the standard ChatGPT/Claude.ai hover-icon pattern. Retry replaces the failed turn and re-submits the same query as a fresh research run; Delete removes the user message, research, and any assistant response for that turn. The History page also gains a Copy-query button on each card and the rerun button is now shown for failed/suspended researches too (previously only successful runs had it). ([#4659](https://github.com/LearningCircuit/local-deep-research/pull/4659))
+- **Star Reviews dashboard polish.** Recent-feedback entries now link to their source research, and a screen-reader live region announces when the dashboard refreshes (e.g. after changing the time period), including which charts have no data yet. ([#4793](https://github.com/LearningCircuit/local-deep-research/pull/4793))
+- **The search-engine selector is now grouped into labelled bands.** Engines are ordered by a trust/cost gradient with section headers — Favorites, Collections, Academic, Local RAG, Books, Code, News, No API key, API key — instead of one alphabetical list. Starring an engine still pins it to the Favorites band at the top. ([#4814](https://github.com/LearningCircuit/local-deep-research/pull/4814))
+- **`/api/v1/health` now reports file-descriptor and thread diagnostics.** Authenticated callers receive a `resources` block (`fd_count`, `fd_soft_limit`, `fd_hard_limit`, `fd_usage_percent`, `thread_count`), and `status` flips to `"warning"` above 70% FD usage — making FD growth observable over HTTP for dashboards and alerting. Anonymous callers still get the basic `status`/`message`/`timestamp`, so the Docker healthcheck is unaffected. FD metrics report `null` on non-Linux platforms. ([#4915](https://github.com/LearningCircuit/local-deep-research/pull/4915))
+- **Unified background reconciler that indexes ANY unindexed document.** A single scheduled job now indexes every unindexed document so nothing is permanently missed: (a) library uploads that the immediate auto-index queue dropped when saturated, and (b) research downloads that were never ingested into your library. Enable the **Document Scheduler** and then **"Auto-index documents in the background"** (both off by default) to turn it on — the sweep only runs while the document scheduler itself is enabled. The reconciler is idempotent (already-indexed documents are skipped), processes a small bounded batch per run so a large backlog self-heals gradually, and honors each collection's own embedding configuration. The two cases are budgeted independently, so a backlog of documents that keep failing to index can never starve the other case (research downloads still get indexed even when many library uploads are failing). This replaces both the old library-only sweep and the separate per-research RAG-indexing pass that used to run inside the document scheduler. The legacy **"Generate RAG Embeddings"** setting is kept as a deprecated alias — it now enables this same reconciler — so existing users keep getting their research downloads indexed with no change required.
+- A collection's public/private (egress) classification can now be changed after creation, from a "Privacy & Egress" toggle on the collection details page — not only at creation time.
+- Add an **Adaptive** egress scope (now the default) that follows your primary search engine: a private primary keeps everything local (and forces local LLM/embeddings inference), a public primary allows public engines, and a meta-picker primary allows both. Existing explicit scopes (Both / Public only / Private only / Strict) remain available. Selecting **Private only** now visibly checks and locks the "require local LLM/embeddings" toggles to reflect that local inference is enforced.
+- Add an egress policy module (`security.egress_policy`) that gates search-engine instantiation against a user-declared scope. New settings `policy.egress_scope` (default `both`, preserves current behavior), `llm.require_local_endpoint`, `embeddings.require_local`, and `llm.allowed_local_hostnames` are reserved. Under `strict` scope the factory blocks any engine that isn't the user's primary, closing the previously silent expansion path. The LangGraph agent tool-list filter, LLM/embeddings PEPs, and UI controls ship in the same release (see the related Stage 1b changelog entries).
+- Collections can now be classified **public** or **private** (default private). A private collection is excluded from runs under Public-only / Adaptive-public egress scope and forces local LLM/embeddings inference under Private-only / Adaptive-private scope, so its contents never reach a cloud model. Flip the classification with the "Public collection" checkbox on the collection create form. Adds the `collections.is_public` column (migration 0011).
+- Collections now have an **"Available to the research agent"** toggle (default on). Turn it off to keep a collection out of the LangGraph research agent's tool list when it isn't needed for agentic research — without deleting it or changing its privacy/egress classification. The flag is exposed on the collection create form and the collection details page, persists via the collections API, and is enforced where the agent builds its specialized search tools (a disabled collection's name never reaches the agent).
+- Library collections now show per-collection indexing status ("X of Y indexed" + a "N pending indexing" badge) with a per-card Reindex action. When the optional scheduled-reconciler setting is available, a toggle to index documents in the background on a schedule also appears (otherwise it stays hidden). (#3939/#4627 follow-up).
+- When the egress policy blocks something, the user now gets a **clear, actionable message** — what was blocked, why, and exactly which setting to change to allow it (e.g. "blocked because your Egress Scope is Private only → change it in Settings → Privacy & Egress, or turn off 'Require local LLM endpoint'") — instead of a terse machine code. A new `security.egress.guidance.denial_guidance()` helper centralises these messages; it's wired into the research start-up engine check, and denials surface the raw reason code alongside for support.
+
+  The `quick_summary` API also gains an opt-in: pass `"allow_default_settings": true` to deliberately run with default settings (no egress policy) when your settings can't be loaded, instead of the default fail-closed `503`. The 503 itself now explains the cause and the fix (retry / re-authenticate / opt in).
+- Wire the egress policy module (added in Stage 1a) into the LLM and embeddings call paths. `get_llm()` now raises `PolicyDeniedError` when `llm.require_local_endpoint=True` and a cloud LLM provider is requested. The OpenAI and Anthropic model-listing endpoints in the settings UI degrade gracefully (empty list + audit log entry) under the same condition. `get_rag_service()` runs a pre-flight check before constructing the service so embedding-policy violations surface immediately rather than after hundreds of chunks have been generated. Defaults are unchanged (`require_local_endpoint=False`, `embeddings.require_local=False`), so existing installs see no behavior change.
+
+### 🐛 Bug Fixes
+
+- REST `/api/v1/generate_report` and `/api/v1/analyze_documents` now load the authenticated user's encrypted-DB settings (API keys, model preference, search tool) instead of silently falling back to application defaults plus `LDR_*` env vars. Like `/quick_summary`, both endpoints now fail closed (HTTP 503) when the settings snapshot cannot be loaded, with `allow_default_settings: true` as the explicit opt-out. ([#3661](https://github.com/LearningCircuit/local-deep-research/pull/3661))
+- Starting a follow-up research with no LLM model configured now fails immediately with a clear "Model is required" error (HTTP 400) instead of spawning a worker that dies and leaves the research stuck IN_PROGRESS. ([#3767](https://github.com/LearningCircuit/local-deep-research/pull/3767))
+- **Star Reviews analytics now count each rating once.** The `/metrics/star-reviews` queries joined `token_usage` directly, which fans out one row per LLM call — so "Recent Ratings" showed the same research duplicated (far fewer than 20 unique entries) and the LLM/search-engine breakdown counts and averages were inflated by the number of calls per research. These queries now collapse token usage to one row per research before joining. The quality-dimensions radar no longer hides real data when only some dimensions are rated, and reports a per-dimension sample size. The rating endpoint also rejects boolean values (which previously passed the 1–5 check) and overly long feedback. ([#3804](https://github.com/LearningCircuit/local-deep-research/pull/3804))
+- The `/api/rate-limiting/cleanup` endpoint now rejects out-of-range or non-integer `days` values with HTTP 400 instead of passing them through to the destructive bulk delete (clamped to `1 <= days <= 365`). ([#3805](https://github.com/LearningCircuit/local-deep-research/pull/3805))
+- Wikipedia search no longer floods the log with per-title tracebacks when the MediaWiki API rate-limits a query — the engine now detects the JSON-decode error pattern, logs a single warning, and bails out of the batch with whatever previews it collected. Fetched pages that come back empty (paywalls, JS-only SPAs, blank 200s) and LLM extractions that return an empty summary now short-circuit to `NOT RELEVANT` without locking the URL into the citation collector, so the agent is free to re-fetch it later under a different focus. ([#3844](https://github.com/LearningCircuit/local-deep-research/pull/3844))
+- `SettingsManager.get_all_settings()` no longer crashes when the DB contains a row whose `type` column holds an enum value no longer in `SettingType` (e.g. legacy `'CHAT'`-typed rows from a removed feature). The handler now catches `LookupError` alongside `SQLAlchemyError` and falls back to defaults-only — strictly safer than crashing every caller (`/settings/api`, benchmark start, research start, MCP). ([#3947](https://github.com/LearningCircuit/local-deep-research/pull/3947))
+- **Submitting an empty value for a password-typed setting is now a no-op** — previously could overwrite the stored secret with `""`. Belt-and-braces fix for the same `[REDACTED]` sentinel issue: a stale browser tab can no longer corrupt API keys by triggering a save. To clear a password setting, clear the source env var or use settings import. ([#3954](https://github.com/LearningCircuit/local-deep-research/pull/3954))
+- Fix Ollama `enable_thinking` setting being silently dropped on the live class
+  construction path. Reasoning models (deepseek-r1, qwen2.5) now correctly
+  receive `reasoning=True/False` per the user's `llm.ollama.enable_thinking`
+  setting. Also apply the 80%-of-context-window `max_tokens` cap on every
+  provider's `create_llm` (was previously only applied in dead procedural
+  code that never ran in production) and unify the optional-API-key
+  placeholder string across all providers (`"not-required"`). Note: users
+  who set `llm.context_window_unrestricted = false` together with a small
+  `llm.context_window_size` will now have `max_tokens` capped at 80% of
+  that window for OpenAI/Anthropic too — previously the cap only applied
+  to Ollama. For LM Studio and llama.cpp the cap is always active, keyed
+  to `llm.local_context_window_size` (default 20480 → effective cap 16384
+  with the default `llm.max_tokens` of 30000). `llm.supports_max_tokens =
+  false` is now honored on the live path (previously the kwarg was sent
+  regardless), and whitespace-only API keys for required-key providers now
+  raise a clear error at construction instead of being sent to the API.
+
+  Ollama's `num_ctx` now resolves through the same shared helper as the
+  context-overflow bookkeeping, so the reported `context_limit` always
+  matches the window the model actually runs with (previously an absent or
+  null `llm.local_context_window_size` produced `num_ctx` 4096/server-default
+  while overflow detection assumed 8192). The ineffective `max_tokens`
+  kwarg is no longer passed to `ChatOllama` (it was silently ignored;
+  `num_ctx` is the effective control). A keyless `openai_endpoint`
+  configuration is now permitted — it previously raised "API key not
+  configured" at construction, blocking keyless local servers (vLLM,
+  text-generation-webui); a warning is logged so endpoints that do
+  require a key remain diagnosable.
+
+  Also fix `llama.cpp` provider availability check: instances behind an
+  authentication proxy now correctly report as available when
+  `llm.llamacpp.api_key` is configured. Previously the probe was sent
+  without credentials and was always rejected, so users with auth-proxied
+  llama-server saw the provider as unavailable regardless of server state. ([#3984](https://github.com/LearningCircuit/local-deep-research/pull/3984))
+- **Settings on mobile no longer scrolls forever.** Every section now starts collapsed at the mobile breakpoint (`max-width: 767px`) — page height drops from > 16,384 px to ~7,300 px on a Pixel 5. Desktop is unchanged; typing in the Settings search box still force-expands every surviving section. ([#4032](https://github.com/LearningCircuit/local-deep-research/pull/4032))
+- Chat Mode post-merge audit follow-ups (#3891). The send-message API now rejects a non-boolean ``trigger_research`` with HTTP 400 instead of silently coercing a truthy value (e.g. ``"false"``) to ``True`` and launching unwanted research. Chat-session deletion now records the (truncated) username alongside the session id in the audit log, so a stolen-token bulk delete leaves a forensic trail. The frontend now logs (instead of silently swallowing) a failed background title-generation request, and the Markdown chat export documents that its output is a human-readable archive that is not safe to feed back into a Markdown/HTML renderer without escaping. The "Clear History" confirmation now warns that it also permanently deletes all chat sessions and their conversations (it always did, but the dialog only mentioned research history). Accessibility: the chat input gains ``aria-required`` and the welcome suggestion buttons gain descriptive ``aria-label``s. Internal cleanup: the duplicated atomic ``UPDATE … RETURNING`` counter logic in ``ChatService`` is extracted into a shared ``_atomic_increment`` helper. New tests cover the 100+-message pagination cap, render-time Jinja2 autoescaping, and the stricter ``trigger_research`` contract. ([#4427](https://github.com/LearningCircuit/local-deep-research/pull/4427))
+- Two pre-existing export/metadata correctness bugs are fixed. RIS citation exports were malformed: the entry builder reused its output accumulator as a scratch variable when reading the title, so each record emitted the raw source text *before* the mandatory leading ``TY  - `` tag and reference managers (Zotero/Mendeley/EndNote) rejected the file. Separately, ``get_subscription_history`` called ``json.loads()`` on ``research_meta`` — a JSON column that SQLAlchemy already deserializes to a dict on read — so the resulting ``TypeError`` was swallowed by a bare ``except`` and every subscription-history item silently lost its headline and topics; it now handles both dict and legacy-string values like the rest of the module. ([#4435](https://github.com/LearningCircuit/local-deep-research/pull/4435))
+- Fixed a cross-user data exposure in multi-user deployments: saving any setting emitted a ``settings_changed`` WebSocket event — carrying the changed setting keys **and their raw values**, including plaintext LLM API keys — to *every* connected client on the shared Socket.IO server, where each browser merged them into its local settings cache. Sockets now join a per-user room on connect and ``settings_changed`` is scoped to the owning user's own browser tabs only; a settings change made outside a request context (start-up defaults, background workers) no longer emits at all instead of falling back to a broadcast. Single-user (desktop) deployments were unaffected. ([#4437](https://github.com/LearningCircuit/local-deep-research/pull/4437))
+- Fixed three more cross-user WebSocket leaks in multi-user deployments (follow-up to the ``settings_changed`` fix): ``parallel_search_started`` — which carried the user's **raw research query** — and ``engine_completed`` were broadcast to every connected client despite having no frontend listener, and are removed entirely; ``search_engine_selected`` (shown in the research log panel) is now scoped to the owning user's per-user room, resolved from the research worker's search context, and is skipped entirely — never broadcast — when no username is available. ([#4446](https://github.com/LearningCircuit/local-deep-research/pull/4446))
+- Fixed benchmark runs with xbench-DeepSearch enabled ignoring the configured number of examples: `XBenchDeepSearchDataset.load()` overrode the base class with its own `num_examples` parameter, so the registry's argument-less `load()` call skipped sampling and queued all 100 (Chinese-language) xbench questions. A run configured for e.g. 50 SimpleQA + 10 xbench questions therefore processed 150 tasks against a recorded total of 60, driving the progress display past 100% and surfacing unexpected Chinese questions. The dataset now samples via the constructor's `num_examples`/`seed` like every other dataset. ([#4451](https://github.com/LearningCircuit/local-deep-research/pull/4451))
+- The research progress display and agent-thinking panel now show the actual search engine name (e.g. "DuckDuckGo") instead of the generic internal id "web_search". The LangGraph strategy keeps the stable tool id in the progress event metadata while surfacing the friendly, brand-correct engine name in the message, and the agent-thinking panel renderer now prefers that human-readable message. ([#4470](https://github.com/LearningCircuit/local-deep-research/pull/4470))
+- Fixed metrics pages showing "No token usage data yet" despite completed researches (#4457). LLM calls whose provider returns no token usage data (OpenAI-compatible servers that omit `usage` on streamed responses, proxies that strip it, Ollama omitting `prompt_eval_count` for fully-cached prompts) were silently skipped, leaving every metrics panel empty; they are now recorded with zero token counts plus a warning log, so call counts, models, phases and response times still appear. Added an opt-in `llm.openai_endpoint.stream_usage` setting that requests usage stats on streamed responses (`stream_options.include_usage`) for servers that support it (LM Studio 0.3.18+, llama.cpp, vLLM, OpenRouter) — off by default because some gateways reject the parameter with a 400. Also fixed background-thread token writes not updating the `ModelUsage` aggregate, and several /metrics panels ignoring the selected time period: the research count, plus the ratings, link and strategy analytics, fell back to a fixed 30-day (or all-time) window when "3 months" or "1 year" was chosen, because the period vocabularies were inconsistent. All metrics endpoints now share one period map. ([#4473](https://github.com/LearningCircuit/local-deep-research/pull/4473))
+- Fixed the v1.7.0 Docker image crashing with SIGILL (exit code 132) on x86 CPUs that support AVX but not AVX2 (e.g. Sandy Bridge / Ivy Bridge era). The ``faiss-cpu`` 1.14.2 wheels dropped the runtime CPU dispatch that 1.13.x shipped (separate generic/AVX2/AVX512 builds) and ship a single ``libfaiss.so`` compiled with AVX2 instructions, so importing faiss at app start-up executed an illegal instruction on AVX-only CPUs. ``faiss-cpu`` is now pinned to the 1.13.x series until upstream wheels restore a non-AVX2 fallback (reported upstream as facebookresearch/faiss#5296; lifting the pin is tracked in #4499). ([#4480](https://github.com/LearningCircuit/local-deep-research/pull/4480))
+- Hardened app startup against unrelated broken packages in the user's site-packages: the text-splitter registry now imports the three splitters it uses directly from their `langchain_text_splitters` submodules instead of the package root, whose `__init__` eagerly imports every optional splitter backend (spacy, nltk, konlpy, ...). Previously, a globally installed spacy paired with an incompatible typer/click combination crashed `ldr-web` at startup with `TypeError: type 'Choice' is not subscriptable`. ([#4490](https://github.com/LearningCircuit/local-deep-research/pull/4490))
+- Research no longer fabricates citations when every search returns nothing: if no sources are found (or a local collection search fails), the answer now states this explicitly instead of inventing references, the progress log shows an error, and failed collection searches are reported instead of being silently treated as "no matching documents". ([#4503](https://github.com/LearningCircuit/local-deep-research/pull/4503))
+- A temporarily unreachable embedding provider (e.g. Ollama not running) no longer quarantines a healthy collection index and silently replaces it with an empty one; the search now fails with a clear provider error and the index survives untouched. ([#4512](https://github.com/LearningCircuit/local-deep-research/pull/4512))
+- Fixed intermittent ~60s UI freezes / "Navigation timeout" failures (#4431) caused by heavy third-party imports (matplotlib via the benchmarks package) and synchronous stderr logging blocking the dev server's request pipeline under load. matplotlib/optuna.visualization are now imported lazily (only when benchmark visualizations are generated), and the stderr log sink uses `enqueue=True` so logging never blocks on I/O while holding loguru's handler lock. Note: `enqueue=True` makes stderr logging asynchronous (a background writer thread), so on an abrupt crash the last few buffered log lines may be lost and ordering relative to other sinks can differ slightly. ([#4536](https://github.com/LearningCircuit/local-deep-research/pull/4536))
+- Fixed chat-initiated research running without the user's database password after the in-memory session-password store expired (24h TTL) or was cleared by a server/container restart while the session cookie was still valid. In that state the research completed (so the user saw results) but every background metric write was silently dropped, leaving the metrics dashboard empty (part of #4457). The chat `send_message` route now applies the same encryption-aware password guard as the direct and follow-up research routes, returning a clear "session expired — log out and log back in" message before any rows are created. Unencrypted installs are unaffected. ([#4571](https://github.com/LearningCircuit/local-deep-research/pull/4571))
+- Fixed a `MemoryError` when listing or clearing history on large libraries — history/report list queries no longer load full report bodies into memory. ([#4574](https://github.com/LearningCircuit/local-deep-research/pull/4574))
+- Fixed the rate-limiting panel on /metrics showing all zeros (no tracked engines, no wait times) for every user. The analytics read the `RateLimitAttempt` table, but raw attempt persistence was disabled to prevent database locking under parallel search, so that table is never populated. The panel now derives engine health, wait-time estimates, success rates, and recent attempt counts from the `RateLimitEstimate` data that rate limiting actually persists. (A few raw-attempt-only metrics — rate-limit-event counts and true per-attempt average wait — cannot be reconstructed and are reported as 0 / the learned base wait.) ([#4576](https://github.com/LearningCircuit/local-deep-research/pull/4576))
+- **The cloud Anthropic model dropdown was empty.** The model-list route fetched Claude models correctly, but the auto-discovered-provider pass that ran afterward re-listed them with the OpenAI SDK (`Authorization: Bearer`), which 401s against the Anthropic API (it requires `x-api-key`) and overwrote the good result with an empty list. `AnthropicProvider` now lists models via the `anthropic` SDK, so the dropdown populates again. ([#4586](https://github.com/LearningCircuit/local-deep-research/pull/4586))
+- Fixed news subscription scheduling regressions from #4492: a manual "Run Now" whose research later failed no longer pushes the subscription out a full refresh interval (it is reset to due so the scheduler retries), the run-now request no longer holds the encrypted-DB session open across the research-start HTTP call, and the subscription folder-update PUT route now keeps `status` authoritative (an `is_active` toggle pauses correctly instead of being ignored by the scheduler). The same folder-update route was also repaired — it previously failed on every call due to a bad relative import and a missing serializer. ([#4587](https://github.com/LearningCircuit/local-deep-research/pull/4587))
+- Fixed two news endpoints that returned HTTP 500 for any real subscription: the subscription history modal (it read a non-existent `refresh_count` column; the run count is now derived from research history) and the organized-subscriptions view (it called `.to_dict()` on plain dicts; it now returns the folder-grouped shape the UI renders). ([#4588](https://github.com/LearningCircuit/local-deep-research/pull/4588))
+- Fixed the RAG collection `embedding_dimension`, which was always stored as NULL (the metadata helper probed a non-existent `provider.embedding_dimension` attribute); it is now derived from the actual embedding model. Also extended the collection-indexing dedup to the bulk `index-all` route, which previously stored no embedding metadata and never cleaned up stale chunks/indices on force-reindex. ([#4593](https://github.com/LearningCircuit/local-deep-research/pull/4593))
+- Fixed the download manager progress popup showing "X / 0 files" instead of the real total. The pending-item count is now taken after the queue is populated, so the denominator reflects the actual number of papers being downloaded. The download manager now also surfaces an explicit alert when there is nothing to download (e.g., all papers already downloaded) or when queueing fails for every selected research session, instead of silently completing with "0 / 0 files". ([#4660](https://github.com/LearningCircuit/local-deep-research/pull/4660))
+- Fixed three rate-limiting API endpoints (`/api/rate-limiting/current`, `/api/rate-limiting/status`, `/api/search-quality`) that always returned empty or per-process-cached data because they read from the in-memory `get_tracker()` singleton, which is freshly built per request and never sees other workers' state. They now read directly from the persisted `RateLimitEstimate` table — the same source the `/metrics` panel was switched to in #4576. Together with that change, every rate-limiting analytics surface is now driven by the same authoritative data. Raw `RateLimitAttempt` writes remain disabled (deliberately, per #4576); only the per-engine estimates the rate limiter actually learns are exposed. The `/api/search-quality` response shape changed in the process (`recent_avg_results` / `min_recent_results` / `max_recent_results` / `sample_size` are gone, replaced by `success_rate`); the benchmark page's status-warning JS was updated to consume the new shape, and the old "Very low results" warning (whose underlying per-attempt search-result-count data was never persisted) is replaced by a "High rate-limit failures" warning driven by `success_rate` and `status`. ([#4721](https://github.com/LearningCircuit/local-deep-research/pull/4721))
+- WeasyPrint (the PDF-export dependency) is now imported lazily on first use instead of at web-server startup. It was previously imported during blueprint registration via `research_routes`, adding ~20s of "cold heavy-import" to server cold start on CPU-constrained CI runners — a contributor to the `Navigation timeout 60000ms` UI-shard flakiness in #4431. PDF export behaviour is unchanged. ([#4734](https://github.com/LearningCircuit/local-deep-research/pull/4734))
+- **Star Reviews "Unknown" search-engine bucket no longer double-counts.** The search-engine breakdown attributed any research that made non-search LLM calls (whose token rows have no `search_engine_selected`) to the "Unknown" bucket *in addition to* its real engine, inflating "Unknown" toward every research. Ratings with a recorded engine now count under that engine only; "Unknown" holds only ratings with no recorded search engine. ([#4786](https://github.com/LearningCircuit/local-deep-research/pull/4786))
+- **Star Reviews merges the "Unknown" buckets.** The LLM and search-engine breakdowns no longer split unidentified entries across separate bars — an empty string, the lowercase `unknown` sentinel written for an unidentified model, and the `Unknown` shown for ratings with no recorded model/engine now collapse into a single "Unknown" bucket (real model names keep their exact casing). ([#4790](https://github.com/LearningCircuit/local-deep-research/pull/4790))
+- **LangGraph agent no longer crashes with `BadRequestError` on reasoning-mode LLMs.** `ProcessingLLMWrapper` (the central `<think>` stripper applied to every LLM returned by `get_llm`) previously let `bind_tools` bypass the stripper via its `__getattr__` shim, so `langgraph_agent_strategy.create_agent()` ended up calling the raw reasoning model. Qwen 3.x and deepseek-r1 then emitted `<think>…</think>` text directly inside tool-call `content`, which the OpenAI-compatible provider rejected with `400 Failed to parse input at pos 0: <think>…`. `bind_tools` is now overridden to re-wrap the tool-bound model so the stripper survives the agent tool-binding step (#4804). ([#4804](https://github.com/LearningCircuit/local-deep-research/pull/4804))
+- **TinyFish search engine no longer logs your search query on request errors.** When a TinyFish Search/Fetch call failed, the request exception (which embeds the query in the URL) was written to the error log; failures now record only a static message plus the HTTP status. Also restores `ruff format` on the TinyFish test files so the pre-commit gate passes. ([#4806](https://github.com/LearningCircuit/local-deep-research/pull/4806))
+- **Star Reviews announces the real load error to screen readers.** A failed refresh now speaks the actual error message via the dashboard's live region instead of a generic "failed to load", and a no-op clear that never re-announced was removed. ([#4828](https://github.com/LearningCircuit/local-deep-research/pull/4828))
+- Server startup no longer eagerly imports the heavy `langchain_text_splitters`/sentence-transformers stack; it is loaded lazily (and thread-safely) on the RAG indexing path instead. This speeds up boot and fixes intermittent UI-test server-start failures, and avoids a rare concurrent-import error that could silently skip document indexing right after a restart. ([#4829](https://github.com/LearningCircuit/local-deep-research/pull/4829))
+- Journal-quality data downloads now follow OpenAlex's 2026-06 "standard-format" snapshot layout (`data/jsonl/<entity>/manifest.json`). The previous paths started returning 404, breaking the OpenAlex sources and institutions downloads. ([#4830](https://github.com/LearningCircuit/local-deep-research/pull/4830))
+- The manual **Run now** button and the manual overdue-subscription sweep now actually start research. Both previously made a server-internal loopback HTTP POST to `/research/api/start`, which sits on a CSRF-protected blueprint and so always failed with HTTP 400 ("CSRF token is missing") — the "Run now" button reported `Failed to start research: 400`. They now invoke the research handler in-process, bypassing the HTTP/CSRF layer while still resolving the user's encrypted-database credentials. (Scheduled subscription runs were unaffected.) ([#4834](https://github.com/LearningCircuit/local-deep-research/pull/4834))
+- A failed `commit()`/`flush()` inside a `get_user_db_session()` block no longer poisons the thread's reused database session. The context manager now rolls the session back (and re-raises) when the block errors, so a single failed transaction can't cascade `PendingRollbackError` into every later operation on that thread. Paths that catch and swallow such a failure while reusing the session are also recovered explicitly: a PDF download or text extraction that fails mid-write, and a multi-file library upload where one bad file would otherwise fail the whole batch. ([#4841](https://github.com/LearningCircuit/local-deep-research/pull/4841))
+- A rare connection-level failure while auto-indexing a freshly downloaded document no longer leaves the request's database session in a state that fails the next operation. The best-effort auto-index step now recovers the shared session if its lookup fails, continuing the session-rollback hardening from the previous release. ([#4862](https://github.com/LearningCircuit/local-deep-research/pull/4862))
+- Under `embeddings.require_local=True` (or any `PRIVATE_ONLY` egress scope, which forces it), searching a library collection whose embedding model was indexed by bare name — e.g. the shipped default `all-MiniLM-L6-v2` — raised `policy_denied: embeddings_model_not_cached` even when the model was present in the local HuggingFace cache. The pre-flight cache check used the bare name as the `repo_id`, but the SentenceTransformer loader requests the namespaced form (`sentence-transformers/all-MiniLM-L6-v2`) and the HF hub cache is keyed on whichever form the loader requests. The check now probes both forms (matching the loader's resolution for both SBERT-curated names and the `basic_transformer_models` allowlist of vanilla transformers like `bert-base-uncased`), so a correctly cached model is admitted. ([#4887](https://github.com/LearningCircuit/local-deep-research/pull/4887))
+- Fixed a circular import that made `import local_deep_research.domain_classifier.models` (and importing the `domain_classifier` package before `database.models`) crash with `ImportError: cannot import name 'DomainClassification' from partially initialized module`. The declarative SQLAlchemy `Base` now lives in a dependency-free `database.base` leaf that model modules import without triggering the full `database.models` package, so import order no longer matters. `Base` identity and `Base.metadata` are unchanged. ([#4910](https://github.com/LearningCircuit/local-deep-research/pull/4910))
+- Under `embeddings.require_local=True` (or any `PRIVATE_ONLY` egress scope), a collection search whose embedding model is configured as an existing **local directory path** was wrongly refused with `policy_denied: embeddings_model_not_cached`. The `require_local` pre-flight probed the path as a HuggingFace `repo_id`, which raised a validation error that was swallowed and treated as a cache miss — even though the SentenceTransformer loader resolves such a path locally and never contacts the hub. The pre-flight now mirrors the loader's `os.path.exists` guard and admits an existing local path (blank/degenerate names still fail closed), so a local model directory is no longer false-denied. ([#4924](https://github.com/LearningCircuit/local-deep-research/pull/4924))
+- **Benchmark YAML exports now quote and escape free-form string fields** (`model`, `model_provider`, `search_engine`, `dataset`, and the evaluator's `model`/`provider`/`endpoint_url`). They were previously interpolated raw, so a value containing a YAML-special character — a colon-space, `#`, quote, or newline — could produce malformed YAML (or inject an unintended key). These fields now go through the existing `yamlEscape()` helper, matching how the settings-snapshot block already serializes strings. Numeric fields and the controlled-format `date_tested`/`ldr_version` are left as-is.
+- Benchmark results now persist reliably instead of intermittently showing "Found 0 results" with no progress. The request-thread and worker-thread result syncs could both insert the same row (or a dataset repeating a question could produce a duplicate `query_hash`), tripping the `benchmark_results` unique constraint; because that failed at commit time it rolled back the whole pending batch, discarding results that had actually completed. Result persistence is now serialized and deduplicated, so concurrent syncs and duplicate questions are handled cleanly.
+- Cap persisted log messages at 5000 chars so long langgraph runs no longer accumulate 10 KB ``ResearchLog`` rows. Mirrors PR #4004's frontend truncation discipline, now applied to the database sink. Also promotes the duplicated ``/history/logs`` pagination-cap literals into a single shared constant (``HISTORY_LOGS_HARD_CAP`` / ``HISTORY_LOGS_DEFAULT_LIMIT``, exposed to the log panel as ``window.LDR_LOG_LIMITS``) so the route clamp and the panel's DOM cap no longer drift. Full diagnostics remain available in stderr/file sinks.
+- Collection indexing now behaves identically whether started via the streaming
+  (SSE) route or the background worker. The two paths had duplicated and drifted:
+  the background worker did not persist a collection's ``embedding_dimension`` (so
+  collections indexed in the background lost it), and the SSE route skipped the
+  force-reindex cleanup that clears old chunks and FAISS indices (so a streamed
+  force-reindex could leave stale, mixed-model vectors behind). The shared
+  embedding-metadata, force-reindex cleanup, and document-query logic is now
+  factored into single helpers used by both paths.
+- Converting research history into the searchable library now pages through reports in bounded batches instead of loading every report body into memory at once, preventing `MemoryError` on large histories.
+- Detailed-mode research reports now honor the ``report.enable_file_backup`` setting. The detailed completion path saved ``report_content`` via a raw ORM write that bypassed the report-storage abstraction, so a user who enabled file backup got on-disk ``.md``/``_metadata.json`` files for quick-mode research but silently never for detailed-mode research. Route the detailed save through ``get_report_storage().save_report()`` exactly like the quick and error paths already do (DRY-review finding H2).
+- Emojis in exported PDFs no longer render as empty "tofu" boxes. The default WeasyPrint stylesheet now lists ``"Noto Color Emoji"`` (with ``"Noto Emoji"`` as a monochrome fallback and ``"Segoe UI Emoji"`` / ``"Apple Color Emoji"`` for native installs) at the tail of both the body and ``code``/``pre`` font-family stacks, and the official Docker image now bundles ``fonts-noto-color-emoji`` alongside ``fonts-noto-cjk``. Non-Docker users on Linux without an emoji font installed still need to install one (e.g. ``apt install fonts-noto-color-emoji``) — see ``docs/faq.md``. This change also fixes a pre-existing flaw in the ``custom_css`` path: ``markdown_to_pdf(custom_css=...)`` previously *replaced* the default stylesheet entirely, silently dropping the CJK fallback from #4055; it now layers custom CSS on top of the default so the font fallbacks survive.
+- Fixed 5 `TestQuickSummary` API tests that broke after the egress fail-closed change (#4300): the `quick_summary` endpoint re-imported `get_user_db_session` locally, shadowing the module-level binding so `patch("...web.api.get_user_db_session")` never intercepted — the tests passed only because the old code swallowed the resulting DB error. Removed the redundant local import (one consistent patch target) and updated the settings-failure test to assert the new fail-closed `503` behaviour instead of the obsolete empty-snapshot fallback.
+- Fixed ScaleSerp silently returning zero results when the API includes a result with a `null` link: the eagerly evaluated `link[:50]` display fallback raised `TypeError`, which the outer handler swallowed along with the entire result set.
+- Follow-up relevance filtering now rejects negative and out-of-range source indices and deduplicates repeated ones, so a malformed LLM response can no longer select the wrong source or list the same source twice.
+- Harden the central `ProcessingLLMWrapper` think-tag stripping against non-string responses. `remove_think_tags` is text-only, so it now runs only when `response.content` is a `str`; non-string content (e.g. provider content-block lists such as Anthropic's, or `None`) is passed through unchanged instead of raising `TypeError`. This removes the wrapper-level crash that the per-site `str(...)` guards used to absorb before they were dropped. Note: a few direct `.content` consumers still assume string content, so robust list-content handling at those call sites is tracked separately.
+- Honor ``?limit`` on ``/api/research/<id>/logs`` (the route the log panel actually fetches). It previously ignored the parameter and returned every row, so a long langgraph run could still ship thousands of rows to the browser despite the panel requesting a bounded tail. The limit now returns the newest N rows (oldest-first, unchanged ordering) clamped to ``HISTORY_LOGS_HARD_CAP``; omitting ``?limit`` preserves the existing return-all contract for direct API callers.
+- Library document indexing, collection document listing, and filesystem sync no longer load every document's full text body into memory at once — the large `text_content` column is now deferred (and a document's "has text" flag is computed in SQL), preventing `MemoryError` on large libraries.
+- Library/collection uploads now actually accept ODT, DOCX, PPTX and XLSX/XLS files. These formats were advertised as supported but failed at runtime with a swallowed ``ModuleNotFoundError`` because their ``unstructured`` parser dependencies (``python-docx``, ``python-pptx``, ``openpyxl``, ``msoffcrypto-tool``, ``xlrd``) were never installed — only PDF worked. Those parsers are now declared dependencies, and the loader registry probes each format's real runtime dependency so it only advertises formats it can actually extract: missing deps now yield a clear "Unsupported format" instead of a silent failure. Legacy binary ``.doc``/``.ppt`` (which need a LibreOffice ``soffice`` binary) and image OCR (which needs ``pytesseract`` + ``tesseract``) are offered only when those tools are present.
+- Link analytics (`GET /api/link-analytics`) now projects only the columns it needs plus a SQL-level "has preview" flag, instead of loading full research-resource rows. This avoids materializing every resource's `content_preview` text on the whole-table scan that runs when no time filter is applied (#4560).
+- Make research-log ordering deterministic when rows share a timestamp. ``ResearchLog.timestamp`` is not unique, so ``order_by(timestamp.desc()).limit(N)`` left the rows surviving a shared-timestamp boundary SQL-undefined — under dense logging the newest-N tail (and the "latest milestone" lookup) could vary between requests. Add ``ResearchLog.id`` as a tie-break across all three log-ordering sites (the ``/api/research/<id>/logs`` route, the ``get_logs_for_research`` helper behind ``/history/logs``, and the latest-milestone query). Follow-up to #4645.
+- News subscriptions now treat the ``status`` column as the single source of truth for whether a subscription is active. Previously the scheduler decided what to run by filtering on the separate ``is_active`` boolean, which ``create_subscription`` never sets (it writes only ``status``), so a subscription created as *paused* kept ``is_active=True`` by default and was still run by the scheduler. The active/overdue query was also spelled three different ways across the scheduler and the news API, and one copy omitted the ``next_refresh is not None`` guard. All of these now go through ``NewsSubscription.active_filter`` / ``due_filter``. Manually running a subscription ("run now") also reads its saved model/provider/strategy/search-engine from the database (it previously read a trimmed dict that dropped those fields, so the manual run ignored the subscription's configured model) and advances the refresh schedule on success so an overdue subscription is not immediately re-run by the scheduler. The shared run payload and refresh-schedule arithmetic are extracted into ``news/subscription_runner.py``. The post-completion refresh-time update in the research service (which previously called a subscription-storage helper that referenced non-existent columns and so always failed silently) now works, and the subscriptions list API now returns each subscription's saved model/provider/strategy/search-engine instead of dropping them.
+- Parse ``LDR_TEST_MODE`` as a proper boolean when deciding whether to relax the SQLCipher KDF floor. Previously ``_get_min_kdf_iterations()`` used a bare truthiness check, so any non-empty value — including ``LDR_TEST_MODE=0`` or ``=false`` — enabled the relaxed test minimum, silently weakening database encryption for anyone setting the flag explicitly to disable it. ``PYTEST_CURRENT_TEST`` stays presence-based (pytest sets it to a non-boolean string).
+- REST `/api/v1/analyze_documents` now rejects unknown body parameters with a clear 400 listing the allowed ones (previously a typo like `max_result` surfaced as an opaque 500), and the `/api/v1/` docs endpoint documents the `allow_default_settings` flag for all three research endpoints.
+- Remove redundant per-site `<think>`-tag stripping now that `get_llm` LLMs are normalized centrally in `ProcessingLLMWrapper`. Direct `self.model.invoke(...).content` is already think-stripped, so 11 scattered stripping calls (6 `remove_think_tags(...)` and 5 `get_llm_response_text(...)`) across the constraint/evidence/knowledge components, the synthesis path, and the document-analysis API were dropped (relying on the single central strip point), keeping explicit handling only where it is load-bearing (agent/`bind_tools` paths, injected LLMs). Empty-answer fallbacks now log a warning so a silent degrade is visible.
+- The "download all" and "queue all undownloaded" library actions now project only the columns they use instead of loading full `ResearchResource` rows, avoiding materializing every resource's `content_preview` text on a whole-table scan for large libraries (#4560).
+- The `/api/history` endpoint now paginates its results (default 200, max 500) instead of loading every research row — and its `research_meta` JSON — into memory at once.
+- The library documents API now clamps the `limit`/`offset` query parameters, so a negative `limit` (which SQLite treats as "no limit") can no longer bypass pagination and load an entire collection into memory.
+- The library domain-filter scan (`get_unique_domains`) now streams document URLs in batches instead of loading every row at once, avoiding excessive memory use on very large libraries (#4560).
+- The rate-limiting reset (`POST /api/rate-limiting/engines/<engine>/reset`) and cleanup (`POST /api/rate-limiting/cleanup`) endpoints now operate on the persisted `RateLimitEstimate` table instead of the per-request `get_tracker()` singleton. Like the read endpoints fixed in #4721, the tracker's mutation path is gated on a research-session context that is absent in an analytics HTTP request, so these endpoints were silent no-ops that never cleared the estimates the `/status` and `/current` panels display. Reset now deletes the engine's stored estimate (so it re-learns), and cleanup deletes estimates not updated within the requested window.
+- The research-session filter dropdown on the library page no longer loads every research row unbounded; it is now capped to the most-recent sessions, avoiding excessive memory/DOM use on very large histories (#4560).
+- The three egress warning banners (public-egress, cloud LLM, cloud embeddings) now have separate dismiss flags. Previously they shared one, so dismissing the fresh-install public-egress notice also permanently hid the critical cloud-LLM and cloud-embeddings warnings.
+- When testing a notification webhook URL through the settings UI, the
+  "Test" button now surfaces the validator's reason instead of a generic
+  "Invalid notification service URL." message. For private/internal IP
+  rejections that the operator can unblock (loopback, RFC1918, CGNAT,
+  link-local, IPv6 private), the error message names
+  `LDR_NOTIFICATIONS_ALLOW_PRIVATE_IPS`; for NAT64-wrapped non-metadata
+  destinations on IPv6-only deployments (RFC 6052 well-known
+  `64:ff9b::/96` or RFC 8215 local-use `64:ff9b:1::/48`) it names
+  `LDR_SECURITY_ALLOW_NAT64` — the only flag that can unblock those, which
+  the hint probes for independently. Cloud-metadata IPs are always blocked and
+  the env-var hint is intentionally suppressed for them — neither flag
+  re-opens metadata, so naming them would mislead the user.
+
+  Also adds an "IPv6-only deployments (NAT64)" subsection to
+  `docs/SearXNG-Setup.md` so operators routing IPv4 through NAT64 know
+  about the opt-in.
+- `get_llm_response_text` now extracts the text from list-type LLM content blocks (Anthropic extended-thinking / tool-use responses, where `message.content` is a list of blocks) instead of stringifying the list to its Python `repr`. This fixes garbled entity, sub-query, and candidate parsing, and stops direct `.content` consumers from crashing on list content.
+
+### 🗑️ Removed
+
+- Dropped the orphaned `cache` and `search_cache` tables and removed their unused
+  `Cache`/`SearchCache` models (migration 0016). Neither table was ever populated
+  by any code path, so existing databases lose no data — the empty tables are
+  removed automatically on the next migration. ([#drop-orphaned-cache-tables](https://github.com/LearningCircuit/local-deep-research/pull/drop-orphaned-cache-tables))
+- Removed the `/api/v1/quick_summary_test` REST endpoint. It was a near-duplicate of `/quick_summary` with hardcoded test defaults; call `/quick_summary` with `search_tool`, `iterations`, and `temperature` set explicitly instead. ([#3661](https://github.com/LearningCircuit/local-deep-research/pull/3661))
+- Removed the benchmark result-reuse feature: new benchmark runs no longer silently import results from previous completed runs with matching search settings. The compatibility check covered only a handful of settings (ignoring evaluation config, dataset versions, and the rest of the configuration), so results produced under different conditions could contaminate a run's accuracy, and the reuse accounting could report more than 100% completion. Every run now researches its own sampled questions and reports only its own results. ([#4498](https://github.com/LearningCircuit/local-deep-research/pull/4498))
+- Removed the DOAJ Seal quality tier (score 8) and the Tier 4 "+1 Seal bonus": DOAJ retired the Seal in April 2025 and removed it from its metadata, so the tier could never be earned again and only ever fired on stale pre-2025 data. DOAJ-listed journals keep their score-5 floor. The `has_doaj_seal` column was dropped from the reference DB (schema version 4 — the DB rebuilds automatically on first use) and the Seal star/count no longer appear on the journal-quality dashboard.
+- Removed the experimental search strategies that were only reachable through the "Show All Strategies" toggle (rapid, parallel, iterative, recursive, adaptive, smart, standard, iterdrag, browsecomp, evidence, the constrained/dual-confidence family, modular, and others), along with the toggle itself. The strategy dropdown now offers source-based, focused-iteration, focused-iteration-standard, topic-organization, mcp, and langgraph-agent. The news feature's internal news-aggregation strategy is unaffected.
+- Removed the now-unused `PathValidator.confine_to_base` helper (and its tests). It was added only to confine the local-folder RAG indexing route, which has been removed, leaving it with no callers.
+- Removed the unused `GET /research/api/config` endpoint. Every field it returned (`version`, `llm_provider`, `search_tool`, `features.notifications`) was read from `current_app.config` keys that are never set, so it only ever returned hardcoded placeholder values. It had no frontend or production callers. Use `GET /research/api/settings/current-config` for live configuration instead.
+- Removed the unused `PricingCache._load_cache` and `PricingCache._save_cache` methods. They were left behind as deprecated no-op stubs (empty `pass` bodies) when the disk-backed pricing cache was replaced with an in-memory bounded `TTLCache`, and they had no callers anywhere in the codebase.
+- Removed the unused ``news.subscription_manager`` storage subsystem
+  (``SQLSubscriptionStorage``, ``SearchSubscription``, ``TopicSubscription``,
+  ``BaseSubscription`` and their factories), the abstract ``SubscriptionStorage``
+  interface, and ``StorageManager.get_user_subscriptions`` /
+  ``get_user_stats``. This code was never reached by any live path -- all real
+  subscription functionality goes through ``news.api`` and the scheduler -- and
+  was broken against the current ``NewsSubscription`` model (it referenced
+  columns such as ``user_id``/``refresh_count``/``results_count`` that do not
+  exist, so it raised ``AttributeError`` whenever called). The package-level
+  exports ``SearchSubscription`` and ``TopicSubscription`` from
+  ``local_deep_research.news`` are removed as part of this; nothing in the
+  codebase imported them.
+- Removed two orphaned RAG HTTP endpoints that had no UI and no other caller: `GET /library/api/rag/index-local` (local-folder indexing — superseded by collection-based indexing) and `POST /library/api/rag/index-research` (which called a method that no longer exists and always errored). The unused `index_local_file` service method and glob-pattern allowlist were removed with them.
+
+### 📝 Other Changes
+
+- The available-models endpoint no longer hand-rolls Ollama/OpenAI/Anthropic model listing that was immediately overwritten by provider auto-discovery — auto-discovery is now the single fetch path. This removes a redundant per-refresh network round-trip to each provider, and local-only model listing now goes through the provider classes (which validate the URL for SSRF and send auth headers, so an auth-protected local Ollama now lists its models). ([#4646](https://github.com/LearningCircuit/local-deep-research/pull/4646))
+- The two Ollama health-check endpoints (`/check/ollama_status`, `/check/ollama_model`) now share a single `/api/tags` probe helper instead of separately re-implementing the fetch + new/old API-format parsing + error classification, so "is Ollama up?" answers consistently across them. No change to the endpoints' responses. ([#4650](https://github.com/LearningCircuit/local-deep-research/pull/4650))
+- Add a comprehensive Puppeteer UI suite for the egress-policy feature: `tests/ui_tests/test_egress_policy_ui.js` (CI-safe, 32 assertions) covers every UX touchpoint — Privacy & Egress panel rendering, all four scopes painting their expected border/text colors (amber/teal/indigo palette verified via computed-style reads, not just DOM state), the STRICT-on-meta-picker guard reverting to "Both" with an explanatory hint, per-research overrides round-tripping through the settings DB, scope cue propagating to /history, /metrics and /chat via base.html's `body[data-scope]`, settings dashboard listing the policy keys, and the require-local toggles persisting across reloads. A second file, `tests/ui_tests/NO_CI_test_egress_policy_live_research.js`, drives end-to-end research runs against a real Ollama + SearXNG instance (defaults to a lab box; overridable via env) and verifies the run-start PEP at `research_routes.py:248` — STRICT+searxng succeeds, STRICT+auto returns 400 with the meta-picker incoherence message, PUBLIC_ONLY refuses a SearXNG whose instance URL resolves to a private IP (`scope_mismatch_public_only`), PRIVATE_ONLY accepts the same SearXNG (URL classifier overrides the static `is_public=True`), and a cloud LLM under `require_local_endpoint=true` is accepted at the precheck but reaches `failed` status as the strategy's `get_llm()` hits the PEP. Both files gate screenshots behind `!process.env.CI` matching the `tests/ui_tests/test_settings_page.js:57` convention; the live test's `NO_CI_` filename prefix means the runner skips it in CI (where the lab endpoints are unreachable). Screenshots land in `tests/ui_tests/screenshots/egress-policy{,-live}/` (already gitignored).
+- CI: set ``LDR_TEST_MODE=1`` on every test/scan workflow that starts a server or initialises the test database, so the intended fast SQLCipher KDF (``LDR_DB_CONFIG_KDF_ITERATIONS=1000``) is actually honoured. ``_get_min_kdf_iterations()`` deliberately ignores generic ``CI``/``TEST_ENV`` and only relaxes for ``PYTEST_CURRENT_TEST``/``LDR_TEST_MODE``; without the latter the requested 1000 was silently clamped to the production minimum (256000), making every DB open and the post-login background backup ~256× slower. Under contended CI runners that turned the backup into a multi-minute, GIL-holding stall that timed out release-gate UI shards (chat/settings navigations) — the failures seen on clean ``main`` while investigating #4430. Also renames the deprecated ``LDR_DB_KDF_ITERATIONS`` env var to its canonical ``LDR_DB_CONFIG_KDF_ITERATIONS`` across all six workflows (docker-tests, nuclei, owasp-zap-scan, playwright-webkit-tests, puppeteer-e2e-tests, docker-multiarch-test).
+- Consolidate the egress-policy code into a dedicated `security/egress/` subpackage (`policy.py` + `audit_hook.py`) with a README documenting the design, scope model, and the full map of enforcement points. No behavior change — module paths moved from `security.egress_policy` / `security.egress_audit_hook` to `security.egress.policy` / `security.egress.audit_hook`.
+- Extracted the benchmark YAML export helpers (`yamlEscape` / `formatSettingValue` / `formatSettingsSnapshot`) from the inline `benchmark_results.html` template into a unit-tested module (`static/js/utils/yaml_export.js`, loaded via `<script src>` like the other shared JS utils). No behavior change — it adds regression coverage (`tests/js/utils/yaml_export.test.js`) for the escaping that guards every benchmark YAML download, so a future change to `yamlEscape` can't silently corrupt exports.
+- Hardened the #4804 `<think>`-stripper fix: corrected the now-stale `bind_tools` comments in the LangGraph agent, made the async wrapper test actually assert stripping, and added a real `create_agent` integration regression test (sync + async).
+- Moved `ProcessingLLMWrapper` out of the `wrap_llm_without_think_tags` closure to module scope in `config/llm_config.py` — it's now importable and `isinstance`-checkable, and defined once instead of rebuilt on every call. Pure refactor; no behavior change.
+- Recolor the egress-scope visual cues to match the actual data-leak risk hierarchy. Previously BOTH (the default) wore a light-amber accent and PUBLIC_ONLY a deeper-amber one, which reads as "PUBLIC_ONLY is more dangerous than the default" — but BOTH is actually the riskiest scope: it permits local engines AND a cloud LLM in the same run, so local library/RAG chunks can land in OpenAI via the LLM call. PUBLIC_ONLY blocks local engines, so local data never enters that pipeline. New palette: BOTH gets no left-border accent at all (the existing "Public search egress enabled" banner does the nagging in text); PUBLIC_ONLY uses sky-blue (#0ea5e9 — informational "you chose public sources only"); PRIVATE_ONLY stays teal (#14a37f); STRICT moves from indigo to violet (#8b5cf6) so it's clearly distinct from the new PUBLIC_ONLY blue for color-blind users. The privacy panel itself defaults to a neutral slate when BOTH is active rather than amber. The risk-honest rationale is documented in the `base.html` style-block comment.
+- Removed dead and inert code from Chat Mode's context manager that shipped without a consumer: the unused ``build_prompt_context()`` and ``_get_recent_messages()`` helpers, the ``conversation_history`` and ``accumulated_sources`` keys in the research-context dict (nothing downstream read them), and the cross-turn source-count tracking (``_extract_sources_from_history``, the ``source_count`` field, and ``update_accumulated_context``'s ``source_count_delta`` parameter) — which never functioned because the producer was never called with sources. The ``chat.max_context_messages`` setting is removed with it, since it only fed the now-deleted recent-message inclusion path. ``chat.max_findings_to_include`` (which still controls how many prior findings carry into a follow-up) is unchanged.
+- Removed the redundant local re-import of `get_settings_manager` inside the `quick_summary` API endpoint (it was already bound at module level), eliminating the same import-shadowing anti-pattern fixed for `get_user_db_session` — so `patch("...web.api.get_settings_manager")` now reliably intercepts. Both helpers are now module-level for one consistent patch surface; only `quick_summary` itself remains a local import (it pulls in the research stack, which would cycle).
+- Tests: fix a GC-timing flake in ``test_base_downloader.py::test_context_manager_calls_close``. The test patched ``close`` on the ``BaseDownloader`` *class*, but ``BaseDownloader.__del__`` also calls ``self.close()`` — so whenever the garbage collector happened to finalize a downloader left over from an earlier test while the patch was live, the shared class-level mock counted an extra call and the assertion failed with "Expected 'close' to have been called once. Called 2 times." (seen failing an unrelated PR's CI, #4415). Patch on the instance instead — the same pattern the neighbouring ``__del__`` test already uses — which is immune to other instances' finalizers (mechanism reproduced deterministically both ways).
+- Tests: give the logpanel ``prunes the oldest entries when count exceeds MAX_LOG_ENTRIES`` vitest case an explicit 20s timeout. #4304 fixed the real root cause (501 ``setTimeout(autoscroll, 0)`` tasks piling on the real timer queue) by faking all timers, but the test's remaining cost is honest O(n²) DOM work — 501 inserts each running ``addLogEntryToPanel``'s full-container ``querySelectorAll`` scans, ~2.6–2.9s in happy-dom on a dev machine — leaving no headroom against the 5s vitest default under parallel CI load. The recurring timeout flake intermittently failed the shared "All Pytest Tests + Coverage" job on unrelated PRs (e.g. #4415). The test must fill to the real 500-entry cap to exercise the prune, so the work can't shrink; a bigger budget is now the right lever (unlike #4299, which proposed it while the timer bug was still live).
+- Tests: lock in the ``LDR_TEST_MODE`` boolean-parsing behaviour for the SQLCipher KDF floor. Adds direct ``_get_min_kdf_iterations()`` unit tests for falsey values (``0``/``false``/``no``/``off`` must NOT relax the floor — the #4564 regression) and the full truthy set (``on``/``enabled`` must relax it, guarding against a narrower parser). Also moves the integration test off the registry's ``min_value`` boundary so it exercises the KDF-floor clamp rather than range validation.
+- Tests: stop the ``QueueProcessorV2`` background thread between tests in the ``reset_singletons`` autouse fixture (it already did this for ``BackgroundJobScheduler``). The module-level ``queue_processor`` singleton was started by the first ``create_app()`` in an xdist worker and never stopped, so one test's processor thread ran for the whole worker — looping through ``SettingsManager`` → SQLCipher connection opens on the shared ``db_manager`` concurrently with every later test, and emitting logs to a closed pytest stderr sink at teardown (the same class of bug the scheduler handling fixes). Tests that exercise the queue patch the singleton, so stopping the real thread does not affect them.
+- The pre-commit CI job now retries only the hook-environment download (the network-flaky step), not the lint run itself. Previously the entire `pre-commit run` was wrapped in a 2-attempt retry, so an auto-fixing hook (e.g. ``ruff-format``) that rewrote a file would fail the first attempt, leave the now-fixed tree in place, and pass the retry — silently turning a formatting violation into a green check while the unformatted code stayed on the branch. Hook downloads are retried via ``pre-commit install-hooks``; the check now runs exactly once and fails honestly.
+- The release gate now imports the SIMD-heavy native dependencies (numpy, pandas, pyarrow, scikit-learn, faiss, torch) under qemu-emulated SandyBridge (AVX without AVX2) and Haswell (AVX2) CPUs. This catches dependency wheels that silently raise the x86-64 instruction-set baseline and would crash with SIGILL on older CPUs — the failure mode that shipped in v1.7.0 when faiss-cpu 1.14.2 executed an AVX2 instruction at import time on AVX-only hosts (#4480). Building the gate surfaced that AVX is already LDR's de-facto minimum CPU requirement: the pandas and scikit-learn wheels crash on pre-AVX CPUs (Nehalem/Westmere era, pre-2011), so those are not part of the gate. The check runs only in the release gate, not on every PR, since full CPU emulation is slow and the failure mode can only ship via dependency bumps.

docs/release_notes/1.9.0.md

@@ -0,0 +1,43 @@
+
+### 🔒 Security
+
+- Raw exception variables (`e`, `exc`, etc.) are no longer interpolated directly into `logger.exception()` messages across all LLM provider, embedding provider, and web search engine paths; each catch site now scrubs the message through `sanitize_error_message()`/`redact_secrets()` before logging, preventing error text from leaking API keys, credentials, or internal URL structure into production logs. Scrubbed exception log lines now include the exception class name so exceptions that stringify to an empty message still produce useful diagnostics. A new `check-sensitive-logging` pre-commit hook enforces this rule going forward, rejecting direct exception-variable interpolation in the secure-logging directories and flagging `exc_info=True` on `warning`/`error`/`critical` calls. ([#4888](https://github.com/LearningCircuit/local-deep-research/pull/4888))
+- `NewsAPIException.to_dict()` now redacts credential shapes from the fields it serializes to the client through the two `@errorhandler(NewsAPIException)` handlers: the human-readable `message` goes through the central `sanitize_error_for_client()`, and string leaves of `details` go through `sanitize_error_message()`. This is a defence-in-depth backstop at the response boundary — the primary defence stays at the `news/api.py` raise sites (curated generic messages, #4843) — so a future raise site, subclass, or external caller that lets a credential-bearing string into `message` or `details` (e.g. the caller-supplied search `query` forwarded in `details`) has it redacted before it ships. Redaction is credential-shape based (Bearer/`Authorization`, `?api_key=`-style params, known token prefixes, `http(s)` URL userinfo), so legitimate text is unchanged; `error_code` / `status_code` are left untouched. It does not attempt to catch DSN userinfo (`postgresql://user:pass@`), SQL, or filesystem paths — those remain the responsibility of the raise-site discipline. ([#4931](https://github.com/LearningCircuit/local-deep-research/pull/4931))
+- The central credential sanitizer (`sanitize_error_message` / `sanitize_error_for_client`) now redacts URL userinfo credentials in *any* scheme (case-insensitive), not just `http(s)`. Raw **URL-form** database connection errors — SQLAlchemy `dialect+driver` DSNs (`postgresql+psycopg2://user:pass@host`, `mysql+pymysql://…`, `mongodb+srv://…`), plain DB schemes, and password-only DSNs (`redis://:pass@host`) — previously leaked their password through any surfaced exception message; the password is now replaced with `[REDACTED]`. Credential-less DSNs with a port (`postgresql://host:5432/db`) and all existing `http(s)` behavior are unchanged. Key=value DSNs (e.g. pyodbc `Server=...;Pwd=...`) have no `://` and remain out of scope for a userinfo regex. This benefits every consumer of the central sanitizer (download service, fetch tool, agent strategies, search engines, and the news error-response backstop). ([#4933](https://github.com/LearningCircuit/local-deep-research/pull/4933))
+- `WebAPIException.to_dict()` now redacts credential shapes from the fields it serializes to the client (`message` via `sanitize_error_for_client()`, `details` string leaves via `sanitize_error_message()`), mirroring the `NewsAPIException` backstop. This is a defence-in-depth measure at the response boundary so that a future raise site, subclass, or external caller that lets a credential-bearing string into `message` or `details` has it redacted before it ships. Legitimate text is unchanged, and `status` / `error_code` / `status_code` are left untouched. ([#4937](https://github.com/LearningCircuit/local-deep-research/pull/4937))
+- The egress guardrail now enforces a two-axis (data-sensitivity × destination-exposure) admissibility rule at the start of a research run (ADR-0007): the run is refused when a sensitive source (a private collection/library/store) would reach an exposing sink (a public search engine or a cloud LLM/embeddings provider), in addition to the existing scope checks. Override by setting Egress Scope to **Unprotected**, marking the collection public, or trusting the destination.
+
+### ✨ New Features
+
+- **New Zotero integration: import a Zotero library or collection into your document library, with optional background auto-sync.** Add your Zotero API key, library type/ID and (optionally) a collection key under **Settings → Zotero**, then use the new **Library → Zotero** page to test the connection, list collections, and **Sync now**. Imported papers are stored, text-extracted and RAG-indexed like any other library document, so they become searchable during research via the collection search engine — no separate search engine to configure. Sync is incremental (new, changed and removed items are reconciled), and items without an attached PDF are imported as metadata/abstract text by default (toggle off to import PDFs only). Enable **Auto-Sync** to refresh in the background on a configurable interval. ([#4723](https://github.com/LearningCircuit/local-deep-research/pull/4723))
+- Zotero integration polish from first real-world testing: standalone PDF
+  attachments (PDFs added without a parent item) now import as documents;
+  the library ID is auto-detected from the API key (usernames resolve
+  too); manual syncs re-examine previously skipped items; imports appear
+  in the main Library view; a live progress bar tracks syncs; the config
+  page autosaves with essentials-first layout and runs the connection
+  test on saving a key; friendlier defaults (text-only PDF storage,
+  integration enabled once a key is set) and actionable error messages. ([#4960](https://github.com/LearningCircuit/local-deep-research/pull/4960))
+- Add a per-category entry counter next to each filter button in the Research Logs panel (All, Milestones, Info, Warning, Errors). Counts reflect entries currently rendered in the DOM and update on insert, prune, and batch load, so users can see at a glance which categories still have entries when the global DOM cap is hit.
+- Add an **Unprotected** egress scope — an explicit opt-out that disables the egress-scope restrictions for a run (the hard SSRF and cloud-metadata blocks still apply), surfaced with a light-red panel and a non-dismissible "protection disabled" banner. The rarely-used **Both** scope is retired: it is removed from the selector, and existing saved `both` values are migrated to **Adaptive** (a residual value is also coerced to Adaptive at read time). If you relied on `both` to run a private collection with a cloud model, mark that collection **public** or choose **Unprotected**.
+- Add per-destination trust to the egress model (ADR-0007): `policy.trusted_inference_providers` and `policy.trusted_search_engines` let you mark specific off-machine LLM/embeddings providers or search engines as trusted, so the two-axis classification treats them as *contained* (e.g. a zero-retention enterprise Anthropic endpoint, or a self-hosted Elasticsearch/Paperless on a public hostname). A banner surfaces active trust entries.
+
+### 🐛 Bug Fixes
+
+- Registration no longer leaves a bricked account behind: if creating the encrypted user database fails after the auth row is committed, both the orphaned auth row and any partial on-disk database files (including the per-database salt) are now cleaned up, so the username is genuinely reusable on retry. ([#4934](https://github.com/LearningCircuit/local-deep-research/pull/4934))
+- The settings write API now rejects malformed keys (a trailing dot like `local_search_chunk_size.`, a leading dot, an empty `..` segment, or blank/whitespace) with HTTP 400 instead of silently persisting them. Such rows made `get_setting()` return a `{"": value}` wrapper dict that rendered as `[object Object]` on settings pages (#4840). The manager (`set_setting`, `create_or_update_setting`) also refuses to create malformed keys, and `import_settings` skips them so a corrupted export can't reintroduce them. Complements the read-side fix in #4852. ([#4935](https://github.com/LearningCircuit/local-deep-research/pull/4935))
+- Registration now recovers from an orphaned per-database salt file (left with no matching database by a prior interrupted registration), so a username that was previously stuck as un-registerable can be registered again. ([#4942](https://github.com/LearningCircuit/local-deep-research/pull/4942))
+- Both settings prefix-read paths — `SettingsManager.get_setting` (DB) and `get_setting_from_snapshot` (the snapshot path used by research threads) — now filter out malformed rows (e.g. a legacy `foo..` or `foo. ` key), so a stray malformed row can no longer turn a leaf read into a `{".": value}` wrapper dict that renders as `[object Object]`. Completes the read side of #4840 beyond #4852's exact single-trailing-dot exclusion; existing malformed rows remain harmless and are not deleted. ([#4946](https://github.com/LearningCircuit/local-deep-research/pull/4946))
+- Fix the 'Info' filter button in the Research Logs panel being a silent no-op. Clicking it left the panel showing every entry (warnings, errors, milestones) because the visibility helper returned true for all log types when the active filter was 'info'. The filter now narrows to info entries only, matching the button's label.
+- Fixed a 30–50 second visible lag between clicking **Stop** and a focused-iteration research actually stopping. `FocusedIterationStrategy.analyze_topic` now calls `self.check_termination()` at the start of every iteration, after the parallel search (before optional verification searches), and just before the final synthesis LLM call, so a cancellation request is detected within the current iteration rather than only at the next iteration's progress emit.
+- Fixes a subtle data-loss bug in the Research Logs panel: identical warning, error, and milestone entries were being collapsed into a single `(N×)` counter, the same way repetitive info entries like "Status: OK" already were. Collapsing diagnostic entries like that strips the recency signal — after the second identical error you can no longer tell *when* the last failure happened, and repeated retries look like a single stuck event instead of forward progress.
+
+  The content-dedup scan now applies only to `info` entries (where collapsing repetitive noise is useful). Warning / error / milestone entries always render. The id-based dedup earlier in the pipeline still catches exact retransmits with the same id, so the bypass is strictly about *content-similar-but-distinct-events* getting the same timestamp.
+
+### 📝 Other Changes
+
+- Fixed broken copy-paste examples in the README and docs (benchmark CLI
+  module path, `rate_limiting reset --engine`, nonexistent `openclaw`
+  search engine) plus two dead external links, and added a pre-commit
+  check that keeps README links, anchors, and examples in sync with the
+  codebase. ([#4950](https://github.com/LearningCircuit/local-deep-research/pull/4950))

docs/release_notes/1.9.1.md

@@ -0,0 +1,30 @@
+
+### 🔒 Security
+
+- User-supplied strings that reach SQL `LIKE` queries — library search and domain filters, domain-classifier sampling, and news subscription filters — now escape `%` and `_` wildcards. Previously a value like `100%` or `a_b` was interpreted as a wildcard and matched unrelated rows; searches and filters now match literally, closing a wildcard-injection/enumeration vector within a user's own database. ([#3094](https://github.com/LearningCircuit/local-deep-research/pull/3094))
+- All LLM provider, embedding provider, and web search engine modules now log through the diagnose-gated `security.secure_logging` wrapper, so exception tracebacks in these paths only appear in logs when diagnose mode is explicitly enabled (`LDR_APP_DEBUG` + `LDR_LOGURU_DIAGNOSE`) — production logs no longer receive full tracebacks that could echo API keys or request internals from provider/engine failures. The `check-sensitive-logging` pre-commit hook now enforces this by construction: raw `loguru` imports and every known bypass route (`logger.opt()`, `@logger.catch`, private logger handles, `traceback`/`sys.exc_info()` interpolation, dynamic loguru/traceback imports, logger rebinding, and `getattr` dodges) are rejected in these directories, `bind()`/`patch()` chain messages get the same exception-scrubbing checks as direct logger calls, and exception detail reached through attribute or subscript access (`e.args`, `e.response.text`, `e.strerror`) is flagged like `str(e)` itself. ([#4976](https://github.com/LearningCircuit/local-deep-research/pull/4976))
+- Fixed a plaintext-secret disclosure in `GET /settings/api/bulk`: a namespace request (e.g. `keys[]=llm`) returned nested `*.api_key` values unredacted because the redaction check looked only at the outer requested key, and `keys[]=%` could dump the whole settings table via an unescaped `LIKE`. Redaction now recurses into setting subtrees and the key lookup escapes LIKE wildcards. ([#5028](https://github.com/LearningCircuit/local-deep-research/pull/5028))
+- `GET /settings/api/bulk` now redacts password-typed settings whose leaf name is a non-classic secret token (e.g. `client_secret`, `secret_key`, `bearer_token`, `api_secret`, `app_secret`). The bulk endpoint redacts by key name only, so these were previously shipped in the clear while the singular GET masked them. ([#5038](https://github.com/LearningCircuit/local-deep-research/pull/5038))
+- Exception-derived strings flowing through the `/api/v1/quick_summary` and `/api/v1/analyze_documents` JSON responses are now scrubbed at the HTTP boundary (and at the strategy layer in `SourceBasedSearchStrategy`) via `sanitize_error_for_client`, closing a CWE-209 information-exposure vector flagged by CodeQL (#8019): a caught exception interpolated into a strategy's error fields could otherwise reach the API client — as `summary` (the key `quick_summary()`/`analyze_documents()` actually return), `formatted_findings`, or per-finding `content` — with credentials or stack-trace text intact. Credentials are scrubbed and length is capped; the `"Error: "` prefix is preserved so error payloads stay recognizable to clients, and the strategy-layer scrub independently covers the web-UI SSE path (`research_service.py` → `ErrorReportGenerator`), which consumes strategy output directly. `/api/v1/generate_report` gets the same boundary scrub purely as a precaution — its current payload (`content`/`metadata`) carries no error fields.
+
+### ✨ New Features
+
+- **Sofya search engine** — Added [Sofya](https://sofya.co) as an optional hosted web-search engine. Sofya's `/v1/search` returns ranked results with SERP snippets and extracted page content (markdown) in a single call, fitting LDR's two-phase retrieval model without a separate fetch round trip. Supports `basic`/`snippets` depth, `general`/`news` topics, recency filtering, and domain include/exclude. Configure your API key under Settings → Search → Sofya (`search.engine.web.sofya.api_key`) or the `LDR_SEARCH_ENGINE_WEB_SOFYA_API_KEY` environment variable; GitHub sign-up grants 1000 free credits/month.
+
+### 🐛 Bug Fixes
+
+- Fixed the news subscription-history endpoint (`/news/api/subscriptions/<id>/history`) raising an internal error for any subscription that had research runs: `get_subscription_history` called `.isoformat()` on the `created_at`/`completed_at` values, which are stored as isoformat strings, so it now returns them directly. ([#3094](https://github.com/LearningCircuit/local-deep-research/pull/3094))
+- Fix collapsed-whitespace bug in SearXNG-sourced titles. The HTML parser was calling `BeautifulSoup.Tag.get_text(strip=True)` to read result titles, which strips leading/trailing whitespace *and* collapses every internal whitespace run to nothing — so multi-word titles like "Word One Word Two Word Three" rendered as "WordOneWordTwoWordThree" in the `## Sources` block. Swapped to `get_text(" ", strip=True)` (the documented BeautifulSoup idiom for "strip edges, replace internal runs with a single space") so word boundaries survive. The same fix was applied to result snippets. Title preservation flows through to the LLM relevance filter and `BaseCandidateExplorer` / `ProgressiveExplorer` candidate-phrase extraction, both of which previously saw single-token titles from SearXNG-sourced collections. ([#4970](https://github.com/LearningCircuit/local-deep-research/pull/4970))
+- Registration/login no longer leave a user silently logged in when post-authentication setup fails partway: the partial session is now rolled back, so a reported failure matches reality (you are not logged in). ([#5006](https://github.com/LearningCircuit/local-deep-research/pull/5006))
+- Fix citation labels rendering as empty `[]` for library/RAG sources. The `DOMAIN_HYPERLINKS`, `DOMAIN_ID_HYPERLINKS`, and `DOMAIN_ID_ALWAYS_HYPERLINKS` modes previously fed `urlparse(url).netloc` straight into the citation label, so any relative URL (most notably RAG / library hits that look like `/library/document/<uuid>`) produced an empty string. With a single relative-URL citation the label collapsed to `[[]](url)` and rendered as a `[]` link with no anchor text; with multiple relative-URL citations the suffix leaked out as `[[-1]](url)`, `[[-2]](url)`. The formatter now falls back to a slugified (lowercased, alphanumeric-only) version of the document title, and as a final fallback to the citation number itself, so the label is always non-empty and the link carries the document name. Real web URLs (arxiv.org, github.com, etc.) are unchanged — the domain still wins when `netloc` is non-empty.
+- Fixed `formatBytes` rendering sizes of 1 TB or larger as `"N undefined"` (e.g. a library blob total of 1 TB showed as `1 undefined`). The shared byte formatter only defined units up to GB; it now covers TB through EB and clamps the unit index at both ends, so very large values no longer index past the end of the unit list and sub-1-byte values no longer produce a negative index.
+- When the Research Logs panel hits its 500-entry DOM cap, the oldest entries are flushed first — which previously meant old warnings and errors got dropped even when the cap was blown by a flood of routine info entries. The prune now walks the cap-excess slots in priority order (info first, then milestones, then warnings, then errors), so the panel keeps its most diagnostic entries even on long research runs.
+
+### 📝 Other Changes
+
+- Removed the `importlib.util` disk-load workaround for `url_classifier` in `citation_formatter.py` (introduced in #4880). Now that `content_fetcher` no longer eagerly imports its heavy dependency tree (#5023), the formatter uses a normal `from ..content_fetcher.url_classifier import URLClassifier, URLType`. Also fixed two nits in the #5023 test file: an off-by-one project-root path in the subprocess tests and `check=True` calls that hid the subprocess stderr from failure reports. ([#4992](https://github.com/LearningCircuit/local-deep-research/pull/4992))
+- Fixed an order-dependent test flake: the security import-fallback tests replaced `local_deep_research.security*` modules with fresh copies, recreating enums like `Sensitivity` and breaking identity checks in tests that ran later on the same worker.
+- Source-tagged citation formatting no longer re-runs the URL-classifier label lookup for every citation occurrence — the pre-computed label cache is consulted lazily, so each source's label resolves once. (The disk-loading import this originally worked around was removed entirely in #4992.)
+- The Sofya egress-labels test now compares enum members by `.value` instead of by identity. Under the Docker test setup the package is simultaneously available from `/app/src` (PYTHONPATH) and the installed wheel, so `SofyaSearchEngine`'s class attribute and the test's `Sensitivity` can resolve to distinct class objects with identical members. Python's `Enum.__eq__` returns `NotImplemented` across distinct classes (so `==` fails just like `is`); only `.value` (or `.name`) survives the dual-class scenario.
+- The `check-pathlib-usage` pre-commit hook now catches `import os as <alias>` followed by `<alias>.path.*`. Previously the AST matcher only flagged the literal name `os`, so an aliased import silently bypassed the check — which is how a `_os.path.join(...)` slipped into #4880 and had to be caught by human review instead.
+- `content_fetcher.ContentFetcher` is now resolved lazily on first attribute access (PEP 562 module `__getattr__`) instead of being eagerly imported in the package `__init__`. `url_classifier` (and any other import that goes through the `content_fetcher` package) no longer drags in the fetcher's transitive deps (`requests`, `playwright`, `bs4`, `lxml`, etc.), so minimal test setups and the citation formatter can stop using the disk-load workaround introduced in #4880. Public API is unchanged.

docs/release_notes/1.9.2.md

@@ -0,0 +1,14 @@
+
+### ✨ New Features
+
+- Research steps in chat mode now expand to show the actual tool output (search results, fetched page content — capped at 4,000 chars) when clicked, and the agent's "selecting next action from …" heartbeat lists every enabled tool instead of sampling 3 with "+N more". ([#5051](https://github.com/LearningCircuit/local-deep-research/pull/5051))
+
+### 🐛 Bug Fixes
+
+- Fix the Language Model dropdown frequently coming up empty on the New Research page (requiring a manual click of the refresh button). Three root causes are addressed: the page fired two concurrent `/available-models` requests on every load, which contended against a cold Ollama; Ollama's model-list query used a too-tight 2-second timeout (raised to 5s) whose failure was silently turned into an empty list; and a transient empty result could linger in the client cache. The list now loads once per page and self-heals from a transient empty response.
+- Fixed numbers in exported PDFs rendering as wide, spaced-out emoji glyphs ("2 0 2 6" instead of "2026"). Digits 0-9 carry the Unicode Emoji property, so listing "Noto Color Emoji" in the PDF font stacks (#4730) made Pango draw every digit with the emoji font. Emoji families are removed from the stacks; emoji still render via fontconfig per-character fallback to the installed emoji font.
+- Validate that a language model is selected before starting a research from the New Research page. Submitting with an empty model now shows an inline "Please select or enter a model." error and focuses the model field instead of kicking off a run that would fail downstream.
+
+### 📝 Other Changes
+
+- Fixed cross-test state leaks from tests that reload or evict production modules: the rate-limiter tests restored a fake config, the history-routes fixture leaked auth-bypassed blueprints, the office-format tests could leak a blocked-dependency loader registry, the settings-logger tests leaked the log level, the security import-fallback helper re-created the whole security tree (stacking permanent audit hooks), and four sibling sites (thread-local-session eviction, auth-db reload, env-definitions re-import, loader-registry import guards) now restore the exact pre-test modules.

docs/search-engines.md

@@ -0,0 +1,250 @@
+# Search Engines Guide
+
+Local Deep Research integrates with multiple search engines to provide comprehensive research capabilities. This guide covers all available search engines, their specializations, and configuration details.
+
+> **Note**: This documentation is maintained by the community and may contain inaccuracies. While we strive to keep it up-to-date, please verify critical information and report any errors via [GitHub Issues](https://github.com/LearningCircuit/local-deep-research/issues).
+
+## Overview
+
+LDR supports three categories of search engines:
+- **Free Search Engines** - No API key required
+- **Premium Search Engines** - Require API keys but offer enhanced features
+- **Custom Sources** - Your own documents and databases
+
+## Search Engine Selection
+
+### Dynamic Engine Selection (Recommended)
+
+The default `langgraph-agent` strategy selects the most appropriate engines dynamically per query: every enabled engine (and any registered retriever or local collection) is exposed to the research agent as a tool, and the agent decides which to call for each sub-question. You only pick a primary engine (`searxng` is the recommended default):
+
+```python
+result = quick_summary(
+    query="What are the latest advances in quantum computing?",
+    search_tool="searxng"  # Primary engine; the langgraph-agent strategy
+                           # can still pull in other enabled engines per query
+)
+```
+
+> **Note**: The former `auto` and `parallel` meta engines were removed — the langgraph-agent strategy replaces them. Stored settings are migrated automatically; explicit `search_tool="auto"` callers should switch to a concrete engine like `searxng`.
+
+## Free Search Engines
+
+### Academic Search Engines
+
+#### arXiv
+- **Specialization**: Scientific papers and preprints
+- **Best for**: Physics, mathematics, computer science, biology
+- **Results**: Direct access to research papers
+- **Rate Limit**: Moderate - automatic retry on limits
+
+#### PubMed
+- **Specialization**: Biomedical and life science literature
+- **Best for**: Medical research, clinical studies, biology
+- **Results**: Abstracts and links to full papers
+- **Rate Limit**: Generous - rarely hits limits
+
+#### Semantic Scholar
+- **Specialization**: Academic literature across all fields
+- **Best for**: Cross-disciplinary research, citation networks
+- **Results**: Paper summaries with citation context
+- **Rate Limit**: Moderate - adaptive rate limiting handles this
+
+### General Purpose
+
+#### Wikipedia
+- **Specialization**: General knowledge and encyclopedic information
+- **Best for**: Background information, concepts, facts
+- **Results**: Well-structured article content
+- **Rate Limit**: Very generous
+
+#### SearXNG (Highly Recommended)
+- **Specialization**: Meta-search engine aggregating multiple sources
+- **Best for**: Comprehensive web search with privacy
+- **Results**: Aggregated results from Google, Bing, DuckDuckGo, etc.
+- **Setup**:
+  ```bash
+  docker pull searxng/searxng
+  docker run -d -p 8080:8080 --name searxng searxng/searxng
+  ```
+- **Configuration**: Set URL to `http://localhost:8080` in settings
+
+#### DuckDuckGo
+- **Specialization**: Privacy-focused web search
+- **Best for**: General web queries without tracking
+- **Results**: Web pages, instant answers
+- **Rate Limit**: Strict - use SearXNG for better reliability
+
+### Technical Search
+
+#### GitHub
+- **Specialization**: Code repositories and documentation
+- **Best for**: Finding code examples, libraries, technical solutions
+- **Results**: Repository information, code snippets, issues
+- **Rate Limit**: Moderate when unauthenticated
+
+#### Elasticsearch
+- **Specialization**: Custom search within your Elasticsearch cluster
+- **Best for**: Searching your own indexed data
+- **Configuration**: See [Elasticsearch Setup Guide](elasticsearch_search_engine.md)
+
+### Historical Search
+
+#### Wayback Machine
+- **Specialization**: Historical web content
+- **Best for**: Finding deleted content, tracking changes over time
+- **Results**: Archived web pages with timestamps
+- **Rate Limit**: Moderate
+
+### News Search
+
+#### The Guardian
+- **Specialization**: News articles and journalism
+- **Best for**: Current events, news analysis
+- **Results**: Recent news articles
+- **Note**: Requires API key (free tier available at https://open-platform.theguardian.com/)
+
+#### Wikinews
+- **Specialization**: Open and collaboratively-written news articles on a wide range of topics
+- **Best for**: Historical and recent news, general news coverage, quick overviews
+- **Results**: News articles written by volunteers with verified sources
+
+## Premium Search Engines
+
+### Tavily
+- **Specialization**: AI-optimized search for LLM applications
+- **Best for**: High-quality, relevant results for AI research
+- **Pricing**: Free tier available, paid plans for higher volume
+- **Configuration**:
+  ```bash
+  # In .env file or web interface
+  LDR_SEARCH_ENGINE_TAVILY_API_KEY=your-key-here
+  ```
+
+### Google (via SerpAPI)
+- **Specialization**: Comprehensive web search
+- **Best for**: Most current and comprehensive results
+- **Pricing**: Paid service with free trial
+- **Configuration**:
+  ```bash
+  LDR_SEARCH_ENGINE_WEB_SERPAPI_API_KEY=your-key-here
+  ```
+
+### Google Programmable Search Engine
+- **Specialization**: Customizable Google search
+- **Best for**: Searching specific sites or topics
+- **Pricing**: Free tier with limits
+- **Configuration**:
+  ```bash
+  LDR_SEARCH_ENGINE_WEB_GOOGLE_PSE_API_KEY=your-key-here
+  LDR_SEARCH_ENGINE_WEB_GOOGLE_PSE_ENGINE_ID=your-engine-id
+  ```
+
+### Brave Search
+- **Specialization**: Independent search index with privacy focus
+- **Best for**: Web search without big tech tracking
+- **Pricing**: Free tier available
+- **Configuration**:
+  ```bash
+  LDR_SEARCH_ENGINE_WEB_BRAVE_API_KEY=your-key-here
+  ```
+
+## Custom Sources
+
+### Local Documents
+- **Specialization**: Search your private documents
+- **Supported formats**: PDF, TXT, MD, DOCX, CSV, and more
+- **Configuration**: See [Configuring Local Search](https://github.com/LearningCircuit/local-deep-research/wiki/Configuring-Local-Search)
+- **Setup**:
+  1. Go to Settings → Search for "local"
+  2. Add document collection paths
+  3. Choose embedding model (CPU or Ollama)
+  4. First search will index documents
+
+### LangChain Retrievers
+- **Specialization**: Any vector store or database
+- **Supported**: FAISS, Chroma, Pinecone, Weaviate, Elasticsearch
+- **Configuration**: See [LangChain Integration Guide](LANGCHAIN_RETRIEVER_INTEGRATION.md)
+
+
+## Search Performance Comparison
+
+| Engine | Speed | Quality | Privacy | Rate Limits |
+|--------|-------|---------|---------|-------------|
+| SearXNG | ★★★★★ | ★★★★☆ | ★★★★★ | ★★★★★ |
+| Wikipedia | ★★★★☆ | ★★★★☆ | ★★★★★ | ★★★★★ |
+| arXiv | ★★★★☆ | ★★★★★ | ★★★★★ | ★★★☆☆ |
+| PubMed | ★★★★☆ | ★★★★★ | ★★★★★ | ★★★★☆ |
+| Tavily | ★★★★☆ | ★★★★★ | ★★★☆☆ | ★★★★☆ |
+| Google (SerpAPI) | ★★★★☆ | ★★★★★ | ★★☆☆☆ | ★★★★★ |
+| Local Documents | ★★★☆☆ | ★★★★★ | ★★★★★ | ★★★★★ |
+
+## Rate Limiting and Reliability
+
+LDR includes intelligent adaptive rate limiting that:
+- Learns optimal wait times for each engine
+- Automatically retries failed requests
+- Prevents your IP from being blocked
+- Maintains high reliability
+
+### Managing Rate Limits
+
+```bash
+# Check rate limit status
+python -m local_deep_research.web_search_engines.rate_limiting status
+
+# Reset rate limits if needed
+python -m local_deep_research.web_search_engines.rate_limiting reset
+```
+
+## Search Strategies
+
+LDR supports multiple search strategies that determine how queries are processed:
+
+- **langgraph-agent**: Agentic research that picks engines dynamically per query (default)
+- **source-based**: Single query, fast results
+- **focused_iteration**: Iterative refinement for accuracy
+
+## Best Practices
+
+1. **For General Research**: Use `searxng` with the default langgraph-agent strategy
+2. **For Academic Research**: Combine `arxiv`, `pubmed`, and `semantic_scholar`
+3. **For Technical Questions**: Use `github` with `searxng`
+4. **For Maximum Privacy**: Use `searxng` with local Ollama models
+5. **For Best Quality**: Use `tavily` or Google with `focused_iteration` strategy
+
+## Troubleshooting
+
+### SearXNG Not Working
+- Verify container is running: `docker ps | grep searxng`
+- Check URL in settings: `http://localhost:8080`
+- Test directly: `curl http://localhost:8080`
+- Check the logs: `docker logs searxng` or view them in the LDR web UI
+
+### Rate Limit Errors
+- Wait a few minutes and retry
+- Use the langgraph-agent strategy, which can route around rate-limited engines
+- Consider adding premium engines for higher limits
+
+### No Results Found
+- Try different search engines
+- Broaden your query
+- Check internet connectivity
+- Verify API keys for premium engines
+
+## Advanced Configuration
+
+### Configuring Search Engines
+
+You can enable/disable specific search engines and adjust their reliability parameters in the settings. This affects which engines the langgraph-agent strategy can choose from and how the system handles rate limiting.
+
+### Multi-Engine Research
+
+The former `auto` and `parallel` meta engines (which fanned a query out over several engines) have been removed. To research across multiple engines, use the default langgraph-agent strategy: it calls any enabled engine as a tool, in parallel where useful, and picks per sub-question which engines to query.
+
+## Related Documentation
+
+- [API Quickstart](api-quickstart.md)
+- [Configuration Guide](env_configuration.md)
+- [Full Configuration Reference](CONFIGURATION.md)
+- [LangChain Integration](LANGCHAIN_RETRIEVER_INTEGRATION.md)
+- [Elasticsearch Setup](elasticsearch_search_engine.md)

docs/security/CODEQL_GUIDE.md

@@ -0,0 +1,193 @@
+# Local CodeQL Analysis Guide
+
+This guide explains how to set up and use CodeQL for local security analysis of the Local Deep Research project, covering both Python backend and JavaScript frontend code.
+
+## Prerequisites
+
+1. **CodeQL CLI**
+   - Download the latest CodeQL bundle from [GitHub](https://github.com/github/codeql-action/releases)
+   - Extract it to a permanent location (e.g., `C:\codeql` on Windows or `/opt/codeql` on Linux)
+   - Add the CodeQL executable to your system PATH
+
+2. **Ollama**
+   - Install Ollama from [ollama.ai](https://ollama.ai)
+   - Pull the required model: `ollama pull deepseek-r1:32b`
+
+3. **Analysis Scripts**
+   - Copy the appropriate analysis script from `docs/security/` to your project root:
+     - Windows: `analyze_sarif.ps1`
+     - Linux/Mac: `analyze_sarif.sh`
+
+## Setup Instructions
+
+### Windows
+
+1. **Install CodeQL**
+   ```powershell
+   # Download and extract CodeQL bundle
+   Invoke-WebRequest -Uri "https://github.com/github/codeql-action/releases/latest/download/codeql-bundle-win64.tar.gz" -OutFile "codeql.zip"
+   tar -xvzf codeql.zip -C C:\codeql
+
+   # Add to PATH (run as administrator)
+   [Environment]::SetEnvironmentVariable("Path", "$env:Path;C:\codeql\codeql", "Machine")
+   ```
+
+2. **Create CodeQL Databases**
+   ```powershell
+   # Navigate to project root/src (src is recommended for codeql analysis, to avoid analyzing venv folder)
+   cd path/to/local-deep-research/src
+
+   # Create Python database
+   codeql database create --language=python --source-root . ./python-db
+
+   # Create JavaScript database (for frontend code)
+   codeql database create --language=javascript --source-root . ./js-db
+   ```
+
+3. **Run Analysis**
+   ```powershell
+   # Run Python CodeQL analysis
+   codeql database analyze ./python-db python-security-and-quality.qls --format=sarif-latest --output=python-results.sarif
+
+   # Run JavaScript CodeQL analysis
+   codeql database analyze ./js-db javascript-security-extended.qls --format=sarif-latest --output=js-results.sarif
+
+   # Merge results (optional)
+   codeql dataset merge --output=combined-db --source=python-db --source=js-db
+   codeql database analyze ./combined-db --format=sarif-latest --output=combined-results.sarif
+
+   # Analyze results with Ollama
+   .\analyze_sarif.ps1
+   ```
+
+### Linux/Mac
+
+1. **Install CodeQL**
+   ```bash
+   # Download and extract CodeQL bundle
+   wget https://github.com/github/codeql-action/releases/latest/download/codeql-bundle-linux64.tar.gz
+   tar -xvzf codeql-bundle-linux64.tar.gz -C /opt/codeql
+
+   # Add to PATH (add to ~/.bashrc or ~/.zshrc)
+   export PATH=$PATH:/opt/codeql/codeql
+   ```
+
+2. **Create CodeQL Databases**
+   ```bash
+   # Navigate to project root/src
+   cd path/to/local-deep-research/src
+
+   # Create Python database
+   codeql database create --language=python --source-root . ./python-db
+
+   # Create JavaScript database
+   codeql database create --language=javascript --source-root . ./js-db
+   ```
+
+3. **Run Analysis**
+   ```bash
+   # Make script executable
+   chmod +x analyze_sarif.sh
+
+   # Run Python CodeQL analysis
+   codeql database analyze ./python-db python-security-and-quality.qls --format=sarif-latest --output=python-results.sarif
+
+   # Run JavaScript CodeQL analysis
+   codeql database analyze ./js-db javascript-security-extended.qls --format=sarif-latest --output=js-results.sarif
+
+   # Merge results (optional)
+   codeql dataset merge --output=combined-db --source=python-db --source=js-db
+   codeql database analyze ./combined-db --format=sarif-latest --output=combined-results.sarif
+
+   # Analyze results with Ollama
+   ./analyze_sarif.sh
+   ```
+
+## Important Queries
+
+The following CodeQL queries are particularly relevant for our project:
+
+1. **Python Security**
+   - `python-security-and-quality.qls`: General security and code quality
+   - `python/sql-injection`: SQL injection vulnerabilities
+   - `python/hardcoded-credentials`: Hardcoded secrets
+   - `python/unsafe-deserialization`: Insecure deserialization
+
+2. **JavaScript Security**
+   - `javascript-security-extended.qls`: Comprehensive security checks
+   - `javascript/xss`: Cross-site scripting vulnerabilities
+   - `javascript/prototype-pollution`: Prototype pollution issues
+   - `javascript/express-misconfigured-cors`: CORS misconfigurations
+   - `javascript/unsafe-dynamic-import`: Unsafe dynamic imports
+   - `javascript/unsafe-eval`: Unsafe eval() usage
+
+3. **Custom Queries**
+   - Logging injection vulnerabilities
+   - Uninitialized variables
+   - API security issues
+   - Frontend security best practices
+
+## Analysis Script Features
+
+The analysis scripts (`analyze_sarif.ps1` and `analyze_sarif.sh`) provide:
+
+1. **Input Validation**
+   - Checks if Ollama is running
+   - Validates SARIF file format
+   - Verifies required dependencies
+
+2. **Error Handling**
+   - Graceful error messages
+   - Color-coded output
+   - Detailed error reporting
+
+3. **Output**
+   - Human-readable analysis
+   - Prioritized findings
+   - Recommended fixes
+   - Language-specific recommendations
+
+## Troubleshooting
+
+1. **Ollama Connection Issues**
+   - Ensure Ollama is running: `ollama serve`
+   - Check if the model is pulled: `ollama list`
+   - Verify the endpoint in the script matches your setup
+
+2. **CodeQL Database Creation**
+   - Clean previous databases: `rm -rf ./python-db ./js-db`
+   - Ensure dependencies are installed (Python and Node.js)
+   - Check for sufficient disk space
+   - For JavaScript: Ensure node_modules is present
+
+3. **Analysis Script Issues**
+   - Verify script permissions (Linux/Mac)
+   - Check PowerShell execution policy (Windows)
+   - Ensure all required tools are installed
+
+## Best Practices
+
+1. **Regular Analysis**
+   - Run analysis before major commits
+   - Schedule regular security scans
+   - Review and address findings promptly
+   - Run both frontend and backend scans
+
+2. **Database Management**
+   - Clean old databases regularly
+   - Keep CodeQL CLI updated
+   - Use appropriate query suites
+   - Consider using RAM disk for large projects
+
+3. **Results Handling**
+   - Document resolved issues
+   - Track false positives
+   - Share findings with the team
+   - Prioritize critical frontend and backend issues
+
+## Additional Resources
+
+- [CodeQL Documentation](https://codeql.github.com/docs/)
+- [Python CodeQL Queries](https://github.com/github/codeql/tree/main/python/ql/src)
+- [JavaScript CodeQL Queries](https://github.com/github/codeql/tree/main/javascript/ql/src)
+- [Ollama Documentation](https://ollama.ai/docs)

docs/security/analyze_sarif.sh

@@ -0,0 +1,111 @@
+#!/bin/bash
+
+# CodeQL SARIF Analysis Script for Unix/Linux
+# This script analyzes CodeQL SARIF results using Ollama for human-readable explanations
+
+# Configuration
+SARIF_PATH="python-results.sarif"
+OUTPUT_PATH="codeql_analysis_results.txt"
+OLLAMA_ENDPOINT="http://localhost:11434/api/generate"
+MODEL="deepseek-r1:32b"
+
+# Colors for output
+RED='\033[0;31m'
+GREEN='\033[0;32m'
+YELLOW='\033[1;33m'
+CYAN='\033[0;36m'
+NC='\033[0m' # No Color
+
+# Function to check if Ollama is running
+check_ollama() {
+    echo -n "Checking Ollama connection... "
+    if curl -s "$OLLAMA_ENDPOINT" > /dev/null; then
+        echo -e "${GREEN}OK${NC}"
+        return 0
+    else
+        echo -e "${RED}FAILED${NC}"
+        echo -e "${RED}Error: Ollama is not running. Please start Ollama first.${NC}"
+        return 1
+    fi
+}
+
+# Function to validate SARIF file
+validate_sarif() {
+    echo -n "Validating SARIF file... "
+    if [ ! -f "$SARIF_PATH" ]; then
+        echo -e "${RED}FAILED${NC}"
+        echo -e "${RED}Error: SARIF file not found at: $SARIF_PATH${NC}"
+        return 1
+    fi
+
+    if jq empty "$SARIF_PATH" 2>/dev/null; then
+        echo -e "${GREEN}OK${NC}"
+        return 0
+    else
+        echo -e "${RED}FAILED${NC}"
+        echo -e "${RED}Error: Invalid SARIF file format${NC}"
+        return 1
+    fi
+}
+
+# Main script
+echo -e "${CYAN}CodeQL SARIF Analysis Tool${NC}"
+echo -e "${CYAN}=========================${NC}"
+
+# Check if required tools are installed
+command -v curl >/dev/null 2>&1 || { echo -e "${RED}Error: curl is required but not installed.${NC}"; exit 1; }
+command -v jq >/dev/null 2>&1 || { echo -e "${RED}Error: jq is required but not installed.${NC}"; exit 1; }
+
+# Check if Ollama is running
+check_ollama || exit 1
+
+# Validate SARIF file
+validate_sarif || exit 1
+
+# Read SARIF content
+echo -n "Reading SARIF file... "
+if SARIF_CONTENT=$(cat "$SARIF_PATH"); then
+    echo -e "${GREEN}OK${NC}"
+else
+    echo -e "${RED}FAILED${NC}"
+    echo -e "${RED}Error: Failed to read SARIF file${NC}"
+    exit 1
+fi
+
+# Prepare the prompt
+PROMPT="Please analyze these security findings from CodeQL analysis.
+Focus on:
+1. Critical vulnerabilities
+2. High priority issues
+3. Potential impact
+4. Recommended fixes
+
+Here is the analysis data:
+$SARIF_CONTENT"
+
+# Prepare the request body
+REQUEST_BODY=$(jq -n \
+    --arg model "$MODEL" \
+    --arg prompt "$PROMPT" \
+    '{"model": $model, "prompt": $prompt}')
+
+# Make the request to Ollama and process the streaming response
+echo -n "Analyzing results with Ollama... "
+if curl -s -X POST "$OLLAMA_ENDPOINT" \
+    -H "Content-Type: application/json" \
+    -d "$REQUEST_BODY" | \
+    while IFS= read -r line; do
+        if [ -n "$line" ]; then
+            echo "$line" | jq -r '.response // empty'
+        fi
+    done > "$OUTPUT_PATH"; then
+    echo -e "${GREEN}OK${NC}"
+else
+    echo -e "${RED}FAILED${NC}"
+    echo -e "${RED}Error: Failed to analyze with Ollama${NC}"
+    exit 1
+fi
+
+echo -e "\n${GREEN}Analysis complete!${NC}"
+echo -e "${YELLOW}Results saved to: $OUTPUT_PATH${NC}"
+echo -e "${YELLOW}You can view the results by opening: $OUTPUT_PATH${NC}"

docs/security/database-backup.md

@@ -0,0 +1,92 @@
+# Database Backup System
+
+Local Deep Research includes an automatic database backup system that creates encrypted backups of your user database after each successful login.
+
+## Overview
+
+- **Automatic**: Backups run in the background after login without blocking the UI
+- **Encrypted**: Backups use the same encryption as your main database
+- **Safe**: Uses SQLCipher's `sqlcipher_export()` for atomic backups that work correctly with WAL mode
+- **Configurable**: Enable/disable and configure retention via settings
+- **Pre-migration**: A backup is automatically created before any database schema migration
+
+## How It Works
+
+1. When you log in successfully, a background backup is scheduled
+2. Only one backup per calendar day is created — subsequent logins the same day are skipped to prevent a corrupted database from overwriting all good backups
+3. The backup runs in a separate thread (non-blocking)
+4. Uses `sqlcipher_export()` to create an encrypted copy preserving all cipher settings
+5. Old backups are automatically cleaned up based on your retention settings
+6. Before database migrations, a backup is always created regardless of the daily limit
+
+## Backup Location
+
+Backups are stored in:
+```
+{data_directory}/encrypted_databases/backups/{user_hash}/
+```
+
+Where `{user_hash}` is the first 16 hex characters of the SHA-256 hash of your username.
+
+Each backup file is named with a timestamp:
+```
+ldr_backup_20250125_143022.db
+```
+
+## Settings
+
+Configure backup behavior in Settings > Backup:
+
+| Setting | Default | Description |
+|---------|---------|-------------|
+| **Enable Auto-Backup** | `true` | Enable/disable automatic backups on login. Disable if disk space is limited. |
+| **Max Backups** | `1` | Maximum number of backup files to keep (1-30) |
+| **Backup Retention (days)** | `7` | Delete backups older than this many days |
+
+**Note**: Each backup is a full encrypted copy of your database and cannot be compressed. With the default of 1 backup, disk usage equals your database size. Users with large databases (e.g., containing uploaded PDFs) should monitor disk usage and can reduce the backup count or disable backups entirely via the **Enable Auto-Backup** setting if disk space is limited.
+
+## Why sqlcipher_export()?
+
+We use SQLCipher's `sqlcipher_export()` instead of `VACUUM INTO` or simple file copy because:
+
+1. **Encryption Preservation**: `VACUUM INTO` does not preserve SQLCipher encryption settings. `sqlcipher_export()` correctly copies data while maintaining the same encryption key and cipher configuration
+2. **WAL Safety**: Regular file copy can corrupt databases using WAL (Write-Ahead Logging) mode
+3. **Atomic Operation**: The backup uses ATTACH + export + DETACH, and is written to a temporary file then atomically renamed
+4. **Integrity Verification**: Each backup is verified with `PRAGMA quick_check` before being finalized
+
+## Restoring from Backup
+
+To restore from a backup:
+
+1. Stop the application
+2. Locate your backup in the backup directory
+3. Copy it to replace your current database file (keep the `.salt` file alongside it)
+4. Restart the application
+
+**Important**: The backup uses the same password as when it was created. If you've changed your password since the backup, you'll need to use the old password to access it.
+
+## Troubleshooting
+
+### Backups not being created
+
+1. Check if backups are enabled in Settings
+2. Check the logs for backup-related errors
+3. Verify sufficient disk space (requires 2x database size)
+
+### Disk space issues
+
+The system checks for available disk space before creating a backup. If you see "Insufficient disk space" errors:
+
+1. Free up disk space
+2. Reduce the max backup count setting
+3. Reduce the retention days setting
+
+### Backup verification failed
+
+If you see "Backup verification failed" in logs, the backup may be corrupted. This can happen if:
+
+1. The disk ran out of space during backup
+2. There was a system crash during backup
+3. The source database is corrupted
+
+In this case, the corrupted backup file is automatically deleted.

docs/strategies/deleted/README.md

@@ -0,0 +1,108 @@
+# Deleted advanced-search-system components — notes archive
+
+This directory captures the **novelty** of components deleted from
+`src/local_deep_research/advanced_search_system/` — strategies, question
+generators, constraint checkers, filters, candidate explorers, evidence
+gatherers. Git already stores the code and its history; the job of these
+notes is the prose explanation of *what was novel and why*, something
+git blame can't reconstruct.
+
+Keep files short. Link to the deletion PR or commit for the code; write
+1-2 sentences per idea, not verbatim copies.
+
+(The directory is named `docs/strategies/deleted/` for historical
+reasons — the hook started narrower. It now covers the whole
+`advanced_search_system/` module.)
+
+## When to add a file here
+
+Whenever a PR deletes one or more `.py` files anywhere under
+`src/local_deep_research/advanced_search_system/`. The
+`require-strategy-deletion-docs` pre-commit hook enforces this — a
+commit that deletes such a file without adding or updating a `.md`
+file in this directory will be blocked.
+
+Exempt: `__init__.py` aggregators anywhere in the tree — deleting one
+by itself does not remove a component. Everything else, including
+`base_strategy.py` / `base_question.py` / other `base_*.py`, requires
+a notes file because deleting a base class is a significant refactor.
+
+## Naming
+
+One file per deletion PR:
+
+```
+pr-<number>-<short-slug>.md
+```
+
+Examples: `pr-3147-dead-strategies.md`, `pr-3205-entity-aware.md`.
+
+If a single PR deletes multiple components, document all of them in
+one file with one `## Component:` section per deletion.
+
+## Template
+
+```markdown
+# PR #<n> — <title>
+
+Strategies/components deleted in PR #<n> (see that PR for the full
+pre-deletion code — this file only summarises what was novel).
+
+## Component: `<ClassName>`
+
+- File deleted: `<path>` (<N> LOC at deletion).
+- Reachability: <one line, e.g. "not in `search_system_factory.py`; only referenced by its own test">.
+- Closest reachable successor: `<SuccessorClassName>` (`<path>`, factory key `"<X>"`).
+
+### Useful ideas from the pre-deletion version
+
+- **<short name>** — <1–2 sentences describing what it did and why
+  it was distinctive, and whether it was validated>.
+- **<short name>** — <...>.
+
+### Why deletion was safe
+
+<2–3 sentences mapping distinctive features to the successor, or
+flagging at-risk items and why losing them is acceptable.>
+
+### Recovery path
+
+<1–2 sentences: prefer "add a flag on the existing class" over
+"restore the deleted file". Or: "do not restore".>
+```
+
+## The guiding principle — reference, don't duplicate
+
+Write the notes **as commentary on the code in git**, not as a mirror
+of it. Example of the shape to aim for:
+
+> `generate_creative_search_angles` asked the LLM for 30–40
+> alternate-angle queries using a detective framing (character / title /
+> genre guessing, reverse searches). Exploratory and never validated
+> against a benchmark. Not replicated in `ModularStrategy`.
+
+One paragraph conveys the idea. A reader who wants the exact prompt can
+follow the PR link or run `git show <sha>:<path>`. We are not
+re-hosting the code.
+
+## What a good bullet mentions
+
+Each novelty bullet should be **brief** and answer:
+
+- What the component did that was different from the successor.
+- Why that difference was interesting (a heuristic, a tuning choice, a
+  prompt-engineering trick, an interface gap it filled).
+- Whether it was validated or exploratory — set expectations for
+  anyone considering a revival.
+
+Bullets worth writing are the ones a future reader can't easily infer
+by diffing the deleted file against its successor.
+
+## What not to include
+
+- **Verbatim copies of prompts, docstrings, or code blocks** — the
+  deletion PR already has them; don't re-host.
+- **Line-by-line diffs against the successor** — the PR diff covers it.
+- **Discussion of review decisions** — those belong on the PR itself.
+- **Apologies or extended rationale** — the "Why deletion was safe"
+  section should be 2–3 sentences, not an essay.

docs/strategies/deleted/pr-3147-dead-strategies.md

@@ -0,0 +1,105 @@
+# PR #3147 — delete 3 dead strategy files + 4 orphaned test files
+
+Components deleted in PR #3147 (see that PR for the full pre-deletion code —
+this file only summarises what was novel).
+
+All three were unreachable from `search_system_factory.py` at deletion, with
+no consumers outside their own test files and the `STRATEGY_IMPORTS` list in
+`tests/strategies/conftest.py`.
+
+---
+
+## Component: `ImprovedEvidenceBasedStrategy`
+
+- File deleted: `src/local_deep_research/advanced_search_system/strategies/improved_evidence_based_strategy.py` (782 LOC at deletion).
+- Reachability: not in `search_system_factory.py`; only referenced by its own two test files and `STRATEGY_IMPORTS`.
+- Closest reachable successor: `EnhancedEvidenceBasedStrategy` (`src/local_deep_research/advanced_search_system/strategies/evidence_based_strategy_v2.py`, factory key `"evidence"`).
+
+### Useful ideas from the pre-deletion version
+
+None uniquely interesting. Every distinctive mechanism had a more capable
+analog in v2 at deletion time — 4-stage discovery vs v2's 5-stage, binary
+failed-query blacklist vs v2's EMA pattern-success tracking, simple
+source-ratio diversity vs v2's entropy-based scoring, plain dict state vs
+v2's `QueryPattern` / `SourceProfile` dataclasses, naive re-search final
+verification vs v2's unused-source-driven verification.
+
+### Why deletion was safe
+
+v2 was the factory's `"evidence"` target at deletion and is a strict
+functional superset. Last 12 months of git history on the Improved file
+showed lint sweeps and audit fixes only — no feature work.
+
+### Recovery path
+
+Do not restore the file. If any piece of the v1-improved flow proves useful,
+add it as a method or option on `EnhancedEvidenceBasedStrategy`.
+
+---
+
+## Component: `LLMDrivenModularStrategy`
+
+- File deleted: `src/local_deep_research/advanced_search_system/strategies/llm_driven_modular_strategy.py` (~820 LOC at deletion).
+- Reachability: not in `search_system_factory.py`; inner classes (`CandidateConfidence`, `LLMConstraintProcessor`, `EarlyRejectionManager`) were duplicated in `ModularStrategy`.
+- Closest reachable successor: `ModularStrategy` (`src/local_deep_research/advanced_search_system/strategies/modular_strategy.py`, factory keys `"modular"` / `"modular-parallel"`). The factory instantiates it with `llm_constraint_processing=True` by default, so "LLM-driven" behavior is already the production path.
+
+### Useful ideas from the pre-deletion version
+
+- **5-phase query generation** — LLMDriven ran a 5-step pipeline
+  (decompose → combinations → creative-angles → optimize → execute) with
+  two extra LLM calls that `ModularStrategy` doesn't make. See
+  `generate_creative_search_angles()` and `optimize_search_combinations()`
+  in the pre-deletion file. The first asks the model for 30–40
+  alternate-angle queries (character / title / genre guessing, reverse
+  searches) via a detective framing; the second reorganises generated
+  queries into priority buckets (`high_priority`, `systematic_granular`,
+  `creative_angles`, `contextual_searches`, `fallback_broad`) and
+  de-dupes. Both are exploratory prompt-engineering tricks that were
+  never validated against a benchmark; the extra cost was never shown
+  to be worth it.
+- **Divergent `EarlyRejectionManager` thresholds** — LLMDriven rejected
+  on `negative > 0.7 OR positive < 0.1` (aggressive dual-check);
+  `ModularStrategy` uses `negative > 0.85` only (conservative
+  single-check). `ModularStrategy`'s threshold is the deliberate
+  production default — it prefers not rejecting on absence-of-positive.
+  Noted here so a future reader doesn't "fix" it by tightening.
+
+### Why deletion was safe
+
+`ModularStrategy` subsumes LLMDriven on every axis except the two
+exploratory methods above: the shared inner classes are near-identical,
+`ModularStrategy`'s explorer factory is more flexible than LLMDriven's
+hardcoded `AdaptiveExplorer`, and only `ModularStrategy` has search
+caching, background evaluation, and candidate recovery.
+
+### Recovery path
+
+If the 5-phase generation becomes interesting for a real workload,
+reintroduce it as an optional mode on `ModularStrategy` (e.g. a
+`llm_creative_angles=True` flag) or as a new `CuratedQueryExplorer`
+variant. Do not resurrect the parallel strategy class.
+
+---
+
+## Component: `DirectSearchStrategy`
+
+- File deleted: `src/local_deep_research/advanced_search_system/strategies/direct_search_strategy.py` (219 LOC at deletion).
+- Reachability: not in `search_system_factory.py`; no dedicated test file; only referenced in `STRATEGY_IMPORTS`.
+- Closest reachable successors: `RapidSearchStrategy` (factory key `"rapid"`) and `StandardSearchStrategy` configured with `iterations=1, questions_per_iteration=1` (factory key `"standard"`).
+
+### Useful ideas from the pre-deletion version
+
+None. The strategy was a strict degenerate case of `StandardSearchStrategy`
+reduced to one iteration with one question — same progress labels, same
+citation-handler call, same findings-repository shape.
+
+### Why deletion was safe
+
+No unique logic, no dedicated tests, no external references. Any "single
+direct search" use case is already covered by an existing strategy
+configuration.
+
+### Recovery path
+
+Do not restore. Configure `RapidSearchStrategy` or
+`StandardSearchStrategy` with minimal iteration settings instead.

docs/strategies/deleted/pr-3184-dead-advanced-search-files.md

@@ -0,0 +1,154 @@
+# PR #3184 — delete 6 dead advanced_search_system files + 17 test files
+
+Components deleted in PR #3184 (see that PR for the full pre-deletion code —
+this file only summarises what was novel).
+
+All six were unreachable from `search_system_factory.py` and had zero
+production imports at deletion; each was orphaned experimental work.
+Multi-round subagent novelty audit confirmed every file before deletion.
+
+---
+
+## Component: `AdaptiveQueryGenerator`
+
+- File deleted: `src/local_deep_research/advanced_search_system/query_generation/adaptive_query_generator.py` (405 LOC at deletion).
+- Reachability: no producers or consumers outside the module.
+- Closest reachable successors: `SmartQueryStrategy` (`src/local_deep_research/advanced_search_system/strategies/smart_query_strategy.py`) for LLM-driven query synthesis, and `EnhancedEvidenceBasedStrategy` (`evidence_based_strategy_v2.py`) for pattern learning.
+
+### Useful ideas from the pre-deletion version
+
+- **Hand-curated query-template library with hardcoded success rates** — a small registry of query shapes (entity+property+location, event+temporal+statistic, name+comparison) seeded at 0.6–0.7 success and updated from usage. Superseded by v2's `QueryPattern` dataclass doing the same thing, though less granular.
+- **LLM-synonym semantic-expansion cache** — for each constraint value, fetched 3 alternative phrasings once and built `(term OR syn1 OR syn2)` queries from them. Not actively exercised anywhere today; v2 has a "semantic_expansion" pattern name but uses static templates.
+- **Constraint-tuple success tracking** — counted which *combinations* of constraint types led to successful candidate discoveries (not just individual constraints) and sorted by frequency. This combination-ordering heuristic is absent in successors.
+- **Type-specific single-constraint fallback templates** — hardcoded fallback queries for each constraint type (e.g. `'"{value}" names list'` for `NAME_PATTERN`, `'"{value}" incidents accidents'` for `EVENT`) guaranteeing query generation even when multi-constraint templates fail. Successors regenerate via LLM, losing the guaranteed-coverage floor.
+- **4-tier on-demand fallback escalation** — simplify → broaden (AND → OR) → decompose to single constraints → LLM-generated alternatives. `SmartQueryStrategy._generate_query_variations` generates variants up front instead, which is cheaper but less reactive.
+
+### Why deletion was safe
+
+v2 and `SmartQueryStrategy` reimplement the most important pieces (pattern learning, semantic expansion concept) at coarser granularity. Type-specific hardcoded templates are replaced by general LLM synthesis — flexible but loses guaranteed-minimal-coverage.
+
+### Recovery path
+
+If the semantic-expansion cache or combination-tuple learning proves valuable in benchmarks, add it as an optional flag on `EnhancedEvidenceBasedStrategy` and wire it into `_update_patterns()`.
+
+---
+
+## Component: `DiversityManager`
+
+- File deleted: `src/local_deep_research/advanced_search_system/source_management/diversity_manager.py` (613 LOC at deletion).
+- Reachability: zero references across `src/` and `tests/`.
+- Closest reachable successor: `EnhancedEvidenceBasedStrategy._calculate_source_diversity` (`evidence_based_strategy_v2.py` ~lines 887-909). The successor does a fraction of what the deleted file did.
+
+### Useful ideas from the pre-deletion version
+
+This was the richest file in the PR by far — nine distinct ideas worth flagging:
+
+- **Multi-dimensional source profiling** — per-source record of URL, domain, type, credibility, specialties, temporal coverage, geographic focus, unified in one profile object. Successor tracks only name + usage stats.
+- **Structured `DiversityMetrics` dataclass** — separate scores for type, temporal, geographic, credibility distribution plus a specialty-coverage breakdown. Successor collapses everything into one entropy aggregate.
+- **Credibility scoring with domain heuristics** — source-type base scores (academic 0.9 / government 0.85 / news 0.7 / wiki 0.75 / blog 0.5 / forum 0.4 / social 0.3) adjusted by domain signals (HTTPS, presence of citations, author info). No equivalent today.
+- **LLM-driven source-type detection** — content-excerpt prompt to classify ambiguous sources instead of pattern-matching URLs. No equivalent today.
+- **Temporal and geographic extraction from content** — regex + content analysis to derive year range and geographic focus. No equivalent today.
+- **Diversity gap identification + actionable recommendations** — explicitly detected missing types / regions / time ranges and generated search modifiers to fill them. No recommendation engine anywhere today.
+- **Source-effectiveness feedback loop** — boosted credibility on constraint satisfaction; tracked per-source evidence quality and timestamps. Successor tracks count + success rate only.
+- **Constraint-aware source selection** — preferred specialised sources per constraint type (academic for properties, IMDB for statistics, Wikipedia for locations). Successor uses generic least-used-first.
+- **Diversity-preserving selection** — refused to add an Nth source of one type if another type was underrepresented. Score-ranking alone doesn't do this.
+
+### Why deletion was safe
+
+Orphaned experimental system. No caller invoked any of `select_diverse_sources`, `recommend_additional_sources`, `calculate_diversity_metrics`, or `track_source_effectiveness`. Production strategies use the simpler entropy-based balancing in v2 plus category-based candidate diversity in `DiversityExplorer`.
+
+### Recovery path
+
+If source-profile coherence or credibility heuristics become necessary, reintroduce the `type_priorities` mapping and credibility-enhancement logic as composable helpers on `EnhancedEvidenceBasedStrategy`, not as a standalone manager.
+
+---
+
+## Component: `CrossConstraintManager`
+
+- File deleted: `src/local_deep_research/advanced_search_system/search_optimization/cross_constraint_manager.py` (624 LOC at deletion).
+- Reachability: zero references anywhere.
+- Closest reachable successor: `EnhancedEvidenceBasedStrategy._analyze_constraint_relationships` (`evidence_based_strategy_v2.py` ~lines 836-850). Does far less.
+
+### Useful ideas from the pre-deletion version
+
+- **Typed constraint relationships** — pairwise `ConstraintRelationship` classified as `complementary / dependent / exclusive` via LLM with a strength score in `[0, 1]`. Successor only tracks untyped adjacency lists.
+- **Three-layer constraint clustering** — type-based → relationship-graph-connected-component → LLM-driven semantic clustering, with a cluster-coherence score (weighted average relationship strength × cluster size). No equivalent today.
+- **Progressive multi-constraint query generation** — built a family of queries ramping up constraint count (top-2, then top-3, …) rather than generating one combined query. Not replicated in successors.
+- **Pairwise cross-validation queries** — dedicated per-pair queries of the form "does candidate satisfy constraint A *and* constraint B together?" Absent today.
+- **Cross-constraint candidate validation** — joint-confidence scoring that explicitly boosted complementary constraint pairs. Successors do per-constraint evidence gathering without joint coupling.
+
+### Why deletion was safe
+
+Pure search-optimization layer that was never wired into the factory. The core multi-constraint mechanics (pairing, combined searches, evidence aggregation) are now inline methods on `EnhancedEvidenceBasedStrategy` and the former `ImprovedEvidenceBasedStrategy`; the relationship-typing + clustering-coherence content is genuinely lost, but with no benchmark data it's not clear it was paying for itself.
+
+### Recovery path
+
+If relationship typing becomes load-bearing, rebuild as a mixin on `EnhancedEvidenceBasedStrategy` reusing the `ConstraintRelationship` / `ConstraintCluster` dataclass shapes from the pre-deletion file. Don't restore the 624-LOC parallel manager.
+
+---
+
+## Component: `IntelligentConstraintRelaxer`
+
+- File deleted: `src/local_deep_research/advanced_search_system/constraint_checking/intelligent_constraint_relaxer.py` (501 LOC at deletion).
+- Reachability: not imported anywhere.
+- Closest reachable successors: `ConstrainedSearchStrategy._rank_constraints_by_restrictiveness` and `ParallelConstrainedStrategy._create_relaxed_combinations`.
+
+### Useful ideas from the pre-deletion version
+
+- **Explicit constraint-type priority hierarchy** — hardcoded priority map (`NAME_PATTERN=10, STATISTIC=3, COMPARISON=1`, etc.) distinguishing "essential, never relax" from "safe to drop first." Successors use a generic restrictiveness score based on length/presence-of-numbers; no semantic-importance knowledge.
+- **Progressive relaxation with `min_constraints=2` floor** — generated relaxations in priority order (drop 1 low-priority, then 2, then 3) while preserving enough structure for the query to mean something. Successors skip to binary phases (strict → relaxed → individual).
+- **Type-specific semantic variations** — four dedicated relaxers (STATISTIC / COMPARISON / TEMPORAL / PROPERTY) producing semantically-aware alternatives: ±10% / ±20% / ±50% tolerance bands for statistics, term mappings like "times more" → "significantly more" for comparisons, year → decade → year-range for temporal. Successors strip constraints to bare values, losing these mappings.
+- **Rejection-impact analysis** — `analyze_relaxation_impact` labelled each relaxation as high/low-impact with a user-facing warning ("High-priority constraints removed. Results may be less accurate."). No equivalent tracing today.
+
+### Why deletion was safe
+
+Ranking by restrictiveness and phased-search fallback together cover the "coarse" relaxation shape without the specialised class. The per-type semantic variations are less sophisticated now, but the codebase prioritises breadth over depth of relaxation.
+
+### Recovery path
+
+Put the constraint-type-to-priority map on `Constraint.is_critical()` (in `base_constraint.py`) as a lookup table. If the per-type variation logic matters later, port the four `_relax_*_constraint` method bodies into a helper used by `ConstrainedSearchStrategy` — don't restore the file.
+
+---
+
+## Component: `BrowseCompAnswerDecoder`
+
+- File deleted: `src/local_deep_research/advanced_search_system/answer_decoding/browsecomp_answer_decoder.py` (422 LOC at deletion).
+- Reachability: exported in `answer_decoding/__init__.py` at deletion but never instantiated anywhere in production code.
+- Closest reachable successor: dataset-level decryption helpers in `src/local_deep_research/benchmarks/datasets/utils.py` and `browsecomp.py` (XOR + hardcoded-key fallback only).
+
+### Useful ideas from the pre-deletion version
+
+- **Multi-scheme encoding detection pipeline** — attempted base64, hex, URL-encoding, ROT13, and Caesar in one pattern-driven sweep. Successor only handles XOR+base64, which is sufficient for current BrowseComp datasets but not obviously future-proof.
+- **Caesar cipher scored by an English-frequency heuristic** — ranked shifts 1–25 using letter-frequency scoring (top-10 letters +2, top-20 +1) plus common-word bonuses, returning only shifts above threshold. More general than the hardcoded-key lookup in the successor.
+- **Plaintext-vs-encoded heuristics** — short-answer detection, English-indicator word presence, year/money/name/percentage regex, character-diversity floor. Useful for avoiding false decoding of legitimate plaintext answers; no equivalent today.
+- **Character-distribution answer validation** — ≥80% printable, no control characters, ≥30% letters, <50% punctuation. Successor's range check (ASCII 32–126 only) is much looser.
+- **`analyze_answer_encoding` diagnostic mode** — returned a full per-scheme breakdown of matches, decodings, and validity. Debug tool, not production-critical.
+
+### Why deletion was safe
+
+The decoder was only a hypothetical extension layer; real BrowseComp decryption happens at dataset load time with XOR+password-derived keys. No benchmark runner or strategy depended on it.
+
+### Recovery path
+
+If a dataset arrives using Caesar, ROT13, or multi-scheme encoding, lift the `_decode_caesar` / `_english_score` functions and the plaintext heuristics from the pre-deletion file into `benchmarks/datasets/utils.py`. Don't restore the whole decoder module.
+
+---
+
+## Component: `EvidenceRequirements` (and its helpers)
+
+- File deleted: `src/local_deep_research/advanced_search_system/evidence/requirements.py` (122 LOC at deletion).
+- Reachability: exported in `evidence/__init__.py` but never imported or instantiated anywhere.
+- Closest reachable successor: `EvidenceEvaluator` (`evidence/evaluator.py`) + `base_evidence.py`.
+
+### Useful ideas from the pre-deletion version
+
+- **Per-constraint-type confidence floors** — distinct minimum-confidence thresholds per constraint type: `STATISTIC=0.8, EVENT=0.7, PROPERTY=0.6, NAME_PATTERN=0.5`. Current code uses a single global threshold (typically 0.85 or factory-default 0.95) for all types, losing this nuance.
+- **Per-constraint preferred/acceptable source guidance** — hand-curated source-type preferences per constraint (e.g. `NAME_PATTERN` → etymology / linguistic_analysis; `STATISTIC` → government databases). Never made it into retrieval or evaluation.
+
+### Why deletion was safe
+
+Never called. `EvidenceEvaluator` uses `EvidenceType.base_confidence` in `base_evidence.py` and per-match score adjustments, bypassing constraint-typed requirements entirely.
+
+### Recovery path
+
+If constraint-aware confidence floors prove useful, add a `constraint_type` kwarg to `EvidenceEvaluator.extract_evidence` and apply the per-type threshold there. Don't restore a standalone class.

docs/strategies/deleted/pr-3205-entity-aware.md

@@ -0,0 +1,88 @@
+# PR #3205 — delete dead entity-aware strategy + orphaned question generator
+
+Components deleted in PR #3205 (see that PR for the full pre-deletion code —
+this file only summarises what was novel).
+
+Both components were unreachable from `search_system_factory.py` at
+deletion; the question generator had no callers outside the strategy
+class, so removing the strategy orphaned it.
+
+---
+
+## Component: `EntityAwareSourceStrategy`
+
+- File deleted: `src/local_deep_research/advanced_search_system/strategies/entity_aware_source_strategy.py` (139 LOC at deletion).
+- Reachability: not in `search_system_factory.py`; not exported from `strategies/__init__.py`; only referenced by its own pure-logic test and `STRATEGY_IMPORTS`.
+- Closest reachable successor: `BrowseCompEntityStrategy` (`src/local_deep_research/advanced_search_system/strategies/browsecomp_entity_strategy.py`, factory key `"browsecomp-entity"`) for constraint-driven entity research. For plain-string entity-ID queries, the fallback is `SourceBasedSearchStrategy` with `StandardQuestionGenerator` or `BrowseCompQuestionGenerator`.
+
+### Useful ideas from the pre-deletion version
+
+- **17-keyword entity-query classifier** — the strategy and its question
+  generator both classified a query as entity-seeking if any of `who /
+  what / which / identify / name / character / person / place /
+  organization / company / author / scientist / inventor / city /
+  country / book / movie` appeared in lowercase. High false-positive
+  rate in practice (`who` / `what` / `which` match most research
+  queries), but the list is a starting point if someone builds a proper
+  classifier later.
+- **Naive capitalised-word NER** — `_format_search_results_as_context`
+  treated every capitalised token of length > 2 as a candidate entity
+  with a ±2-word context window. The file's own TODO flagged this as
+  a placeholder. Documented here only so the next person who reaches
+  for this shortcut knows it was tried and found inadequate
+  (matches section headings, sentence-initial words, title-cased junk).
+
+### Why deletion was safe
+
+Zero user-facing path. The strategy added only a question-generator
+swap and the naive NER over its parent `SourceBasedSearchStrategy` —
+no algorithmic novelty. The real prompt engineering lived in the
+question generator (next section).
+
+### Interface gap worth noting
+
+`BrowseCompEntityStrategy` requires pre-built `Constraint` objects.
+`EntityAwareSourceStrategy` was the only (unreachable) path that
+accepted plain-string entity-ID queries. That gap is already unserved
+in production; if it matters, it's a future feature request, not a
+revert.
+
+### Recovery path
+
+Do not restore this 139-LOC wrapper. If plain-string entity-ID queries
+matter, add a helper that extracts `Constraint` objects from a free-form
+query (reuse `ConstraintAnalyzer`) and routes to `BrowseCompEntityStrategy`.
+
+---
+
+## Component: `EntityAwareQuestionGenerator`
+
+- File deleted: `src/local_deep_research/advanced_search_system/questions/entity_aware_question.py` (178 LOC at deletion).
+- Reachability: only consumer was `EntityAwareSourceStrategy`; orphaned once that was removed. Also dropped from `questions/__init__.py`'s `__all__`.
+- Closest reachable successor: `BrowseCompQuestionGenerator` (`src/local_deep_research/advanced_search_system/questions/browsecomp_question.py`). It covers entity-combination queries algorithmically (extract entities, progressively combine 2–3 of them, quote when beneficial) rather than through the explicit prompt templates below.
+
+### Useful ideas from the pre-deletion version
+
+- **Prompt templates teaching multi-constraint quoted searches** —
+  `generate_questions` and `generate_sub_questions` are the only
+  prompts in the codebase that explicitly tell the model to combine
+  multiple identifying constraints into a single quoted search, with
+  a concrete worked example (a fictional 1960s-1980s TV character who
+  breaks the fourth wall). If you need to prompt for entity-ID
+  queries later, start from these rather than re-deriving — they're a
+  reasonable first cut, just never validated.
+- **Liability to avoid on revival** — `generate_questions` returned
+  an empty list whenever the 17-keyword gate didn't match, which
+  halted follow-up research entirely for non-entity queries. Don't
+  reintroduce that fallback.
+
+### Why deletion was safe
+
+Zero remaining consumers after `EntityAwareSourceStrategy` was removed.
+
+### Recovery path
+
+If the prompt engineering proves useful, subclass
+`BrowseCompQuestionGenerator` (or add an optional mode to it) rather
+than restoring a parallel generator class. Drop the empty-list
+fallback if you do.

docs/strategies/deleted/pr-4781-remove-dead-knowledge-generators.md

@@ -0,0 +1,59 @@
+# PR #4781 — remove dead knowledge-generator modules
+
+Components deleted in this PR (see the PR diff / `git show <sha>:<path>` for the
+full pre-deletion code — this file only summarises what was novel). Follows up on
+the `section_links` vulture false positive whitelisted in #4755.
+
+## Component: `BaseKnowledgeGenerator`
+
+- File deleted: `advanced_search_system/knowledge/base_knowledge.py` (149 LOC).
+- Reachability: **none in production**. Nothing under `src/` imports it,
+  `knowledge/__init__.py` re-exports nothing, and the only live module in the
+  package — `followup_context_manager.FollowUpContextHandler` — is standalone and
+  explicitly does *not* inherit from it. Referenced only by its own unit tests.
+- Closest reachable successor: none. The "generate / compress knowledge"
+  abstraction was never wired into a strategy; strategies do their own synthesis
+  and citation formatting inline (e.g. `base_strategy._format_citations`,
+  `langgraph_agent_strategy`).
+
+## Component: `StandardKnowledge`
+
+- File deleted: `advanced_search_system/knowledge/standard_knowledge.py` (156 LOC).
+- Reachability: same as above — the only concrete subclass of
+  `BaseKnowledgeGenerator`, never instantiated outside its own tests.
+- Closest reachable successor: none (see above).
+
+### Useful ideas from the pre-deletion version
+
+- **`compress_knowledge`** — single LLM call to compress accumulated knowledge,
+  asked to retain facts/citations and drop redundancy, returning the *original*
+  text on failure (fail-open). Exploratory; never benchmarked, never called by a
+  strategy. Its `section_links` param was unused in the body — the leaf that
+  vulture flagged and #4755 whitelisted.
+- **`generate_knowledge` / `generate_sub_knowledge`** — generic LLM synthesis of
+  knowledge from query+context, and a per-sub-question variant. Conceptually
+  superseded by the strategies' own iterative synthesis steps; nothing
+  distinctive in the prompts.
+- **IEEE-style `format_citations` (implemented on `StandardKnowledge`; abstract
+  on the base) + concrete base helpers** (`_validate_links`,
+  `_validate_knowledge`, `_extract_key_points` on `BaseKnowledgeGenerator`) —
+  small validation/formatting utilities. The IEEE citation formatting overlaps
+  with the strategies' existing `_format_citations`.
+
+All of the above were **exploratory**: no production caller, no benchmark or
+live test fixture — only unit tests that mocked the model.
+
+### Why deletion was safe
+
+Zero production reachability: nothing imports either module, the package's one
+live class is independent, and search strategies already own their synthesis and
+citation formatting. Removing the modules also retires the now-redundant
+`section_links` vulture whitelist entry (the false positive it suppressed lived
+only in these files). The `pid` whitelist entry is unrelated and stays —
+`faiss_safe_load.safe_load_faiss` is live.
+
+### Recovery path
+
+Do not restore the files. If knowledge compression/synthesis is ever wanted as a
+shared step, add it as a method or small helper on the active strategy rather
+than reviving an abstract base + concrete impl that nothing instantiated.

docs/strategies/deleted/pr-remove-experimental-search-strategies.md

@@ -0,0 +1,160 @@
+# PR — remove experimental search strategies not in the frontend
+
+Components deleted in this PR (see the PR diff / `git show <sha>:<path>` for
+the full pre-deletion code — this file only summarises what was novel).
+
+All deleted strategies were reachable only through `create_strategy()` when
+selected via the "Show All Strategies" toggle (`search.show_all_strategies`,
+also removed) plus their own tests; none were on the default dropdown. The
+surviving, factory-reachable strategies are `SourceBasedSearchStrategy`
+(`source-based`), `FocusedIterationStrategy` (`focused-iteration` /
+`focused-iteration-standard`), `TopicOrganizationStrategy`
+(`topic-organization`), `MCPSearchStrategy` (`mcp`), `LangGraphAgentStrategy`
+(`langgraph-agent`), and the internal `NewsAggregationStrategy` (`news`).
+
+---
+
+## Component group: constraint / evidence / dual-confidence chain
+
+Files deleted (all under `src/local_deep_research/advanced_search_system/strategies/`):
+`evidence_based_strategy.py`, `evidence_based_strategy_v2.py`,
+`constrained_search_strategy.py`, `parallel_constrained_strategy.py`,
+`early_stop_constrained_strategy.py`, `smart_query_strategy.py`,
+`dual_confidence_strategy.py`, `dual_confidence_with_rejection.py`,
+`concurrent_dual_confidence_strategy.py`, `constraint_parallel_strategy.py`,
+plus the support packages `constraint_checking/` and `answer_decoding/`.
+
+- Reachability: a single-rooted inheritance chain
+  (`EvidenceBasedStrategy` → `ConstrainedSearchStrategy` → … →
+  `DualConfidenceWithRejectionStrategy` → `{Concurrent…, ConstraintParallel}`)
+  reachable only via the show-all dropdown and their own tests.
+- Closest reachable successor: none direct — this was the BrowseComp
+  entity-finding lineage. `FocusedIterationStrategy` covers factual Q&A in
+  production and still uses the surviving BrowseComp *question generators*
+  (`questions/browsecomp_question.py`, `questions/flexible_browsecomp_question.py`).
+
+### Useful ideas from the pre-deletion versions
+
+- **Dual-confidence scoring** — scored candidates with separate positive /
+  negative / uncertainty signals (`uncertainty_penalty`, `negative_weight`)
+  rather than a single relevance score, with an early-rejection threshold to
+  prune candidates that strongly violate a critical constraint. A genuinely
+  distinctive evaluation idea; only ever exercised on BrowseComp-style
+  entity puzzles, not on general research.
+- **Concurrent search + evaluation** — `ConcurrentDualConfidenceStrategy` ran
+  candidate search and constraint evaluation on overlapping thread pools with
+  a quality-plateau early stop, instead of strict search-then-evaluate rounds.
+- **`constraint_checking/` checkers** — pluggable checker hierarchy
+  (strict / threshold / dual-confidence / rejection-engine) over a shared
+  `BaseConstraintChecker` interface. The pluggability was the interesting part;
+  it had no consumer outside this strategy chain.
+
+### Why deletion was safe
+
+The whole lineage was self-contained (each class only inherited from the one
+above it) and never on the default dropdown. The reusable, validated pieces —
+the BrowseComp question generators and the `candidate_exploration/` explorers —
+live in separate packages that survive and are still used by
+`FocusedIterationStrategy`.
+
+### Recovery path
+
+Do not restore the files. If dual-confidence scoring proves worth reviving,
+add it as an evaluation option on a surviving strategy rather than
+resurrecting the inheritance chain.
+
+---
+
+## Component group: query-decomposition strategies
+
+Files deleted: `recursive_decomposition_strategy.py`,
+`adaptive_decomposition_strategy.py`, `smart_decomposition_strategy.py`.
+
+- Reachability: factory keys `recursive`, `adaptive`, `smart`; show-all only.
+- Closest reachable successor: `SourceBasedSearchStrategy` / the
+  decomposition question generator (`questions/decomposition_question.py`,
+  which survives).
+
+### Useful ideas
+
+- **Adaptive step typing** — `AdaptiveDecompositionStrategy` modelled a run as
+  typed steps (`StepType`: constraint-extraction / initial-search /
+  verification / refinement / synthesis) with a confidence heuristic
+  (verified-facts / constraints, +0.1 for having candidates, capped 0.95).
+  A tidy state model that was never benchmarked against the simpler strategies.
+
+### Why deletion was safe
+
+Exploratory; the surviving `DecompositionQuestionGenerator` keeps the
+sub-query generation idea available to any strategy.
+
+### Recovery path
+
+Do not restore. Add a decomposition option to a surviving strategy if needed.
+
+---
+
+## Component group: simple / single-pass strategies
+
+Files deleted: `rapid_search_strategy.py`, `parallel_search_strategy.py`,
+`standard_strategy.py`, `iterdrag_strategy.py`,
+`iterative_reasoning_strategy.py` (factory keys `iterative` and
+`iterative-reasoning`).
+
+- Reachability: show-all only, plus benchmark defaults that pointed at
+  `iterdrag` (now repointed to `source-based`).
+- Closest reachable successor: `SourceBasedSearchStrategy` (single/parallel
+  passes) and `FocusedIterationStrategy` (iterative reasoning with knowledge
+  accumulation).
+
+### Useful ideas
+
+- **IterDRAG** — interleaved iterative retrieval and generation; was the
+  historical benchmark default. Its retrieval-augmented loop is subsumed by
+  focused-iteration's knowledge accumulation.
+- **`RapidSearchStrategy`** — deliberately single-iteration for latency-bound
+  use; the same effect is achievable with `source-based` and
+  `max_iterations=1`.
+
+### Why deletion was safe
+
+All were thin variants of behavior the two surviving general strategies
+provide via parameters.
+
+### Recovery path
+
+Prefer flags on `source-based` / `focused-iteration` over restoring the files.
+
+---
+
+## Component group: meta / specialised strategies
+
+Files deleted: `iterative_refinement_strategy.py`,
+`browsecomp_optimized_strategy.py`, `browsecomp_entity_strategy.py`,
+`modular_strategy.py` (factory keys `modular` / `modular-parallel`).
+
+- Reachability: show-all only.
+- Closest reachable successor: `FocusedIterationStrategy` (BrowseComp
+  optimisation + question generation) and `TopicOrganizationStrategy`.
+
+### Useful ideas
+
+- **`IterativeRefinementStrategy`** — a *wrapper* meta-strategy: it ran an
+  inner strategy (default `source-based`), then used LLM gap-analysis to drive
+  follow-up refinement rounds. The wrap-any-strategy interface was the novel
+  part; it added cost without a benchmarked win.
+- **`ModularStrategy`** — composed constraint-checker + candidate-explorer
+  modules selected by string keys. The module-composition idea is interesting
+  but only ever wired to the now-deleted constraint checkers.
+- **`BrowseCompEntityStrategy`** — entity-focused knowledge-graph building for
+  BrowseComp puzzles; the question-generation half survives in `questions/`.
+
+### Why deletion was safe
+
+None were on the default dropdown; their validated sub-pieces (BrowseComp
+question generators, `candidate_exploration/`) survive independently.
+
+### Recovery path
+
+Do not restore. Re-expose the refinement/modular ideas as options on a
+surviving strategy if a benchmark justifies it.

docs/strategies/deleted/pr-remove-mcp-strategy.md

@@ -0,0 +1,59 @@
+# PR — remove the MCP (ReAct) search strategy
+
+Component deleted in this PR (see the PR diff / `git show <sha>:<path>` for the
+full pre-deletion code — this file only summarises what was novel). Tracked by
+issue #4548.
+
+## Component: `MCPSearchStrategy`
+
+- Files deleted:
+  - `src/local_deep_research/advanced_search_system/strategies/mcp_strategy.py`
+    (1876 LOC) — the strategy itself.
+  - `src/local_deep_research/mcp/client.py` (486 LOC) — the MCP **client**
+    (`MCPClient` / `MCPClientManager` / `run_async`), whose only consumer was
+    `MCPSearchStrategy`. `src/local_deep_research/mcp/server.py` (LDR-as-an-MCP-
+    **provider**, FastMCP) is unrelated and **survives**.
+- Reachability: factory keys `mcp` / `agentic` (`agentic` was a pure alias),
+  the `search.search_strategy` dropdown, and its own tests. Never the default
+  (the default is `langgraph-agent`). `mcp.servers` (default `[]`) was its only
+  setting.
+- Closest reachable successor: `LangGraphAgentStrategy`
+  (`advanced_search_system/strategies/langgraph_agent_strategy.py`, factory key
+  `langgraph-agent`). The factory now routes the `mcp` / `agentic` aliases to it
+  with a deprecation warning, and migration `0014` rewrites persisted
+  references.
+
+### Useful ideas from the pre-deletion version
+
+- **ReAct tool-calling loop over external MCP servers** — connected to
+  user-configured stdio MCP servers (`mcp.servers`), discovered their tools at
+  runtime, and let the LLM call them in a Reason+Act loop. This external-MCP-
+  server tool integration is the **genuinely unique capability**;
+  `langgraph-agent` only exposes LDR's own search engines as tools, not
+  arbitrary MCP servers. It is the one capability removal actually loses —
+  tracked as a follow-up to re-home it on `langgraph-agent` if demand appears.
+- **`focused_research`-as-a-tool** — exposed LDR's focused-iteration research as
+  a callable tool inside the agent loop. Not lost: `langgraph-agent`'s
+  `research_subtopic` tool already provides recursive sub-research (via parallel
+  subagents).
+- **`_mcp_scope_blocked` egress gating** — a fail-closed scope check that
+  blocked MCP tool discovery *and* execution under `STRICT` / `PRIVATE_ONLY`,
+  because MCP tools run in stdio subprocesses whose egress the PEP audit hook
+  cannot inspect. The same fail-closed pattern survives in
+  `notifications.manager` and `journal_reputation_filter`.
+
+### Why deletion was safe
+
+`langgraph-agent` (the default) is a near functional superset for agentic
+research over LDR's own engines — same specialized search tools, content fetch,
+parallel subagents, and egress filtering. The only capability genuinely lost is
+connecting to arbitrary external MCP servers as tool sources, which had an empty
+default (`mcp.servers = []`) and no benchmark/test fixtures exercising it. All
+scope-gating patterns survive in sibling modules.
+
+### Recovery path
+
+Do not restore the files. If external-MCP-server tool support is revived, add it
+as a tool-source adapter on `LangGraphAgentStrategy` (a dynamic MCP→LangChain
+tool adapter plus a port of the fail-closed `_mcp_scope_blocked` egress gate)
+rather than a parallel strategy + client.

docs/troubleshooting-openai-api-key.md

@@ -0,0 +1,255 @@
+# Troubleshooting OpenAI API Key Configuration
+
+This guide helps troubleshoot common issues with OpenAI API key configuration in Local Deep Research v1.0+.
+
+## Quick Test
+
+Run the end-to-end test to verify your configuration:
+
+```bash
+# Using command line arguments
+python tests/test_openai_api_key_e2e.py \
+  --username YOUR_USERNAME \
+  --password YOUR_PASSWORD \
+  --api-key YOUR_OPENAI_API_KEY
+
+# Using environment variables
+export LDR_USERNAME=your_username
+export LDR_PASSWORD=your_password
+export OPENAI_API_KEY=sk-your-api-key
+python tests/test_openai_api_key_e2e.py
+```
+
+## Common Issues and Solutions
+
+### 1. "No API key found"
+
+**Symptoms:**
+- Error message about missing API key
+- Research fails to start
+
+**Solutions:**
+
+1. **Via Web Interface:**
+   - Login to LDR web interface
+   - Go to Settings
+   - Select "OpenAI" as LLM Provider
+   - Enter your API key in the "OpenAI API Key" field
+   - Click Save
+
+2. **Via Environment Variable:**
+   ```bash
+   export OPENAI_API_KEY=sk-your-api-key
+   python -m local_deep_research.web.app
+   ```
+
+3. **Programmatically:**
+   ```python
+   from local_deep_research.settings import SettingsManager
+   from local_deep_research.database.session_context import get_user_db_session
+
+   with get_user_db_session(username="user", password="pass") as session:
+       settings_manager = SettingsManager(session)
+       settings_manager.set_setting("llm.provider", "openai")
+       settings_manager.set_setting("llm.openai.api_key", "sk-your-api-key")
+   ```
+
+### 2. "Invalid API key"
+
+**Symptoms:**
+- 401 Unauthorized errors
+- "Incorrect API key provided" messages
+
+**Solutions:**
+
+1. **Verify API Key Format:**
+   - OpenAI keys start with `sk-`
+   - Should be around 51 characters long
+   - No extra spaces or quotes
+
+2. **Check API Key Validity:**
+   ```bash
+   # Test directly with curl
+   curl https://api.openai.com/v1/models \
+     -H "Authorization: Bearer YOUR_API_KEY"
+   ```
+
+3. **Regenerate API Key:**
+   - Go to https://platform.openai.com/api-keys
+   - Create a new API key
+   - Update in LDR settings
+
+### 3. "Rate limit exceeded"
+
+**Symptoms:**
+- 429 errors
+- "You exceeded your current quota" messages
+
+**Solutions:**
+
+1. **Check OpenAI Usage:**
+   - Visit https://platform.openai.com/usage
+   - Verify you have available credits
+
+2. **Add Payment Method:**
+   - OpenAI requires payment info for API access
+   - Add at https://platform.openai.com/account/billing
+
+3. **Use Different Model:**
+   ```python
+   settings_manager.set_setting("llm.model", "gpt-3.5-turbo")  # Cheaper
+   # Instead of gpt-4 which is more expensive
+   ```
+
+### 4. "Settings not persisting"
+
+**Symptoms:**
+- API key needs to be re-entered after restart
+- Settings revert to defaults
+
+**Solutions:**
+
+1. **Ensure Proper Shutdown:**
+   - Use Ctrl+C to stop server (not kill -9)
+   - Wait for "Server stopped" message
+
+2. **Check Database Permissions:**
+   ```bash
+   ls -la encrypted_databases/
+   # Should show your user database with write permissions
+   ```
+
+3. **Verify Settings Save:**
+   ```python
+   # After setting, verify it was saved
+   saved_key = settings_manager.get_setting("llm.openai.api_key")
+   print(f"Saved key: {'*' * 20 if saved_key else 'Not saved'}")
+   ```
+
+### 5. "API key not being used"
+
+**Symptoms:**
+- Settings show OpenAI configured but different LLM is used
+- API key is saved but not applied
+
+**Solutions:**
+
+1. **Check Provider Setting:**
+   ```python
+   provider = settings_manager.get_setting("llm.provider")
+   print(f"Current provider: {provider}")  # Should be "openai"
+   ```
+
+2. **Verify Settings Snapshot:**
+   ```python
+   settings_snapshot = settings_manager.get_all_settings()
+   print("Provider:", settings_snapshot.get("llm.provider", {}).get("value"))
+   print("API Key:", "Set" if settings_snapshot.get("llm.openai.api_key", {}).get("value") else "Not set")
+   ```
+
+3. **Force Provider Selection:**
+   ```python
+   # In research call
+   result = quick_summary(
+       query="Test",
+       settings_snapshot=settings_snapshot,
+       provider="openai",  # Force OpenAI
+       model_name="gpt-3.5-turbo"
+   )
+   ```
+
+## Testing Your Configuration
+
+### 1. Simple API Test
+
+```python
+from local_deep_research.config.llm_config import get_llm
+from local_deep_research.settings import SettingsManager
+from local_deep_research.database.session_context import get_user_db_session
+
+with get_user_db_session(username="user", password="pass") as session:
+    settings_manager = SettingsManager(session)
+    settings_snapshot = settings_manager.get_all_settings()
+
+    # Test LLM initialization
+    try:
+        llm = get_llm(settings_snapshot=settings_snapshot)
+        print("✓ LLM initialized successfully")
+
+        # Test response
+        from langchain.schema import HumanMessage
+        response = llm.invoke([HumanMessage(content="Say hello")])
+        print(f"✓ Response: {response.content}")
+    except Exception as e:
+        print(f"✗ Error: {e}")
+```
+
+### 2. Full Research Test
+
+```python
+from local_deep_research.api.research_functions import quick_summary
+
+result = quick_summary(
+    query="What is OpenAI?",
+    settings_snapshot=settings_snapshot,
+    iterations=1,
+    questions_per_iteration=1
+)
+
+print(f"Research ID: {result['research_id']}")
+print(f"Summary: {result['summary'][:200]}...")
+```
+
+## Advanced Configuration
+
+### Using Azure OpenAI
+
+```python
+settings_manager.set_setting("llm.provider", "openai")
+settings_manager.set_setting("llm.openai.api_key", "your-azure-key")
+settings_manager.set_setting("llm.openai.api_base", "https://your-resource.openai.azure.com/")
+settings_manager.set_setting("llm.model", "your-deployment-name")
+```
+
+### Using OpenAI-Compatible Endpoints
+
+```python
+settings_manager.set_setting("llm.provider", "openai")
+settings_manager.set_setting("llm.openai.api_key", "your-api-key")
+settings_manager.set_setting("llm.openai.api_base", "https://your-endpoint.com/v1")
+```
+
+### Organization ID
+
+```python
+settings_manager.set_setting("llm.openai.organization", "org-your-org-id")
+```
+
+## Getting Help
+
+1. **Run Diagnostic Test:**
+   ```bash
+   python tests/test_openai_api_key_e2e.py --verbose
+   ```
+
+2. **Check Logs:**
+   ```bash
+   # Look for OpenAI-related errors
+   grep -i "openai\|api.*key" logs/ldr.log
+   ```
+
+3. **Community Support:**
+   - GitHub Issues: https://github.com/LearningCircuit/local-deep-research/issues
+   - Discord: https://discord.gg/ttcqQeFcJ3
+
+4. **API Key Best Practices:**
+   - Never commit API keys to version control
+   - Use environment variables for production
+   - Rotate keys regularly
+   - Set usage limits in OpenAI dashboard
+
+## See Also
+
+- [Full Configuration Reference](CONFIGURATION.md) - All settings, defaults, and environment variables
+- [Configuration Guide](env_configuration.md) - Environment variable setup guide
+- [Troubleshooting](troubleshooting.md) - General troubleshooting

docs/troubleshooting.md

@@ -0,0 +1,699 @@
+# Troubleshooting Guide
+
+This guide covers common issues and their solutions.
+
+## Table of Contents
+
+- [LLM Connection Issues](#llm-connection-issues)
+- [Search Engine Issues](#search-engine-issues)
+- [Rate Limiting](#rate-limiting)
+- [Database Issues](#database-issues)
+- [WebSocket/Real-time Updates](#websocketreal-time-updates)
+- [Docker Issues](#docker-issues)
+- [API Issues](#api-issues)
+- [Performance Issues](#performance-issues)
+- [Resource Exhaustion](#resource-exhaustion)
+
+---
+
+## LLM Connection Issues
+
+### Ollama Not Connecting
+
+**Symptoms:**
+- "Failed to connect to Ollama"
+- "Connection refused" errors
+- Empty responses from LLM
+
+**Solutions:**
+
+1. **Verify Ollama is running:**
+   ```bash
+   curl http://localhost:11434/api/tags
+   ```
+
+2. **Check the URL configuration:**
+   - Default: `http://localhost:11434`
+   - For Docker: Use `http://host.docker.internal:11434` or your host IP
+   - Settings location: `llm.ollama.url`
+
+3. **Verify the model is pulled:**
+   ```bash
+   ollama list
+   ollama pull llama3.2
+   ```
+
+4. **Check Docker networking:**
+   ```bash
+   # If running LDR in Docker, Ollama on host:
+   docker run --add-host=host.docker.internal:host-gateway ...
+   ```
+
+### OpenAI API Errors
+
+**Symptoms:**
+- "Invalid API key"
+- "Rate limit exceeded"
+- "Model not found"
+
+**Solutions:**
+
+1. **Verify API key format:**
+   - Should start with `sk-`
+   - Check for leading/trailing whitespace
+
+2. **Check API key permissions:**
+   - Ensure key has access to the model you're using
+   - Verify organization ID if using org-scoped keys
+
+3. **Rate limits:**
+   - Wait and retry for rate limit errors
+   - Consider using a higher tier API key
+   - Reduce `questions_per_iteration` setting
+
+4. **Model availability:**
+   - Verify model name is correct (e.g., `gpt-4`, not `gpt4`)
+   - Check if model is available in your region
+
+### OpenRouter Issues
+
+**Symptoms:**
+- Authentication failures
+- Model not available
+
+**Solutions:**
+
+1. **API key format:**
+   ```
+   # Settings
+   llm.openrouter.api_key = <your-key-here>
+   ```
+
+2. **Model naming:**
+   - Use full model paths: `anthropic/claude-3-opus`
+   - Check available models at openrouter.ai/docs
+
+---
+
+## Search Engine Issues
+
+### DuckDuckGo Returning No Results
+
+**Symptoms:**
+- Empty search results
+- "No results found" consistently
+
+**Solutions:**
+
+1. **Rate limiting:** DuckDuckGo aggressively rate limits. Solutions:
+   - Switch to SearXNG or another engine
+   - Increase wait time in rate limiting settings
+   - Use `search.rate_limiting.profile = conservative`
+
+2. **Check network:** Verify you can access DuckDuckGo directly
+
+3. **Try alternative engines:**
+   ```python
+   # In settings
+   search.tool = "searxng"  # or "brave", "tavily", etc.
+   ```
+
+### SearXNG Setup Issues
+
+**Symptoms:**
+- "Connection refused"
+- "404 Not Found"
+
+**Solutions:**
+
+1. **Verify SearXNG is running:**
+   ```bash
+   curl http://localhost:8080/search?q=test&format=json
+   ```
+
+2. **Check URL configuration:**
+   - Settings UI: **Settings → Search → SearXNG → Instance URL**
+   - Setting key: `search.engine.web.searxng.default_params.instance_url`
+   - Env var: `LDR_SEARCH_ENGINE_WEB_SEARXNG_DEFAULT_PARAMS_INSTANCE_URL`
+
+3. **Ensure JSON format is enabled** in SearXNG settings
+
+4. **Docker networking:** From inside the LDR container, `localhost` is the container itself, not your host. Use `http://searxng:8080` if SearXNG is a sibling service in Docker Compose, or `http://host.docker.internal:8080` if SearXNG runs on the host (Mac/Windows/WSL2 — see the [Windows/WSL2 FAQ entry](faq.md#port-5000-not-accessible-on-windows) for the full recipe).
+
+### API Key Issues for Search Engines
+
+**Symptoms:**
+- "API key required"
+- "Unauthorized" errors
+
+**Solutions:**
+
+1. **Verify key is set:**
+   - Check in Settings > Search > [Engine Name]
+   - Or via environment variable
+
+2. **Engine-specific settings:**
+   | Engine | Setting Key |
+   |--------|-------------|
+   | Brave | `search.engine.brave.api_key` |
+   | Tavily | `search.engine.tavily.api_key` |
+   | Serper | `search.engine.serper.api_key` |
+   | SerpAPI | `search.engine.serpapi.api_key` |
+
+---
+
+## Rate Limiting
+
+### "Rate limit exceeded" Errors
+
+**Symptoms:**
+- Searches failing with rate limit errors
+- Long waits between searches
+- Inconsistent search performance
+
+**Solutions:**
+
+1. **View current rate limit status:**
+   ```bash
+   python -m local_deep_research.web_search_engines.rate_limiting status
+   ```
+
+2. **Reset rate limits for an engine:**
+   ```bash
+   python -m local_deep_research.web_search_engines.rate_limiting reset --engine duckduckgo
+   ```
+
+3. **Adjust rate limiting profile:**
+   ```
+   # Options: conservative, balanced, aggressive
+   search.rate_limiting.profile = conservative
+   ```
+
+4. **Use the langgraph-agent strategy** (the default) to distribute load — it
+   selects engines dynamically per query and can route around rate-limited ones.
+
+### Rate Limiting CLI Commands
+
+```bash
+# View status
+python -m local_deep_research.web_search_engines.rate_limiting status
+python -m local_deep_research.web_search_engines.rate_limiting status --engine arxiv
+
+# Reset learned rates
+python -m local_deep_research.web_search_engines.rate_limiting reset --engine duckduckgo
+
+# Clean old data
+python -m local_deep_research.web_search_engines.rate_limiting cleanup --days 30
+
+# Export data
+python -m local_deep_research.web_search_engines.rate_limiting export --format csv
+```
+
+---
+
+## Database Issues
+
+### "Database is locked" Errors
+
+**Symptoms:**
+- SQLite lock errors
+- Operations timing out
+- Concurrent access failures
+
+**This is likely a bug.** If you encounter persistent "database is locked" errors, please:
+
+1. **Collect logs:**
+   - Check the application logs for error details
+   - Note what action triggered the error
+
+2. **Report the issue:**
+   - Open an issue at [GitHub Issues](https://github.com/LearningCircuit/local-deep-research/issues)
+   - Include the logs and steps to reproduce
+
+**Temporary workarounds:**
+
+1. **Check for zombie processes:**
+   ```bash
+   ps aux | grep python
+   # Kill any stuck LDR processes
+   ```
+
+2. **Restart the application** to release any held locks
+
+### Encryption/SQLCipher Issues
+
+**Symptoms:**
+- "file is not a database"
+- "database disk image is malformed"
+- Cannot open user database
+
+**Solutions:**
+
+1. **Verify SQLCipher is installed:**
+   ```bash
+   pip show sqlcipher3-binary
+   ```
+
+2. **Check password/key:**
+   - User databases are encrypted with derived keys
+   - Password changes require re-encryption
+
+3. **For corrupted databases:**
+   - Check `~/.local/share/local-deep-research/users/` for backups
+   - Consider creating a new user account
+
+4. **Integrity check:**
+   - Use the `/auth/integrity-check` endpoint
+   - Or run manual SQLite integrity checks
+
+### Migration Issues
+
+**Symptoms:**
+- Schema version mismatch
+- Missing tables or columns
+
+**Solutions:**
+
+1. **Check version:**
+   ```python
+   from local_deep_research import __version__
+   print(__version__)
+   ```
+
+2. **Run migrations** (if applicable):
+   - Migrations are typically automatic on startup
+   - Check logs for migration errors
+
+---
+
+## WebSocket/Real-time Updates
+
+### Progress Updates Not Showing
+
+**Symptoms:**
+- Research starts but no progress shown
+- UI appears stuck
+- Results appear suddenly at end
+
+**Solutions:**
+
+1. **Check browser console** for WebSocket errors
+
+2. **Verify SocketIO connection:**
+   - Open browser DevTools > Network > WS
+   - Look for `/socket.io` connections
+
+3. **Authentication / expired session:**
+   - WebSocket connections require an authenticated session.
+   - If your session has expired, or the server was restarted while your tab was open, the handshake is rejected and the UI may silently fall back to HTTP polling with no progress shown.
+   - Log out and back in to restore the connection.
+
+4. **Firewall/proxy issues:**
+   - WebSocket needs persistent connections
+   - Some proxies don't support WebSocket
+   - Try direct connection (no proxy)
+
+5. **Fallback to polling:**
+   - The client automatically falls back to HTTP polling
+   - Check if polling requests are working
+
+### Connection Drops
+
+**Symptoms:**
+- Frequent disconnections
+- "transport close" errors
+
+**Solutions:**
+
+1. **Check network stability**
+
+2. **Adjust timeout settings:**
+   - Default ping timeout: 20 seconds
+   - Default ping interval: 5 seconds
+
+3. **For reverse proxy setups:**
+   ```nginx
+   # Nginx example (see docs/deployment/reverse-proxy.md for the full config)
+   location /socket.io {
+       proxy_pass http://127.0.0.1:5000;   # not "localhost" — may resolve to ::1 first
+       proxy_http_version 1.1;
+       proxy_set_header Upgrade $http_upgrade;
+       proxy_set_header Connection "upgrade";
+       proxy_set_header Host $host;
+       proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+       # Required when terminating TLS at the proxy: without X-Forwarded-Proto
+       # the same-origin WebSocket check (the default) sees http:// and rejects
+       # the browser's https Origin. Also needed for secure cookies / HSTS.
+       proxy_set_header X-Forwarded-Proto $scheme;
+       proxy_read_timeout 86400;
+   }
+   ```
+
+   See [Deploying behind a reverse proxy](deployment/reverse-proxy.md) for the
+   complete nginx/Caddy configuration (uploads, redirects, SSE timeouts).
+
+---
+
+## Docker Issues
+
+### Port Conflicts
+
+#### macOS AirPlay (Port 5000)
+
+**Symptom:** "Address already in use" error or container starts but http://localhost:5000 is unreachable
+
+**Diagnose:**
+```bash
+lsof -i :5000
+sudo lsof -i :5000  # May need sudo for system services
+# If "ControlCe" or "AirPlayXPC" appears → AirPlay is the cause
+```
+
+**Solutions:**
+
+1. **Disable AirPlay Receiver** (macOS 12 Monterey and later):
+   - System Settings → General → AirDrop & Handoff → Toggle OFF "AirPlay Receiver"
+
+2. **Use a different port** (recommended if you need AirPlay):
+   ```yaml
+   # docker-compose.yml
+   ports:
+     - "8080:5000"  # Access at http://localhost:8080
+   ```
+
+   Or with Docker CLI:
+   ```bash
+   docker run -p 8080:5000 ...
+   ```
+
+**Note:** Other services that may use port 5000 include Flask development servers, Synology DSM, and some VPN software. The diagnostic commands above will help identify the culprit.
+
+---
+
+### Container Won't Start
+
+**Symptoms:**
+- Container exits immediately
+- "exec format error"
+- Port already in use
+
+**Solutions:**
+
+1. **Check logs:**
+   ```bash
+   docker logs local-deep-research
+   ```
+
+2. **Port conflicts:**
+   ```bash
+   # Check what's using port 5000
+   lsof -i :5000
+
+   # Use different port
+   docker run -p 8080:5000 ...
+   ```
+
+3. **Architecture mismatch:**
+   - Ensure image matches your CPU architecture (amd64/arm64)
+
+### GPU Not Working
+
+**Symptoms:**
+- Ollama running on CPU instead of GPU
+- "CUDA not available"
+
+**Solutions:**
+
+1. **Use GPU-specific compose file:**
+   ```bash
+   docker compose -f docker-compose.yml -f docker-compose.gpu.override.yml up
+   ```
+
+2. **Verify NVIDIA runtime:**
+   ```bash
+   docker run --rm --gpus all nvidia/cuda:11.0-base nvidia-smi
+   ```
+
+3. **Install nvidia-container-toolkit:**
+   ```bash
+   # Ubuntu/Debian
+   sudo apt-get install -y nvidia-container-toolkit
+   sudo systemctl restart docker
+   ```
+
+### Volume/Permission Issues
+
+**Symptoms:**
+- "Permission denied" errors
+- Data not persisting
+
+**Solutions:**
+
+1. **Check volume ownership:**
+   ```bash
+   ls -la ~/.local/share/local-deep-research/
+   ```
+
+2. **Fix permissions:**
+   ```bash
+   sudo chown -R $(id -u):$(id -g) ~/.local/share/local-deep-research/
+   ```
+
+---
+
+## API Issues
+
+### CSRF Token Errors
+
+**Symptoms:**
+- "CSRF token missing"
+- "CSRF validation failed"
+
+**Solutions:**
+
+1. **Fetch token before requests:**
+   ```python
+   # Get CSRF token from server
+   resp = session.get("http://localhost:5000/auth/csrf-token")
+   csrf = resp.json()["csrf_token"]
+
+   # Include in requests
+   session.post(
+       "http://localhost:5000/api/v1/quick_summary",
+       json={"query": "..."},
+       headers={"X-CSRFToken": csrf}
+   )
+   ```
+
+2. **Use the LDRClient** which handles CSRF automatically:
+   ```python
+   from local_deep_research.api.client import LDRClient
+
+   with LDRClient() as client:
+       client.login(username, password)
+       result = client.quick_research("query")
+   ```
+
+### Authentication Failures
+
+**Symptoms:**
+- "Login required"
+- Session expires unexpectedly
+
+**Solutions:**
+
+1. **Verify credentials:**
+   - Username is case-sensitive
+   - Check for password special characters
+
+2. **Session issues:**
+   - Clear cookies and re-login
+   - Check session timeout settings
+
+3. **For API access:**
+   - Consider using API keys instead of sessions
+   - Check `api.enabled` setting
+
+---
+
+## Performance Issues
+
+### Slow Research
+
+**Symptoms:**
+- Research taking too long
+- High memory usage
+- Timeouts
+
+**Solutions:**
+
+1. **Reduce iterations:**
+   ```
+   search.iterations = 2  # Instead of default 4
+   ```
+
+2. **Reduce questions per iteration:**
+   ```
+   search.questions_per_iteration = 3  # Instead of 5
+   ```
+
+3. **Use a lighter strategy:**
+   ```
+   search.search_strategy = source-based  # Instead of the agentic default
+   ```
+
+4. **Limit search results:**
+   ```
+   search.max_results = 5  # Instead of 10
+   ```
+
+5. **Use snippet-only mode:**
+   ```
+   search.snippets_only = true  # Skip full content retrieval
+   ```
+
+### Memory Issues
+
+**Symptoms:**
+- Out of memory errors
+- System becomes unresponsive
+
+**Solutions:**
+
+1. **Limit concurrent research:**
+   - Reduce queue size
+   - Wait for research to complete before starting new ones
+
+2. **Use smaller models:**
+   - `llama3.2:3b` instead of larger variants
+   - Quantized models (Q4, Q5)
+
+3. **Increase swap space** (Linux):
+   ```bash
+   sudo fallocate -l 8G /swapfile
+   sudo chmod 600 /swapfile
+   sudo mkswap /swapfile
+   sudo swapon /swapfile
+   ```
+
+---
+
+## Resource Exhaustion
+
+### File Descriptor Exhaustion
+
+**Symptoms:**
+- `sqlite3.OperationalError: unable to open database file`
+- `OSError: [Errno 24] Too many open files`
+- Cascading failures across unrelated operations (logging, HTTP requests, WebSocket connections fail simultaneously)
+
+**Why it happens:**
+
+Each SQLCipher WAL-mode connection uses 2 file descriptors (main db + WAL), plus 1 shared SHM fd per database. With per-user encrypted databases, the QueuePool alone uses `users × (pool_size × 2 + 1)` FDs at steady state (41 per user with defaults), up to `users × ((20 + 40) × 2 + 1) = users × 121` under load. The default Linux soft ulimit of 1024 is tight for multi-user deployments.
+
+**Diagnosis:**
+
+```bash
+# Inside Docker (PID 1 is the app due to exec in entrypoint)
+ls /proc/1/fd | wc -l
+cat /proc/1/limits | grep "open files"
+
+# Bare-metal Linux
+ls /proc/$(pgrep -fo ldr-web)/fd | wc -l
+
+# Detailed view — show database-related FDs
+lsof -p <PID> | grep -E '\.db|\.wal|\.shm'
+```
+
+**Solutions:**
+
+1. The app includes automatic credential cleanup (~5 minutes) and periodic pool disposal (every 30 minutes) — this normally handles cleanup transparently
+2. **Docker:** The daemon default FD limit (typically 1M+) is appropriate. Do not set a lower `nofile` ulimit — this was intentionally removed from `docker-compose.yml`
+3. **Bare-metal Linux:** The default soft limit of 1024 may be too low. Increase it:
+   ```bash
+   ulimit -n 65536
+   ```
+4. Restart the application to release all file descriptors
+
+For the technical details of the cleanup architecture, see [Architecture - Thread & Resource Lifecycle](./architecture.md#thread--resource-lifecycle).
+
+---
+
+## Debug Logging
+
+> **Security note:** Log files are unencrypted and may contain sensitive information such as research queries. Ensure appropriate file permissions.
+
+### Enable File Logging
+
+By default, LDR logs to the console. To enable persistent file logging:
+
+```bash
+export LDR_ENABLE_FILE_LOGGING=true
+```
+
+### Log File Locations
+
+| Platform | Path |
+|----------|------|
+| Linux | `~/.local/share/local-deep-research/logs/` |
+| macOS | `~/Library/Application Support/local-deep-research/logs/` |
+| Windows | `%USERPROFILE%\AppData\Local\local-deep-research\logs\` |
+| Custom | Set `LDR_DATA_DIR` environment variable |
+
+**Log files:**
+- `ldr_web.log` - Main application log
+- Logs rotate at 10MB with 7-day retention (compressed)
+
+### Docker Logging
+
+```bash
+# Live log stream
+docker compose logs -f local-deep-research
+
+# Last 100 lines
+docker compose logs --tail 100 local-deep-research
+
+# Follow logs with timestamps
+docker compose logs -f -t local-deep-research
+```
+
+### Verbose File Logging
+
+To capture DEBUG-level output to log files:
+
+```bash
+export LDR_ENABLE_FILE_LOGGING=true
+```
+
+Log files will include DEBUG-level messages. See log file locations above.
+
+**Security note:** Log files are unencrypted and may contain sensitive information such as research queries. Ensure appropriate file permissions.
+
+---
+
+## Getting Help
+
+If you're still experiencing issues:
+
+1. **Check logs:**
+   - Console output
+   - Log files (see [Debug Logging](#debug-logging) above)
+
+2. **Search existing issues:**
+   - [GitHub Issues](https://github.com/LearningCircuit/local-deep-research/issues)
+
+3. **Create a new issue** with:
+   - LDR version
+   - Operating system
+   - Docker/native installation
+   - Steps to reproduce
+   - Relevant logs
+
+---
+
+## See Also
+
+- [Architecture Overview](./architecture/OVERVIEW.md) - System architecture
+- [FAQ](./faq.md) - Frequently asked questions
+- [Search Engines Guide](./search-engines.md) - Detailed engine documentation
+- [Architecture - Thread & Resource Lifecycle](./architecture.md#thread--resource-lifecycle) - Resource cleanup layers and FD budget