learningcircuit--local-deep-research/docs/release_notes/1.8.1.md

### 💥 Breaking Changes

- Removed the `mcp` / `agentic` (ReAct) search strategy. Existing selections are automatically migrated to `langgraph-agent` (a near functional superset); connecting to external MCP servers as research tool sources is no longer supported. (LDR still ships its own MCP **server** for exposing research to assistants like Claude — that is unaffected.) ([#4548](https://github.com/LearningCircuit/local-deep-research/pull/4548))
- **Breaking (programmatic API):** `get_llm(provider=…)` and `get_embeddings(provider=…)` now **fail closed** when called with no settings snapshot (`settings_snapshot=None`) for any provider that is not a known local-default. Snapshot-less callers may instantiate only the localhost-default providers — LLM: `ollama`, `lmstudio`, `llamacpp`; embeddings: `sentence_transformers`, `ollama` — plus LLMs you registered in-process via the programmatic API (`quick_summary(llms={…})`). Any other provider (incl. `openai`, `anthropic`, `google`, `openrouter`, and any future cloud provider) raises `PolicyDeniedError` instead of silently constructing a cloud client. This closes a snapshot-less egress hole; programmatic callers that relied on building a cloud provider without a snapshot must now pass a `settings_snapshot` (or register the LLM in-process). The allowlists are intentionally tight so unknown/new providers fail closed by default.
- **The `auto` and `parallel` meta search engines have been removed** (including the `meta` and `parallel_scientific` aliases). The default langgraph-agent strategy replaces them — it selects search engines dynamically per query, so a separate LLM-based engine picker is redundant. Stored values (`search.tool` setting, news subscriptions, queued researches, saved benchmark configs) are migrated automatically on upgrade (migration 0013; `search.tool` values pointing at a removed engine become `searxng`). **What to do:** API callers passing `search_tool="auto"` or `"parallel"` must pick a concrete engine (e.g. `searxng`), and `LDR_SEARCH_TOOL=auto` environment overrides must be updated likewise.
- **Upgrade behaviour change:** the default egress scope is now `adaptive` (was effectively `both`). Adaptive follows your primary search engine, so on upgrade a config with a *concrete* primary may narrow: a public primary (e.g. SearXNG/arXiv) now excludes local collections (public-only behaviour), and a *private collection* primary forces local LLM + embeddings. To keep the previous "any engine, cloud inference" behaviour, set Egress Scope to **Both**. See docs/egress-modes.md.

### 🔒 Security

- News recommendation logging now sanitizes externally-derived topic strings (and search error text) before interpolating them into log records, closing a log-injection vector where crafted news content could forge log lines or inject terminal escape sequences. ([#3767](https://github.com/LearningCircuit/local-deep-research/pull/3767))
- **`/settings/api` no longer leaks env-overridden API keys to authenticated users.** The bulk settings JSON endpoint now redacts password-typed values (`llm.openai.api_key`, search-engine API keys, etc.) so they come back as `[REDACTED]` instead of plaintext. Metadata is preserved so the UI still renders correctly. Note: the settings form template still pre-fills password inputs server-side — that's a separate UX issue and not addressed here. ([#3947](https://github.com/LearningCircuit/local-deep-research/pull/3947))
- **Settings form no longer pre-fills password inputs with the stored value**, eliminating an authenticated View-Source disclosure of API keys / OAuth tokens. Password fields render empty with a placeholder indicating configuration state and `autocomplete="new-password"` to prevent browser caching. ([#3954](https://github.com/LearningCircuit/local-deep-research/pull/3954))
- Extended `redact_secrets()` coverage to credential-bearing exception handlers in the encrypted-database, web queue, and scheduler subsystems — sites where the user's SQLCipher master password is in lexical scope. Unlike API keys (rotatable), the SQLCipher master password is unrecoverable (TRUST.md §5), so a leak via a rendered traceback or upstream exception message is a permanent compromise. Covers `database/encrypted_db.py` (6 sites in `create_user_database` / `open_user_database` / `change_password`), `web/queue/processor_v2.py:_start_research_directly`, and 11 sites across `scheduler/background.py` (subscription scheduling, document processing, RAG indexing, overdue-subscription handling, subscription research trigger). See #4182. ([#4182](https://github.com/LearningCircuit/local-deep-research/pull/4182))
- Egress policy hardening (PR #4300, Round 6 review): 24 fixes across DNS classification (process-global socket timeout removed, threading.RLock around cache mutations), PDP coverage (nested-dict settings unwrap, NAT64 metadata reclassification, fetch_content tool gate), engine/RAG/data coverage (MetaSearchEngine Wikipedia fallback routed through factory, all 5 direct LibraryRAGService sites pre-flight-checked, SearchCache hash includes scope to close cache-bypass), and UI/cross-cutting work (per-research policy overrides on the research form, settings dashboard renders the policy keys, benchmark snapshot threading, policy_audit WebSocket filter, audit log on policy-key changes, dismiss-flag rename to follow the dismiss_* convention). 23 regression tests added.

  A second adversarial review round added 13 more fixes: closed two egress bypasses (Elasticsearch `cloud_id` is now refused under private scope; OpenAI/discovered-provider model-list probes are scope-gated in both the settings and RAG model endpoints); closed a metadata-SSRF gap (literal cloud-metadata IPs are now classified PUBLIC on the literal-IP branch of `_classify_host`, matching the DNS branch); ADAPTIVE now resolves a registered local-retriever primary to PRIVATE_ONLY (was BOTH, which leaked the private corpus to cloud inference); the PEP-578 audit-hook backstop is now armed on non-web entry points (CLI / news scheduler / programmatic API) and re-armed on ThreadPoolExecutor pool workers; fixed the SearXNG `url_setting` key so a local SearXNG is no longer over-blocked under PRIVATE_ONLY; `allowed_local_hostnames` no longer rejects unresolvable intranet hosts on save; the denied-fetch quota no longer counts benign parse failures; and `DownloadService` fails closed when the settings backend errors. 10 regression tests added.

  Follow-up regression fix: user-registered in-process LLMs (the programmatic API's `llms={...}` and plugins) are no longer denied with `provider_url_unset` when a run resolves to require-local inference (e.g. ADAPTIVE with a registered-retriever primary). The PDP now distinguishes user registrations from auto-discovered built-in providers — a custom name shadowing a built-in cloud provider stays blocked, and snapshot-less calls keep failing closed for everything else. 3 regression tests added.

  Third adversarial review round (5×20 agents) confirmed and fixed 6 more issues (4 of 11 verified findings were declined as not-exploitable or design-intent, 1 was a false positive): the `evaluate_url` cloud-metadata block now normalizes alternate IPv4 encodings (octal `0251.0376.0251.0376`, hex, integer) so an IMDS literal can't read as an allowed public host under PUBLIC_ONLY/BOTH and always denies with the explicit `blocked_metadata_ip` reason; `dangerous_scheme` (javascript:/data:/file: hrefs) no longer counts toward the per-run denied-fetch quota (matching `unsupported_scheme`, so a document full of data: URIs can't starve later legitimate fetches); `evaluate_llm_endpoint` and `evaluate_embeddings` now percent-decode the host before classification (consistency with `evaluate_url`, so a legitimate percent-encoded local endpoint isn't wrongly denied under require-local); `context_from_snapshot` fails closed with `ValueError` on a non-dict snapshot instead of a swallowable `AttributeError`; and the run-scoped DNS classification cache is now first-writer-wins so concurrent disagreeing lookups (round-robin DNS) can't flip a hostname's classification mid-run. 11 regression tests added.

  Round 2 of the 5×20 review (audit-hook + thread-context propagation) confirmed and fixed 4 issues (of 21 findings: 6 false positives, 1 already-fixed, several lower-severity backstop-completeness gaps deferred with the primary PEPs as the live gate): the PEP-578 audit hook now decodes bytes-encoded socket addresses before classifying (CPython fires `socket.connect` with a bytes host, which previously passed straight through the PRIVATE_ONLY/STRICT backstop); `set_active_context` rejects an unresolved ADAPTIVE-scope context (it would have silently no-op'd the hook); `active_egress_context` saves and restores the previous context so a nested activation no longer wipes the parent's; and the default `langgraph-agent` strategy now re-arms the audit-hook backstop inside each subagent ThreadPoolExecutor worker (threading.local isn't inherited by pool workers). 4 regression tests added.

  Round 3 of the 5×20 review (call-site PEPs / data-egress paths) confirmed the empty-snapshot fail-open as the one genuinely exploitable issue and fixed it (most of the 18 findings were lower-severity than first rated — UI-display consistency, config-time validations already backstopped by execution-time PEPs, documented accepted-risk redirects, or backward-compatible empty-snapshot defaults): the `quick_summary` REST endpoint now fails CLOSED (HTTP 503) when the user's settings snapshot can't be loaded, instead of continuing with an empty `{}` snapshot that silently downgraded a configured PRIVATE_ONLY/require-local user to the permissive BOTH scope. News-subscription policy validation now also surfaces an incoherent egress config (STRICT scope + meta-picker primary, which raises ValueError) as a fail-fast misconfiguration error at create/update time rather than silently skipping the check. 1 regression test added.

  Round 4 of the 5×20 review (adversarial bypass hunting) found no new exploitable bypass — every candidate was a known/documented accepted-risk (DNS-rebinding and redirect TOCTOU, per-context fetch quota, connect-only audit-hook scope), a fail-safe over-restriction, or an operator-trust-boundary case (a programmatic caller controls its own snapshot). One cheap defense-in-depth completion was applied: `evaluate_url` now also blocks cloud-metadata endpoints reachable by hostname (GCP `metadata.google.internal` / `metadata.goog`, any case) and strips an insignificant trailing dot before the metadata checks, so `metadata.google.internal.` and `169.254.169.254.` can no longer read as an allowed public host (closing the hostname/trailing-dot gaps the round-1 octal/hex/integer fix didn't cover). 2 regression tests added.

  Round 5 of the 5×20 review (quality / docs / UI / completeness) fixed 5 issues: policy-audit denial logs now redact URLs (scheme://host:port only) via the existing `redact_url_for_log` at all four sites (ContentFetcher, DownloadService, and both full_search paths), so a denied URL carrying userinfo credentials or an API-key query param is no longer written verbatim to the audit log; the ADAPTIVE warning-banner render path (`/api/warnings`) now resolves scope with `allow_dns=False` so a settings-page load can't block up to the DNS timeout on a synchronous getaddrinfo for a URL-engine primary; `CONFIGURATION.md` was regenerated so its Settings List documents the egress keys and their `LDR_*` env vars (the hand-written prose, which the on-merge auto-regeneration would have wiped, moved into the hand-maintained egress-modes.md with the cross-link fixed); plus a missing `_retriever_is_local` return annotation and an added BOTH-scope allow test. 3 regression tests added. ([#4300](https://github.com/LearningCircuit/local-deep-research/pull/4300))
- FAISS RAG indexes are now loaded through a restricted unpickler that only permits the two classes a legitimate docstore contains, instead of `FAISS.load_local(..., allow_dangerous_deserialization=True)`. A tampered `index.pkl` (the docstore pickle that is actually deserialized) can no longer execute arbitrary code on load — closing the pickle-deserialization RCE for every case, including a first-encounter index with no integrity record and a `.pkl`-only swap that leaves the `.faiss` checksum unchanged. The fix needs no integrity record, schema change, or re-indexing. The loader is fail-closed: if a future LangChain/Pydantic upgrade changes the docstore pickle format to use other classes, affected indexes raise `UnpicklingError` on load (and are quarantined and rebuilt) rather than silently deserializing — recover by re-indexing the affected collection, or by widening the loader's allow-list to the new class. ([#4632](https://github.com/LearningCircuit/local-deep-research/pull/4632))
- **WebSocket/Socket.IO connections now default to same-origin only.** When `LDR_SECURITY_WEBSOCKET_ALLOWED_ORIGINS` is unset or empty, cross-origin WebSocket connections are rejected instead of being allowed from any origin — closing a Cross-Site WebSocket Hijacking gap as defense-in-depth (the session cookie's `SameSite=Lax` already blocked the classic cross-site case) and matching the same-origin default already used for HTTP CORS. To run a cross-origin front-end, set `LDR_SECURITY_WEBSOCKET_ALLOWED_ORIGINS` (and `LDR_SECURITY_CORS_ALLOWED_ORIGINS`) to its origin. If you terminate TLS at a reverse proxy, forward `X-Forwarded-Proto` so the same-origin check sees `https` — otherwise the WebSocket handshake is rejected (see the troubleshooting guide). Set `*` to restore the previous allow-all behavior on trusted local/dev networks. A rejected WebSocket handshake is now logged with a warning naming the origin and how to allow it, so a misconfigured origin is diagnosable instead of a silently frozen progress UI. ([#4807](https://github.com/LearningCircuit/local-deep-research/pull/4807))
- **Local file/RAG path validation no longer 500s or blocks legitimate paths.** Browsing to a restricted path on a non-root deployment returned a 500 (an uncaught permission error from a symlink pre-check) instead of a clean "invalid path" response, and legitimately symlinked index folders (Docker/Kubernetes bind mounts, macOS `/tmp`) were wrongly rejected; both are fixed, and the system-directory block now also holds on macOS (where `/etc` is itself a symlink to `/private/etc`). Separately, a `LDR_DATA_DIR` path containing an apostrophe (e.g. `/home/O'Brien/ldr`) is now fully supported: it no longer crashes startup, and encrypted-database backups handle it too (the apostrophe is safely escaped in the backup's `ATTACH DATABASE` statement). ([#4808](https://github.com/LearningCircuit/local-deep-research/pull/4808))
- **Restricted-directory access blocks no longer log the user's path.** When local file/RAG path validation blocks access to a system directory, the error log now names only which restricted directory was hit (e.g. `/etc`) instead of the user's full resolved path, which could contain a username. ([#4820](https://github.com/LearningCircuit/local-deep-research/pull/4820))
- **Local-folder RAG indexing no longer logs the path you submitted when it's rejected.** A failed path validation on the local-folder indexing endpoint logged the raw submitted path (which can contain a username); it now logs a generic "Path validation failed" message instead. The `Invalid path` response is unchanged. ([#4825](https://github.com/LearningCircuit/local-deep-research/pull/4825))
- **Local-folder RAG indexing logs the folder/file name instead of the full path.** The "indexing complete" log and the per-file indexing-error log now record only the folder/file basename rather than the full filesystem path (which can contain a username), while keeping the same diagnostic detail. ([#4831](https://github.com/LearningCircuit/local-deep-research/pull/4831))
- **API error responses no longer leak raw exception text or server config (CWE-209).** Several response paths embedded internal exception detail into messages returned to the client: the `/api/news/*` endpoints (database/SQLAlchemy errors), the start-research and news-subscription egress prechecks (raw `ValueError`s), the research failure handler (unrecognized exceptions *and* LLM/search/provider "configuration error" detail — including server-level endpoints and file paths, since settings can come from `LDR_*` environment overrides) via `/api/research/<id>/status` and the persisted error report, and the benchmark status endpoint (`GET /benchmark/api/status/<id>`, which returned a failed run's raw `str(e)`). All now return generic, category-appropriate messages with an actionable hint where one applies, while the full cause is retained server-side in the logs — hardening shared/multi-user deployments where a non-admin user must not see another tenant's or the server's internal configuration. ([#4843](https://github.com/LearningCircuit/local-deep-research/pull/4843))
- Local-folder RAG indexing now validates the supplied file glob patterns against an allowlist, rejecting any pattern that could escape the indexed folder (e.g. `../../etc/*` or an absolute `/etc/*`). Previously only the base folder was validated, so on multi-tenant/shared deployments a crafted pattern could read server-side files into the requesting user's library. ([#4846](https://github.com/LearningCircuit/local-deep-research/pull/4846))
- Local-folder RAG indexing now confines globbed matches to the indexed folder: a symlink inside the folder that points outside it (a linked file, or a linked directory recursive globbing descends into) is skipped rather than read. Previously such a symlink could expose files outside the folder, since `glob` follows symlinks and only the base folder was validated. Complements the glob-pattern validation in #4846; together they close the folder-escape surface on this endpoint. ([#4848](https://github.com/LearningCircuit/local-deep-research/pull/4848))
- **`GET /history/report/<id>` no longer leaks the settings snapshot (API keys).** The report route returned the persisted `research_meta` wholesale in its response metadata, and `research_meta` includes `settings_snapshot` — which holds all application settings, including API keys, tokens, and base URLs. It now strips `settings_snapshot` via the same `strip_settings_snapshot()` helper its four sibling routes already use, while preserving every other (non-sensitive) metadata field the report view needs. ([#4853](https://github.com/LearningCircuit/local-deep-research/pull/4853))
- Symlink-loop confinement for RAG local indexing now works on Python 3.13+. The check relied on `Path.resolve()` raising on a symlink loop, which newer Python no longer does; it now resolves with `strict=True` so a planted in-base symlink loop is correctly skipped rather than silently kept. ([#4864](https://github.com/LearningCircuit/local-deep-research/pull/4864))
- **Benchmark YAML downloads no longer leak the evaluation `endpoint_url` in the default export.** The default (summary) download is meant to be safe to share, but it still emitted the evaluator's `endpoint_url`, which can be a private/internal host. It's now gated behind the existing **Export → "Include settings snapshot"** opt-in — matching the settings-snapshot gating — so the default summary stays shareable and reproducibility-minded users still get it on demand.
- **Closed three remaining password-leak paths in the settings module.** `GET /settings/api/<key>` and `GET /settings/api/bulk` now redact password-typed values to `[REDACTED]` (matching the bulk `/settings/api` endpoint redacted in the previous release). `POST /save_settings` (the JS-disabled form-encoded fallback) now treats an empty value for a password setting as a no-op, so no submission path can wipe a stored API key with empty string.
- Add a process-wide PEP 578 `sys.audit` hook (`security/egress_audit_hook.py`) that gates `socket.connect` against the active EgressContext. This is the secondary line of defense — every explicit PEP we wired (ContentFetcher, evaluate_llm_endpoint, MCP download_content, journal_reputation_filter, …) remains the primary line and still fires first. The hook catches what those PEPs cannot: a third-party library that opens its own connection, a new code path added without policy awareness, a langchain tool registered by an MCP server, prompt-injection steering a tool into raw `requests.get`, or anything else that ultimately calls `socket.connect`. The hook is installed once on first import of `local_deep_research.security` (idempotent — PEP 578 hooks cannot be removed) and is INACTIVE by default: with no `EgressContext` registered on the current thread, every connect passes through unmodified, so test runners, import-time helpers, and scripts that touch a socket are unaffected. Workers opt in by calling `set_active_context(ctx)` (or using the `active_egress_context` context manager); the research worker in `web/services/research_service.py` now does this after building the snapshot, and the centralized `database/thread_local_session._ThreadCleanup` exit handler clears it on worker shutdown so a pooled thread cannot leak one run's scope into the next task. The hook gates only AF_INET / AF_INET6 — AF_UNIX, AF_NETLINK and friends pass through. It does NOT defend against an adversary with code execution in the LDR process (they can clear the active context or add a passthrough hook themselves); for that, layer OS-level controls per SECURITY.md. 18 unit tests in `tests/security/test_egress_audit_hook.py` pin the contract: install idempotency, get/set/clear, context-manager cleanup on exception, per-scope behaviour against real raw sockets (PRIVATE_ONLY blocks 8.8.8.8 and permits 127.0.0.1; PUBLIC_ONLY mirrors; STRICT blocks public hosts; BOTH allows either), IPv6 loopback bracket handling, AF_UNIX pass-through, and per-thread isolation under concurrency.
- Bumped the Docker base image from python:3.14.5-slim to python:3.14.6-slim, picking up the upstream CPython fixes for CVE-2026-9669 (bz2 decompressor reuse out-of-bounds write), CVE-2026-7774 (tarfile data_filter path-traversal bypass), and CVE-2026-3276 (unicodedata.normalize CPU exhaustion). Also suppressed three base-image CVEs with no fix available in Debian trixie (graphite2 CVE-2026-50593, expat CVE-2026-50219, perl HTTP::Tiny CVE-2026-7010 — none reachable in this container) in .grype.yaml, and removed five suppressions for python CVEs that the 3.14.5/3.14.6 releases fixed.
- Closed a residual SQLCipher-password leak left after the #4182 sweep: `open_user_database` redacted the migration-failure log line but still re-raised `DatabaseInitializationError` with the unredacted original error embedded in its message and chained via `from init_err`. The caller (`thread_local_session`) logs that typed error with `logger.exception`, which re-rendered the chain — and the password its frame locals carry under `diagnose=True` — defeating the redaction. The typed error now carries the redacted message and breaks the chain with `from None` (ADR-0003). See #4182.
- Closed another `diagnose=True` frame-locals leak of the SQLCipher master password from the #4182 sweep: `ThreadSafeMetricsWriter.get_session` opened a per-thread metrics session with `password` live in the frame and logged failures with `logger.exception`, so a rendered traceback could persist the plaintext password (unrecoverable — TRUST.md §5). It now logs with `logger.warning` (no traceback); the exception still propagates to the caller via `raise`, so nothing is swallowed. The module is added to the `test_password_redaction_invariant` allow-list so CI permanently blocks any regression. See #4182.
- Closed the last `diagnose=True` frame-locals leak of the SQLCipher master password from the #4182 sweep: the two consumers of `DatabaseInitializationError` — `thread_local_session.get_session` and `ResourceStatusTracker.__init__` — caught the error and logged it with `logger.exception` while `password` was a live local in the frame, so a rendered traceback could persist the plaintext password (unrecoverable — TRUST.md §5). Both now log with `logger.warning` (no traceback); the redacted detail is already logged at the raise site. The two modules are added to the `test_password_redaction_invariant` allow-list so CI permanently blocks any regression. See #4182.
- Completed the #4182 logging chokepoint: the optional file sink (`LDR_ENABLE_FILE_LOGGING`) now also runs with `diagnose=False`, matching the database and frontend sinks. Previously it still honored `LDR_LOGURU_DIAGNOSE`, so an operator with debug + diagnose + file logging all enabled could persist frame-local credentials (including the unrecoverable SQLCipher master password — TRUST.md §5) into an unencrypted log file. Frame-local exception dumps now render only to the ephemeral stderr sink; no persisted or shipped sink (DB, browser, file) ever renders them. See #4182.
- Completed the analytics-page innerHTML hardening pass: coerced the `frequency_rank` value (`#N` domain rank badge) in `link_analytics.html` with `Number()` — the one numeric sibling left un-coerced next to `usageCount`/`usagePercentage`/`researchDiversity` — and escaped `research_id` in `cost_analytics.html`'s "most expensive research" list (`encodeURIComponent()` in the `href`, `window.escapeHtml()` in the link text, mirroring `star_reviews.html`). Both are defense-in-depth on raw `${...}` interpolation into `innerHTML` with no DOMPurify barrier; neither is currently exploitable (rank is a server-side int, `research_id` is a `uuid4` and the cost-analytics render path is presently disabled), but together they make the PR's "across analytics pages" scope complete.
- Egress policy hardening (Round 7 review): close four PEP gaps surfaced by a 120-agent re-audit and apply the same fix to four sibling sites. (1) LLM gate at config/llm_config.py now uses a known-local allow-list (`ollama`/`lmstudio`/`llamacpp`) when no snapshot is supplied — previously the gate silently no-op'd, letting cloud LLMs instantiate from snapshot-less callers under `llm.require_local_endpoint=true`. (2) `JournalReputationFilter` forwards `settings_snapshot` to `get_llm()` and `create_default` lets `PolicyDeniedError` propagate instead of silently returning None. (3) MCP `_discover_mcp_tools` / `_execute_mcp_tool` block under STRICT / PRIVATE_ONLY (and on absent snapshot), with a policy_audit log and a UI progress signal. (4) `_execute_download_content` narrows its bare `except Exception` so corrupted scope values fail closed instead of dropping to SSRF-only. The same bare-except fail-open pattern is fixed consistently in `notifications/manager.py`, `research_library/services/download_service.py` (paired with a `_policy_locked` short-circuit in the URL check), `web_search_engines/search_engine_base.py` (disables `include_full_content` when policy can't be evaluated), and `web/routes/research_routes.py` (STRICT + meta-picker now surfaces as 400 at run-start). Adds a Flask `PolicyDeniedError` handler so denials escaping a synchronous PEP return a clean 400 with the decision reason. Regression tests: parametrized no-snapshot fail-closed for every cloud provider plus ambiguous (`openai_endpoint`) and hypothetical (`groq`, `mistral`, `cohere`) entries; allow-list passes for the three known-local providers; MCP scope gate (STRICT/PRIVATE_ONLY/no-snapshot/corrupted policy) verifies the connection manager is never touched and the empty list is cached; download_content refuses without constructing ContentFetcher; journal_reputation_filter forwards snapshot and propagates PolicyDeniedError.
- Egress-policy follow-ups after the 5×20 review: (1) ContentFetcher now relaxes a downloader's `SafeSession` to allow private IPs under `PRIVATE_ONLY`, mirroring `policy_aware_validate_url`, so a private/lab URL the policy already approved is no longer rejected by the downloader's own strict SSRF re-validation; (2) the PEP-578 audit-hook backstop is now re-armed in the `NewsAggregationStrategy` analysis ThreadPoolExecutor worker (threading.local isn't inherited by pool workers) and in the document scheduler's worker thread (built from the user's saved settings), so scheduled downloads and news-analysis LLM calls keep defense-in-depth parity under `PRIVATE_ONLY`/`STRICT`. The primary snapshot-based PEPs already gated these paths; this restores the secondary net. (MCP search runs synchronously on the already-armed strategy thread, so it needed no change.)
- Extracted `updateEnhancedDomainList` from the `link_analytics.html` inline script into `static/js/pages/link_analytics_render.js` (surgical extraction mirroring PR #4584) so it can be tested in isolation. Added 7 Vitest XSS regression tests at `tests/js/pages/link-analytics-xss.test.js` covering the escape sites fixed in #3095 (Number()-coercion of `research_diversity`/`usage_count`/`usage_percentage`, `escapeHtml` of research-link queries and classification fields, `encodeURI`/`encodeURIComponent` of domain names and research IDs, and the inline-script wrapper pattern at `link_analytics.html:797-802`). These tests run in PR CI via the existing Vitest job. Added a new `link-analytics` Puppeteer shard for full-page verification of `/metrics/links` in the release pipeline (strict-mode only), with runtime assertions that no `<script>` element is injected and that the "Recent Researches (N total)" header shows a numeric `N`.
- Hardened the SQLCipher-master-password frame-locals leak class (#4182) at two levels. (1) Sink level: `log_utils` now forces `diagnose=False` on the database sink (which persists into the user's own encrypted DB) and the frontend sink (which ships to the browser), so loguru can never render frame-local credentials into a durable or remote sink even when an operator enables `LDR_LOGURU_DIAGNOSE` for local debugging — a single chokepoint covering every credential-bearing exception handler app-wide. (2) Targeted sweep: the credential-centric session helpers `metrics/search_tracker`, `metrics/token_counter`, `database/library_init`, `database/backup/backup_executor`, `web_search_engines/rate_limiting/tracker`, and `research_library/.../research_history_indexer` now log failures with `logger.warning` instead of `logger.exception`, and are added to the `test_password_redaction_invariant` allow-list so CI permanently blocks regressions. Big multi-purpose route/service handlers are intentionally left logging full tracebacks for unrelated errors — they are covered by the sink-level guard. See #4182.
- Operator-configured LLM provider URLs (`llm.ollama.url`, `llm.lmstudio.url`,
  `llm.llamacpp.url`, `llm.openai_endpoint.url`) are now validated against the
  same SSRF rules as outbound HTTP before the LangChain SDK constructor runs.
  Closes an auth-gated SSRF gap where the SDK's internal `httpx` client would
  otherwise bypass the existing `safe_requests` guard.

  Follow-up review closed two more sites in the same vuln class: the
  OpenAI-compatible base class's model-listing (`list_models_for_api`, used by
  the custom-endpoint / LM Studio / llama.cpp providers) now validates
  `base_url` before constructing the OpenAI client and returns an empty list on
  a blocked URL, and `OpenAIProvider` now validates `llm.openai.api_base` before
  handing it to `ChatOpenAI`.
- RAG auto-indexing now applies backpressure to its background worker pool. The `ThreadPoolExecutor` previously had an unbounded internal queue; under sustained upload bursts (possible under the configurable upload rate cap from #3935), this could queue thousands of indexing jobs and exhaust memory. The pool is now capped at 100 in-flight + queued jobs; submissions over the cap are dropped with a clear log warning ("Auto-index queue saturated… trigger a manual reindex if needed"). Documents still upload successfully — only the *automatic* indexing is skipped, and users can manually reindex via the UI.
- Routed every search-engine site that scrubbed errors via `redact_secrets` through a single `BaseSearchEngine._scrub_error()` helper (regex sanitize + literal redaction), replacing ~60 hand-copied call sites. This closes the per-site drift class that could drop a credential, and upgrades several engines whose logged errors previously skipped the regex sanitization pass.
- Search, fetch, and agent tool error messages are now scrubbed of credentials before they reach the LLM and the user-visible research output — a search-engine or LLM exception can embed a request URL carrying an API key, so these are now routed through the same `sanitize_error_for_client` helper used for download errors.
- The MCP search strategy now filters its specialized search-engine tool list by the active egress scope (parity with the LangGraph strategy): forbidden engines never appear in the tool schema the LLM sees, and a policy denial at execution surfaces as an audit log line instead of a generic tool error.
- The ``custom_endpoint`` URL supplied to ``/api/start_research``, ``/api/news/subscriptions`` (POST and PUT/PATCH), and ``/api/followup/start`` is now validated for SSRF at the request boundary (rejecting cloud-metadata / link-local targets, always blocked by ``ssrf_validator``) before any research is queued. This is fail-fast defense-in-depth — the OpenAI-compatible provider re-validates the same URL via ``assert_base_url_safe`` before the LangChain client is built — but the route-layer check rejects early, before a DB row is written or a thread is spawned, and keeps the endpoint out of the logs. The endpoint is normalized exactly as the provider normalizes it, so scheme-less local endpoints (``localhost:11434``, ``192.168.1.10:8000``) are accepted; private IPs and localhost still pass, so local LLM providers (Ollama, LM Studio, vLLM) are unaffected.
- The error-message credential scrubber now also redacts GitHub (`ghp_`/`github_pat_`), AWS (`AKIA`/`ASIA`/…), Slack (`xox*-`/`xapp-`), Google OAuth (`ya29.`) tokens and JWTs — extending coverage to more common credential formats before error text reaches clients or logs.
- The per-run denied-fetch quota (`MAX_DENIED_FETCHES_PER_RUN`) is now aggregated per *run* instead of per `EgressContext`. Each call site (content fetcher, full-content search, download service, the audit hook) builds its own `EgressContext`, so the counter previously reset whenever a new engine/fetcher was constructed — letting a malicious indexed document evade the exhaustion guard by spreading denied fetches across contexts. The counter is now anchored to the run's armed audit-hook context (shared across the whole run, re-armed identically on pool workers), with a safe fallback to the local context for snapshot-less / programmatic callers.
- The server now logs a loud warning at startup when the SQLCipher KDF is weakened below the production floor (reached only in test mode, e.g. ``LDR_TEST_MODE``) while user databases already exist. New databases would be created with this weak at-rest work factor, and — separately — any existing database created at a higher KDF can no longer be opened: its on-disk key is unchanged, but the server now derives a different, weaker key, so decryption fails and login returns a generic "Invalid username or password" 401 for every affected user (the KDF-mismatch symptom class behind PR #4775). Silent on fresh deployments and when the effective KDF is at/above the floor.
- The shared error-message credential scrubber now also redacts OAuth `access_token`/`refresh_token` and other sensitive URL query parameters, Azure `subscription-key`, `Authorization:`/`x-api-key:` headers, and Google (`AIza…`) API keys — strengthening credential redaction for every code path that surfaces error text to clients or logs.
- Under the PRIVATE_ONLY egress scope (and Adaptive resolving to private), notification dispatch via non-http Apprise vendor schemes (`slack://`, `discord://`, `telegram://`, …) is now refused — those send to external vendor APIs and can't be verified local. Address a self-hosted notifier by its `http(s)://` URL instead, which is allowed as a private host. Other scopes are unchanged.
- Wrapped the `research_diversity` interpolation in the link-analytics "Recent Researches (N total)" header in `Number()`, matching the same coercion already applied to the sibling `🔍 N researches` badge in the same template. Closes a defense-in-depth gap (raw `${researchDiversity}` into `innerHTML` with no DOMPurify) left inconsistent by PR #3095. Not currently exploitable — the value is a Python `len(...)` int server-side — but aligns with the PR's stated scope of coercing numeric fields used in template-literal text positions.

### ✨ New Features

- **Star Reviews dashboard gains six new charts.** The `/metrics/star-reviews` page now shows a doughnut distribution, rating-volume overlay on the trends chart, research-mode comparison, LLM satisfaction breakdown (stacked bars), quality-dimensions radar, and a recent-feedback section. Two bugs were also fixed: recent ratings displayed "Invalid Date" and showed generic fallbacks instead of actual query text and mode. ([#3804](https://github.com/LearningCircuit/local-deep-research/pull/3804))
- **Benchmark YAML downloads now show the LDR version and `date_tested` from when the benchmark *ran*, not at download time.** Previously a v1.6.5 run downloaded on v1.6.7 incorrectly stamped v1.6.7 — making cross-run comparison impossible. The full per-key settings snapshot active at the run (API keys redacted) is now captured at run time and is available **opt-in** via the new **Export → "Include settings snapshot"** checkbox; by default the YAML contains only the summary configuration, so it's safe to share without exposing internal URLs or local paths. Benchmarks created before this release show `ldr_version: unknown (pre-0014 run)` and, when settings are requested, `settings: null` — no backfill, but no breakage either. ([#3844](https://github.com/LearningCircuit/local-deep-research/pull/3844))
- Add TinyFish as an optional hosted search engine with structured search results and browser-rendered content extraction. ([#4057](https://github.com/LearningCircuit/local-deep-research/pull/4057))
- Added a "Sampling Seed" field to the benchmark configuration (default 42): the same seed and example counts now select the same benchmark questions on every run, so accuracy numbers are comparable across runs and configurations. The seed is stored with each benchmark run. Leave the field empty to draw a fresh random sample each run (the previous behavior). ([#4516](https://github.com/LearningCircuit/local-deep-research/pull/4516))
- Library searches now warn in the research log when results are incomplete because one or more collections failed, naming the affected collections; failed-collection error messages use collection names instead of internal ids. ([#4520](https://github.com/LearningCircuit/local-deep-research/pull/4520))
- **New "Anthropic-Compatible Endpoint" LLM provider** for self-hosted services that speak the Anthropic Messages API (`/v1/messages`) rather than the OpenAI chat-completions format. Select it under Settings → LLM, set `llm.anthropic_endpoint.url` (and an optional `llm.anthropic_endpoint.api_key` — keyless gateways are supported), and LDR builds a `ChatAnthropic` client pointed at your endpoint. The base URL is SSRF-validated before any request, and a real `ANTHROPIC_API_KEY` in the environment is never sent to a self-hosted endpoint. The official cloud Anthropic provider is unchanged. ([#4581](https://github.com/LearningCircuit/local-deep-research/pull/4581))
- Add per-message **Copy**, **Retry**, and **Delete** actions to chat — the standard ChatGPT/Claude.ai hover-icon pattern. Retry replaces the failed turn and re-submits the same query as a fresh research run; Delete removes the user message, research, and any assistant response for that turn. The History page also gains a Copy-query button on each card and the rerun button is now shown for failed/suspended researches too (previously only successful runs had it). ([#4659](https://github.com/LearningCircuit/local-deep-research/pull/4659))
- **Star Reviews dashboard polish.** Recent-feedback entries now link to their source research, and a screen-reader live region announces when the dashboard refreshes (e.g. after changing the time period), including which charts have no data yet. ([#4793](https://github.com/LearningCircuit/local-deep-research/pull/4793))
- **The search-engine selector is now grouped into labelled bands.** Engines are ordered by a trust/cost gradient with section headers — Favorites, Collections, Academic, Local RAG, Books, Code, News, No API key, API key — instead of one alphabetical list. Starring an engine still pins it to the Favorites band at the top. ([#4814](https://github.com/LearningCircuit/local-deep-research/pull/4814))
- **`/api/v1/health` now reports file-descriptor and thread diagnostics.** Authenticated callers receive a `resources` block (`fd_count`, `fd_soft_limit`, `fd_hard_limit`, `fd_usage_percent`, `thread_count`), and `status` flips to `"warning"` above 70% FD usage — making FD growth observable over HTTP for dashboards and alerting. Anonymous callers still get the basic `status`/`message`/`timestamp`, so the Docker healthcheck is unaffected. FD metrics report `null` on non-Linux platforms. ([#4915](https://github.com/LearningCircuit/local-deep-research/pull/4915))
- **Unified background reconciler that indexes ANY unindexed document.** A single scheduled job now indexes every unindexed document so nothing is permanently missed: (a) library uploads that the immediate auto-index queue dropped when saturated, and (b) research downloads that were never ingested into your library. Enable the **Document Scheduler** and then **"Auto-index documents in the background"** (both off by default) to turn it on — the sweep only runs while the document scheduler itself is enabled. The reconciler is idempotent (already-indexed documents are skipped), processes a small bounded batch per run so a large backlog self-heals gradually, and honors each collection's own embedding configuration. The two cases are budgeted independently, so a backlog of documents that keep failing to index can never starve the other case (research downloads still get indexed even when many library uploads are failing). This replaces both the old library-only sweep and the separate per-research RAG-indexing pass that used to run inside the document scheduler. The legacy **"Generate RAG Embeddings"** setting is kept as a deprecated alias — it now enables this same reconciler — so existing users keep getting their research downloads indexed with no change required.
- A collection's public/private (egress) classification can now be changed after creation, from a "Privacy & Egress" toggle on the collection details page — not only at creation time.
- Add an **Adaptive** egress scope (now the default) that follows your primary search engine: a private primary keeps everything local (and forces local LLM/embeddings inference), a public primary allows public engines, and a meta-picker primary allows both. Existing explicit scopes (Both / Public only / Private only / Strict) remain available. Selecting **Private only** now visibly checks and locks the "require local LLM/embeddings" toggles to reflect that local inference is enforced.
- Add an egress policy module (`security.egress_policy`) that gates search-engine instantiation against a user-declared scope. New settings `policy.egress_scope` (default `both`, preserves current behavior), `llm.require_local_endpoint`, `embeddings.require_local`, and `llm.allowed_local_hostnames` are reserved. Under `strict` scope the factory blocks any engine that isn't the user's primary, closing the previously silent expansion path. The LangGraph agent tool-list filter, LLM/embeddings PEPs, and UI controls ship in the same release (see the related Stage 1b changelog entries).
- Collections can now be classified **public** or **private** (default private). A private collection is excluded from runs under Public-only / Adaptive-public egress scope and forces local LLM/embeddings inference under Private-only / Adaptive-private scope, so its contents never reach a cloud model. Flip the classification with the "Public collection" checkbox on the collection create form. Adds the `collections.is_public` column (migration 0011).
- Collections now have an **"Available to the research agent"** toggle (default on). Turn it off to keep a collection out of the LangGraph research agent's tool list when it isn't needed for agentic research — without deleting it or changing its privacy/egress classification. The flag is exposed on the collection create form and the collection details page, persists via the collections API, and is enforced where the agent builds its specialized search tools (a disabled collection's name never reaches the agent).
- Library collections now show per-collection indexing status ("X of Y indexed" + a "N pending indexing" badge) with a per-card Reindex action. When the optional scheduled-reconciler setting is available, a toggle to index documents in the background on a schedule also appears (otherwise it stays hidden). (#3939/#4627 follow-up).
- When the egress policy blocks something, the user now gets a **clear, actionable message** — what was blocked, why, and exactly which setting to change to allow it (e.g. "blocked because your Egress Scope is Private only → change it in Settings → Privacy & Egress, or turn off 'Require local LLM endpoint'") — instead of a terse machine code. A new `security.egress.guidance.denial_guidance()` helper centralises these messages; it's wired into the research start-up engine check, and denials surface the raw reason code alongside for support.

  The `quick_summary` API also gains an opt-in: pass `"allow_default_settings": true` to deliberately run with default settings (no egress policy) when your settings can't be loaded, instead of the default fail-closed `503`. The 503 itself now explains the cause and the fix (retry / re-authenticate / opt in).
- Wire the egress policy module (added in Stage 1a) into the LLM and embeddings call paths. `get_llm()` now raises `PolicyDeniedError` when `llm.require_local_endpoint=True` and a cloud LLM provider is requested. The OpenAI and Anthropic model-listing endpoints in the settings UI degrade gracefully (empty list + audit log entry) under the same condition. `get_rag_service()` runs a pre-flight check before constructing the service so embedding-policy violations surface immediately rather than after hundreds of chunks have been generated. Defaults are unchanged (`require_local_endpoint=False`, `embeddings.require_local=False`), so existing installs see no behavior change.

### 🐛 Bug Fixes

- REST `/api/v1/generate_report` and `/api/v1/analyze_documents` now load the authenticated user's encrypted-DB settings (API keys, model preference, search tool) instead of silently falling back to application defaults plus `LDR_*` env vars. Like `/quick_summary`, both endpoints now fail closed (HTTP 503) when the settings snapshot cannot be loaded, with `allow_default_settings: true` as the explicit opt-out. ([#3661](https://github.com/LearningCircuit/local-deep-research/pull/3661))
- Starting a follow-up research with no LLM model configured now fails immediately with a clear "Model is required" error (HTTP 400) instead of spawning a worker that dies and leaves the research stuck IN_PROGRESS. ([#3767](https://github.com/LearningCircuit/local-deep-research/pull/3767))
- **Star Reviews analytics now count each rating once.** The `/metrics/star-reviews` queries joined `token_usage` directly, which fans out one row per LLM call — so "Recent Ratings" showed the same research duplicated (far fewer than 20 unique entries) and the LLM/search-engine breakdown counts and averages were inflated by the number of calls per research. These queries now collapse token usage to one row per research before joining. The quality-dimensions radar no longer hides real data when only some dimensions are rated, and reports a per-dimension sample size. The rating endpoint also rejects boolean values (which previously passed the 1–5 check) and overly long feedback. ([#3804](https://github.com/LearningCircuit/local-deep-research/pull/3804))
- The `/api/rate-limiting/cleanup` endpoint now rejects out-of-range or non-integer `days` values with HTTP 400 instead of passing them through to the destructive bulk delete (clamped to `1 <= days <= 365`). ([#3805](https://github.com/LearningCircuit/local-deep-research/pull/3805))
- Wikipedia search no longer floods the log with per-title tracebacks when the MediaWiki API rate-limits a query — the engine now detects the JSON-decode error pattern, logs a single warning, and bails out of the batch with whatever previews it collected. Fetched pages that come back empty (paywalls, JS-only SPAs, blank 200s) and LLM extractions that return an empty summary now short-circuit to `NOT RELEVANT` without locking the URL into the citation collector, so the agent is free to re-fetch it later under a different focus. ([#3844](https://github.com/LearningCircuit/local-deep-research/pull/3844))
- `SettingsManager.get_all_settings()` no longer crashes when the DB contains a row whose `type` column holds an enum value no longer in `SettingType` (e.g. legacy `'CHAT'`-typed rows from a removed feature). The handler now catches `LookupError` alongside `SQLAlchemyError` and falls back to defaults-only — strictly safer than crashing every caller (`/settings/api`, benchmark start, research start, MCP). ([#3947](https://github.com/LearningCircuit/local-deep-research/pull/3947))
- **Submitting an empty value for a password-typed setting is now a no-op** — previously could overwrite the stored secret with `""`. Belt-and-braces fix for the same `[REDACTED]` sentinel issue: a stale browser tab can no longer corrupt API keys by triggering a save. To clear a password setting, clear the source env var or use settings import. ([#3954](https://github.com/LearningCircuit/local-deep-research/pull/3954))
- Fix Ollama `enable_thinking` setting being silently dropped on the live class
  construction path. Reasoning models (deepseek-r1, qwen2.5) now correctly
  receive `reasoning=True/False` per the user's `llm.ollama.enable_thinking`
  setting. Also apply the 80%-of-context-window `max_tokens` cap on every
  provider's `create_llm` (was previously only applied in dead procedural
  code that never ran in production) and unify the optional-API-key
  placeholder string across all providers (`"not-required"`). Note: users
  who set `llm.context_window_unrestricted = false` together with a small
  `llm.context_window_size` will now have `max_tokens` capped at 80% of
  that window for OpenAI/Anthropic too — previously the cap only applied
  to Ollama. For LM Studio and llama.cpp the cap is always active, keyed
  to `llm.local_context_window_size` (default 20480 → effective cap 16384
  with the default `llm.max_tokens` of 30000). `llm.supports_max_tokens =
  false` is now honored on the live path (previously the kwarg was sent
  regardless), and whitespace-only API keys for required-key providers now
  raise a clear error at construction instead of being sent to the API.

  Ollama's `num_ctx` now resolves through the same shared helper as the
  context-overflow bookkeeping, so the reported `context_limit` always
  matches the window the model actually runs with (previously an absent or
  null `llm.local_context_window_size` produced `num_ctx` 4096/server-default
  while overflow detection assumed 8192). The ineffective `max_tokens`
  kwarg is no longer passed to `ChatOllama` (it was silently ignored;
  `num_ctx` is the effective control). A keyless `openai_endpoint`
  configuration is now permitted — it previously raised "API key not
  configured" at construction, blocking keyless local servers (vLLM,
  text-generation-webui); a warning is logged so endpoints that do
  require a key remain diagnosable.

  Also fix `llama.cpp` provider availability check: instances behind an
  authentication proxy now correctly report as available when
  `llm.llamacpp.api_key` is configured. Previously the probe was sent
  without credentials and was always rejected, so users with auth-proxied
  llama-server saw the provider as unavailable regardless of server state. ([#3984](https://github.com/LearningCircuit/local-deep-research/pull/3984))
- **Settings on mobile no longer scrolls forever.** Every section now starts collapsed at the mobile breakpoint (`max-width: 767px`) — page height drops from > 16,384 px to ~7,300 px on a Pixel 5. Desktop is unchanged; typing in the Settings search box still force-expands every surviving section. ([#4032](https://github.com/LearningCircuit/local-deep-research/pull/4032))
- Chat Mode post-merge audit follow-ups (#3891). The send-message API now rejects a non-boolean ``trigger_research`` with HTTP 400 instead of silently coercing a truthy value (e.g. ``"false"``) to ``True`` and launching unwanted research. Chat-session deletion now records the (truncated) username alongside the session id in the audit log, so a stolen-token bulk delete leaves a forensic trail. The frontend now logs (instead of silently swallowing) a failed background title-generation request, and the Markdown chat export documents that its output is a human-readable archive that is not safe to feed back into a Markdown/HTML renderer without escaping. The "Clear History" confirmation now warns that it also permanently deletes all chat sessions and their conversations (it always did, but the dialog only mentioned research history). Accessibility: the chat input gains ``aria-required`` and the welcome suggestion buttons gain descriptive ``aria-label``s. Internal cleanup: the duplicated atomic ``UPDATE … RETURNING`` counter logic in ``ChatService`` is extracted into a shared ``_atomic_increment`` helper. New tests cover the 100+-message pagination cap, render-time Jinja2 autoescaping, and the stricter ``trigger_research`` contract. ([#4427](https://github.com/LearningCircuit/local-deep-research/pull/4427))
- Two pre-existing export/metadata correctness bugs are fixed. RIS citation exports were malformed: the entry builder reused its output accumulator as a scratch variable when reading the title, so each record emitted the raw source text *before* the mandatory leading ``TY  - `` tag and reference managers (Zotero/Mendeley/EndNote) rejected the file. Separately, ``get_subscription_history`` called ``json.loads()`` on ``research_meta`` — a JSON column that SQLAlchemy already deserializes to a dict on read — so the resulting ``TypeError`` was swallowed by a bare ``except`` and every subscription-history item silently lost its headline and topics; it now handles both dict and legacy-string values like the rest of the module. ([#4435](https://github.com/LearningCircuit/local-deep-research/pull/4435))
- Fixed a cross-user data exposure in multi-user deployments: saving any setting emitted a ``settings_changed`` WebSocket event — carrying the changed setting keys **and their raw values**, including plaintext LLM API keys — to *every* connected client on the shared Socket.IO server, where each browser merged them into its local settings cache. Sockets now join a per-user room on connect and ``settings_changed`` is scoped to the owning user's own browser tabs only; a settings change made outside a request context (start-up defaults, background workers) no longer emits at all instead of falling back to a broadcast. Single-user (desktop) deployments were unaffected. ([#4437](https://github.com/LearningCircuit/local-deep-research/pull/4437))
- Fixed three more cross-user WebSocket leaks in multi-user deployments (follow-up to the ``settings_changed`` fix): ``parallel_search_started`` — which carried the user's **raw research query** — and ``engine_completed`` were broadcast to every connected client despite having no frontend listener, and are removed entirely; ``search_engine_selected`` (shown in the research log panel) is now scoped to the owning user's per-user room, resolved from the research worker's search context, and is skipped entirely — never broadcast — when no username is available. ([#4446](https://github.com/LearningCircuit/local-deep-research/pull/4446))
- Fixed benchmark runs with xbench-DeepSearch enabled ignoring the configured number of examples: `XBenchDeepSearchDataset.load()` overrode the base class with its own `num_examples` parameter, so the registry's argument-less `load()` call skipped sampling and queued all 100 (Chinese-language) xbench questions. A run configured for e.g. 50 SimpleQA + 10 xbench questions therefore processed 150 tasks against a recorded total of 60, driving the progress display past 100% and surfacing unexpected Chinese questions. The dataset now samples via the constructor's `num_examples`/`seed` like every other dataset. ([#4451](https://github.com/LearningCircuit/local-deep-research/pull/4451))
- The research progress display and agent-thinking panel now show the actual search engine name (e.g. "DuckDuckGo") instead of the generic internal id "web_search". The LangGraph strategy keeps the stable tool id in the progress event metadata while surfacing the friendly, brand-correct engine name in the message, and the agent-thinking panel renderer now prefers that human-readable message. ([#4470](https://github.com/LearningCircuit/local-deep-research/pull/4470))
- Fixed metrics pages showing "No token usage data yet" despite completed researches (#4457). LLM calls whose provider returns no token usage data (OpenAI-compatible servers that omit `usage` on streamed responses, proxies that strip it, Ollama omitting `prompt_eval_count` for fully-cached prompts) were silently skipped, leaving every metrics panel empty; they are now recorded with zero token counts plus a warning log, so call counts, models, phases and response times still appear. Added an opt-in `llm.openai_endpoint.stream_usage` setting that requests usage stats on streamed responses (`stream_options.include_usage`) for servers that support it (LM Studio 0.3.18+, llama.cpp, vLLM, OpenRouter) — off by default because some gateways reject the parameter with a 400. Also fixed background-thread token writes not updating the `ModelUsage` aggregate, and several /metrics panels ignoring the selected time period: the research count, plus the ratings, link and strategy analytics, fell back to a fixed 30-day (or all-time) window when "3 months" or "1 year" was chosen, because the period vocabularies were inconsistent. All metrics endpoints now share one period map. ([#4473](https://github.com/LearningCircuit/local-deep-research/pull/4473))
- Fixed the v1.7.0 Docker image crashing with SIGILL (exit code 132) on x86 CPUs that support AVX but not AVX2 (e.g. Sandy Bridge / Ivy Bridge era). The ``faiss-cpu`` 1.14.2 wheels dropped the runtime CPU dispatch that 1.13.x shipped (separate generic/AVX2/AVX512 builds) and ship a single ``libfaiss.so`` compiled with AVX2 instructions, so importing faiss at app start-up executed an illegal instruction on AVX-only CPUs. ``faiss-cpu`` is now pinned to the 1.13.x series until upstream wheels restore a non-AVX2 fallback (reported upstream as facebookresearch/faiss#5296; lifting the pin is tracked in #4499). ([#4480](https://github.com/LearningCircuit/local-deep-research/pull/4480))
- Hardened app startup against unrelated broken packages in the user's site-packages: the text-splitter registry now imports the three splitters it uses directly from their `langchain_text_splitters` submodules instead of the package root, whose `__init__` eagerly imports every optional splitter backend (spacy, nltk, konlpy, ...). Previously, a globally installed spacy paired with an incompatible typer/click combination crashed `ldr-web` at startup with `TypeError: type 'Choice' is not subscriptable`. ([#4490](https://github.com/LearningCircuit/local-deep-research/pull/4490))
- Research no longer fabricates citations when every search returns nothing: if no sources are found (or a local collection search fails), the answer now states this explicitly instead of inventing references, the progress log shows an error, and failed collection searches are reported instead of being silently treated as "no matching documents". ([#4503](https://github.com/LearningCircuit/local-deep-research/pull/4503))
- A temporarily unreachable embedding provider (e.g. Ollama not running) no longer quarantines a healthy collection index and silently replaces it with an empty one; the search now fails with a clear provider error and the index survives untouched. ([#4512](https://github.com/LearningCircuit/local-deep-research/pull/4512))
- Fixed intermittent ~60s UI freezes / "Navigation timeout" failures (#4431) caused by heavy third-party imports (matplotlib via the benchmarks package) and synchronous stderr logging blocking the dev server's request pipeline under load. matplotlib/optuna.visualization are now imported lazily (only when benchmark visualizations are generated), and the stderr log sink uses `enqueue=True` so logging never blocks on I/O while holding loguru's handler lock. Note: `enqueue=True` makes stderr logging asynchronous (a background writer thread), so on an abrupt crash the last few buffered log lines may be lost and ordering relative to other sinks can differ slightly. ([#4536](https://github.com/LearningCircuit/local-deep-research/pull/4536))
- Fixed chat-initiated research running without the user's database password after the in-memory session-password store expired (24h TTL) or was cleared by a server/container restart while the session cookie was still valid. In that state the research completed (so the user saw results) but every background metric write was silently dropped, leaving the metrics dashboard empty (part of #4457). The chat `send_message` route now applies the same encryption-aware password guard as the direct and follow-up research routes, returning a clear "session expired — log out and log back in" message before any rows are created. Unencrypted installs are unaffected. ([#4571](https://github.com/LearningCircuit/local-deep-research/pull/4571))
- Fixed a `MemoryError` when listing or clearing history on large libraries — history/report list queries no longer load full report bodies into memory. ([#4574](https://github.com/LearningCircuit/local-deep-research/pull/4574))
- Fixed the rate-limiting panel on /metrics showing all zeros (no tracked engines, no wait times) for every user. The analytics read the `RateLimitAttempt` table, but raw attempt persistence was disabled to prevent database locking under parallel search, so that table is never populated. The panel now derives engine health, wait-time estimates, success rates, and recent attempt counts from the `RateLimitEstimate` data that rate limiting actually persists. (A few raw-attempt-only metrics — rate-limit-event counts and true per-attempt average wait — cannot be reconstructed and are reported as 0 / the learned base wait.) ([#4576](https://github.com/LearningCircuit/local-deep-research/pull/4576))
- **The cloud Anthropic model dropdown was empty.** The model-list route fetched Claude models correctly, but the auto-discovered-provider pass that ran afterward re-listed them with the OpenAI SDK (`Authorization: Bearer`), which 401s against the Anthropic API (it requires `x-api-key`) and overwrote the good result with an empty list. `AnthropicProvider` now lists models via the `anthropic` SDK, so the dropdown populates again. ([#4586](https://github.com/LearningCircuit/local-deep-research/pull/4586))
- Fixed news subscription scheduling regressions from #4492: a manual "Run Now" whose research later failed no longer pushes the subscription out a full refresh interval (it is reset to due so the scheduler retries), the run-now request no longer holds the encrypted-DB session open across the research-start HTTP call, and the subscription folder-update PUT route now keeps `status` authoritative (an `is_active` toggle pauses correctly instead of being ignored by the scheduler). The same folder-update route was also repaired — it previously failed on every call due to a bad relative import and a missing serializer. ([#4587](https://github.com/LearningCircuit/local-deep-research/pull/4587))
- Fixed two news endpoints that returned HTTP 500 for any real subscription: the subscription history modal (it read a non-existent `refresh_count` column; the run count is now derived from research history) and the organized-subscriptions view (it called `.to_dict()` on plain dicts; it now returns the folder-grouped shape the UI renders). ([#4588](https://github.com/LearningCircuit/local-deep-research/pull/4588))
- Fixed the RAG collection `embedding_dimension`, which was always stored as NULL (the metadata helper probed a non-existent `provider.embedding_dimension` attribute); it is now derived from the actual embedding model. Also extended the collection-indexing dedup to the bulk `index-all` route, which previously stored no embedding metadata and never cleaned up stale chunks/indices on force-reindex. ([#4593](https://github.com/LearningCircuit/local-deep-research/pull/4593))
- Fixed the download manager progress popup showing "X / 0 files" instead of the real total. The pending-item count is now taken after the queue is populated, so the denominator reflects the actual number of papers being downloaded. The download manager now also surfaces an explicit alert when there is nothing to download (e.g., all papers already downloaded) or when queueing fails for every selected research session, instead of silently completing with "0 / 0 files". ([#4660](https://github.com/LearningCircuit/local-deep-research/pull/4660))
- Fixed three rate-limiting API endpoints (`/api/rate-limiting/current`, `/api/rate-limiting/status`, `/api/search-quality`) that always returned empty or per-process-cached data because they read from the in-memory `get_tracker()` singleton, which is freshly built per request and never sees other workers' state. They now read directly from the persisted `RateLimitEstimate` table — the same source the `/metrics` panel was switched to in #4576. Together with that change, every rate-limiting analytics surface is now driven by the same authoritative data. Raw `RateLimitAttempt` writes remain disabled (deliberately, per #4576); only the per-engine estimates the rate limiter actually learns are exposed. The `/api/search-quality` response shape changed in the process (`recent_avg_results` / `min_recent_results` / `max_recent_results` / `sample_size` are gone, replaced by `success_rate`); the benchmark page's status-warning JS was updated to consume the new shape, and the old "Very low results" warning (whose underlying per-attempt search-result-count data was never persisted) is replaced by a "High rate-limit failures" warning driven by `success_rate` and `status`. ([#4721](https://github.com/LearningCircuit/local-deep-research/pull/4721))
- WeasyPrint (the PDF-export dependency) is now imported lazily on first use instead of at web-server startup. It was previously imported during blueprint registration via `research_routes`, adding ~20s of "cold heavy-import" to server cold start on CPU-constrained CI runners — a contributor to the `Navigation timeout 60000ms` UI-shard flakiness in #4431. PDF export behaviour is unchanged. ([#4734](https://github.com/LearningCircuit/local-deep-research/pull/4734))
- **Star Reviews "Unknown" search-engine bucket no longer double-counts.** The search-engine breakdown attributed any research that made non-search LLM calls (whose token rows have no `search_engine_selected`) to the "Unknown" bucket *in addition to* its real engine, inflating "Unknown" toward every research. Ratings with a recorded engine now count under that engine only; "Unknown" holds only ratings with no recorded search engine. ([#4786](https://github.com/LearningCircuit/local-deep-research/pull/4786))
- **Star Reviews merges the "Unknown" buckets.** The LLM and search-engine breakdowns no longer split unidentified entries across separate bars — an empty string, the lowercase `unknown` sentinel written for an unidentified model, and the `Unknown` shown for ratings with no recorded model/engine now collapse into a single "Unknown" bucket (real model names keep their exact casing). ([#4790](https://github.com/LearningCircuit/local-deep-research/pull/4790))
- **LangGraph agent no longer crashes with `BadRequestError` on reasoning-mode LLMs.** `ProcessingLLMWrapper` (the central `<think>` stripper applied to every LLM returned by `get_llm`) previously let `bind_tools` bypass the stripper via its `__getattr__` shim, so `langgraph_agent_strategy.create_agent()` ended up calling the raw reasoning model. Qwen 3.x and deepseek-r1 then emitted `<think>…</think>` text directly inside tool-call `content`, which the OpenAI-compatible provider rejected with `400 Failed to parse input at pos 0: <think>…`. `bind_tools` is now overridden to re-wrap the tool-bound model so the stripper survives the agent tool-binding step (#4804). ([#4804](https://github.com/LearningCircuit/local-deep-research/pull/4804))
- **TinyFish search engine no longer logs your search query on request errors.** When a TinyFish Search/Fetch call failed, the request exception (which embeds the query in the URL) was written to the error log; failures now record only a static message plus the HTTP status. Also restores `ruff format` on the TinyFish test files so the pre-commit gate passes. ([#4806](https://github.com/LearningCircuit/local-deep-research/pull/4806))
- **Star Reviews announces the real load error to screen readers.** A failed refresh now speaks the actual error message via the dashboard's live region instead of a generic "failed to load", and a no-op clear that never re-announced was removed. ([#4828](https://github.com/LearningCircuit/local-deep-research/pull/4828))
- Server startup no longer eagerly imports the heavy `langchain_text_splitters`/sentence-transformers stack; it is loaded lazily (and thread-safely) on the RAG indexing path instead. This speeds up boot and fixes intermittent UI-test server-start failures, and avoids a rare concurrent-import error that could silently skip document indexing right after a restart. ([#4829](https://github.com/LearningCircuit/local-deep-research/pull/4829))
- Journal-quality data downloads now follow OpenAlex's 2026-06 "standard-format" snapshot layout (`data/jsonl/<entity>/manifest.json`). The previous paths started returning 404, breaking the OpenAlex sources and institutions downloads. ([#4830](https://github.com/LearningCircuit/local-deep-research/pull/4830))
- The manual **Run now** button and the manual overdue-subscription sweep now actually start research. Both previously made a server-internal loopback HTTP POST to `/research/api/start`, which sits on a CSRF-protected blueprint and so always failed with HTTP 400 ("CSRF token is missing") — the "Run now" button reported `Failed to start research: 400`. They now invoke the research handler in-process, bypassing the HTTP/CSRF layer while still resolving the user's encrypted-database credentials. (Scheduled subscription runs were unaffected.) ([#4834](https://github.com/LearningCircuit/local-deep-research/pull/4834))
- A failed `commit()`/`flush()` inside a `get_user_db_session()` block no longer poisons the thread's reused database session. The context manager now rolls the session back (and re-raises) when the block errors, so a single failed transaction can't cascade `PendingRollbackError` into every later operation on that thread. Paths that catch and swallow such a failure while reusing the session are also recovered explicitly: a PDF download or text extraction that fails mid-write, and a multi-file library upload where one bad file would otherwise fail the whole batch. ([#4841](https://github.com/LearningCircuit/local-deep-research/pull/4841))
- A rare connection-level failure while auto-indexing a freshly downloaded document no longer leaves the request's database session in a state that fails the next operation. The best-effort auto-index step now recovers the shared session if its lookup fails, continuing the session-rollback hardening from the previous release. ([#4862](https://github.com/LearningCircuit/local-deep-research/pull/4862))
- Under `embeddings.require_local=True` (or any `PRIVATE_ONLY` egress scope, which forces it), searching a library collection whose embedding model was indexed by bare name — e.g. the shipped default `all-MiniLM-L6-v2` — raised `policy_denied: embeddings_model_not_cached` even when the model was present in the local HuggingFace cache. The pre-flight cache check used the bare name as the `repo_id`, but the SentenceTransformer loader requests the namespaced form (`sentence-transformers/all-MiniLM-L6-v2`) and the HF hub cache is keyed on whichever form the loader requests. The check now probes both forms (matching the loader's resolution for both SBERT-curated names and the `basic_transformer_models` allowlist of vanilla transformers like `bert-base-uncased`), so a correctly cached model is admitted. ([#4887](https://github.com/LearningCircuit/local-deep-research/pull/4887))
- Fixed a circular import that made `import local_deep_research.domain_classifier.models` (and importing the `domain_classifier` package before `database.models`) crash with `ImportError: cannot import name 'DomainClassification' from partially initialized module`. The declarative SQLAlchemy `Base` now lives in a dependency-free `database.base` leaf that model modules import without triggering the full `database.models` package, so import order no longer matters. `Base` identity and `Base.metadata` are unchanged. ([#4910](https://github.com/LearningCircuit/local-deep-research/pull/4910))
- Under `embeddings.require_local=True` (or any `PRIVATE_ONLY` egress scope), a collection search whose embedding model is configured as an existing **local directory path** was wrongly refused with `policy_denied: embeddings_model_not_cached`. The `require_local` pre-flight probed the path as a HuggingFace `repo_id`, which raised a validation error that was swallowed and treated as a cache miss — even though the SentenceTransformer loader resolves such a path locally and never contacts the hub. The pre-flight now mirrors the loader's `os.path.exists` guard and admits an existing local path (blank/degenerate names still fail closed), so a local model directory is no longer false-denied. ([#4924](https://github.com/LearningCircuit/local-deep-research/pull/4924))
- **Benchmark YAML exports now quote and escape free-form string fields** (`model`, `model_provider`, `search_engine`, `dataset`, and the evaluator's `model`/`provider`/`endpoint_url`). They were previously interpolated raw, so a value containing a YAML-special character — a colon-space, `#`, quote, or newline — could produce malformed YAML (or inject an unintended key). These fields now go through the existing `yamlEscape()` helper, matching how the settings-snapshot block already serializes strings. Numeric fields and the controlled-format `date_tested`/`ldr_version` are left as-is.
- Benchmark results now persist reliably instead of intermittently showing "Found 0 results" with no progress. The request-thread and worker-thread result syncs could both insert the same row (or a dataset repeating a question could produce a duplicate `query_hash`), tripping the `benchmark_results` unique constraint; because that failed at commit time it rolled back the whole pending batch, discarding results that had actually completed. Result persistence is now serialized and deduplicated, so concurrent syncs and duplicate questions are handled cleanly.
- Cap persisted log messages at 5000 chars so long langgraph runs no longer accumulate 10 KB ``ResearchLog`` rows. Mirrors PR #4004's frontend truncation discipline, now applied to the database sink. Also promotes the duplicated ``/history/logs`` pagination-cap literals into a single shared constant (``HISTORY_LOGS_HARD_CAP`` / ``HISTORY_LOGS_DEFAULT_LIMIT``, exposed to the log panel as ``window.LDR_LOG_LIMITS``) so the route clamp and the panel's DOM cap no longer drift. Full diagnostics remain available in stderr/file sinks.
- Collection indexing now behaves identically whether started via the streaming
  (SSE) route or the background worker. The two paths had duplicated and drifted:
  the background worker did not persist a collection's ``embedding_dimension`` (so
  collections indexed in the background lost it), and the SSE route skipped the
  force-reindex cleanup that clears old chunks and FAISS indices (so a streamed
  force-reindex could leave stale, mixed-model vectors behind). The shared
  embedding-metadata, force-reindex cleanup, and document-query logic is now
  factored into single helpers used by both paths.
- Converting research history into the searchable library now pages through reports in bounded batches instead of loading every report body into memory at once, preventing `MemoryError` on large histories.
- Detailed-mode research reports now honor the ``report.enable_file_backup`` setting. The detailed completion path saved ``report_content`` via a raw ORM write that bypassed the report-storage abstraction, so a user who enabled file backup got on-disk ``.md``/``_metadata.json`` files for quick-mode research but silently never for detailed-mode research. Route the detailed save through ``get_report_storage().save_report()`` exactly like the quick and error paths already do (DRY-review finding H2).
- Emojis in exported PDFs no longer render as empty "tofu" boxes. The default WeasyPrint stylesheet now lists ``"Noto Color Emoji"`` (with ``"Noto Emoji"`` as a monochrome fallback and ``"Segoe UI Emoji"`` / ``"Apple Color Emoji"`` for native installs) at the tail of both the body and ``code``/``pre`` font-family stacks, and the official Docker image now bundles ``fonts-noto-color-emoji`` alongside ``fonts-noto-cjk``. Non-Docker users on Linux without an emoji font installed still need to install one (e.g. ``apt install fonts-noto-color-emoji``) — see ``docs/faq.md``. This change also fixes a pre-existing flaw in the ``custom_css`` path: ``markdown_to_pdf(custom_css=...)`` previously *replaced* the default stylesheet entirely, silently dropping the CJK fallback from #4055; it now layers custom CSS on top of the default so the font fallbacks survive.
- Fixed 5 `TestQuickSummary` API tests that broke after the egress fail-closed change (#4300): the `quick_summary` endpoint re-imported `get_user_db_session` locally, shadowing the module-level binding so `patch("...web.api.get_user_db_session")` never intercepted — the tests passed only because the old code swallowed the resulting DB error. Removed the redundant local import (one consistent patch target) and updated the settings-failure test to assert the new fail-closed `503` behaviour instead of the obsolete empty-snapshot fallback.
- Fixed ScaleSerp silently returning zero results when the API includes a result with a `null` link: the eagerly evaluated `link[:50]` display fallback raised `TypeError`, which the outer handler swallowed along with the entire result set.
- Follow-up relevance filtering now rejects negative and out-of-range source indices and deduplicates repeated ones, so a malformed LLM response can no longer select the wrong source or list the same source twice.
- Harden the central `ProcessingLLMWrapper` think-tag stripping against non-string responses. `remove_think_tags` is text-only, so it now runs only when `response.content` is a `str`; non-string content (e.g. provider content-block lists such as Anthropic's, or `None`) is passed through unchanged instead of raising `TypeError`. This removes the wrapper-level crash that the per-site `str(...)` guards used to absorb before they were dropped. Note: a few direct `.content` consumers still assume string content, so robust list-content handling at those call sites is tracked separately.
- Honor ``?limit`` on ``/api/research/<id>/logs`` (the route the log panel actually fetches). It previously ignored the parameter and returned every row, so a long langgraph run could still ship thousands of rows to the browser despite the panel requesting a bounded tail. The limit now returns the newest N rows (oldest-first, unchanged ordering) clamped to ``HISTORY_LOGS_HARD_CAP``; omitting ``?limit`` preserves the existing return-all contract for direct API callers.
- Library document indexing, collection document listing, and filesystem sync no longer load every document's full text body into memory at once — the large `text_content` column is now deferred (and a document's "has text" flag is computed in SQL), preventing `MemoryError` on large libraries.
- Library/collection uploads now actually accept ODT, DOCX, PPTX and XLSX/XLS files. These formats were advertised as supported but failed at runtime with a swallowed ``ModuleNotFoundError`` because their ``unstructured`` parser dependencies (``python-docx``, ``python-pptx``, ``openpyxl``, ``msoffcrypto-tool``, ``xlrd``) were never installed — only PDF worked. Those parsers are now declared dependencies, and the loader registry probes each format's real runtime dependency so it only advertises formats it can actually extract: missing deps now yield a clear "Unsupported format" instead of a silent failure. Legacy binary ``.doc``/``.ppt`` (which need a LibreOffice ``soffice`` binary) and image OCR (which needs ``pytesseract`` + ``tesseract``) are offered only when those tools are present.
- Link analytics (`GET /api/link-analytics`) now projects only the columns it needs plus a SQL-level "has preview" flag, instead of loading full research-resource rows. This avoids materializing every resource's `content_preview` text on the whole-table scan that runs when no time filter is applied (#4560).
- Make research-log ordering deterministic when rows share a timestamp. ``ResearchLog.timestamp`` is not unique, so ``order_by(timestamp.desc()).limit(N)`` left the rows surviving a shared-timestamp boundary SQL-undefined — under dense logging the newest-N tail (and the "latest milestone" lookup) could vary between requests. Add ``ResearchLog.id`` as a tie-break across all three log-ordering sites (the ``/api/research/<id>/logs`` route, the ``get_logs_for_research`` helper behind ``/history/logs``, and the latest-milestone query). Follow-up to #4645.
- News subscriptions now treat the ``status`` column as the single source of truth for whether a subscription is active. Previously the scheduler decided what to run by filtering on the separate ``is_active`` boolean, which ``create_subscription`` never sets (it writes only ``status``), so a subscription created as *paused* kept ``is_active=True`` by default and was still run by the scheduler. The active/overdue query was also spelled three different ways across the scheduler and the news API, and one copy omitted the ``next_refresh is not None`` guard. All of these now go through ``NewsSubscription.active_filter`` / ``due_filter``. Manually running a subscription ("run now") also reads its saved model/provider/strategy/search-engine from the database (it previously read a trimmed dict that dropped those fields, so the manual run ignored the subscription's configured model) and advances the refresh schedule on success so an overdue subscription is not immediately re-run by the scheduler. The shared run payload and refresh-schedule arithmetic are extracted into ``news/subscription_runner.py``. The post-completion refresh-time update in the research service (which previously called a subscription-storage helper that referenced non-existent columns and so always failed silently) now works, and the subscriptions list API now returns each subscription's saved model/provider/strategy/search-engine instead of dropping them.
- Parse ``LDR_TEST_MODE`` as a proper boolean when deciding whether to relax the SQLCipher KDF floor. Previously ``_get_min_kdf_iterations()`` used a bare truthiness check, so any non-empty value — including ``LDR_TEST_MODE=0`` or ``=false`` — enabled the relaxed test minimum, silently weakening database encryption for anyone setting the flag explicitly to disable it. ``PYTEST_CURRENT_TEST`` stays presence-based (pytest sets it to a non-boolean string).
- REST `/api/v1/analyze_documents` now rejects unknown body parameters with a clear 400 listing the allowed ones (previously a typo like `max_result` surfaced as an opaque 500), and the `/api/v1/` docs endpoint documents the `allow_default_settings` flag for all three research endpoints.
- Remove redundant per-site `<think>`-tag stripping now that `get_llm` LLMs are normalized centrally in `ProcessingLLMWrapper`. Direct `self.model.invoke(...).content` is already think-stripped, so 11 scattered stripping calls (6 `remove_think_tags(...)` and 5 `get_llm_response_text(...)`) across the constraint/evidence/knowledge components, the synthesis path, and the document-analysis API were dropped (relying on the single central strip point), keeping explicit handling only where it is load-bearing (agent/`bind_tools` paths, injected LLMs). Empty-answer fallbacks now log a warning so a silent degrade is visible.
- The "download all" and "queue all undownloaded" library actions now project only the columns they use instead of loading full `ResearchResource` rows, avoiding materializing every resource's `content_preview` text on a whole-table scan for large libraries (#4560).
- The `/api/history` endpoint now paginates its results (default 200, max 500) instead of loading every research row — and its `research_meta` JSON — into memory at once.
- The library documents API now clamps the `limit`/`offset` query parameters, so a negative `limit` (which SQLite treats as "no limit") can no longer bypass pagination and load an entire collection into memory.
- The library domain-filter scan (`get_unique_domains`) now streams document URLs in batches instead of loading every row at once, avoiding excessive memory use on very large libraries (#4560).
- The rate-limiting reset (`POST /api/rate-limiting/engines/<engine>/reset`) and cleanup (`POST /api/rate-limiting/cleanup`) endpoints now operate on the persisted `RateLimitEstimate` table instead of the per-request `get_tracker()` singleton. Like the read endpoints fixed in #4721, the tracker's mutation path is gated on a research-session context that is absent in an analytics HTTP request, so these endpoints were silent no-ops that never cleared the estimates the `/status` and `/current` panels display. Reset now deletes the engine's stored estimate (so it re-learns), and cleanup deletes estimates not updated within the requested window.
- The research-session filter dropdown on the library page no longer loads every research row unbounded; it is now capped to the most-recent sessions, avoiding excessive memory/DOM use on very large histories (#4560).
- The three egress warning banners (public-egress, cloud LLM, cloud embeddings) now have separate dismiss flags. Previously they shared one, so dismissing the fresh-install public-egress notice also permanently hid the critical cloud-LLM and cloud-embeddings warnings.
- When testing a notification webhook URL through the settings UI, the
  "Test" button now surfaces the validator's reason instead of a generic
  "Invalid notification service URL." message. For private/internal IP
  rejections that the operator can unblock (loopback, RFC1918, CGNAT,
  link-local, IPv6 private), the error message names
  `LDR_NOTIFICATIONS_ALLOW_PRIVATE_IPS`; for NAT64-wrapped non-metadata
  destinations on IPv6-only deployments (RFC 6052 well-known
  `64:ff9b::/96` or RFC 8215 local-use `64:ff9b:1::/48`) it names
  `LDR_SECURITY_ALLOW_NAT64` — the only flag that can unblock those, which
  the hint probes for independently. Cloud-metadata IPs are always blocked and
  the env-var hint is intentionally suppressed for them — neither flag
  re-opens metadata, so naming them would mislead the user.

  Also adds an "IPv6-only deployments (NAT64)" subsection to
  `docs/SearXNG-Setup.md` so operators routing IPv4 through NAT64 know
  about the opt-in.
- `get_llm_response_text` now extracts the text from list-type LLM content blocks (Anthropic extended-thinking / tool-use responses, where `message.content` is a list of blocks) instead of stringifying the list to its Python `repr`. This fixes garbled entity, sub-query, and candidate parsing, and stops direct `.content` consumers from crashing on list content.

### 🗑️ Removed

- Dropped the orphaned `cache` and `search_cache` tables and removed their unused
  `Cache`/`SearchCache` models (migration 0016). Neither table was ever populated
  by any code path, so existing databases lose no data — the empty tables are
  removed automatically on the next migration. ([#drop-orphaned-cache-tables](https://github.com/LearningCircuit/local-deep-research/pull/drop-orphaned-cache-tables))
- Removed the `/api/v1/quick_summary_test` REST endpoint. It was a near-duplicate of `/quick_summary` with hardcoded test defaults; call `/quick_summary` with `search_tool`, `iterations`, and `temperature` set explicitly instead. ([#3661](https://github.com/LearningCircuit/local-deep-research/pull/3661))
- Removed the benchmark result-reuse feature: new benchmark runs no longer silently import results from previous completed runs with matching search settings. The compatibility check covered only a handful of settings (ignoring evaluation config, dataset versions, and the rest of the configuration), so results produced under different conditions could contaminate a run's accuracy, and the reuse accounting could report more than 100% completion. Every run now researches its own sampled questions and reports only its own results. ([#4498](https://github.com/LearningCircuit/local-deep-research/pull/4498))
- Removed the DOAJ Seal quality tier (score 8) and the Tier 4 "+1 Seal bonus": DOAJ retired the Seal in April 2025 and removed it from its metadata, so the tier could never be earned again and only ever fired on stale pre-2025 data. DOAJ-listed journals keep their score-5 floor. The `has_doaj_seal` column was dropped from the reference DB (schema version 4 — the DB rebuilds automatically on first use) and the Seal star/count no longer appear on the journal-quality dashboard.
- Removed the experimental search strategies that were only reachable through the "Show All Strategies" toggle (rapid, parallel, iterative, recursive, adaptive, smart, standard, iterdrag, browsecomp, evidence, the constrained/dual-confidence family, modular, and others), along with the toggle itself. The strategy dropdown now offers source-based, focused-iteration, focused-iteration-standard, topic-organization, mcp, and langgraph-agent. The news feature's internal news-aggregation strategy is unaffected.
- Removed the now-unused `PathValidator.confine_to_base` helper (and its tests). It was added only to confine the local-folder RAG indexing route, which has been removed, leaving it with no callers.
- Removed the unused `GET /research/api/config` endpoint. Every field it returned (`version`, `llm_provider`, `search_tool`, `features.notifications`) was read from `current_app.config` keys that are never set, so it only ever returned hardcoded placeholder values. It had no frontend or production callers. Use `GET /research/api/settings/current-config` for live configuration instead.
- Removed the unused `PricingCache._load_cache` and `PricingCache._save_cache` methods. They were left behind as deprecated no-op stubs (empty `pass` bodies) when the disk-backed pricing cache was replaced with an in-memory bounded `TTLCache`, and they had no callers anywhere in the codebase.
- Removed the unused ``news.subscription_manager`` storage subsystem
  (``SQLSubscriptionStorage``, ``SearchSubscription``, ``TopicSubscription``,
  ``BaseSubscription`` and their factories), the abstract ``SubscriptionStorage``
  interface, and ``StorageManager.get_user_subscriptions`` /
  ``get_user_stats``. This code was never reached by any live path -- all real
  subscription functionality goes through ``news.api`` and the scheduler -- and
  was broken against the current ``NewsSubscription`` model (it referenced
  columns such as ``user_id``/``refresh_count``/``results_count`` that do not
  exist, so it raised ``AttributeError`` whenever called). The package-level
  exports ``SearchSubscription`` and ``TopicSubscription`` from
  ``local_deep_research.news`` are removed as part of this; nothing in the
  codebase imported them.
- Removed two orphaned RAG HTTP endpoints that had no UI and no other caller: `GET /library/api/rag/index-local` (local-folder indexing — superseded by collection-based indexing) and `POST /library/api/rag/index-research` (which called a method that no longer exists and always errored). The unused `index_local_file` service method and glob-pattern allowlist were removed with them.

### 📝 Other Changes

- The available-models endpoint no longer hand-rolls Ollama/OpenAI/Anthropic model listing that was immediately overwritten by provider auto-discovery — auto-discovery is now the single fetch path. This removes a redundant per-refresh network round-trip to each provider, and local-only model listing now goes through the provider classes (which validate the URL for SSRF and send auth headers, so an auth-protected local Ollama now lists its models). ([#4646](https://github.com/LearningCircuit/local-deep-research/pull/4646))
- The two Ollama health-check endpoints (`/check/ollama_status`, `/check/ollama_model`) now share a single `/api/tags` probe helper instead of separately re-implementing the fetch + new/old API-format parsing + error classification, so "is Ollama up?" answers consistently across them. No change to the endpoints' responses. ([#4650](https://github.com/LearningCircuit/local-deep-research/pull/4650))
- Add a comprehensive Puppeteer UI suite for the egress-policy feature: `tests/ui_tests/test_egress_policy_ui.js` (CI-safe, 32 assertions) covers every UX touchpoint — Privacy & Egress panel rendering, all four scopes painting their expected border/text colors (amber/teal/indigo palette verified via computed-style reads, not just DOM state), the STRICT-on-meta-picker guard reverting to "Both" with an explanatory hint, per-research overrides round-tripping through the settings DB, scope cue propagating to /history, /metrics and /chat via base.html's `body[data-scope]`, settings dashboard listing the policy keys, and the require-local toggles persisting across reloads. A second file, `tests/ui_tests/NO_CI_test_egress_policy_live_research.js`, drives end-to-end research runs against a real Ollama + SearXNG instance (defaults to a lab box; overridable via env) and verifies the run-start PEP at `research_routes.py:248` — STRICT+searxng succeeds, STRICT+auto returns 400 with the meta-picker incoherence message, PUBLIC_ONLY refuses a SearXNG whose instance URL resolves to a private IP (`scope_mismatch_public_only`), PRIVATE_ONLY accepts the same SearXNG (URL classifier overrides the static `is_public=True`), and a cloud LLM under `require_local_endpoint=true` is accepted at the precheck but reaches `failed` status as the strategy's `get_llm()` hits the PEP. Both files gate screenshots behind `!process.env.CI` matching the `tests/ui_tests/test_settings_page.js:57` convention; the live test's `NO_CI_` filename prefix means the runner skips it in CI (where the lab endpoints are unreachable). Screenshots land in `tests/ui_tests/screenshots/egress-policy{,-live}/` (already gitignored).
- CI: set ``LDR_TEST_MODE=1`` on every test/scan workflow that starts a server or initialises the test database, so the intended fast SQLCipher KDF (``LDR_DB_CONFIG_KDF_ITERATIONS=1000``) is actually honoured. ``_get_min_kdf_iterations()`` deliberately ignores generic ``CI``/``TEST_ENV`` and only relaxes for ``PYTEST_CURRENT_TEST``/``LDR_TEST_MODE``; without the latter the requested 1000 was silently clamped to the production minimum (256000), making every DB open and the post-login background backup ~256× slower. Under contended CI runners that turned the backup into a multi-minute, GIL-holding stall that timed out release-gate UI shards (chat/settings navigations) — the failures seen on clean ``main`` while investigating #4430. Also renames the deprecated ``LDR_DB_KDF_ITERATIONS`` env var to its canonical ``LDR_DB_CONFIG_KDF_ITERATIONS`` across all six workflows (docker-tests, nuclei, owasp-zap-scan, playwright-webkit-tests, puppeteer-e2e-tests, docker-multiarch-test).
- Consolidate the egress-policy code into a dedicated `security/egress/` subpackage (`policy.py` + `audit_hook.py`) with a README documenting the design, scope model, and the full map of enforcement points. No behavior change — module paths moved from `security.egress_policy` / `security.egress_audit_hook` to `security.egress.policy` / `security.egress.audit_hook`.
- Extracted the benchmark YAML export helpers (`yamlEscape` / `formatSettingValue` / `formatSettingsSnapshot`) from the inline `benchmark_results.html` template into a unit-tested module (`static/js/utils/yaml_export.js`, loaded via `<script src>` like the other shared JS utils). No behavior change — it adds regression coverage (`tests/js/utils/yaml_export.test.js`) for the escaping that guards every benchmark YAML download, so a future change to `yamlEscape` can't silently corrupt exports.
- Hardened the #4804 `<think>`-stripper fix: corrected the now-stale `bind_tools` comments in the LangGraph agent, made the async wrapper test actually assert stripping, and added a real `create_agent` integration regression test (sync + async).
- Moved `ProcessingLLMWrapper` out of the `wrap_llm_without_think_tags` closure to module scope in `config/llm_config.py` — it's now importable and `isinstance`-checkable, and defined once instead of rebuilt on every call. Pure refactor; no behavior change.
- Recolor the egress-scope visual cues to match the actual data-leak risk hierarchy. Previously BOTH (the default) wore a light-amber accent and PUBLIC_ONLY a deeper-amber one, which reads as "PUBLIC_ONLY is more dangerous than the default" — but BOTH is actually the riskiest scope: it permits local engines AND a cloud LLM in the same run, so local library/RAG chunks can land in OpenAI via the LLM call. PUBLIC_ONLY blocks local engines, so local data never enters that pipeline. New palette: BOTH gets no left-border accent at all (the existing "Public search egress enabled" banner does the nagging in text); PUBLIC_ONLY uses sky-blue (#0ea5e9 — informational "you chose public sources only"); PRIVATE_ONLY stays teal (#14a37f); STRICT moves from indigo to violet (#8b5cf6) so it's clearly distinct from the new PUBLIC_ONLY blue for color-blind users. The privacy panel itself defaults to a neutral slate when BOTH is active rather than amber. The risk-honest rationale is documented in the `base.html` style-block comment.
- Removed dead and inert code from Chat Mode's context manager that shipped without a consumer: the unused ``build_prompt_context()`` and ``_get_recent_messages()`` helpers, the ``conversation_history`` and ``accumulated_sources`` keys in the research-context dict (nothing downstream read them), and the cross-turn source-count tracking (``_extract_sources_from_history``, the ``source_count`` field, and ``update_accumulated_context``'s ``source_count_delta`` parameter) — which never functioned because the producer was never called with sources. The ``chat.max_context_messages`` setting is removed with it, since it only fed the now-deleted recent-message inclusion path. ``chat.max_findings_to_include`` (which still controls how many prior findings carry into a follow-up) is unchanged.
- Removed the redundant local re-import of `get_settings_manager` inside the `quick_summary` API endpoint (it was already bound at module level), eliminating the same import-shadowing anti-pattern fixed for `get_user_db_session` — so `patch("...web.api.get_settings_manager")` now reliably intercepts. Both helpers are now module-level for one consistent patch surface; only `quick_summary` itself remains a local import (it pulls in the research stack, which would cycle).
- Tests: fix a GC-timing flake in ``test_base_downloader.py::test_context_manager_calls_close``. The test patched ``close`` on the ``BaseDownloader`` *class*, but ``BaseDownloader.__del__`` also calls ``self.close()`` — so whenever the garbage collector happened to finalize a downloader left over from an earlier test while the patch was live, the shared class-level mock counted an extra call and the assertion failed with "Expected 'close' to have been called once. Called 2 times." (seen failing an unrelated PR's CI, #4415). Patch on the instance instead — the same pattern the neighbouring ``__del__`` test already uses — which is immune to other instances' finalizers (mechanism reproduced deterministically both ways).
- Tests: give the logpanel ``prunes the oldest entries when count exceeds MAX_LOG_ENTRIES`` vitest case an explicit 20s timeout. #4304 fixed the real root cause (501 ``setTimeout(autoscroll, 0)`` tasks piling on the real timer queue) by faking all timers, but the test's remaining cost is honest O(n²) DOM work — 501 inserts each running ``addLogEntryToPanel``'s full-container ``querySelectorAll`` scans, ~2.6–2.9s in happy-dom on a dev machine — leaving no headroom against the 5s vitest default under parallel CI load. The recurring timeout flake intermittently failed the shared "All Pytest Tests + Coverage" job on unrelated PRs (e.g. #4415). The test must fill to the real 500-entry cap to exercise the prune, so the work can't shrink; a bigger budget is now the right lever (unlike #4299, which proposed it while the timer bug was still live).
- Tests: lock in the ``LDR_TEST_MODE`` boolean-parsing behaviour for the SQLCipher KDF floor. Adds direct ``_get_min_kdf_iterations()`` unit tests for falsey values (``0``/``false``/``no``/``off`` must NOT relax the floor — the #4564 regression) and the full truthy set (``on``/``enabled`` must relax it, guarding against a narrower parser). Also moves the integration test off the registry's ``min_value`` boundary so it exercises the KDF-floor clamp rather than range validation.
- Tests: stop the ``QueueProcessorV2`` background thread between tests in the ``reset_singletons`` autouse fixture (it already did this for ``BackgroundJobScheduler``). The module-level ``queue_processor`` singleton was started by the first ``create_app()`` in an xdist worker and never stopped, so one test's processor thread ran for the whole worker — looping through ``SettingsManager`` → SQLCipher connection opens on the shared ``db_manager`` concurrently with every later test, and emitting logs to a closed pytest stderr sink at teardown (the same class of bug the scheduler handling fixes). Tests that exercise the queue patch the singleton, so stopping the real thread does not affect them.
- The pre-commit CI job now retries only the hook-environment download (the network-flaky step), not the lint run itself. Previously the entire `pre-commit run` was wrapped in a 2-attempt retry, so an auto-fixing hook (e.g. ``ruff-format``) that rewrote a file would fail the first attempt, leave the now-fixed tree in place, and pass the retry — silently turning a formatting violation into a green check while the unformatted code stayed on the branch. Hook downloads are retried via ``pre-commit install-hooks``; the check now runs exactly once and fails honestly.
- The release gate now imports the SIMD-heavy native dependencies (numpy, pandas, pyarrow, scikit-learn, faiss, torch) under qemu-emulated SandyBridge (AVX without AVX2) and Haswell (AVX2) CPUs. This catches dependency wheels that silently raise the x86-64 instruction-set baseline and would crash with SIGILL on older CPUs — the failure mode that shipped in v1.7.0 when faiss-cpu 1.14.2 executed an AVX2 instruction at import time on AVX-only hosts (#4480). Building the gate surfaced that AVX is already LDR's de-facto minimum CPU requirement: the pandas and scikit-learn wheels crash on pre-AVX CPUs (Nehalem/Westmere era, pre-2011), so those are not part of the gate. The check runs only in the release gate, not on every PR, since full CPU emulation is slow and the failure mode can only ship via dependency bumps.