chore: import upstream snapshot with attribution

2026-07-13 13:08:55 +08:00
commit 7a0da7932b
2985 changed files with 1049377 additions and 0 deletions
@@ -0,0 +1,74 @@
+# Local Deep Research v0.2.0 Release Notes
+
+We're excited to announce Local Deep Research v0.2.0, a major update that brings significant improvements to research capabilities, performance, and user experience.
+
+## Major Enhancements
+
+### New Search Strategies
+- **Parallel Search**: Lightning-fast research that processes multiple questions simultaneously
+- **Iterative Deep Search**: Enhanced exploration of complex topics with improved follow-up questions
+- **Cross-Engine Filtering**: Smart result ranking across multiple search engines for higher quality information
+
+### Improved Search Integrations
+- **Enhanced SearxNG Support**: Better integration with self-hosted SearxNG instances
+- **Improved GitHub Integration**: More effective search and analysis of code repositories
+- **Better Source Selection**: Refined logic for choosing the most appropriate search engines per query
+
+### Technical Improvements
+- **Unified Database**: All settings and history now in a single `ldr.db` database
+- **Improved Ollama Integration**: Better reliability and error handling with local models
+- **Enhanced Error Recovery**: More graceful handling of connectivity issues and API errors
+
+### User Experience
+- **Enhanced Logging Panel**: Improved visibility with duplicate detection and better filtering
+- **Streamlined Settings UI**: Reorganized settings interface with better organization
+- **Research Progress Tracking**: More detailed real-time updates during research
+
+### Development Improvements
+- **PDM Support**: Switched to PDM for dependency management
+- **Pre-commit Hooks**: Added linting and code quality checks
+- **Code Security**: Added CodeQL integration with analysis scripts
+- **Improved Documentation**: Better development guides and setup instructions
+
+## API Changes
+
+- Renamed and consolidated some API functions for consistency
+- Added support for additional parameters in research configuration
+- Improved error handling and response formatting
+
+## Migration Notes
+
+- The application now uses a unified database (`ldr.db`) that will automatically migrate data from older databases
+- If upgrading from v0.1.x, your settings and research history will be automatically migrated on first run
+- The `llm_config.py` file has been deprecated in favor of direct environment variable configuration
+
+## Bug Fixes
+
+- Fixed issues with settings persistence across sessions
+- Resolved UI rendering problems in the history and results pages
+- Fixed socket.io event handling and client disconnection issues
+- Improved handling of large document collections
+- Fixed API endpoint URL inconsistencies
+
+## Contributors
+This release represents the combined efforts of multiple contributors :
+- @djpetti, @HashedViking, @LearningCircuit (core contributors to this release; sorted alphabetically)
+- @dim-tsoukalas, @scottvr (sorted alphabetically)
+
+## Get Involved
+
+- Join our [Discord](https://discord.gg/ttcqQeFcJ3) for support and discussions
+- Follow our [Subreddit](https://www.reddit.com/r/LocalDeepResearch/) for announcements and updates
+- Report bugs and request features on our [GitHub Issues](https://github.com/LearningCircuit/local-deep-research/issues)
+
+---
+
+## Installation
+
+Download the [Windows Installer](https://github.com/LearningCircuit/local-deep-research/releases/download/v0.2.0/LocalDeepResearch_Setup.exe) or install via pip:
+
+```bash
+pip install local-deep-research
+```
+
+Requires Ollama or other LLM provider. See the [README](https://github.com/LearningCircuit/local-deep-research/blob/main/README.md) for complete setup instructions.
@@ -0,0 +1,55 @@
+# Local Deep Research v0.4.0 Release Notes
+
+We're excited to announce Local Deep Research v0.4.0, bringing significant improvements to search capabilities, model integrations, and overall system performance.
+
+## Major Enhancements
+
+### LLM Improvements
+- **Custom OpenAI Endpoint Support**: Added support for custom OpenAI-compatible endpoints
+- **Dynamic Model Fetching**: Improved model discovery for both OpenAI and Anthropic using their official packages
+- **Increased Context Window**: Enhanced default context window size and maximum limits
+
+### Search Enhancements
+- **Journal Quality Assessment**: Added capability to estimate journal reputation and quality for academic sources
+- **Enhanced SearXNG Integration**: Fixed API key handling and prioritized SearXNG in auto search
+- **Elasticsearch Improvements**: Added English translations to Chinese content in Elasticsearch files
+
+### User Experience
+- **Search Engine Visibility**: Added display of selected search engine during research
+- **Better API Key Management**: Improved handling of search engine API keys from database settings
+- **Custom Context Windows**: Added user-configurable context window size for LLMs
+
+### System Improvements
+- **Logging System Upgrade**: Migrated to `loguru` for improved logging capabilities
+- **Memory Optimization**: Fixed high memory usage when journal quality filtering is enabled
+- **Resumable Benchmarks**: Added support for resuming interrupted benchmark runs
+
+## Bug Fixes
+- Fixed broken SearXNG API key setting
+- Memory usage optimizations for journal quality filtering
+- Cleanup of OpenAI endpoint model loading features
+- Various fixes for evaluation scripts
+- Improved settings manager reliability
+
+## Development Improvements
+- Added test coverage for settings manager
+- Cleaner code organization for LLM integration
+- Enhanced API key handling from database settings
+
+## New Contributors
+- @JayLiu7319 contributed support for Custom OpenAI Endpoint models
+
+## Full Changelog
+For complete details of all changes, see the [full changelog](https://github.com/LearningCircuit/local-deep-research/compare/v0.3.12...v0.4.0).
+
+---
+
+## Installation
+
+Download the [Windows Installer](https://github.com/LearningCircuit/local-deep-research/releases/download/v0.4.0/LocalDeepResearch_Setup.exe) or install via pip:
+
+```bash
+pip install local-deep-research
+```
+
+Requires Ollama or other LLM provider. See the [README](https://github.com/LearningCircuit/local-deep-research/blob/main/README.md) for complete setup instructions.
@@ -0,0 +1,62 @@
+# Journal Quality Redesign — release notes (pending)
+
+Staging notes for the journal-quality redesign shipped by [#3081](https://github.com/LearningCircuit/local-deep-research/pull/3081). Fold into the next tagged version's release-notes file when cutting the release.
+
+## Major Features
+
+### Tiered journal quality scoring
+
+The journal-reputation filter now uses a five-tier pipeline (predatory check → OpenAlex snapshot → DOAJ → institution affiliation salvage → optional LLM) instead of calling an LLM per source. A bundled read-only reference database of ~280K academic venues + ~120K institutions powers Tiers 1–3 in 100–300 µs per lookup, eliminating the multi-second per-source latency of the previous LLM-only path.
+
+See `docs/journal-quality.md` for the full scoring table, data-source licenses, and the Tier-1 predatory-whitelist override.
+
+### Journal dashboard
+
+New analytics view at **Analytics → Journals** lists every source in the reference DB with server-side filtering, pagination, and sortable columns (h-index, quartile, impact factor, quality score, DOAJ status, predatory flag). Works on ~200K rows without loading them all into memory thanks to the shared read-only SQLite reference DB (`mode=ro&immutable=1`, `chmod 0o444` after build).
+
+### Quality tags in research output
+
+Each source in the research report now carries a compact quality tag:
+
+```
+[1] Physical Review Letters [Q1 ★★★★★]
+[2] Some Niche Journal [Q3 ★★]
+```
+
+Predatory-flagged sources that are not whitelisted are auto-removed before the report is assembled.
+
+## BREAKING — `journals` table columns removed
+
+The per-user encrypted `journals` table no longer stores the following columns:
+
+`issn`, `issn_list`, `publisher`, `openalex_source_id`, `source_type`, `h_index`, `impact_factor`, `works_count`, `cited_by_count`, `is_in_doaj`, `has_doaj_seal`, `is_predatory`, `predatory_source`, `is_indexed_in_scopus`, `data_version`, `sjr_quartile`.
+
+These metrics are now served exclusively by the bundled read-only reference DB (`journal_quality.db`). The per-user `journals` table is now a Tier-4 LLM cache only and retains: `id`, `name`, `name_lower`, `quality`, `score_source`, `quality_model`, `quality_analysis_time`.
+
+### Impact
+
+- If you have custom SQL tooling that queries the removed columns on the per-user DB, point it at the reference DB via the new accessor (`journal_quality/db.py`) or the `/api/journals` dashboard endpoint instead.
+- The upgrade itself preserves every row in the `journals` table — only the dropped columns disappear. `quality`, `name`, `name_lower`, and the LLM-cache columns stay.
+- No action is needed for users who do not write their own SQL.
+
+## New data downloads on first use
+
+The journal-quality reference DB builds itself on first access from five third-party snapshots (OpenAlex, DOAJ, Stop Predatory Journals, JabRef abbreviations, OpenAlex Institutions — all CC0 or MIT licensed). The build streams several hundred MB from the OpenAlex S3 bulk dump plus smaller snapshots from the other sources, unpacks to ~1 GB of intermediate working set, and typically takes 1–2 minutes on a normal connection. All five sources fetch **in parallel** via a ThreadPoolExecutor — wall-clock is bound by the slowest source (the OpenAlex sources snapshot at ~30–60 s) rather than the sum. On fresh installs the first research request returns immediately with `[preprint — not in journal catalog]` / `[journal quality data is downloading…]` placeholder tags; a second search a minute later picks up the real Q-tier scores. The dashboard's Journals page (`/metrics/journals`) shows per-source progress bars during the build.
+
+## Settings
+
+One new opt-in toggle and four opt-out per-engine toggles:
+
+- `search.journal_reputation.enable_llm_scoring` (default `false`, **opt-in**) — if enabled, Tier 3.6 and Tier 4 use SearXNG + the LLM for unknown journals.
+- `search.engine.web.arxiv.journal_reputation.enabled` (default `true`, opt-out)
+- `search.engine.web.openalex.journal_reputation.enabled` (default `true`, opt-out)
+- `search.engine.web.semantic_scholar.journal_reputation.enabled` (default `true`, opt-out)
+- `search.engine.web.nasa_ads.journal_reputation.enabled` (default `true`, opt-out) — per-engine toggles for running the reputation filter over each academic-search engine's results; disable individually to skip filtering for a given engine.
+
+## Operational notes
+
+- The first user request after upgrade pays a one-time migration cost on the per-user `journals` table (SQLite batch rebuild). Typical libraries complete in under a second; very large libraries (100k+ journals) may stall 2–5 seconds.
+- The first visit to **Analytics → Journals** before the reference DB has finished building shows the page frame with a per-source download banner rather than a data grid; `/api/journals` returns HTTP 503 with `{"status": "error", "message": "Journal reference database not available."}` until the build completes. A research request run in parallel kicks off the build in a background daemon thread, so warming the filter path first is the fastest way to make the dashboard live on a fresh install.
+- Windows installs now enforce read-only on the reference DB via `SetFileAttributesW`, matching the POSIX `chmod 0o444` behavior.
+- The bulk data download fails fast with a clear error if free disk space falls below 2 GB.
+- Transient network failures during the bulk download retry up to three times with exponential backoff (1/2/4 s), respecting `Retry-After` on HTTP 429.
@@ -0,0 +1,85 @@
+
+### 💥 Breaking Changes
+
+- **`llm.model` no longer auto-fills `gemma3:12b`.** Pre-1.7 installs silently downloaded a multi-GB Ollama binary on first use. The field is now empty by default — pick a model in **Settings → LLM**, or research will fail loudly with a clear `ValueError`. Existing users with a saved `llm.model` setting are unaffected; fresh installs see an inline warning in Settings (`aria-live="polite"`) and a clear error on research start. Affects all built-in providers (`openai`, `anthropic`, `ollama`, `lmstudio`, `llamacpp`, `google`, `openrouter`, `ionos`, `xai`). ([#3670](https://github.com/LearningCircuit/local-deep-research/pull/3670))
+- **`llamacpp` provider migrated from in-process to HTTP.** No longer loads `.gguf` files in-process via `llama-cpp-python` — connects to `llama-server`'s OpenAI-compatible HTTP endpoint instead. Removed settings: `llm.llamacpp_model_path`, `llm.llamacpp_n_batch`, `llm.llamacpp_n_gpu_layers`, `llm.llamacpp_f16_kv`. New settings: `llm.llamacpp.url` (default `http://localhost:8080/v1`), `llm.llamacpp.api_key` (optional, for auth proxies). Migration: run `llama-server -m <your-model.gguf>` and the default URL picks it up; alternatively use the `openai_endpoint` provider pointed at the same URL. ([#3670](https://github.com/LearningCircuit/local-deep-research/pull/3670))
+
+### 🔒 Security
+
+- Fix LM Studio model detection (#3800) — the settings dropdown was silently dropping models for every auto-discovered provider except Ollama / OpenAI / Anthropic / OpenAI-Compatible-Endpoint, including LM Studio, Llama.cpp, Google Gemini, xAI, OpenRouter, and IONOS. While investigating, also fixed a separate credential-leak class where the route's auto-discovery loop sent every other provider's configured API key in the `Authorization` header to the local LM Studio / llama-server endpoint on every settings-page load. ([#3800](https://github.com/LearningCircuit/local-deep-research/pull/3800))
+- Fix SSRF parser-differential bypass (GHSA-g23j-2vwm-5c25). URLs containing
+  backslash, whitespace, or ASCII control bytes are now rejected upfront by the
+  SSRF validator and notification-URL validator; hostname extraction switched
+  from `urllib.parse.urlparse` to `urllib3.util.parse_url` so the validator and
+  the HTTP client agree on destination by construction. Credit: @Fushuling,
+  @RacerZ-fighting.
+- Hardened SSRF defenses against AWS ECS task metadata
+  (`169.254.170.2`, `169.254.170.23`), Tencent Cloud (`169.254.0.23`),
+  and AlibabaCloud (`100.100.100.200`) metadata endpoints — these are
+  now always blocked alongside the existing AWS IMDS / Azure / OCI /
+  DigitalOcean entry (`169.254.169.254`). Redacted credentials, path,
+  and query from URL-rejection logs (operators with grep/regex tooling
+  on the rejection log lines will see authority-only `scheme://host:port`
+  instead of full URLs going forward).
+- SSRF defense-in-depth: block IPv6 transition prefixes that can wrap
+  private IPv4 destinations on hosts with kernel sit0/NAT64 routes.
+
+  - `2002::/16` (6to4, RFC 3056 — deprecated by RFC 7526)
+  - `64:ff9b::/96` (NAT64 well-known, RFC 6052)
+  - `64:ff9b:1::/48` (NAT64 local-use, RFC 8215 — same SSRF threat class
+    as the WKP; missing it earned a HackerOne bounty against
+    ssrf_filter)
+  - `2001::/32` (Teredo, RFC 4380)
+  - `100::/64` (IPv6 discard, RFC 6666)
+  - `::/96` (IPv4-Compatible IPv6, RFC 4291 §2.5.5.1 — DEPRECATED 2006;
+    same SSRF threat class as the transition prefixes)
+
+  The metadata-IP block is hardened against IPv6-wrapped IMDS access:
+  when an IPv6 destination falls in a NAT64 prefix, the embedded IPv4 is
+  extracted and matched against `ALWAYS_BLOCKED_METADATA_IPS`, so
+  `[64:ff9b::a9fe:a9fe]` cannot reach 169.254.169.254 even on a NAT64
+  host.
+
+  Operators on IPv6-only deployments using DNS64+NAT64 (where outbound
+  IPv4 traffic is synthesized through `64:ff9b::/96`) can opt back in via
+  the env-only setting `security.allow_nat64`
+  (`LDR_SECURITY_ALLOW_NAT64=true`). The opt-in is scoped strictly to
+  the two NAT64 prefixes — 6to4, Teredo, and discard remain blocked
+  unconditionally, and the IMDS embedded-IPv4 carve-out still applies.
+
+### ✨ New Features
+
+- **LM Studio optional API key.** New `llm.lmstudio.api_key` setting for LM Studio instances with authentication enabled. ([#3670](https://github.com/LearningCircuit/local-deep-research/pull/3670))
+- **Release notes now use [towncrier](https://towncrier.readthedocs.io) news fragments.** Each PR drops one tiny `changelog.d/<id>.<category>.md` file (categories: `breaking`, `security`, `feature`, `bugfix`, `removal`, `misc`). At release prep, the maintainer runs `pdm run towncrier build --version <X.Y.Z> --yes`, which renders the fragments into `docs/release_notes/<X.Y.Z>.md` and removes them. Replaces the previous shared-file workflow, which scaled poorly at LDR's PR throughput. See `changelog.d/README.md` for the convention. ([#3773](https://github.com/LearningCircuit/local-deep-research/pull/3773))
+- **New citation format: source-tagged with global numbering.** Reports can now render citations as `[arxiv-1]`, `[openai.com-2]`, `[arxiv-3]` — the source tag identifies what kind of source each citation is (short URLClassifier tag for known academic sources, cleaned domain for generic web URLs, or the collection name for local RAG/library hits, e.g. `[my-papers-4]`), while the number stays the original global counter so labels never collide and inline citations match the bibliography order. Opt in via `report.citation_format → "Source-tagged with global numbering"`; the existing default remains `Numbers with hyperlinks [1]`. ([#4012](https://github.com/LearningCircuit/local-deep-research/pull/4012))
+- Context-overflow warnings (front-page banner, result-page banner, completion toast) now show a clickable action that takes you straight to the relevant diagnostics — historical warnings link to the context-overflow metrics page, while per-research warnings link to that research's overflow section on its details page.
+- Context-overflow warnings now include structured fields (`research_id`, `model`, `provider`) and detect both input-only and total-context (input+output) overflow. Four distinct tags (`[provider-confirmed]`, `[total-context]`, `[estimated]`, `[estimated-total-context]`) make different overflow causes distinguishable in logs and alerts. Hosted providers (OpenAI, Anthropic, OpenRouter) get estimation-based detection so their truncation events are no longer invisible. The high-context VRAM warning now links directly to the context-overflow metrics page.
+- OpenAI-compatible runtime errors (LM Studio / vLLM / llama.cpp server / OpenRouter / custom endpoint) now surface a message that names the provider, configured base URL, and model. The existing `Error type: <code>` token convention is extended with seven `openai_*` codes (`openai_connection_refused`, `openai_timeout`, `openai_auth`, `openai_permission_denied`, `openai_model_not_found`, `openai_bad_request`, `openai_unknown`), and `ErrorReporter` maps each to the right `ErrorCategory`. The original exception text is preserved in a `Details:` suffix; userinfo embedded in a base URL (`https://u:key@host/v1`) is stripped before display.
+- The `/metrics/context-overflow` page is reorganised around per-research truncation diagnosis: redundant aggregate sections (Token Usage Overview, Token Usage Over Time, Model Usage Breakdown, Context Limit Distribution doughnut, Token Usage by Phase) are dropped in favour of empty-state messages and a recoloured and re-shaped scatter chart that buckets points by context-utilisation ratio (green circle / amber triangle / red diamond — shape gives a redundant signal for colourblind users) rather than per-limit dashed reference lines. Tooltips now explicitly label requests with unknown context limits or unknown token counts. The main `/metrics` dashboard gains a small Context Overflow summary panel linking to the full diagnostics page; on aggregation failure the panel shows `—` rather than a misleading `0%`.
+- Upload rate limits are now configurable via `LDR_SECURITY_RATE_LIMIT_UPLOAD_USER` and `LDR_SECURITY_RATE_LIMIT_UPLOAD_IP`. The default cap is raised from `10 per minute;100 per hour` to `60 per minute;1000 per hour`, unblocking bulk RAG library uploads.
+
+### 🐛 Bug Fixes
+
+- Fix progress page log panel sizing so it uses CSS layout instead of JavaScript height calculations. ([#2606](https://github.com/LearningCircuit/local-deep-research/pull/2606))
+- Progress page current-task text now preserves line breaks and wraps long generated task messages instead of collapsing them into a hard-to-read wall of text. ([#2609](https://github.com/LearningCircuit/local-deep-research/pull/2609))
+- Live research log entries now keep the same newest-first visual order as loaded log history instead of appearing at the bottom of the log panel. ([#2610](https://github.com/LearningCircuit/local-deep-research/pull/2610))
+- **Journal-quality dashboard ("Your Research") no longer shows orphan papers from deleted research sessions.** Deleting a research session previously left behind `Paper` rows whose `PaperAppearance` had cascade-deleted, so the dashboard kept counting them. The aggregate top-200 query and the predatory-blocked count now exclude papers that have no remaining appearances. Per-research views were already correct. ([#3544](https://github.com/LearningCircuit/local-deep-research/pull/3544))
+- Fix the model dropdown in Settings to populate for every auto-discovered LLM provider, not just Ollama / OpenAI / Anthropic / OpenAI-Compatible-Endpoint. LM Studio, Llama.cpp, Google Gemini, xAI, OpenRouter, and IONOS were silently dropped by the frontend (`processModelData`) even when the backend returned their models correctly. ([#3800](https://github.com/LearningCircuit/local-deep-research/pull/3800))
+- Fix file descriptor exhaustion (#3816) caused by `ChatOllama._async_client` not being released alongside `_client`. `_close_base_llm()` now closes both the sync and async httpx clients owned by `ChatOllama`, addressing the eventpoll-FD growth seen in long-running deployments. ([#3816](https://github.com/LearningCircuit/local-deep-research/pull/3816))
+- Fix the research details "View Journals" button so it opens the scoped journal-quality dashboard instead of a missing page. ([#3825](https://github.com/LearningCircuit/local-deep-research/pull/3825))
+- Fix Download Manager failure after 20–30 PDFs with `UNIQUE constraint failed: documents.document_hash` and the cascading PendingRollbackError that prevented further downloads. ([#3827](https://github.com/LearningCircuit/local-deep-research/pull/3827))
+- Fix login/registration over `http://localhost:5001` on Docker Desktop.
+  The session cookie's `Secure` flag is now driven purely by the request
+  protocol (HTTPS) rather than by source-IP heuristics, so HTTP requests
+  from any IP — including Docker Desktop's NAT gateway — now keep their
+  cookies and CSRF tokens. ([#3849](https://github.com/LearningCircuit/local-deep-research/pull/3849))
+- Fix Ollama embedding indexing aborting with `"input length exceeds the context length"` (HTTP 500) by exposing a configurable `embeddings.ollama.num_ctx` setting. The new field appears on the embedding settings page when the Ollama provider is selected and ships with a default of 8192 tokens, which covers chunk sizes up to ~5000 characters for `nomic-embed-text`, `mxbai-embed-large`, `bge-m3`, and `qwen3-embedding`. ([#3870](https://github.com/LearningCircuit/local-deep-research/pull/3870))
+- LangGraph agent strategy error messages now include the exception type, and the `'str' object has no attribute 'model_dump'` failure (issue #3897) is translated into a rich on-result-page recommendation listing the three possible causes — proxy/shim, serving stack, or model — with concrete remediation steps for each. ([#3926](https://github.com/LearningCircuit/local-deep-research/pull/3926))
+- The Context Overflow summary panel on `/metrics` now honors the dashboard's research-mode filter (Quick / Detailed / All) the same way the rest of the panels do.
+
+### 📝 Other Changes
+
+- Added `LDR_DISABLE_RATE_LIMITING` as the canonical name for disabling HTTP rate limiting, matching the `LDR_` env-var convention. The legacy `DISABLE_RATE_LIMITING` still works but emits a deprecation warning. Note: this is distinct from `LDR_RATE_LIMITING_ENABLED`, which controls the adaptive search-engine rate limiter — a different subsystem (#3905).
+- Migrated CI workflows to the canonical `LDR_DISABLE_RATE_LIMITING` env var (added in #3936) and closed a test-isolation gap in `test_enabled_by_default` that did not clear the canonical var.
+- Removed unused chart-rendering JavaScript on the context-overflow analytics page (token-usage-over-time stacked bar, model-token-stats table, context-limits-distribution doughnut, and the configured-limits list). These functions stopped being called when the page was reorganised to focus on truncation diagnosis (#3792); deleting them now drops ~200 lines of dead code without any user-visible change.
+- The prerelease Docker workflow now also re-points the floating `prerelease` tag at each new RC manifest. Testers can pin `image: localdeepresearch/local-deep-research:prerelease` in their compose file and run `docker compose pull && docker compose up -d` to fetch the latest RC without editing the tag each cycle. Versioned tags (`prerelease-vX.Y.Z-<sha>`) are still published for reproducible testing.
@@ -0,0 +1,28 @@
+
+### 🔒 Security
+
+- Extended the cloud-metadata IP absolute-block in `NotificationURLValidator` to Apprise plugin schemes (signal, gotify, ntfy/ntfys, mattermost, rocketchat, matrix, json, xml, form, mailto). Previously only the http/https branch ran the IMDS guard, so a notification URL like `signal://169.254.169.254/...` would round-trip to Apprise even though Apprise translates it into a plain HTTP request against that host. LAN/loopback reach for self-hosted plugin endpoints (the #4006 use case) is unchanged.
+- Fix Google Gemini API key leak in `list_models_for_api` error log. The
+  URL constructed at request time embeds the key as a `?key=...` query
+  parameter (per Google's documented API), so when the underlying request
+  failed, the `requests` exception message — and therefore the
+  `logger.exception(...)` traceback — contained the full URL with the key.
+  The except handler now redacts the key from the message and uses
+  `logger.warning` (no exception chain).
+
+### 🐛 Bug Fixes
+
+- **JavaScript rendering disabled by default in the production Docker image (#3826).** The headless-browser fallback in the content fetcher (Crawl4AI/Playwright) was previously attempted on every fetch even though the default Docker production image ships without a Chromium binary - each attempt failed loudly and contributed to the memory growth reported in issue #3826. A new `web.enable_javascript_rendering` setting (default `false`) gates the fallback. In limited (mostly accidental) internal benchmark comparisons - between dev instances that happened to have Chromium installed and routine Docker runs that did not - JS rendering did not measurably improve research quality, and most regular benchmark runs are on Docker without Chromium anyway, so the change does not regress observed quality. To enable: install Chromium in the runtime environment (`playwright install --with-deps chromium`) and toggle the setting in the UI. ([#3826](https://github.com/LearningCircuit/local-deep-research/pull/3826))
+- **Embeddings now work against OpenAI-compatible local servers (LM Studio, vLLM, llama.cpp).** Previously the OpenAI embeddings provider only listed itself as available when an API key was configured, and the relevant `embeddings.openai.*` settings were not registered for the UI to surface — so users who ran an OpenAI-compatible local server had no way to point the embeddings tab at it. The provider now also reports available when only `embeddings.openai.base_url` is set, substitutes a placeholder API key for keyless endpoints (matching the LM Studio LLM provider's behavior), and ships a new `settings_openai_embeddings.json` defaults file that registers `embeddings.openai.api_key`, `embeddings.openai.base_url`, `embeddings.openai.model`, and `embeddings.openai.dimensions`. The model-list lookup now routes through the configured `base_url` so model discovery also targets the local server. ([#3883](https://github.com/LearningCircuit/local-deep-research/pull/3883))
+- **"Start Research" button no longer silently does nothing for users with a non-multiple-of-512 context window value.** The hidden `context_window` input on the research form had a `step="512"` constraint, so any stored value not aligned to that grid (e.g. `25000`) failed HTML5 form validation. Because the input lives inside a `display:none` container that is only shown for local providers, the browser could not focus the invalid field to report the error, so submission was aborted with no visible message and no log line — the click appeared to do nothing. Relaxed the step to `1` so any in-range integer is accepted; the value is still bounded by `min`/`max` and persisted unchanged. ([#3909](https://github.com/LearningCircuit/local-deep-research/pull/3909))
+- **Chinese/Japanese/Korean text now renders in exported PDFs.** The default PDF stylesheet hard-coded a Latin-only font stack, so any CJK characters in the research result were dropped silently from the download even though they displayed correctly in the browser. The minimal CSS now includes a broad CJK fallback chain (Noto Sans CJK, PingFang, Hiragino, Apple SD Gothic Neo, Microsoft YaHei/JhengHei, Yu Gothic, Malgun Gothic, SimSun) covering Windows, macOS, and Linux desktops out of the box, and the Docker image now installs `fonts-noto-cjk` so the slim base image has glyph coverage. Linux pip/server installs still need a CJK font package on the host — see [install-pip.md](../install-pip.md) and the [FAQ](../faq.md#chinese--japanese--korean-text-is-missing-from-exported-pdfs) for the per-distro commands. ([#4055](https://github.com/LearningCircuit/local-deep-research/pull/4055))
+- Cross-engine filtering now keeps LLM ranking fallbacks within the subset of results the model actually evaluated, preventing unevaluated search results from leaking into downstream synthesis.
+- Fix file-descriptor leak (`anon_inode:[eventpoll]`) in `_close_base_llm` when invoked while an asyncio loop is running. The previous code path skipped the async-client close in that case and documented that the "loop owner" would close instead — but no loop-owner cleanup actually exists in the project, so the inner `httpx.AsyncClient` (and its `epoll_create` FD) was silently abandoned. The same skip-path fires under the default `langgraph-agent` strategy because LangGraph dispatches some tool steps via asyncio internally, so close calls reached from a sync `finally` can still land inside a live loop. Cleanup now runs in a brief daemon thread that owns its own loop, so `asyncio.run(aclose())` works regardless of the caller's loop state; a bounded 5-second `join` keeps it from holding up shutdown if the Ollama server is unresponsive. This is the gap left by #3855's coverage of #3816 — same leak class, different code path.
+
+  Also fixes a smaller secondary leak in the Docker `HEALTHCHECK`: `urllib.request.urlopen('http://localhost:5000/api/v1/health')` had no `timeout=` argument, so when the app slowed down (FD exhaustion or otherwise) Docker's 10s healthcheck timeout SIGKILL'd the `sh -c` wrapper but the python child got reparented to PID 1 and hung forever, each contributing a `pidfd` and a TCP socket against the app. Adding `timeout=8` lets the child return/raise before Docker's wall so it exits cleanly and gets reaped.
+
+### 📝 Other Changes
+
+- Added a "Testing a Release Candidate (Prerelease Image)" subsection to the developer guide (`docs/developing.md`) covering both the floating `:prerelease` and immutable `prerelease-vX.Y.Z-<sha>` Docker Hub tags, with a side-by-side compose snippet that runs the RC on port 5001 with isolated volumes so a broken migration cannot affect a production instance.
+- Pool-dispose failures in the periodic WAL/SHM handle-release workaround (ADR-0004) now log at ``WARNING`` instead of ``DEBUG`` so silent drift becomes visible. Documented in ``docs/CONFIGURATION.md`` that ``LDR_APP_DEBUG=true`` also enables Loguru ``diagnose=True``, which can materialise sensitive locals into exception traces — do not enable in production.
+- Refactored environment settings to use specialized exception classes, improving error observability and alignment with `TRY003` standards.
@@ -0,0 +1,22 @@
+
+### 🔒 Security
+
+- RAG collection upload endpoint now validates per-file size (50MB limit) and per-request file count (200 limit), matching the research upload endpoint. Previously only the request-level `MAX_CONTENT_LENGTH` (10GB) was enforced, allowing a single oversized file or a request with thousands of zero-byte files to reach the per-file processing loop. Pre-flight `Content-Length` check rejects oversized files before reading bytes into memory.
+
+### ✨ New Features
+
+- **New citation format: source-tagged with global numbering.** Reports can now render citations as `[arxiv-1]`, `[openai.com-2]`, `[arxiv-3]` — the source tag identifies what kind of source each citation is (short URLClassifier tag for known academic sources, cleaned domain for generic web URLs, or the collection name for local RAG/library hits, e.g. `[my-papers-4]`), while the number stays the original global counter so labels never collide and inline citations match the bibliography order. Opt in via `report.citation_format → "Source-tagged with global numbering"`; the existing default remains `Numbers with hyperlinks [1]`. ([#4012](https://github.com/LearningCircuit/local-deep-research/pull/4012))
+- **Add LaTeX math rendering support.** Mathematical formulas written with `$...$` (inline) and `$$...$$` (display) notation are now rendered using KaTeX in research reports.
+- Per-PDF library storage cap (`research_library.max_pdf_size_mb`) default raised from 100 MB to 3 GB so it no longer silently truncates large academic PDFs after the recent upload-validator bump. The setting's UI ceiling is also raised from 500 MB to 10 GB, and the `PDFStorageManager` / `download_service` fallback defaults are updated to match. Deployments that want a tighter bound can still lower the setting via the UI.
+- Per-file upload cap (`FileUploadValidator.MAX_FILE_SIZE`) is now configurable and defaults to 3 GB (was 50 MB) so large academic datasets and PDFs fit out of the box. Deployments that want a tighter bound can lower it via the `LDR_SECURITY_UPLOAD_MAX_FILE_SIZE_MB` environment variable or the `security.upload_max_file_size_mb` setting. Memory usage stays bounded by the existing 5 MB spool-to-disk threshold on multipart requests.
+- The library RAG indexer now prunes old quarantined FAISS index files (`<hash>.faiss.corrupt-*` and matching `.pkl.corrupt-*`) at quarantine time, keeping only the 5 most recent per base path. Prevents the `rag_indices/` cache directory from filling up on systems that experience recurring corruption, while preserving recent diagnostic artefacts. Follow-up to #4197 / #4200.
+
+### 🐛 Bug Fixes
+
+- Fixed `UnicodeDecodeError` on Windows when loading settings, security config, benchmark results, and Vite manifest files. All text-mode `open()` calls now use explicit UTF-8 encoding, and JSON config files use `utf-8-sig` to handle BOM-prefixed files from Windows editors. Fixes #3743. ([#3797](https://github.com/LearningCircuit/local-deep-research/pull/3797))
+- Fixed embedding-model dropdown showing "No models available" with LM Studio (and other OpenAI-compatible local servers) when an embedding model whose name didn't include the literal `embedding` token was loaded (e.g. `nomic-embed-text-v1.5`). The OpenAI and Ollama embedding providers no longer guess from the model name — every model the endpoint reports is shown so the user can pick the one they actually loaded. Ollama still tags models when its `/api/show` capabilities response is available. Fixes #4195. ([#4195](https://github.com/LearningCircuit/local-deep-research/pull/4195))
+- Closed a follow-up race condition in the library RAG indexer where two concurrent workers indexing different documents into the same collection could lose each other's embeddings from the FAISS file — last writer's `save_local` overwrote the earlier writer's chunks (the chunks survived in the DB, but the index file was missing them until a force-reindex rebuilt it). The save path now reloads from disk under the per-`(user, index_path)` lock before adding, so concurrent writers' chunks are merged instead of overwriting each other. Follow-up to #4197/#4200.
+- Fixed a data-loss bug where the library RAG index file (`.faiss`) was silently deleted when integrity verification failed during concurrent auto-indexing — destroying hundreds of previously-indexed documents in one go. Corrupted index files are now quarantined to `<path>.corrupt-<ns>` so they remain recoverable, and concurrent indexers are serialised by a per-`(user, index_path)` lock so the race that produced the checksum mismatch no longer occurs. Also drops the per-document `PRAGMA wal_checkpoint(FULL)` that contributed to `database is locked` errors under bulk-download concurrency. Fixes #4197.
+- Paginate the ``/history/logs/<id>`` endpoint (default 500, clamped to 5000) so loading a long research's history no longer materialises every ResearchLog row server-side or pushes a 50+ MB JSON response that the browser ultimately prunes to 500 entries anyway. Complements the live-socket truncation in PR #4004.
+- Three module-level per-user lock dicts (`_user_init_locks` in `database/library_init.py`, `_user_locks` in `database/backup/backup_service.py`, `_user_critical_locks` on `QueueProcessorV2`) previously accumulated one `threading.Lock` entry per username over the process lifetime with no removal hook on user-close. Bounded by total user count (~900 bytes/user across all three), so not visible on typical self-hosted instances — but long-lived multi-user deployments with user-account churn would see slow memory creep. The three modules now expose `pop_user_*_lock(username)` helpers and a shared `_pop_per_user_locks` call in the connection-cleanup module fires them from both the idle-cleanup sweeper and the logout / password-change paths, matching the cleanup already done for scheduler-job registrations and session-password store.
+- Tighten the `model_dump` error-pattern regex introduced in #3926 so the rich proxy/shim hint only fires for the canonical `object has no attribute 'model_dump'` AttributeError, not for unrelated traces that happen to contain the substring `model_dump`.
@@ -0,0 +1,7 @@
+
+### 🐛 Bug Fixes
+
+- Citation handlers and the report structure generator now route LLM responses through `get_llm_response_text`, stripping `<think>` reasoning blocks from synthesized answers and reports (previously leaked verbatim with reasoning models) and normalizing string/message responses in one place.
+- Cross-engine filtering now deduplicates repeated LLM-ranked result indices so the same source does not appear multiple times with different citation numbers.
+- Fix `AttributeError` in citation handlers during follow-up research when LLMs return string responses instead of message objects.
+- Restore `str()` coercion in the precision citation handler's key-fact extraction so it always returns a string (fixes a mypy `no-any-return` failure introduced in #3884).
@@ -0,0 +1,60 @@
+# 1.6.3 — release notes
+
+LLM provider rework shipped by [#3670](https://github.com/LearningCircuit/local-deep-research/pull/3670). 1.6.3 was tagged without release notes; this file documents its breaking changes retroactively.
+
+## BREAKING — `llm.model` no longer auto-fills `gemma3:12b`
+
+Before this release, leaving the **Language Model** field empty in Settings caused fresh installs to silently download a 7–12 GB Ollama binary (`gemma3:12b`) the user had not asked for. The default is now an empty string (`""`), and `get_llm()` raises `ValueError("LLM model not configured...")` when the field is unset for any provider that requires a model name.
+
+### Impact
+
+- **Fresh installs**: open Settings → LLM, pick a provider, and choose a model name (e.g. `llama3.1:8b` for Ollama, `gpt-4o-mini` for OpenAI, `claude-3-5-sonnet-20241022` for Anthropic). The model field shows a yellow inline warning while empty so the requirement is discoverable.
+- **Existing installs**: any user who already had `llm.model` saved (including `gemma3:12b`) keeps their value — `import_settings(overwrite=False)` at `settings/manager.py:1129` preserves any non-`None` DB value, so only fresh DBs and never-written rows pick up the new empty default.
+- The corresponding HTTP endpoint `/api/check/ollama_model` now returns HTTP 400 with `error_type: "model_not_configured"` when neither the query parameter nor `llm.model` is set, instead of silently substituting `gemma3:12b`.
+
+## BREAKING — `llamacpp` provider switched to HTTP
+
+The `llamacpp` provider no longer loads `.gguf` files in-process via `langchain_community.llms.LlamaCpp`. It now talks HTTP to llama.cpp's OpenAI-compatible `llama-server`, mirroring the LM Studio integration.
+
+### Migration steps for existing llamacpp users
+
+1. Install `llama.cpp` (https://github.com/ggerganov/llama.cpp) if you do not already have it.
+2. Start the server:
+   ```
+   llama-server -m /path/to/your/model.gguf
+   ```
+   Default port is `8080`.
+3. Open Settings → LLM and confirm `llm.llamacpp.url` is `http://localhost:8080/v1` (or wherever your server runs). Set `llm.llamacpp.api_key` only if your server is behind an auth proxy.
+4. Set `llm.model` to the model name `llama-server` is hosting (e.g. `llama-3.1-8b-instruct`).
+
+### Removed settings
+
+The four old in-process settings are gone:
+
+- `llm.llamacpp_model_path`
+- `llm.llamacpp_n_gpu_layers`
+- `llm.llamacpp_n_batch`
+- `llm.llamacpp_f16_kv`
+
+Existing rows for these keys remain in the per-user DB but are no longer read by any code path; they will be silently ignored. Tuning that previously lived in these settings now belongs to the `llama-server` command line.
+
+### New settings
+
+- `llm.llamacpp.url` — HTTP endpoint URL (default `http://localhost:8080/v1`).
+- `llm.llamacpp.api_key` — optional, only needed if you put an auth proxy in front of `llama-server`.
+
+## Cross-provider default-model cleanup
+
+All built-in providers (`anthropic`, `google`, `openrouter`, `xai`, `ionos`, `openai`, `ollama`, `lmstudio`, `llamacpp`, `openai_endpoint`) now have `default_model = ""` and refuse to silently substitute. Where a user-specified model is missing, the central check in `get_llm()` raises a `ValueError` with an actionable message naming the offending setting key. 73 tests were updated to pass `model_name=...` explicitly rather than relying on the old defaults.
+
+## Better error message when local LLM servers are not running
+
+When a user selects `llamacpp`, `lmstudio`, `openai_endpoint` (or another OpenAI-compatible provider) and the configured server is not reachable, the raw `openai`/`httpx` connection error is rewritten — by the friendly-error layer added in #4027 — into a message that names the provider and base URL, e.g. `Cannot reach <provider> at <url>. Check that the server is running and the URL is correct.` so the user sees something actionable instead of a stack trace.
+
+## Optional API keys for `lmstudio` and `llamacpp`
+
+Both providers now support an optional `api_key` setting (`llm.lmstudio.api_key`, `llm.llamacpp.api_key`) for users who put their local server behind an auth proxy. If left blank, the providers use a placeholder key that bare `llama-server` / LM Studio ignore. No action required for default setups.
+
+## Settings UI
+
+The **LLM Provider** dropdown now lists `llama.cpp (Local)` as a selectable option (it was reachable only by manual config-edit in 1.6.2). The empty-`llm.model` warning live-updates on both keystroke and dropdown selection, mirroring the API-key-not-configured pattern in `openai_base.py`.
@@ -0,0 +1,33 @@
+# 1.6.8 — release notes
+
+## Bug fixes
+
+- **(#3747) Restored login for databases created before v1.4.0.**
+  Users whose encrypted user database was created before 2026-03-25
+  (v1.4.0, when Alembic migrations were introduced) could not log in
+  after upgrading: their databases lacked the `alembic_version` row, and
+  the migration runner attempted to apply migrations from scratch against
+  a legacy column shape. Migration `0007`'s index backfill then failed on
+  missing columns (e.g. `settings.category`), leaving the database in a
+  corrupted intermediate state.
+
+  This release detects pre-Alembic databases on first launch, stamps them
+  at the correct baseline (revision `0001`), and lets the remaining
+  migrations apply cleanly. Look for the `BUG-3747:` log line at startup
+  to confirm the recovery path engaged.
+
+  Affected users just need to update to `1.6.8` (or `:latest` / `:1.6`)
+  and restart — the recovery is automatic on the next launch.
+
+  ### Hardening
+
+  - `stamp_database()` is now race-tolerant: concurrent stampers (e.g. two
+    same-user logins arriving simultaneously) no longer trigger
+    `OperationalError` / `IntegrityError` on the duplicate
+    `alembic_version` insert. The race-tolerance is narrowly scoped to
+    `alembic_version`-related errors, so disk-full / corruption / unrelated
+    `SQLITE_BUSY` errors continue to propagate.
+  - `run_migrations()` refuses to operate on what looks like an auth-DB
+    shape (only `users` table, optionally with `alembic_version`) — defense
+    in depth against accidentally routing the auth engine through the
+    user-DB migration runner.
@@ -0,0 +1,81 @@
+
+### 🔒 Security
+
+- Extended `redact_secrets()` coverage from the Google provider (#4070) to every credential-bearing exception-logging call site: the OpenAI-compatible provider base (`list_models`), the custom OpenAI endpoint provider, the OpenAI embedding provider, and the web search-engine subsystem (`BaseSearchEngine.run()` plus the main HTTP-call error paths in Tavily, Exa, Serper, Google PSE, Mojeek, PubMed, Semantic Scholar, Guardian, ScaleSerp, SerpAPI, and GitHub). API keys embedded in upstream SDK error messages or echoed-back URLs/headers no longer leak into logs. Guardian and ScaleSerp chain explicit-value redaction with their existing regex sanitizers for defense in depth. Consolidates #4168, #4175, and #4181. See #4131. ([#4426](https://github.com/LearningCircuit/local-deep-research/pull/4426))
+- Extended `redact_secrets()` coverage to the NASA ADS search engine, which was missed by the original #4131 sweep. Its `_get_previews` catch-all used `logger.exception`, whose traceback frames hold `self.headers` (the `Authorization: Bearer <key>` value) and would render the API key under loguru diagnose mode. The except block now uses a redacted `logger.warning`. Follow-up to #4131.
+- Loguru ``diagnose=True`` exception rendering (which dumps the ``repr()`` of every local variable in every traceback frame) is now gated behind a separate explicit ``LDR_LOGURU_DIAGNOSE`` opt-in instead of riding on ``LDR_APP_DEBUG``. Previously, enabling ``LDR_APP_DEBUG`` for general debug output also turned on localvar dumps, so any exception could write frame-local credentials — API keys, the SQLCipher master password, ``Authorization`` headers — into every log sink. ``LDR_APP_DEBUG`` now controls log *level* only; ``diagnose`` stays off unless the operator additionally sets ``LDR_LOGURU_DIAGNOSE=true``, and an explicit warning is emitted when they do.
+- The MCP server subprocess (``ldr-mcp``) now pins ``diagnose=False`` on its stderr sink. loguru defaults ``diagnose=True``, which renders ``repr()`` of every traceback frame's locals on exception, so the many ``logger.exception(...)`` paths in the MCP request handlers were writing frame-local credentials (``api_key``, ``Authorization`` headers, search-engine secrets) into the MCP client's stderr log on any failure. Companion to the ``LDR_LOGURU_DIAGNOSE`` lockdown for the web/CLI logger; the MCP subprocess has no debug mode, so the gate is unconditionally off.
+- The benchmark CLI (``ldr-benchmark`` / ``benchmarks/cli/benchmark_commands.py``) now pins ``diagnose=False`` on both of its ``logger.add(sys.stderr, ...)`` calls. loguru defaults ``diagnose=True``, which renders ``repr()`` of every traceback frame's locals on exception, so the many ``logger.exception(...)`` paths in the benchmarks package (LLM grader calls, search-engine runners, dataset loaders) were writing frame-local credentials — LLM ``api_key``, ``Authorization`` headers, search-engine secrets — to stderr on any failure. Companion to the ``LDR_LOGURU_DIAGNOSE`` lockdown for the web/CLI logger (#4384) and the MCP subprocess (#4394); the benchmark CLI also bypasses ``config_logger``, so the env-var gate did not reach it.
+- The chat `PATCH` (rename/archive) and `DELETE` session routes now carry the same per-user rate limit as the other state-changing chat endpoints. They were previously bounded only by the global limiter, leaving an uneven abuse surface across the session API.
+- The example scripts under ``examples/`` (``example_browsecomp.py`` and the three ``api_usage/programmatic/`` snippets) now pass ``diagnose=False`` to their ``logger.add(sys.stderr, ...)`` calls. Loguru defaults ``diagnose=True``, which renders ``repr()`` of every local in every traceback frame on exception. These files are templates that users copy into their own scripts, so the previous default propagated the same credential-in-traceback leak fixed for the application logger and MCP subprocess (#4185, #4384) into anyone's downstream code.
+
+### ✨ New Features
+
+- **Chat Mode** — Interactive multi-turn research conversations. Each session accumulates context across turns (entities, topics, source count), streams research steps and citations live as the answer is built, and persists in your per-user database (encrypted by default). Sessions can be archived, reactivated, deleted, or exported as Markdown. Reach it from the **Chat** sidebar link or `/chat/`. See [features.md#chat-mode](../features.md#chat-mode) for the full feature description. ([#2953](https://github.com/LearningCircuit/local-deep-research/pull/2953))
+- **"Summarizing…" indicator for Chat follow-ups** — When you send a follow-up question, the thinking indicator now reads "Summarizing previous conversation…" while the earlier turns are condensed into context for the new research, instead of showing blank dots. It clears the moment research starts streaming its progress.
+- **Configurable follow-up context in Chat Mode** — Follow-up questions now build on the earlier conversation instead of starting cold. By default, the prior conversation is condensed into a short summary focused on your new question (using your configured LLM) and passed to the follow-up research as its "previous findings", keeping multi-turn research on-topic and within context limits. You can change this with the new **Follow-up Context Mode** setting (`chat.followup_context_mode`):
+
+  - `summary` (default) — an LLM summary of the conversation, focused on your new question
+  - `raw` — the most recent research findings, truncated
+  - `full` — the entire conversation transcript
+  - `none` — no prior context (just your new question and the original topic)
+
+  Only `summary` makes an extra LLM call per follow-up; the other modes add no model cost.
+- Chat Mode: the thinking bubble now shows what the agent is currently reasoning about between tool calls — the LLM's intermediate prose (e.g. "I should compare the published benchmark results next…") appears above the bouncing dots and overwrites with each new step, instead of leaving a static indicator for the entire research duration.
+- Chat mode now shows an elapsed-seconds counter ("Stopping research… (Ns)") in the progress task line after the user clicks Stop, so it's clear the click registered even when the worker thread is blocked inside an LLM HTTP call (thinking-mode models like Qwen 3 and DeepSeek-R1 can take 30+ seconds to surface a stopping checkpoint because no chunks yield during ``<think>`` blocks). The counter only appears after 3 seconds elapsed — quick stops show the plain "Stopping research…" label without a number. The counter is cleared automatically whenever research stops, completes, or errors out. This is a UX-only mitigation; reducing the underlying termination latency would require deeper work such as per-token streaming or signal-based HTTP interruption.
+- Chat mode now streams inline citation hyperlinks *as the answer is written* rather than only after the final save. Each chunk sent to the client is run through the same citation formatter the final answer uses, with a small carry buffer that holds incomplete `[N` tokens straddling chunk boundaries until the closing `]` arrives — so the user sees `[[arxiv.org-1]](url)` (or whichever format their ``report.citation_format`` setting selects) appearing live, and there is no visible format-flip when the saved answer replaces the streaming bubble on completion. The server-side partial-content buffer still stores the raw chunk so terminate-mid-stream saves aren't double-formatted on resume.
+- Chat-mode live milestones now use friendlier wording for the agent's tool calls and observations. ``Tool: search_pubmed — "covid"`` becomes ``🔍 Searching PubMed: "covid"``, ``Tool: fetch_url — "https://…"`` becomes ``📖 Reading the page: "https://…"``, and tool results show as ``📄 From {engine}: …`` so the thinking-text reads like a narration of what the agent is doing rather than a dump of internal tool names. Engines without an explicit display name fall back to a cleaned ``Title Case`` form of the raw tool name, so newly added search engines work without a code change.
+
+### 🐛 Bug Fixes
+
+- Added a `weakref.finalize`-based safety net inside the Ollama
+  embeddings factory so that programmatic API callers and example
+  scripts that construct `OllamaEmbeddings` directly — bypassing
+  the managed RAG service lifecycle — still release their underlying
+  httpx clients when the instance is garbage-collected.
+- Chat Mode correctness, settings, and accessibility fixes. (1) The streaming citation path in ``base_citation_handler._invoke_with_streaming`` now routes its joined chunks through ``get_llm_response_text`` before returning, so a reasoning model's ``<think>…</think>`` block no longer leaks into the persisted chat answer — ``.stream()`` bypasses ``ProcessingLLMWrapper.invoke`` (the only place tags were stripped), and the streamed result now matches the non-streaming ``invoke()`` contract. (2) ``SettingsManager.set_setting``'s self-heal block only re-points a row's ``type`` when the key matches a known prefix; keys outside the dispatch map (``focused_iteration.*``, ``langgraph_agent.*``, which ship as ``type=SEARCH``) are no longer silently demoted to ``APP`` on every edit. (3) ``report_assembly_service.assemble_full_report`` no longer wraps ``_build_sources_markdown`` in a blanket ``except`` that dropped the entire Sources block on error — failures propagate to the caller's existing 500 path instead of rendering a report that looks complete but is silently missing all sources. (4) The chat ``PATCH /sessions`` route now checks the boolean returns of ``update_session_title`` / ``reactivate_session`` / ``archive_session`` and returns 500 when a DB write failed but the session still exists, instead of reporting ``success: true`` with the stale row. (5) ``ChatService.delete_session`` sets the in-memory termination flags only **after** the delete commits, so a failed commit can no longer kill the in-flight research of a session that still exists. (6) ``build_research_context`` now populates ``original_query`` (the session's first user message) so the contextual follow-up strategy keeps its topic anchor instead of reading an empty string. (7) The mid-stream termination paths set ``streaming_state["_termination_handled"]`` so the ``ResearchTerminatedException`` handler no longer calls ``handle_termination`` a second time — eliminating the duplicate SUSPENDED status update, duplicate final socket emit, and doubled test-mode sleep. (8) The chat page route uses ``render_template_with_defaults`` so ``?v={{ version }}`` asset cache-busting is no longer empty. (9) The direct-mode capacity-reject path in ``processor_v2._start_research_directly`` now increments ``QueueStatus.queued_tasks`` (via ``add_task_metadata``) for the re-queued research — previously the counter stayed 0, so ``_process_user_queue`` could treat the queue as empty and leave the row undispatched until later user activity re-pumped the counter. (10) The streaming chat bubble no longer sets a nested ``role="status"`` inside the ``role="log"`` message list, which had made screen readers announce every streamed chunk twice. A regression test asserts the streaming path strips ``<think>`` from the returned synthesis.
+- Chat Mode data-integrity fixes — message ordering now respects ``sequence_number`` for same-millisecond rows, the "Load older" cursor uses a composite ``(created_at, id)`` key so boundary rows are no longer silently dropped, Markdown export pages through the full conversation (previously truncated to 50 messages), and the partial-save idempotency flag is set only after the database write succeeds so a transient failure no longer permanently loses the assistant response.
+- Chat Mode fixes for error/stopped-state button visibility, streaming citation carry-over, and accessibility. ``handleResearchError`` and ``handleResearchSuspended`` in ``chat.js`` now call ``showSessionButtons()`` so the edit-title / export controls reappear after a Stop or a failed research — previously only the successful-completion path restored them, leaving the buttons hidden until page reload after a stop or error. The streaming completion finalizer now flushes the citation carry buffer (``_flush_carry`` exposed via ``streaming_state``) before emitting the ``is_final`` sentinel, so an LLM stream that ends mid-token like ``"[12"`` no longer silently drops the leading bracket from the client's accumulated text. ``_PARTIAL_BRACKET_RE`` accepts the lenticular opener ``【`` in addition to ASCII ``[`` — some LLMs emit Chinese-style citation brackets and the citation formatter already recognises them, so the carry buffer needed to hold those back the same way. The error-path ``add_message`` in ``research_service.py`` now passes ``allow_archived=True`` matching the completion and stop-and-partial paths, so a session archived mid-flight no longer silently swallows the "research failed" message. ``ChatService.get_in_progress_research_id`` now re-raises ``DB_EXCEPTIONS`` instead of returning ``None``: a transient DB error during session load no longer looks identical to "no research running" and the route handler returns a 500 the client can surface as an error banner. Chat-css gains visible ``:focus-visible`` rings on send / stop / edit-title / ``#chat-page .ldr-btn``, a real focus ring on ``#chat-input`` (replacing the invisible 15%-opacity wrapper shadow), ``prefers-reduced-motion`` gates on the streaming caret pulse and thinking-dot bounce animations, ``:focus-within`` reveal for the per-message timestamp meta (previously hover-only and therefore invisible to keyboard users), a 44×44 minimum touch target for the mobile send button (WCAG 2.5.5), and the responsive breakpoint aligned to ``767px`` so the chat container's top-bar height assumption no longer disagrees with the rest of the project at exactly 768px. Minor cleanup: ``timedelta`` is now imported at module scope in ``chat/routes.py`` instead of inside ``send_message``, ``ChatContextManager`` is no longer re-exported from ``chat/__init__.py`` (every caller imports from ``chat.context`` directly), and the docs/features.md chat-mode section now mentions Markdown export.
+- Chat Mode input-validation and resource-bound hardening. ``send_message`` now parses the numeric search settings (``search.iterations``, ``search.questions_per_iteration``) **before** the atomic write that commits the user message + IN_PROGRESS research row — a malformed (non-numeric) settings value now returns a clean 400 instead of raising an unhandled 500 after the rows are committed, which previously orphaned them and left the session unusable via the per-session 409 guard. The inline-citation carry buffer in ``research_service._make_chat_stream_callback`` is now capped at 64 bytes: a never-closing ``[`` followed by an endless digit run from a misbehaving LLM is flushed raw past the cap instead of growing the buffer without bound. The client-side ``streamedContent`` accumulator in ``chat.js`` is likewise capped at 256 KB (mirroring the server's ``_MAX_PARTIAL_BUFFER_BYTES``) with a one-time "(Response truncated — exceeded display limit.)" notice, so a model with no ``max_tokens`` can't OOM the browser tab. Two secondary citation regexes that the earlier lenticular-bracket work had missed now also accept ``【N】``: the source-list parser in ``citation_formatter.py`` (RIS export) and the citation-strip in ``benchmarks/graders.py``. Regression tests were added for the carry-buffer flush + overflow contract and the LLM-title newline strip.
+- Chat Mode reliability fixes. The log panel now loads historical logs when expanded on a chat page (the toggle previously held a stale null research id). Delete / clear-history failures now surface an error notification instead of silently doing nothing. When a completed research's answer can't be loaded into the chat, the message now links to the full report (which is still saved) rather than a dead-end "no report available". The header "New Chat" / "Export" buttons are now styled (their base CSS wasn't loaded on the chat page). Stopping a research while it is still starting up now reliably terminates it instead of being silently ignored, and a late error event can no longer overwrite an already-rendered completed answer.
+- Chat Mode rendering and log-safety fixes. ``chat.css``'s `.ldr-chat-container` height now uses `calc(100dvh - …)` (with `100vh` retained as the previous-line fallback, matching the pattern already used in ``mobile-navigation.css``) so iOS Safari no longer buries the chat input below the collapsible URL bar. ``chat.js``'s ``handleResearchComplete`` else-branch (the one that fires when streaming didn't reach its `is_final` sentinel — most commonly after a flush-then-disconnect race introduced by the `_flush_carry` change) now reuses the in-place-swap pattern from the streaming-complete branch when a partial bubble is already on screen: it removes the streaming class and renders the formatted message into the same `.ldr-chat-message-text` element instead of `.remove()` + `addMessageToUI`. Eliminates the 5-8.5 s vanish-then-refetch flicker. The original detach-and-reinsert path is preserved for the no-partial-bubble case where no flicker would occur anyway. ``ChatService.regenerate_title_with_llm`` now strips `\n` and `\r` from the LLM-returned title before storage; the title is interpolated into loguru f-strings (the "title already set" log line one above this fix), so an embedded newline would otherwise forge what looked like a second log entry in aggregators. Also keeps ``document.title`` and ``chatTitle.textContent`` visually clean.
+- Chat Mode report-assembly, layout, and accessibility fixes. ``format_links_to_markdown`` (``utilities/search_utilities.py``) now coerces citation indices to ``str`` before ``sorted()`` so mixed ``int``/``str`` values arriving from different strategies (e.g. ``recursive_decomposition_strategy.py`` emits ``int``, ``_build_sources_markdown``'s fallback emits ``str``) no longer crash report assembly with ``TypeError: '<' not supported between instances of 'str' and 'int'``; the sort still produces numeric order. ``chat.css`` desktop and mobile chat-container heights now subtract the real ``.ldr-top-bar`` heights (60px / 50px) instead of the previous over-reserved 80px / 60px, ending the viewport overflow that clipped the chat input area. ``research_service.py`` ``add_progress_step`` failure path now zeroes ``event_data`` inside its ``except`` block so a DB-lock that swallows the persistence call also suppresses the live socket emit — preserving the documented live/reload symmetry invariant (a chat step the client sees live is the same set the persisted log returns on reload). Two icon-only delete buttons in ``history.js`` gained ``aria-label`` / ``title`` for WCAG 2.1 Level A compliance.
+- Chat Mode: when research completes but the streaming-bubble swap path doesn't end up with a visible assistant response (transient socket drop, race during session switch, missed ``is_final`` chunk, etc.), the chat now silently re-renders from the DB-authoritative ``/messages`` endpoint instead of leaving the user staring at an empty page until they manually refresh.
+- Citation formatter: ``apply_inline_hyperlinks`` (the fallback path that chat-mode answers always hit because the langgraph-agent synthesis doesn't emit a ``## Sources`` block in its prose) now dispatches on ``CitationFormatter.mode`` instead of hard-coding ``NUMBER_HYPERLINKS``. The user's ``report.citation_format`` setting is now honored for chat-mode citations — picking "Domain ID Hyperlinks" or "Source-Tagged Hyperlinks" actually produces ``[[arxiv.org-1]](url)`` / ``[[arxiv-1]](url)`` instead of every chat answer coming out as ``[[1]](url)``.
+- Citation formatter: ``apply_inline_hyperlinks`` now accepts either ``"url"`` or ``"link"`` as the destination key on each source dict. Searxng-sourced results carry the destination under ``"link"`` (``search_engine_searxng.py``), which the fallback hyperlink path silently dropped — so chat-mode answers that did not emit a ``## Sources`` block in their synthesis shipped with plain ``[N]`` brackets instead of clickable citations even though the Sources section beneath the answer was fully populated.
+- Fix report structure parsing crashing with `IndexError` on a numbered section line without a period (e.g. `1 Introduction`), and stop truncating section names that contain periods (`1. U.S. Policy` is now kept whole instead of becoming `U`). The section number is now split on the first period only, with a guard for malformed lines.
+- Fixes for bugs that affect the **non-chat** research path as well as chat. (1) ``LangGraphAgentStrategy.analyze_topic`` no longer overwrites its ``query`` parameter with a truncated tool-call argument inside the tool-call display loop — because ``langgraph-agent`` is the default strategy, after the first ``web_search`` the original research question was silently replaced by a ≤80-char search arg, which then fed the citation re-synthesis, the fallback synthesis, and the recorded ``question`` field, steering the final answer at the wrong question. (2) The queue dispatcher (``processor_v2._start_queued_researches``) now has a dedicated ``except SystemAtCapacityError`` clause: a transient at-capacity rejection is left QUEUED for the next tick instead of being counted toward ``SPAWN_RETRY_LIMIT`` and wrongly marked FAILED after a few ticks under load. (3) ``cleanup_research_resources`` now reports the real terminal status (passed in by the caller — SUSPENDED on user termination, FAILED on error) instead of a hard-coded ``COMPLETED``, so stopping a research no longer emits a spurious "completed" socket signal (which produced a stray "[Stopped]" bubble in chat and a misleading 100%/Completed flip on the standard progress page); chat.js's completion/suspension handlers also cross-guard each other. (4) Chat-triggered research now inserts a ``UserActiveResearch`` row, so it counts toward the per-user ``app.max_concurrent_researches`` cap the same way UI-launched research does — previously chat research was invisible to the cap (queried but never recorded), letting multiple chat tabs bypass it. The row is committed atomically with the research row, cleaned up on spawn failure, and removed on normal completion by the existing completion-sweep middleware.
+
+  Regression tests added for all four.
+- GitHub search relevance ranking now rejects negative, out-of-range, and non-integer result indices and deduplicates repeated ones, so a malformed LLM response can no longer select the wrong result, list the same repository twice, or discard all results.
+- Library RAG service is now closed at the end of every HTTP request
+  that uses it — including streaming endpoints. Together with the
+  embedding-manager close path, this stops file-descriptor accumulation
+  under sustained library indexing/search traffic.
+- News subscriptions run via "Run now" or the overdue-subscription check no longer force the LLM to ``ollama``/``llama3`` when the subscription has no explicit model configured. The run paths in ``news/flask_api.py`` hardcoded those values, which (being truthy) overrode the user's configured ``llm.provider``/``llm.model`` — so a user on OpenAI/Anthropic whose subscription carried no model would silently get an Ollama run, typically failing. The hardcoded defaults are removed; an unset model now passes through as ``None`` so ``start_research`` falls back to the user's settings.
+- Stopped a per-RAG-request file-descriptor leak introduced when the
+  embeddings provider migrated to `langchain_ollama.OllamaEmbeddings`.
+  The library RAG service now closes the underlying httpx clients on
+  teardown, preventing eventpoll FD accumulation under sustained
+  indexing/search load.
+- Strip `<think>` reasoning blocks from the main synthesis path (`synthesize_findings` and the standard knowledge generator), not just the citation handlers — reasoning-model output no longer leaks `<think>…</think>` into final answers. Also fixes precision/forced answer extractors emitting a stray `". <content>"` when the model returns an empty (or think-only) response.
+- The WebSocket subscribe ownership check now recognizes benchmark runs in addition to normal research. The new per-user ownership gate (and the removal of the cross-user broadcast fallback) only matched ``ResearchHistory`` UUIDs, so the benchmark page — which subscribes with an integer ``BenchmarkRun`` id — was rejected and its live progress (current-task detail and the SearXNG rate-limit warning) was silently dropped. ``_user_owns_research`` now also accepts the user's own ``BenchmarkRun`` rows from their per-user encrypted database, restoring benchmark live progress without widening the authorization boundary.
+- The central LLM wrapper now normalizes string-returning providers into a message object and applies `<think>`-tag stripping to async (`ainvoke`) calls too, so any LLM obtained from `get_llm` yields a consistent, think-free `.content` — eliminating `'str' object has no attribute 'content'` crashes at the source. Message objects keep their `tool_calls`/`reasoning_content` (only `.content` is rewritten).
+- The chat ``send_message`` endpoint now uses the shared ``@require_json_body`` decorator like the other state-changing chat POSTs. It was the only one validating the body inline, so a non-JSON content type slipped past the consistent 400 contract; requiring an ``application/json`` body also hardens CSRF on the heaviest chat endpoint (it launches a research run). Behavior for valid requests is unchanged.
+- The queued-research dispatch loop now reverts its queue-counter claim when a global-capacity reject re-queues a research. Previously, ``_start_queued_researches`` marked the task ``processing`` (decrementing ``queued_tasks``, incrementing ``active_tasks``) and, on ``SystemAtCapacityError``, only reset the row's ``is_processing`` flag — leaking a slot into ``active_tasks`` on every capacity-rejected retry. Under sustained capacity pressure ``queued_tasks`` would drift to 0, at which point the per-user queue processor treated the queue as empty and stopped dispatching the still-present rows. ``update_task_status`` now supports a ``processing`` → ``queued`` transition that restores the counts, and the capacity-reject path uses it.
+- `GET /api/report/<id>` now returns the `sources` field from the structured `research_resources` table instead of the dead `all_links_of_system` metadata key. After the chat-mode-v2 report refactor that key is never written, so the field returned an empty list for every research created since — even though the assembled `content` and the news feed already read sources from the table. (#3665)
+
+### 📝 Other Changes
+
+- **`research_history.report_content` shape — answer-only at persistence time.** Internal/library change tied to the Chat Mode work: `report_content` now stores the synthesized answer (LLM prose + inline citations) rather than the full `format_findings` blob (which previously embedded `## Sources` and `## Research Metrics` sections). User-facing views are unchanged — `assemble_full_report()` in `web/services/report_assembly_service.py` reconstructs the legacy display shape on demand for the history page, exports, and the chat "View full research" link, and legacy rows that still contain the inline sections are detected and not double-appended. **Direct callers of `storage.get_report()` / `storage.get_report_with_metadata()` should switch to `assemble_full_report()` if they need the legacy shape.**
+- CI: fix the freshly-merged ``check-shadow-tests`` pre-commit hook using ``os.path.basename()``, which the repo's own ``check-pathlib-usage`` hook forbids. Because PR pre-commit runs ``--all-files`` against the PR-merged-into-main tree, this one line on main turned every open PR's pre-commit red. Switched to ``pathlib.Path(path).name`` (behaviour verified identical to ``os.path.basename`` across separator/edge cases).
+- CI: split ``tests/web/routes/test_settings_routes_coverage.py`` out of the parallel pytest run into a dedicated serial step (same pattern as the existing ``fd_canary`` step). The 94 tests in that file all use the ``authenticated_client`` fixture (register + login + SQLCipher KDF + 10 Alembic migrations per test), and under ``-n auto`` they accumulate enough connection / FD pressure inside the assigned xdist worker to deadlock it silently late in the run — the worker dies with ``[gw0] node down: Not properly terminated`` (no timeout printed), its coverage data is dropped, and the 50% fail-under gate then fails the job even though every test that completed actually passed. Skipping individual tests in this file just relocates the death (confirmed empirically on this PR's earlier iteration). The serial step keeps all 94 tests running, appends into the main run's ``.coverage`` data via ``--cov-append``, and runs the final coverage report + 50% gate after both contributions have merged. Defence-in-depth: pairs with #4393's 180s/thread timeout.
+- CI: switch ``pytest-timeout`` from ``signal`` to ``thread`` method and raise the global per-test timeout from 60s to 180s. On Python 3.14 the SIGALRM-based interrupt was firing inside ``weakref.py`` cleanup, corrupting xdist workers and dropping their coverage data — which then put total coverage below the 50% gate even when every test that completed actually passed. The thread method raises a Python-level exception in the main thread instead of interrupting at an arbitrary safe-point, and 180s gives the heavy ``authenticated_client`` register+login fixture (SQLCipher KDF + Alembic migrations) headroom under ``-n auto`` + coverage contention.
+- Chat Mode polish and test-quality improvements. ``chat/routes.py`` adopts the shared ``@require_json_body(error_format="success")`` decorator on the three POST/PATCH endpoints (``create_session``, ``generate_session_title``, ``update_session``), replacing 4-block inline ``isinstance(data, dict)`` guards. ``UserActiveResearch`` stale-row reclaim is extracted from both ``chat.routes.send_message`` and ``research_routes.start_research`` into a shared ``reclaim_stale_user_active_research(db, username, *, grace_cutoff_dt=None, logger=None)`` helper in ``web/routes/globals.py`` — chat passes a 30s grace cutoff (because chat send can race with its own concurrent sibling), research_routes passes None (matching its pre-existing behaviour). ``ChatService.delete_session`` now imports ``set_termination_flag`` at module top-level instead of inside the function body (no circular import requires the deferred import). The dummy ``"I understand. What else would you like to know?"`` assistant bubble that ``chat.js`` injected when the server returned no ``research_id`` is gone — the thinking indicator is now cleared without injecting a placeholder reply. ``chat.js::loadSession`` now focuses the input on completion, matching ``startNewChat``. Three brittle test patterns are replaced: the seven copy-pasted ``captured_username`` + ``@contextmanager`` blocks in ``test_chat_user_isolation.py`` are consolidated under a single ``username_capturing_db`` fixture; the 16 inline ``SocketIOService._instance = None`` reset lines in ``test_chat_socket_events.py`` become a single autouse fixture that runs in setup AND teardown regardless of test outcome; ``test_chat_settings_integration.py``'s three tautological dict-assertion tests are replaced with four real tests that exercise ``_load_settings`` against patched ``SettingsManager`` and validate every snapshot-extraction key still exists in ``default_settings.json``. ``test_chat_send_message_reclaim.py``'s three source-grep tests against ``chat/routes.py`` are replaced with eight behaviour tests that drive the new shared reclaim helper against a real SQLite session (covering grace-window respect, live-thread skip, username scoping, and the without-cutoff immediate-reclaim mode used by research_routes). ``test_chat_service.py`` drops three ``if hasattr(...)`` assertion guards that would have silently passed when a real model regressed; the ``get_in_progress_research_id`` DB-error test inverts to assert propagation now that the service re-raises instead of returning None. ``test_chat_api.py``'s ordering assertion no longer hides behind ``if len(sessions) >= 2:``.
+- Chat Mode polish fixes. The agent-thinking milestone for ``research_subtopic`` now reads ``🔬 Investigating subtopic: "topic1, topic2"`` instead of empty quotes — the display code now picks up the tool's actual ``subtopics: list[str]`` argument and stringifies the list. ``send_message``'s ``UserActiveResearch`` reclaim block now emits a ``logger.warning`` mirroring the ``ResearchHistory`` reclaim above it, so operators can trace why a per-user concurrency cap was released. ``_validate_title`` strips Unicode format / line-separator characters (``Cf``/``Zl``/``Zp`` — zero-width spaces ``U+200B-U+200D`` and BOM ``U+FEFF``) before the emptiness check, so a title of 500 zero-width chars is rejected instead of saving a session that looks blank in the UI. ``regenerate_title_with_llm`` is now idempotent: it skips the LLM call if the session title no longer matches the non-LLM fallback, so a concurrent tab's edit (or the user's manual edit completing first) is not overwritten by a stale LLM round-trip. The dead ``build_research_context_for_session()`` wrapper in ``chat/context.py`` is removed along with its dedicated tests — production code uses ``ChatContextManager`` directly.
+- Chat follow-up research now logs which prior-context mode ran and how many characters of context it built. Previously the context-building step (especially the LLM summary) produced no log output, so a follow-up's preparation looked like an unexplained pause.
+- Clarified the auto-generated `app.debug` / `LDR_APP_DEBUG` description in `default_settings.json` (and therefore `docs/CONFIGURATION.md`) to reflect the split introduced when `LDR_LOGURU_DIAGNOSE` was added: `LDR_APP_DEBUG` now raises log level only and explicitly does NOT enable Loguru's local-variable dumps on its own. Companion to the `LDR_LOGURU_DIAGNOSE` security fix.
+- Document a known limitation: LangGraph `create_agent`/`bind_tools` (in the `langgraph-agent` and `mcp` strategies) resolve to the base LLM via `ProcessingLLMWrapper.__getattr__`, bypassing the wrapper's `<think>`-tag stripping. Added in-code notes at the three call sites. This is a cosmetic leak only (reasoning-model `<think>` blocks may appear in agent output); it does not affect direct `invoke()` calls, which still go through the wrapper.
+- Documented that the News API `findings` field is the answer-only report content after the chat-mode-v2 refactor (intentional): structured top-N source links live in the separate `links` array rather than an embedded `## Sources` blob. No behavior change. (#3665)
+- Test fixtures and dev scripts that re-configure loguru (``tests/conftest.py`` loguru-caplog fixture, the two ``tests/api_tests/`` ``__main__`` runners, ``tests/test_openai_api_key_e2e.py``, ``tests/settings/test_manager_behavior.py``, ``tests/journal_quality/test_db.py``, ``tests/performance/mcp/echo_server.py``) now pass ``diagnose=False`` to their ``logger.add(...)`` calls, matching the production policy locked in by #4384 / #4394. loguru defaults ``diagnose=True``, which renders ``repr()`` of every traceback frame's local on exception; pinning it off keeps frame-local credentials (api keys, SQLCipher passwords, ``Authorization`` headers) out of pytest output and CI logs even when a test crashes mid-fixture. Hygiene change only — no behavior change for green tests.
+- The news-feed "time ago" formatter no longer swallows unparseable timestamps behind a silent "Recently" label. Since `created_at` is always written as a valid ISO timestamp, a value that won't parse means a corrupt row — that row is now logged and skipped by the feed builder instead of rendering with a misleading time.
@@ -0,0 +1,278 @@
+
+### 💥 Breaking Changes
+
+- Removed the `mcp` / `agentic` (ReAct) search strategy. Existing selections are automatically migrated to `langgraph-agent` (a near functional superset); connecting to external MCP servers as research tool sources is no longer supported. (LDR still ships its own MCP **server** for exposing research to assistants like Claude — that is unaffected.) ([#4548](https://github.com/LearningCircuit/local-deep-research/pull/4548))
+- **Breaking (programmatic API):** `get_llm(provider=…)` and `get_embeddings(provider=…)` now **fail closed** when called with no settings snapshot (`settings_snapshot=None`) for any provider that is not a known local-default. Snapshot-less callers may instantiate only the localhost-default providers — LLM: `ollama`, `lmstudio`, `llamacpp`; embeddings: `sentence_transformers`, `ollama` — plus LLMs you registered in-process via the programmatic API (`quick_summary(llms={…})`). Any other provider (incl. `openai`, `anthropic`, `google`, `openrouter`, and any future cloud provider) raises `PolicyDeniedError` instead of silently constructing a cloud client. This closes a snapshot-less egress hole; programmatic callers that relied on building a cloud provider without a snapshot must now pass a `settings_snapshot` (or register the LLM in-process). The allowlists are intentionally tight so unknown/new providers fail closed by default.
+- **The `auto` and `parallel` meta search engines have been removed** (including the `meta` and `parallel_scientific` aliases). The default langgraph-agent strategy replaces them — it selects search engines dynamically per query, so a separate LLM-based engine picker is redundant. Stored values (`search.tool` setting, news subscriptions, queued researches, saved benchmark configs) are migrated automatically on upgrade (migration 0013; `search.tool` values pointing at a removed engine become `searxng`). **What to do:** API callers passing `search_tool="auto"` or `"parallel"` must pick a concrete engine (e.g. `searxng`), and `LDR_SEARCH_TOOL=auto` environment overrides must be updated likewise.
+- **Upgrade behaviour change:** the default egress scope is now `adaptive` (was effectively `both`). Adaptive follows your primary search engine, so on upgrade a config with a *concrete* primary may narrow: a public primary (e.g. SearXNG/arXiv) now excludes local collections (public-only behaviour), and a *private collection* primary forces local LLM + embeddings. To keep the previous "any engine, cloud inference" behaviour, set Egress Scope to **Both**. See docs/egress-modes.md.
+
+### 🔒 Security
+
+- News recommendation logging now sanitizes externally-derived topic strings (and search error text) before interpolating them into log records, closing a log-injection vector where crafted news content could forge log lines or inject terminal escape sequences. ([#3767](https://github.com/LearningCircuit/local-deep-research/pull/3767))
+- **`/settings/api` no longer leaks env-overridden API keys to authenticated users.** The bulk settings JSON endpoint now redacts password-typed values (`llm.openai.api_key`, search-engine API keys, etc.) so they come back as `[REDACTED]` instead of plaintext. Metadata is preserved so the UI still renders correctly. Note: the settings form template still pre-fills password inputs server-side — that's a separate UX issue and not addressed here. ([#3947](https://github.com/LearningCircuit/local-deep-research/pull/3947))
+- **Settings form no longer pre-fills password inputs with the stored value**, eliminating an authenticated View-Source disclosure of API keys / OAuth tokens. Password fields render empty with a placeholder indicating configuration state and `autocomplete="new-password"` to prevent browser caching. ([#3954](https://github.com/LearningCircuit/local-deep-research/pull/3954))
+- Extended `redact_secrets()` coverage to credential-bearing exception handlers in the encrypted-database, web queue, and scheduler subsystems — sites where the user's SQLCipher master password is in lexical scope. Unlike API keys (rotatable), the SQLCipher master password is unrecoverable (TRUST.md §5), so a leak via a rendered traceback or upstream exception message is a permanent compromise. Covers `database/encrypted_db.py` (6 sites in `create_user_database` / `open_user_database` / `change_password`), `web/queue/processor_v2.py:_start_research_directly`, and 11 sites across `scheduler/background.py` (subscription scheduling, document processing, RAG indexing, overdue-subscription handling, subscription research trigger). See #4182. ([#4182](https://github.com/LearningCircuit/local-deep-research/pull/4182))
+- Egress policy hardening (PR #4300, Round 6 review): 24 fixes across DNS classification (process-global socket timeout removed, threading.RLock around cache mutations), PDP coverage (nested-dict settings unwrap, NAT64 metadata reclassification, fetch_content tool gate), engine/RAG/data coverage (MetaSearchEngine Wikipedia fallback routed through factory, all 5 direct LibraryRAGService sites pre-flight-checked, SearchCache hash includes scope to close cache-bypass), and UI/cross-cutting work (per-research policy overrides on the research form, settings dashboard renders the policy keys, benchmark snapshot threading, policy_audit WebSocket filter, audit log on policy-key changes, dismiss-flag rename to follow the dismiss_* convention). 23 regression tests added.
+
+  A second adversarial review round added 13 more fixes: closed two egress bypasses (Elasticsearch `cloud_id` is now refused under private scope; OpenAI/discovered-provider model-list probes are scope-gated in both the settings and RAG model endpoints); closed a metadata-SSRF gap (literal cloud-metadata IPs are now classified PUBLIC on the literal-IP branch of `_classify_host`, matching the DNS branch); ADAPTIVE now resolves a registered local-retriever primary to PRIVATE_ONLY (was BOTH, which leaked the private corpus to cloud inference); the PEP-578 audit-hook backstop is now armed on non-web entry points (CLI / news scheduler / programmatic API) and re-armed on ThreadPoolExecutor pool workers; fixed the SearXNG `url_setting` key so a local SearXNG is no longer over-blocked under PRIVATE_ONLY; `allowed_local_hostnames` no longer rejects unresolvable intranet hosts on save; the denied-fetch quota no longer counts benign parse failures; and `DownloadService` fails closed when the settings backend errors. 10 regression tests added.
+
+  Follow-up regression fix: user-registered in-process LLMs (the programmatic API's `llms={...}` and plugins) are no longer denied with `provider_url_unset` when a run resolves to require-local inference (e.g. ADAPTIVE with a registered-retriever primary). The PDP now distinguishes user registrations from auto-discovered built-in providers — a custom name shadowing a built-in cloud provider stays blocked, and snapshot-less calls keep failing closed for everything else. 3 regression tests added.
+
+  Third adversarial review round (5×20 agents) confirmed and fixed 6 more issues (4 of 11 verified findings were declined as not-exploitable or design-intent, 1 was a false positive): the `evaluate_url` cloud-metadata block now normalizes alternate IPv4 encodings (octal `0251.0376.0251.0376`, hex, integer) so an IMDS literal can't read as an allowed public host under PUBLIC_ONLY/BOTH and always denies with the explicit `blocked_metadata_ip` reason; `dangerous_scheme` (javascript:/data:/file: hrefs) no longer counts toward the per-run denied-fetch quota (matching `unsupported_scheme`, so a document full of data: URIs can't starve later legitimate fetches); `evaluate_llm_endpoint` and `evaluate_embeddings` now percent-decode the host before classification (consistency with `evaluate_url`, so a legitimate percent-encoded local endpoint isn't wrongly denied under require-local); `context_from_snapshot` fails closed with `ValueError` on a non-dict snapshot instead of a swallowable `AttributeError`; and the run-scoped DNS classification cache is now first-writer-wins so concurrent disagreeing lookups (round-robin DNS) can't flip a hostname's classification mid-run. 11 regression tests added.
+
+  Round 2 of the 5×20 review (audit-hook + thread-context propagation) confirmed and fixed 4 issues (of 21 findings: 6 false positives, 1 already-fixed, several lower-severity backstop-completeness gaps deferred with the primary PEPs as the live gate): the PEP-578 audit hook now decodes bytes-encoded socket addresses before classifying (CPython fires `socket.connect` with a bytes host, which previously passed straight through the PRIVATE_ONLY/STRICT backstop); `set_active_context` rejects an unresolved ADAPTIVE-scope context (it would have silently no-op'd the hook); `active_egress_context` saves and restores the previous context so a nested activation no longer wipes the parent's; and the default `langgraph-agent` strategy now re-arms the audit-hook backstop inside each subagent ThreadPoolExecutor worker (threading.local isn't inherited by pool workers). 4 regression tests added.
+
+  Round 3 of the 5×20 review (call-site PEPs / data-egress paths) confirmed the empty-snapshot fail-open as the one genuinely exploitable issue and fixed it (most of the 18 findings were lower-severity than first rated — UI-display consistency, config-time validations already backstopped by execution-time PEPs, documented accepted-risk redirects, or backward-compatible empty-snapshot defaults): the `quick_summary` REST endpoint now fails CLOSED (HTTP 503) when the user's settings snapshot can't be loaded, instead of continuing with an empty `{}` snapshot that silently downgraded a configured PRIVATE_ONLY/require-local user to the permissive BOTH scope. News-subscription policy validation now also surfaces an incoherent egress config (STRICT scope + meta-picker primary, which raises ValueError) as a fail-fast misconfiguration error at create/update time rather than silently skipping the check. 1 regression test added.
+
+  Round 4 of the 5×20 review (adversarial bypass hunting) found no new exploitable bypass — every candidate was a known/documented accepted-risk (DNS-rebinding and redirect TOCTOU, per-context fetch quota, connect-only audit-hook scope), a fail-safe over-restriction, or an operator-trust-boundary case (a programmatic caller controls its own snapshot). One cheap defense-in-depth completion was applied: `evaluate_url` now also blocks cloud-metadata endpoints reachable by hostname (GCP `metadata.google.internal` / `metadata.goog`, any case) and strips an insignificant trailing dot before the metadata checks, so `metadata.google.internal.` and `169.254.169.254.` can no longer read as an allowed public host (closing the hostname/trailing-dot gaps the round-1 octal/hex/integer fix didn't cover). 2 regression tests added.
+
+  Round 5 of the 5×20 review (quality / docs / UI / completeness) fixed 5 issues: policy-audit denial logs now redact URLs (scheme://host:port only) via the existing `redact_url_for_log` at all four sites (ContentFetcher, DownloadService, and both full_search paths), so a denied URL carrying userinfo credentials or an API-key query param is no longer written verbatim to the audit log; the ADAPTIVE warning-banner render path (`/api/warnings`) now resolves scope with `allow_dns=False` so a settings-page load can't block up to the DNS timeout on a synchronous getaddrinfo for a URL-engine primary; `CONFIGURATION.md` was regenerated so its Settings List documents the egress keys and their `LDR_*` env vars (the hand-written prose, which the on-merge auto-regeneration would have wiped, moved into the hand-maintained egress-modes.md with the cross-link fixed); plus a missing `_retriever_is_local` return annotation and an added BOTH-scope allow test. 3 regression tests added. ([#4300](https://github.com/LearningCircuit/local-deep-research/pull/4300))
+- FAISS RAG indexes are now loaded through a restricted unpickler that only permits the two classes a legitimate docstore contains, instead of `FAISS.load_local(..., allow_dangerous_deserialization=True)`. A tampered `index.pkl` (the docstore pickle that is actually deserialized) can no longer execute arbitrary code on load — closing the pickle-deserialization RCE for every case, including a first-encounter index with no integrity record and a `.pkl`-only swap that leaves the `.faiss` checksum unchanged. The fix needs no integrity record, schema change, or re-indexing. The loader is fail-closed: if a future LangChain/Pydantic upgrade changes the docstore pickle format to use other classes, affected indexes raise `UnpicklingError` on load (and are quarantined and rebuilt) rather than silently deserializing — recover by re-indexing the affected collection, or by widening the loader's allow-list to the new class. ([#4632](https://github.com/LearningCircuit/local-deep-research/pull/4632))
+- **WebSocket/Socket.IO connections now default to same-origin only.** When `LDR_SECURITY_WEBSOCKET_ALLOWED_ORIGINS` is unset or empty, cross-origin WebSocket connections are rejected instead of being allowed from any origin — closing a Cross-Site WebSocket Hijacking gap as defense-in-depth (the session cookie's `SameSite=Lax` already blocked the classic cross-site case) and matching the same-origin default already used for HTTP CORS. To run a cross-origin front-end, set `LDR_SECURITY_WEBSOCKET_ALLOWED_ORIGINS` (and `LDR_SECURITY_CORS_ALLOWED_ORIGINS`) to its origin. If you terminate TLS at a reverse proxy, forward `X-Forwarded-Proto` so the same-origin check sees `https` — otherwise the WebSocket handshake is rejected (see the troubleshooting guide). Set `*` to restore the previous allow-all behavior on trusted local/dev networks. A rejected WebSocket handshake is now logged with a warning naming the origin and how to allow it, so a misconfigured origin is diagnosable instead of a silently frozen progress UI. ([#4807](https://github.com/LearningCircuit/local-deep-research/pull/4807))
+- **Local file/RAG path validation no longer 500s or blocks legitimate paths.** Browsing to a restricted path on a non-root deployment returned a 500 (an uncaught permission error from a symlink pre-check) instead of a clean "invalid path" response, and legitimately symlinked index folders (Docker/Kubernetes bind mounts, macOS `/tmp`) were wrongly rejected; both are fixed, and the system-directory block now also holds on macOS (where `/etc` is itself a symlink to `/private/etc`). Separately, a `LDR_DATA_DIR` path containing an apostrophe (e.g. `/home/O'Brien/ldr`) is now fully supported: it no longer crashes startup, and encrypted-database backups handle it too (the apostrophe is safely escaped in the backup's `ATTACH DATABASE` statement). ([#4808](https://github.com/LearningCircuit/local-deep-research/pull/4808))
+- **Restricted-directory access blocks no longer log the user's path.** When local file/RAG path validation blocks access to a system directory, the error log now names only which restricted directory was hit (e.g. `/etc`) instead of the user's full resolved path, which could contain a username. ([#4820](https://github.com/LearningCircuit/local-deep-research/pull/4820))
+- **Local-folder RAG indexing no longer logs the path you submitted when it's rejected.** A failed path validation on the local-folder indexing endpoint logged the raw submitted path (which can contain a username); it now logs a generic "Path validation failed" message instead. The `Invalid path` response is unchanged. ([#4825](https://github.com/LearningCircuit/local-deep-research/pull/4825))
+- **Local-folder RAG indexing logs the folder/file name instead of the full path.** The "indexing complete" log and the per-file indexing-error log now record only the folder/file basename rather than the full filesystem path (which can contain a username), while keeping the same diagnostic detail. ([#4831](https://github.com/LearningCircuit/local-deep-research/pull/4831))
+- **API error responses no longer leak raw exception text or server config (CWE-209).** Several response paths embedded internal exception detail into messages returned to the client: the `/api/news/*` endpoints (database/SQLAlchemy errors), the start-research and news-subscription egress prechecks (raw `ValueError`s), the research failure handler (unrecognized exceptions *and* LLM/search/provider "configuration error" detail — including server-level endpoints and file paths, since settings can come from `LDR_*` environment overrides) via `/api/research/<id>/status` and the persisted error report, and the benchmark status endpoint (`GET /benchmark/api/status/<id>`, which returned a failed run's raw `str(e)`). All now return generic, category-appropriate messages with an actionable hint where one applies, while the full cause is retained server-side in the logs — hardening shared/multi-user deployments where a non-admin user must not see another tenant's or the server's internal configuration. ([#4843](https://github.com/LearningCircuit/local-deep-research/pull/4843))
+- Local-folder RAG indexing now validates the supplied file glob patterns against an allowlist, rejecting any pattern that could escape the indexed folder (e.g. `../../etc/*` or an absolute `/etc/*`). Previously only the base folder was validated, so on multi-tenant/shared deployments a crafted pattern could read server-side files into the requesting user's library. ([#4846](https://github.com/LearningCircuit/local-deep-research/pull/4846))
+- Local-folder RAG indexing now confines globbed matches to the indexed folder: a symlink inside the folder that points outside it (a linked file, or a linked directory recursive globbing descends into) is skipped rather than read. Previously such a symlink could expose files outside the folder, since `glob` follows symlinks and only the base folder was validated. Complements the glob-pattern validation in #4846; together they close the folder-escape surface on this endpoint. ([#4848](https://github.com/LearningCircuit/local-deep-research/pull/4848))
+- **`GET /history/report/<id>` no longer leaks the settings snapshot (API keys).** The report route returned the persisted `research_meta` wholesale in its response metadata, and `research_meta` includes `settings_snapshot` — which holds all application settings, including API keys, tokens, and base URLs. It now strips `settings_snapshot` via the same `strip_settings_snapshot()` helper its four sibling routes already use, while preserving every other (non-sensitive) metadata field the report view needs. ([#4853](https://github.com/LearningCircuit/local-deep-research/pull/4853))
+- Symlink-loop confinement for RAG local indexing now works on Python 3.13+. The check relied on `Path.resolve()` raising on a symlink loop, which newer Python no longer does; it now resolves with `strict=True` so a planted in-base symlink loop is correctly skipped rather than silently kept. ([#4864](https://github.com/LearningCircuit/local-deep-research/pull/4864))
+- **Benchmark YAML downloads no longer leak the evaluation `endpoint_url` in the default export.** The default (summary) download is meant to be safe to share, but it still emitted the evaluator's `endpoint_url`, which can be a private/internal host. It's now gated behind the existing **Export → "Include settings snapshot"** opt-in — matching the settings-snapshot gating — so the default summary stays shareable and reproducibility-minded users still get it on demand.
+- **Closed three remaining password-leak paths in the settings module.** `GET /settings/api/<key>` and `GET /settings/api/bulk` now redact password-typed values to `[REDACTED]` (matching the bulk `/settings/api` endpoint redacted in the previous release). `POST /save_settings` (the JS-disabled form-encoded fallback) now treats an empty value for a password setting as a no-op, so no submission path can wipe a stored API key with empty string.
+- Add a process-wide PEP 578 `sys.audit` hook (`security/egress_audit_hook.py`) that gates `socket.connect` against the active EgressContext. This is the secondary line of defense — every explicit PEP we wired (ContentFetcher, evaluate_llm_endpoint, MCP download_content, journal_reputation_filter, …) remains the primary line and still fires first. The hook catches what those PEPs cannot: a third-party library that opens its own connection, a new code path added without policy awareness, a langchain tool registered by an MCP server, prompt-injection steering a tool into raw `requests.get`, or anything else that ultimately calls `socket.connect`. The hook is installed once on first import of `local_deep_research.security` (idempotent — PEP 578 hooks cannot be removed) and is INACTIVE by default: with no `EgressContext` registered on the current thread, every connect passes through unmodified, so test runners, import-time helpers, and scripts that touch a socket are unaffected. Workers opt in by calling `set_active_context(ctx)` (or using the `active_egress_context` context manager); the research worker in `web/services/research_service.py` now does this after building the snapshot, and the centralized `database/thread_local_session._ThreadCleanup` exit handler clears it on worker shutdown so a pooled thread cannot leak one run's scope into the next task. The hook gates only AF_INET / AF_INET6 — AF_UNIX, AF_NETLINK and friends pass through. It does NOT defend against an adversary with code execution in the LDR process (they can clear the active context or add a passthrough hook themselves); for that, layer OS-level controls per SECURITY.md. 18 unit tests in `tests/security/test_egress_audit_hook.py` pin the contract: install idempotency, get/set/clear, context-manager cleanup on exception, per-scope behaviour against real raw sockets (PRIVATE_ONLY blocks 8.8.8.8 and permits 127.0.0.1; PUBLIC_ONLY mirrors; STRICT blocks public hosts; BOTH allows either), IPv6 loopback bracket handling, AF_UNIX pass-through, and per-thread isolation under concurrency.
+- Bumped the Docker base image from python:3.14.5-slim to python:3.14.6-slim, picking up the upstream CPython fixes for CVE-2026-9669 (bz2 decompressor reuse out-of-bounds write), CVE-2026-7774 (tarfile data_filter path-traversal bypass), and CVE-2026-3276 (unicodedata.normalize CPU exhaustion). Also suppressed three base-image CVEs with no fix available in Debian trixie (graphite2 CVE-2026-50593, expat CVE-2026-50219, perl HTTP::Tiny CVE-2026-7010 — none reachable in this container) in .grype.yaml, and removed five suppressions for python CVEs that the 3.14.5/3.14.6 releases fixed.
+- Closed a residual SQLCipher-password leak left after the #4182 sweep: `open_user_database` redacted the migration-failure log line but still re-raised `DatabaseInitializationError` with the unredacted original error embedded in its message and chained via `from init_err`. The caller (`thread_local_session`) logs that typed error with `logger.exception`, which re-rendered the chain — and the password its frame locals carry under `diagnose=True` — defeating the redaction. The typed error now carries the redacted message and breaks the chain with `from None` (ADR-0003). See #4182.
+- Closed another `diagnose=True` frame-locals leak of the SQLCipher master password from the #4182 sweep: `ThreadSafeMetricsWriter.get_session` opened a per-thread metrics session with `password` live in the frame and logged failures with `logger.exception`, so a rendered traceback could persist the plaintext password (unrecoverable — TRUST.md §5). It now logs with `logger.warning` (no traceback); the exception still propagates to the caller via `raise`, so nothing is swallowed. The module is added to the `test_password_redaction_invariant` allow-list so CI permanently blocks any regression. See #4182.
+- Closed the last `diagnose=True` frame-locals leak of the SQLCipher master password from the #4182 sweep: the two consumers of `DatabaseInitializationError` — `thread_local_session.get_session` and `ResourceStatusTracker.__init__` — caught the error and logged it with `logger.exception` while `password` was a live local in the frame, so a rendered traceback could persist the plaintext password (unrecoverable — TRUST.md §5). Both now log with `logger.warning` (no traceback); the redacted detail is already logged at the raise site. The two modules are added to the `test_password_redaction_invariant` allow-list so CI permanently blocks any regression. See #4182.
+- Completed the #4182 logging chokepoint: the optional file sink (`LDR_ENABLE_FILE_LOGGING`) now also runs with `diagnose=False`, matching the database and frontend sinks. Previously it still honored `LDR_LOGURU_DIAGNOSE`, so an operator with debug + diagnose + file logging all enabled could persist frame-local credentials (including the unrecoverable SQLCipher master password — TRUST.md §5) into an unencrypted log file. Frame-local exception dumps now render only to the ephemeral stderr sink; no persisted or shipped sink (DB, browser, file) ever renders them. See #4182.
+- Completed the analytics-page innerHTML hardening pass: coerced the `frequency_rank` value (`#N` domain rank badge) in `link_analytics.html` with `Number()` — the one numeric sibling left un-coerced next to `usageCount`/`usagePercentage`/`researchDiversity` — and escaped `research_id` in `cost_analytics.html`'s "most expensive research" list (`encodeURIComponent()` in the `href`, `window.escapeHtml()` in the link text, mirroring `star_reviews.html`). Both are defense-in-depth on raw `${...}` interpolation into `innerHTML` with no DOMPurify barrier; neither is currently exploitable (rank is a server-side int, `research_id` is a `uuid4` and the cost-analytics render path is presently disabled), but together they make the PR's "across analytics pages" scope complete.
+- Egress policy hardening (Round 7 review): close four PEP gaps surfaced by a 120-agent re-audit and apply the same fix to four sibling sites. (1) LLM gate at config/llm_config.py now uses a known-local allow-list (`ollama`/`lmstudio`/`llamacpp`) when no snapshot is supplied — previously the gate silently no-op'd, letting cloud LLMs instantiate from snapshot-less callers under `llm.require_local_endpoint=true`. (2) `JournalReputationFilter` forwards `settings_snapshot` to `get_llm()` and `create_default` lets `PolicyDeniedError` propagate instead of silently returning None. (3) MCP `_discover_mcp_tools` / `_execute_mcp_tool` block under STRICT / PRIVATE_ONLY (and on absent snapshot), with a policy_audit log and a UI progress signal. (4) `_execute_download_content` narrows its bare `except Exception` so corrupted scope values fail closed instead of dropping to SSRF-only. The same bare-except fail-open pattern is fixed consistently in `notifications/manager.py`, `research_library/services/download_service.py` (paired with a `_policy_locked` short-circuit in the URL check), `web_search_engines/search_engine_base.py` (disables `include_full_content` when policy can't be evaluated), and `web/routes/research_routes.py` (STRICT + meta-picker now surfaces as 400 at run-start). Adds a Flask `PolicyDeniedError` handler so denials escaping a synchronous PEP return a clean 400 with the decision reason. Regression tests: parametrized no-snapshot fail-closed for every cloud provider plus ambiguous (`openai_endpoint`) and hypothetical (`groq`, `mistral`, `cohere`) entries; allow-list passes for the three known-local providers; MCP scope gate (STRICT/PRIVATE_ONLY/no-snapshot/corrupted policy) verifies the connection manager is never touched and the empty list is cached; download_content refuses without constructing ContentFetcher; journal_reputation_filter forwards snapshot and propagates PolicyDeniedError.
+- Egress-policy follow-ups after the 5×20 review: (1) ContentFetcher now relaxes a downloader's `SafeSession` to allow private IPs under `PRIVATE_ONLY`, mirroring `policy_aware_validate_url`, so a private/lab URL the policy already approved is no longer rejected by the downloader's own strict SSRF re-validation; (2) the PEP-578 audit-hook backstop is now re-armed in the `NewsAggregationStrategy` analysis ThreadPoolExecutor worker (threading.local isn't inherited by pool workers) and in the document scheduler's worker thread (built from the user's saved settings), so scheduled downloads and news-analysis LLM calls keep defense-in-depth parity under `PRIVATE_ONLY`/`STRICT`. The primary snapshot-based PEPs already gated these paths; this restores the secondary net. (MCP search runs synchronously on the already-armed strategy thread, so it needed no change.)
+- Extracted `updateEnhancedDomainList` from the `link_analytics.html` inline script into `static/js/pages/link_analytics_render.js` (surgical extraction mirroring PR #4584) so it can be tested in isolation. Added 7 Vitest XSS regression tests at `tests/js/pages/link-analytics-xss.test.js` covering the escape sites fixed in #3095 (Number()-coercion of `research_diversity`/`usage_count`/`usage_percentage`, `escapeHtml` of research-link queries and classification fields, `encodeURI`/`encodeURIComponent` of domain names and research IDs, and the inline-script wrapper pattern at `link_analytics.html:797-802`). These tests run in PR CI via the existing Vitest job. Added a new `link-analytics` Puppeteer shard for full-page verification of `/metrics/links` in the release pipeline (strict-mode only), with runtime assertions that no `<script>` element is injected and that the "Recent Researches (N total)" header shows a numeric `N`.
+- Hardened the SQLCipher-master-password frame-locals leak class (#4182) at two levels. (1) Sink level: `log_utils` now forces `diagnose=False` on the database sink (which persists into the user's own encrypted DB) and the frontend sink (which ships to the browser), so loguru can never render frame-local credentials into a durable or remote sink even when an operator enables `LDR_LOGURU_DIAGNOSE` for local debugging — a single chokepoint covering every credential-bearing exception handler app-wide. (2) Targeted sweep: the credential-centric session helpers `metrics/search_tracker`, `metrics/token_counter`, `database/library_init`, `database/backup/backup_executor`, `web_search_engines/rate_limiting/tracker`, and `research_library/.../research_history_indexer` now log failures with `logger.warning` instead of `logger.exception`, and are added to the `test_password_redaction_invariant` allow-list so CI permanently blocks regressions. Big multi-purpose route/service handlers are intentionally left logging full tracebacks for unrelated errors — they are covered by the sink-level guard. See #4182.
+- Operator-configured LLM provider URLs (`llm.ollama.url`, `llm.lmstudio.url`,
+  `llm.llamacpp.url`, `llm.openai_endpoint.url`) are now validated against the
+  same SSRF rules as outbound HTTP before the LangChain SDK constructor runs.
+  Closes an auth-gated SSRF gap where the SDK's internal `httpx` client would
+  otherwise bypass the existing `safe_requests` guard.
+
+  Follow-up review closed two more sites in the same vuln class: the
+  OpenAI-compatible base class's model-listing (`list_models_for_api`, used by
+  the custom-endpoint / LM Studio / llama.cpp providers) now validates
+  `base_url` before constructing the OpenAI client and returns an empty list on
+  a blocked URL, and `OpenAIProvider` now validates `llm.openai.api_base` before
+  handing it to `ChatOpenAI`.
+- RAG auto-indexing now applies backpressure to its background worker pool. The `ThreadPoolExecutor` previously had an unbounded internal queue; under sustained upload bursts (possible under the configurable upload rate cap from #3935), this could queue thousands of indexing jobs and exhaust memory. The pool is now capped at 100 in-flight + queued jobs; submissions over the cap are dropped with a clear log warning ("Auto-index queue saturated… trigger a manual reindex if needed"). Documents still upload successfully — only the *automatic* indexing is skipped, and users can manually reindex via the UI.
+- Routed every search-engine site that scrubbed errors via `redact_secrets` through a single `BaseSearchEngine._scrub_error()` helper (regex sanitize + literal redaction), replacing ~60 hand-copied call sites. This closes the per-site drift class that could drop a credential, and upgrades several engines whose logged errors previously skipped the regex sanitization pass.
+- Search, fetch, and agent tool error messages are now scrubbed of credentials before they reach the LLM and the user-visible research output — a search-engine or LLM exception can embed a request URL carrying an API key, so these are now routed through the same `sanitize_error_for_client` helper used for download errors.
+- The MCP search strategy now filters its specialized search-engine tool list by the active egress scope (parity with the LangGraph strategy): forbidden engines never appear in the tool schema the LLM sees, and a policy denial at execution surfaces as an audit log line instead of a generic tool error.
+- The ``custom_endpoint`` URL supplied to ``/api/start_research``, ``/api/news/subscriptions`` (POST and PUT/PATCH), and ``/api/followup/start`` is now validated for SSRF at the request boundary (rejecting cloud-metadata / link-local targets, always blocked by ``ssrf_validator``) before any research is queued. This is fail-fast defense-in-depth — the OpenAI-compatible provider re-validates the same URL via ``assert_base_url_safe`` before the LangChain client is built — but the route-layer check rejects early, before a DB row is written or a thread is spawned, and keeps the endpoint out of the logs. The endpoint is normalized exactly as the provider normalizes it, so scheme-less local endpoints (``localhost:11434``, ``192.168.1.10:8000``) are accepted; private IPs and localhost still pass, so local LLM providers (Ollama, LM Studio, vLLM) are unaffected.
+- The error-message credential scrubber now also redacts GitHub (`ghp_`/`github_pat_`), AWS (`AKIA`/`ASIA`/…), Slack (`xox*-`/`xapp-`), Google OAuth (`ya29.`) tokens and JWTs — extending coverage to more common credential formats before error text reaches clients or logs.
+- The per-run denied-fetch quota (`MAX_DENIED_FETCHES_PER_RUN`) is now aggregated per *run* instead of per `EgressContext`. Each call site (content fetcher, full-content search, download service, the audit hook) builds its own `EgressContext`, so the counter previously reset whenever a new engine/fetcher was constructed — letting a malicious indexed document evade the exhaustion guard by spreading denied fetches across contexts. The counter is now anchored to the run's armed audit-hook context (shared across the whole run, re-armed identically on pool workers), with a safe fallback to the local context for snapshot-less / programmatic callers.
+- The server now logs a loud warning at startup when the SQLCipher KDF is weakened below the production floor (reached only in test mode, e.g. ``LDR_TEST_MODE``) while user databases already exist. New databases would be created with this weak at-rest work factor, and — separately — any existing database created at a higher KDF can no longer be opened: its on-disk key is unchanged, but the server now derives a different, weaker key, so decryption fails and login returns a generic "Invalid username or password" 401 for every affected user (the KDF-mismatch symptom class behind PR #4775). Silent on fresh deployments and when the effective KDF is at/above the floor.
+- The shared error-message credential scrubber now also redacts OAuth `access_token`/`refresh_token` and other sensitive URL query parameters, Azure `subscription-key`, `Authorization:`/`x-api-key:` headers, and Google (`AIza…`) API keys — strengthening credential redaction for every code path that surfaces error text to clients or logs.
+- Under the PRIVATE_ONLY egress scope (and Adaptive resolving to private), notification dispatch via non-http Apprise vendor schemes (`slack://`, `discord://`, `telegram://`, …) is now refused — those send to external vendor APIs and can't be verified local. Address a self-hosted notifier by its `http(s)://` URL instead, which is allowed as a private host. Other scopes are unchanged.
+- Wrapped the `research_diversity` interpolation in the link-analytics "Recent Researches (N total)" header in `Number()`, matching the same coercion already applied to the sibling `🔍 N researches` badge in the same template. Closes a defense-in-depth gap (raw `${researchDiversity}` into `innerHTML` with no DOMPurify) left inconsistent by PR #3095. Not currently exploitable — the value is a Python `len(...)` int server-side — but aligns with the PR's stated scope of coercing numeric fields used in template-literal text positions.
+
+### ✨ New Features
+
+- **Star Reviews dashboard gains six new charts.** The `/metrics/star-reviews` page now shows a doughnut distribution, rating-volume overlay on the trends chart, research-mode comparison, LLM satisfaction breakdown (stacked bars), quality-dimensions radar, and a recent-feedback section. Two bugs were also fixed: recent ratings displayed "Invalid Date" and showed generic fallbacks instead of actual query text and mode. ([#3804](https://github.com/LearningCircuit/local-deep-research/pull/3804))
+- **Benchmark YAML downloads now show the LDR version and `date_tested` from when the benchmark *ran*, not at download time.** Previously a v1.6.5 run downloaded on v1.6.7 incorrectly stamped v1.6.7 — making cross-run comparison impossible. The full per-key settings snapshot active at the run (API keys redacted) is now captured at run time and is available **opt-in** via the new **Export → "Include settings snapshot"** checkbox; by default the YAML contains only the summary configuration, so it's safe to share without exposing internal URLs or local paths. Benchmarks created before this release show `ldr_version: unknown (pre-0014 run)` and, when settings are requested, `settings: null` — no backfill, but no breakage either. ([#3844](https://github.com/LearningCircuit/local-deep-research/pull/3844))
+- Add TinyFish as an optional hosted search engine with structured search results and browser-rendered content extraction. ([#4057](https://github.com/LearningCircuit/local-deep-research/pull/4057))
+- Added a "Sampling Seed" field to the benchmark configuration (default 42): the same seed and example counts now select the same benchmark questions on every run, so accuracy numbers are comparable across runs and configurations. The seed is stored with each benchmark run. Leave the field empty to draw a fresh random sample each run (the previous behavior). ([#4516](https://github.com/LearningCircuit/local-deep-research/pull/4516))
+- Library searches now warn in the research log when results are incomplete because one or more collections failed, naming the affected collections; failed-collection error messages use collection names instead of internal ids. ([#4520](https://github.com/LearningCircuit/local-deep-research/pull/4520))
+- **New "Anthropic-Compatible Endpoint" LLM provider** for self-hosted services that speak the Anthropic Messages API (`/v1/messages`) rather than the OpenAI chat-completions format. Select it under Settings → LLM, set `llm.anthropic_endpoint.url` (and an optional `llm.anthropic_endpoint.api_key` — keyless gateways are supported), and LDR builds a `ChatAnthropic` client pointed at your endpoint. The base URL is SSRF-validated before any request, and a real `ANTHROPIC_API_KEY` in the environment is never sent to a self-hosted endpoint. The official cloud Anthropic provider is unchanged. ([#4581](https://github.com/LearningCircuit/local-deep-research/pull/4581))
+- Add per-message **Copy**, **Retry**, and **Delete** actions to chat — the standard ChatGPT/Claude.ai hover-icon pattern. Retry replaces the failed turn and re-submits the same query as a fresh research run; Delete removes the user message, research, and any assistant response for that turn. The History page also gains a Copy-query button on each card and the rerun button is now shown for failed/suspended researches too (previously only successful runs had it). ([#4659](https://github.com/LearningCircuit/local-deep-research/pull/4659))
+- **Star Reviews dashboard polish.** Recent-feedback entries now link to their source research, and a screen-reader live region announces when the dashboard refreshes (e.g. after changing the time period), including which charts have no data yet. ([#4793](https://github.com/LearningCircuit/local-deep-research/pull/4793))
+- **The search-engine selector is now grouped into labelled bands.** Engines are ordered by a trust/cost gradient with section headers — Favorites, Collections, Academic, Local RAG, Books, Code, News, No API key, API key — instead of one alphabetical list. Starring an engine still pins it to the Favorites band at the top. ([#4814](https://github.com/LearningCircuit/local-deep-research/pull/4814))
+- **`/api/v1/health` now reports file-descriptor and thread diagnostics.** Authenticated callers receive a `resources` block (`fd_count`, `fd_soft_limit`, `fd_hard_limit`, `fd_usage_percent`, `thread_count`), and `status` flips to `"warning"` above 70% FD usage — making FD growth observable over HTTP for dashboards and alerting. Anonymous callers still get the basic `status`/`message`/`timestamp`, so the Docker healthcheck is unaffected. FD metrics report `null` on non-Linux platforms. ([#4915](https://github.com/LearningCircuit/local-deep-research/pull/4915))
+- **Unified background reconciler that indexes ANY unindexed document.** A single scheduled job now indexes every unindexed document so nothing is permanently missed: (a) library uploads that the immediate auto-index queue dropped when saturated, and (b) research downloads that were never ingested into your library. Enable the **Document Scheduler** and then **"Auto-index documents in the background"** (both off by default) to turn it on — the sweep only runs while the document scheduler itself is enabled. The reconciler is idempotent (already-indexed documents are skipped), processes a small bounded batch per run so a large backlog self-heals gradually, and honors each collection's own embedding configuration. The two cases are budgeted independently, so a backlog of documents that keep failing to index can never starve the other case (research downloads still get indexed even when many library uploads are failing). This replaces both the old library-only sweep and the separate per-research RAG-indexing pass that used to run inside the document scheduler. The legacy **"Generate RAG Embeddings"** setting is kept as a deprecated alias — it now enables this same reconciler — so existing users keep getting their research downloads indexed with no change required.
+- A collection's public/private (egress) classification can now be changed after creation, from a "Privacy & Egress" toggle on the collection details page — not only at creation time.
+- Add an **Adaptive** egress scope (now the default) that follows your primary search engine: a private primary keeps everything local (and forces local LLM/embeddings inference), a public primary allows public engines, and a meta-picker primary allows both. Existing explicit scopes (Both / Public only / Private only / Strict) remain available. Selecting **Private only** now visibly checks and locks the "require local LLM/embeddings" toggles to reflect that local inference is enforced.
+- Add an egress policy module (`security.egress_policy`) that gates search-engine instantiation against a user-declared scope. New settings `policy.egress_scope` (default `both`, preserves current behavior), `llm.require_local_endpoint`, `embeddings.require_local`, and `llm.allowed_local_hostnames` are reserved. Under `strict` scope the factory blocks any engine that isn't the user's primary, closing the previously silent expansion path. The LangGraph agent tool-list filter, LLM/embeddings PEPs, and UI controls ship in the same release (see the related Stage 1b changelog entries).
+- Collections can now be classified **public** or **private** (default private). A private collection is excluded from runs under Public-only / Adaptive-public egress scope and forces local LLM/embeddings inference under Private-only / Adaptive-private scope, so its contents never reach a cloud model. Flip the classification with the "Public collection" checkbox on the collection create form. Adds the `collections.is_public` column (migration 0011).
+- Collections now have an **"Available to the research agent"** toggle (default on). Turn it off to keep a collection out of the LangGraph research agent's tool list when it isn't needed for agentic research — without deleting it or changing its privacy/egress classification. The flag is exposed on the collection create form and the collection details page, persists via the collections API, and is enforced where the agent builds its specialized search tools (a disabled collection's name never reaches the agent).
+- Library collections now show per-collection indexing status ("X of Y indexed" + a "N pending indexing" badge) with a per-card Reindex action. When the optional scheduled-reconciler setting is available, a toggle to index documents in the background on a schedule also appears (otherwise it stays hidden). (#3939/#4627 follow-up).
+- When the egress policy blocks something, the user now gets a **clear, actionable message** — what was blocked, why, and exactly which setting to change to allow it (e.g. "blocked because your Egress Scope is Private only → change it in Settings → Privacy & Egress, or turn off 'Require local LLM endpoint'") — instead of a terse machine code. A new `security.egress.guidance.denial_guidance()` helper centralises these messages; it's wired into the research start-up engine check, and denials surface the raw reason code alongside for support.
+
+  The `quick_summary` API also gains an opt-in: pass `"allow_default_settings": true` to deliberately run with default settings (no egress policy) when your settings can't be loaded, instead of the default fail-closed `503`. The 503 itself now explains the cause and the fix (retry / re-authenticate / opt in).
+- Wire the egress policy module (added in Stage 1a) into the LLM and embeddings call paths. `get_llm()` now raises `PolicyDeniedError` when `llm.require_local_endpoint=True` and a cloud LLM provider is requested. The OpenAI and Anthropic model-listing endpoints in the settings UI degrade gracefully (empty list + audit log entry) under the same condition. `get_rag_service()` runs a pre-flight check before constructing the service so embedding-policy violations surface immediately rather than after hundreds of chunks have been generated. Defaults are unchanged (`require_local_endpoint=False`, `embeddings.require_local=False`), so existing installs see no behavior change.
+
+### 🐛 Bug Fixes
+
+- REST `/api/v1/generate_report` and `/api/v1/analyze_documents` now load the authenticated user's encrypted-DB settings (API keys, model preference, search tool) instead of silently falling back to application defaults plus `LDR_*` env vars. Like `/quick_summary`, both endpoints now fail closed (HTTP 503) when the settings snapshot cannot be loaded, with `allow_default_settings: true` as the explicit opt-out. ([#3661](https://github.com/LearningCircuit/local-deep-research/pull/3661))
+- Starting a follow-up research with no LLM model configured now fails immediately with a clear "Model is required" error (HTTP 400) instead of spawning a worker that dies and leaves the research stuck IN_PROGRESS. ([#3767](https://github.com/LearningCircuit/local-deep-research/pull/3767))
+- **Star Reviews analytics now count each rating once.** The `/metrics/star-reviews` queries joined `token_usage` directly, which fans out one row per LLM call — so "Recent Ratings" showed the same research duplicated (far fewer than 20 unique entries) and the LLM/search-engine breakdown counts and averages were inflated by the number of calls per research. These queries now collapse token usage to one row per research before joining. The quality-dimensions radar no longer hides real data when only some dimensions are rated, and reports a per-dimension sample size. The rating endpoint also rejects boolean values (which previously passed the 1–5 check) and overly long feedback. ([#3804](https://github.com/LearningCircuit/local-deep-research/pull/3804))
+- The `/api/rate-limiting/cleanup` endpoint now rejects out-of-range or non-integer `days` values with HTTP 400 instead of passing them through to the destructive bulk delete (clamped to `1 <= days <= 365`). ([#3805](https://github.com/LearningCircuit/local-deep-research/pull/3805))
+- Wikipedia search no longer floods the log with per-title tracebacks when the MediaWiki API rate-limits a query — the engine now detects the JSON-decode error pattern, logs a single warning, and bails out of the batch with whatever previews it collected. Fetched pages that come back empty (paywalls, JS-only SPAs, blank 200s) and LLM extractions that return an empty summary now short-circuit to `NOT RELEVANT` without locking the URL into the citation collector, so the agent is free to re-fetch it later under a different focus. ([#3844](https://github.com/LearningCircuit/local-deep-research/pull/3844))
+- `SettingsManager.get_all_settings()` no longer crashes when the DB contains a row whose `type` column holds an enum value no longer in `SettingType` (e.g. legacy `'CHAT'`-typed rows from a removed feature). The handler now catches `LookupError` alongside `SQLAlchemyError` and falls back to defaults-only — strictly safer than crashing every caller (`/settings/api`, benchmark start, research start, MCP). ([#3947](https://github.com/LearningCircuit/local-deep-research/pull/3947))
+- **Submitting an empty value for a password-typed setting is now a no-op** — previously could overwrite the stored secret with `""`. Belt-and-braces fix for the same `[REDACTED]` sentinel issue: a stale browser tab can no longer corrupt API keys by triggering a save. To clear a password setting, clear the source env var or use settings import. ([#3954](https://github.com/LearningCircuit/local-deep-research/pull/3954))
+- Fix Ollama `enable_thinking` setting being silently dropped on the live class
+  construction path. Reasoning models (deepseek-r1, qwen2.5) now correctly
+  receive `reasoning=True/False` per the user's `llm.ollama.enable_thinking`
+  setting. Also apply the 80%-of-context-window `max_tokens` cap on every
+  provider's `create_llm` (was previously only applied in dead procedural
+  code that never ran in production) and unify the optional-API-key
+  placeholder string across all providers (`"not-required"`). Note: users
+  who set `llm.context_window_unrestricted = false` together with a small
+  `llm.context_window_size` will now have `max_tokens` capped at 80% of
+  that window for OpenAI/Anthropic too — previously the cap only applied
+  to Ollama. For LM Studio and llama.cpp the cap is always active, keyed
+  to `llm.local_context_window_size` (default 20480 → effective cap 16384
+  with the default `llm.max_tokens` of 30000). `llm.supports_max_tokens =
+  false` is now honored on the live path (previously the kwarg was sent
+  regardless), and whitespace-only API keys for required-key providers now
+  raise a clear error at construction instead of being sent to the API.
+
+  Ollama's `num_ctx` now resolves through the same shared helper as the
+  context-overflow bookkeeping, so the reported `context_limit` always
+  matches the window the model actually runs with (previously an absent or
+  null `llm.local_context_window_size` produced `num_ctx` 4096/server-default
+  while overflow detection assumed 8192). The ineffective `max_tokens`
+  kwarg is no longer passed to `ChatOllama` (it was silently ignored;
+  `num_ctx` is the effective control). A keyless `openai_endpoint`
+  configuration is now permitted — it previously raised "API key not
+  configured" at construction, blocking keyless local servers (vLLM,
+  text-generation-webui); a warning is logged so endpoints that do
+  require a key remain diagnosable.
+
+  Also fix `llama.cpp` provider availability check: instances behind an
+  authentication proxy now correctly report as available when
+  `llm.llamacpp.api_key` is configured. Previously the probe was sent
+  without credentials and was always rejected, so users with auth-proxied
+  llama-server saw the provider as unavailable regardless of server state. ([#3984](https://github.com/LearningCircuit/local-deep-research/pull/3984))
+- **Settings on mobile no longer scrolls forever.** Every section now starts collapsed at the mobile breakpoint (`max-width: 767px`) — page height drops from > 16,384 px to ~7,300 px on a Pixel 5. Desktop is unchanged; typing in the Settings search box still force-expands every surviving section. ([#4032](https://github.com/LearningCircuit/local-deep-research/pull/4032))
+- Chat Mode post-merge audit follow-ups (#3891). The send-message API now rejects a non-boolean ``trigger_research`` with HTTP 400 instead of silently coercing a truthy value (e.g. ``"false"``) to ``True`` and launching unwanted research. Chat-session deletion now records the (truncated) username alongside the session id in the audit log, so a stolen-token bulk delete leaves a forensic trail. The frontend now logs (instead of silently swallowing) a failed background title-generation request, and the Markdown chat export documents that its output is a human-readable archive that is not safe to feed back into a Markdown/HTML renderer without escaping. The "Clear History" confirmation now warns that it also permanently deletes all chat sessions and their conversations (it always did, but the dialog only mentioned research history). Accessibility: the chat input gains ``aria-required`` and the welcome suggestion buttons gain descriptive ``aria-label``s. Internal cleanup: the duplicated atomic ``UPDATE … RETURNING`` counter logic in ``ChatService`` is extracted into a shared ``_atomic_increment`` helper. New tests cover the 100+-message pagination cap, render-time Jinja2 autoescaping, and the stricter ``trigger_research`` contract. ([#4427](https://github.com/LearningCircuit/local-deep-research/pull/4427))
+- Two pre-existing export/metadata correctness bugs are fixed. RIS citation exports were malformed: the entry builder reused its output accumulator as a scratch variable when reading the title, so each record emitted the raw source text *before* the mandatory leading ``TY  - `` tag and reference managers (Zotero/Mendeley/EndNote) rejected the file. Separately, ``get_subscription_history`` called ``json.loads()`` on ``research_meta`` — a JSON column that SQLAlchemy already deserializes to a dict on read — so the resulting ``TypeError`` was swallowed by a bare ``except`` and every subscription-history item silently lost its headline and topics; it now handles both dict and legacy-string values like the rest of the module. ([#4435](https://github.com/LearningCircuit/local-deep-research/pull/4435))
+- Fixed a cross-user data exposure in multi-user deployments: saving any setting emitted a ``settings_changed`` WebSocket event — carrying the changed setting keys **and their raw values**, including plaintext LLM API keys — to *every* connected client on the shared Socket.IO server, where each browser merged them into its local settings cache. Sockets now join a per-user room on connect and ``settings_changed`` is scoped to the owning user's own browser tabs only; a settings change made outside a request context (start-up defaults, background workers) no longer emits at all instead of falling back to a broadcast. Single-user (desktop) deployments were unaffected. ([#4437](https://github.com/LearningCircuit/local-deep-research/pull/4437))
+- Fixed three more cross-user WebSocket leaks in multi-user deployments (follow-up to the ``settings_changed`` fix): ``parallel_search_started`` — which carried the user's **raw research query** — and ``engine_completed`` were broadcast to every connected client despite having no frontend listener, and are removed entirely; ``search_engine_selected`` (shown in the research log panel) is now scoped to the owning user's per-user room, resolved from the research worker's search context, and is skipped entirely — never broadcast — when no username is available. ([#4446](https://github.com/LearningCircuit/local-deep-research/pull/4446))
+- Fixed benchmark runs with xbench-DeepSearch enabled ignoring the configured number of examples: `XBenchDeepSearchDataset.load()` overrode the base class with its own `num_examples` parameter, so the registry's argument-less `load()` call skipped sampling and queued all 100 (Chinese-language) xbench questions. A run configured for e.g. 50 SimpleQA + 10 xbench questions therefore processed 150 tasks against a recorded total of 60, driving the progress display past 100% and surfacing unexpected Chinese questions. The dataset now samples via the constructor's `num_examples`/`seed` like every other dataset. ([#4451](https://github.com/LearningCircuit/local-deep-research/pull/4451))
+- The research progress display and agent-thinking panel now show the actual search engine name (e.g. "DuckDuckGo") instead of the generic internal id "web_search". The LangGraph strategy keeps the stable tool id in the progress event metadata while surfacing the friendly, brand-correct engine name in the message, and the agent-thinking panel renderer now prefers that human-readable message. ([#4470](https://github.com/LearningCircuit/local-deep-research/pull/4470))
+- Fixed metrics pages showing "No token usage data yet" despite completed researches (#4457). LLM calls whose provider returns no token usage data (OpenAI-compatible servers that omit `usage` on streamed responses, proxies that strip it, Ollama omitting `prompt_eval_count` for fully-cached prompts) were silently skipped, leaving every metrics panel empty; they are now recorded with zero token counts plus a warning log, so call counts, models, phases and response times still appear. Added an opt-in `llm.openai_endpoint.stream_usage` setting that requests usage stats on streamed responses (`stream_options.include_usage`) for servers that support it (LM Studio 0.3.18+, llama.cpp, vLLM, OpenRouter) — off by default because some gateways reject the parameter with a 400. Also fixed background-thread token writes not updating the `ModelUsage` aggregate, and several /metrics panels ignoring the selected time period: the research count, plus the ratings, link and strategy analytics, fell back to a fixed 30-day (or all-time) window when "3 months" or "1 year" was chosen, because the period vocabularies were inconsistent. All metrics endpoints now share one period map. ([#4473](https://github.com/LearningCircuit/local-deep-research/pull/4473))
+- Fixed the v1.7.0 Docker image crashing with SIGILL (exit code 132) on x86 CPUs that support AVX but not AVX2 (e.g. Sandy Bridge / Ivy Bridge era). The ``faiss-cpu`` 1.14.2 wheels dropped the runtime CPU dispatch that 1.13.x shipped (separate generic/AVX2/AVX512 builds) and ship a single ``libfaiss.so`` compiled with AVX2 instructions, so importing faiss at app start-up executed an illegal instruction on AVX-only CPUs. ``faiss-cpu`` is now pinned to the 1.13.x series until upstream wheels restore a non-AVX2 fallback (reported upstream as facebookresearch/faiss#5296; lifting the pin is tracked in #4499). ([#4480](https://github.com/LearningCircuit/local-deep-research/pull/4480))
+- Hardened app startup against unrelated broken packages in the user's site-packages: the text-splitter registry now imports the three splitters it uses directly from their `langchain_text_splitters` submodules instead of the package root, whose `__init__` eagerly imports every optional splitter backend (spacy, nltk, konlpy, ...). Previously, a globally installed spacy paired with an incompatible typer/click combination crashed `ldr-web` at startup with `TypeError: type 'Choice' is not subscriptable`. ([#4490](https://github.com/LearningCircuit/local-deep-research/pull/4490))
+- Research no longer fabricates citations when every search returns nothing: if no sources are found (or a local collection search fails), the answer now states this explicitly instead of inventing references, the progress log shows an error, and failed collection searches are reported instead of being silently treated as "no matching documents". ([#4503](https://github.com/LearningCircuit/local-deep-research/pull/4503))
+- A temporarily unreachable embedding provider (e.g. Ollama not running) no longer quarantines a healthy collection index and silently replaces it with an empty one; the search now fails with a clear provider error and the index survives untouched. ([#4512](https://github.com/LearningCircuit/local-deep-research/pull/4512))
+- Fixed intermittent ~60s UI freezes / "Navigation timeout" failures (#4431) caused by heavy third-party imports (matplotlib via the benchmarks package) and synchronous stderr logging blocking the dev server's request pipeline under load. matplotlib/optuna.visualization are now imported lazily (only when benchmark visualizations are generated), and the stderr log sink uses `enqueue=True` so logging never blocks on I/O while holding loguru's handler lock. Note: `enqueue=True` makes stderr logging asynchronous (a background writer thread), so on an abrupt crash the last few buffered log lines may be lost and ordering relative to other sinks can differ slightly. ([#4536](https://github.com/LearningCircuit/local-deep-research/pull/4536))
+- Fixed chat-initiated research running without the user's database password after the in-memory session-password store expired (24h TTL) or was cleared by a server/container restart while the session cookie was still valid. In that state the research completed (so the user saw results) but every background metric write was silently dropped, leaving the metrics dashboard empty (part of #4457). The chat `send_message` route now applies the same encryption-aware password guard as the direct and follow-up research routes, returning a clear "session expired — log out and log back in" message before any rows are created. Unencrypted installs are unaffected. ([#4571](https://github.com/LearningCircuit/local-deep-research/pull/4571))
+- Fixed a `MemoryError` when listing or clearing history on large libraries — history/report list queries no longer load full report bodies into memory. ([#4574](https://github.com/LearningCircuit/local-deep-research/pull/4574))
+- Fixed the rate-limiting panel on /metrics showing all zeros (no tracked engines, no wait times) for every user. The analytics read the `RateLimitAttempt` table, but raw attempt persistence was disabled to prevent database locking under parallel search, so that table is never populated. The panel now derives engine health, wait-time estimates, success rates, and recent attempt counts from the `RateLimitEstimate` data that rate limiting actually persists. (A few raw-attempt-only metrics — rate-limit-event counts and true per-attempt average wait — cannot be reconstructed and are reported as 0 / the learned base wait.) ([#4576](https://github.com/LearningCircuit/local-deep-research/pull/4576))
+- **The cloud Anthropic model dropdown was empty.** The model-list route fetched Claude models correctly, but the auto-discovered-provider pass that ran afterward re-listed them with the OpenAI SDK (`Authorization: Bearer`), which 401s against the Anthropic API (it requires `x-api-key`) and overwrote the good result with an empty list. `AnthropicProvider` now lists models via the `anthropic` SDK, so the dropdown populates again. ([#4586](https://github.com/LearningCircuit/local-deep-research/pull/4586))
+- Fixed news subscription scheduling regressions from #4492: a manual "Run Now" whose research later failed no longer pushes the subscription out a full refresh interval (it is reset to due so the scheduler retries), the run-now request no longer holds the encrypted-DB session open across the research-start HTTP call, and the subscription folder-update PUT route now keeps `status` authoritative (an `is_active` toggle pauses correctly instead of being ignored by the scheduler). The same folder-update route was also repaired — it previously failed on every call due to a bad relative import and a missing serializer. ([#4587](https://github.com/LearningCircuit/local-deep-research/pull/4587))
+- Fixed two news endpoints that returned HTTP 500 for any real subscription: the subscription history modal (it read a non-existent `refresh_count` column; the run count is now derived from research history) and the organized-subscriptions view (it called `.to_dict()` on plain dicts; it now returns the folder-grouped shape the UI renders). ([#4588](https://github.com/LearningCircuit/local-deep-research/pull/4588))
+- Fixed the RAG collection `embedding_dimension`, which was always stored as NULL (the metadata helper probed a non-existent `provider.embedding_dimension` attribute); it is now derived from the actual embedding model. Also extended the collection-indexing dedup to the bulk `index-all` route, which previously stored no embedding metadata and never cleaned up stale chunks/indices on force-reindex. ([#4593](https://github.com/LearningCircuit/local-deep-research/pull/4593))
+- Fixed the download manager progress popup showing "X / 0 files" instead of the real total. The pending-item count is now taken after the queue is populated, so the denominator reflects the actual number of papers being downloaded. The download manager now also surfaces an explicit alert when there is nothing to download (e.g., all papers already downloaded) or when queueing fails for every selected research session, instead of silently completing with "0 / 0 files". ([#4660](https://github.com/LearningCircuit/local-deep-research/pull/4660))
+- Fixed three rate-limiting API endpoints (`/api/rate-limiting/current`, `/api/rate-limiting/status`, `/api/search-quality`) that always returned empty or per-process-cached data because they read from the in-memory `get_tracker()` singleton, which is freshly built per request and never sees other workers' state. They now read directly from the persisted `RateLimitEstimate` table — the same source the `/metrics` panel was switched to in #4576. Together with that change, every rate-limiting analytics surface is now driven by the same authoritative data. Raw `RateLimitAttempt` writes remain disabled (deliberately, per #4576); only the per-engine estimates the rate limiter actually learns are exposed. The `/api/search-quality` response shape changed in the process (`recent_avg_results` / `min_recent_results` / `max_recent_results` / `sample_size` are gone, replaced by `success_rate`); the benchmark page's status-warning JS was updated to consume the new shape, and the old "Very low results" warning (whose underlying per-attempt search-result-count data was never persisted) is replaced by a "High rate-limit failures" warning driven by `success_rate` and `status`. ([#4721](https://github.com/LearningCircuit/local-deep-research/pull/4721))
+- WeasyPrint (the PDF-export dependency) is now imported lazily on first use instead of at web-server startup. It was previously imported during blueprint registration via `research_routes`, adding ~20s of "cold heavy-import" to server cold start on CPU-constrained CI runners — a contributor to the `Navigation timeout 60000ms` UI-shard flakiness in #4431. PDF export behaviour is unchanged. ([#4734](https://github.com/LearningCircuit/local-deep-research/pull/4734))
+- **Star Reviews "Unknown" search-engine bucket no longer double-counts.** The search-engine breakdown attributed any research that made non-search LLM calls (whose token rows have no `search_engine_selected`) to the "Unknown" bucket *in addition to* its real engine, inflating "Unknown" toward every research. Ratings with a recorded engine now count under that engine only; "Unknown" holds only ratings with no recorded search engine. ([#4786](https://github.com/LearningCircuit/local-deep-research/pull/4786))
+- **Star Reviews merges the "Unknown" buckets.** The LLM and search-engine breakdowns no longer split unidentified entries across separate bars — an empty string, the lowercase `unknown` sentinel written for an unidentified model, and the `Unknown` shown for ratings with no recorded model/engine now collapse into a single "Unknown" bucket (real model names keep their exact casing). ([#4790](https://github.com/LearningCircuit/local-deep-research/pull/4790))
+- **LangGraph agent no longer crashes with `BadRequestError` on reasoning-mode LLMs.** `ProcessingLLMWrapper` (the central `<think>` stripper applied to every LLM returned by `get_llm`) previously let `bind_tools` bypass the stripper via its `__getattr__` shim, so `langgraph_agent_strategy.create_agent()` ended up calling the raw reasoning model. Qwen 3.x and deepseek-r1 then emitted `<think>…</think>` text directly inside tool-call `content`, which the OpenAI-compatible provider rejected with `400 Failed to parse input at pos 0: <think>…`. `bind_tools` is now overridden to re-wrap the tool-bound model so the stripper survives the agent tool-binding step (#4804). ([#4804](https://github.com/LearningCircuit/local-deep-research/pull/4804))
+- **TinyFish search engine no longer logs your search query on request errors.** When a TinyFish Search/Fetch call failed, the request exception (which embeds the query in the URL) was written to the error log; failures now record only a static message plus the HTTP status. Also restores `ruff format` on the TinyFish test files so the pre-commit gate passes. ([#4806](https://github.com/LearningCircuit/local-deep-research/pull/4806))
+- **Star Reviews announces the real load error to screen readers.** A failed refresh now speaks the actual error message via the dashboard's live region instead of a generic "failed to load", and a no-op clear that never re-announced was removed. ([#4828](https://github.com/LearningCircuit/local-deep-research/pull/4828))
+- Server startup no longer eagerly imports the heavy `langchain_text_splitters`/sentence-transformers stack; it is loaded lazily (and thread-safely) on the RAG indexing path instead. This speeds up boot and fixes intermittent UI-test server-start failures, and avoids a rare concurrent-import error that could silently skip document indexing right after a restart. ([#4829](https://github.com/LearningCircuit/local-deep-research/pull/4829))
+- Journal-quality data downloads now follow OpenAlex's 2026-06 "standard-format" snapshot layout (`data/jsonl/<entity>/manifest.json`). The previous paths started returning 404, breaking the OpenAlex sources and institutions downloads. ([#4830](https://github.com/LearningCircuit/local-deep-research/pull/4830))
+- The manual **Run now** button and the manual overdue-subscription sweep now actually start research. Both previously made a server-internal loopback HTTP POST to `/research/api/start`, which sits on a CSRF-protected blueprint and so always failed with HTTP 400 ("CSRF token is missing") — the "Run now" button reported `Failed to start research: 400`. They now invoke the research handler in-process, bypassing the HTTP/CSRF layer while still resolving the user's encrypted-database credentials. (Scheduled subscription runs were unaffected.) ([#4834](https://github.com/LearningCircuit/local-deep-research/pull/4834))
+- A failed `commit()`/`flush()` inside a `get_user_db_session()` block no longer poisons the thread's reused database session. The context manager now rolls the session back (and re-raises) when the block errors, so a single failed transaction can't cascade `PendingRollbackError` into every later operation on that thread. Paths that catch and swallow such a failure while reusing the session are also recovered explicitly: a PDF download or text extraction that fails mid-write, and a multi-file library upload where one bad file would otherwise fail the whole batch. ([#4841](https://github.com/LearningCircuit/local-deep-research/pull/4841))
+- A rare connection-level failure while auto-indexing a freshly downloaded document no longer leaves the request's database session in a state that fails the next operation. The best-effort auto-index step now recovers the shared session if its lookup fails, continuing the session-rollback hardening from the previous release. ([#4862](https://github.com/LearningCircuit/local-deep-research/pull/4862))
+- Under `embeddings.require_local=True` (or any `PRIVATE_ONLY` egress scope, which forces it), searching a library collection whose embedding model was indexed by bare name — e.g. the shipped default `all-MiniLM-L6-v2` — raised `policy_denied: embeddings_model_not_cached` even when the model was present in the local HuggingFace cache. The pre-flight cache check used the bare name as the `repo_id`, but the SentenceTransformer loader requests the namespaced form (`sentence-transformers/all-MiniLM-L6-v2`) and the HF hub cache is keyed on whichever form the loader requests. The check now probes both forms (matching the loader's resolution for both SBERT-curated names and the `basic_transformer_models` allowlist of vanilla transformers like `bert-base-uncased`), so a correctly cached model is admitted. ([#4887](https://github.com/LearningCircuit/local-deep-research/pull/4887))
+- Fixed a circular import that made `import local_deep_research.domain_classifier.models` (and importing the `domain_classifier` package before `database.models`) crash with `ImportError: cannot import name 'DomainClassification' from partially initialized module`. The declarative SQLAlchemy `Base` now lives in a dependency-free `database.base` leaf that model modules import without triggering the full `database.models` package, so import order no longer matters. `Base` identity and `Base.metadata` are unchanged. ([#4910](https://github.com/LearningCircuit/local-deep-research/pull/4910))
+- Under `embeddings.require_local=True` (or any `PRIVATE_ONLY` egress scope), a collection search whose embedding model is configured as an existing **local directory path** was wrongly refused with `policy_denied: embeddings_model_not_cached`. The `require_local` pre-flight probed the path as a HuggingFace `repo_id`, which raised a validation error that was swallowed and treated as a cache miss — even though the SentenceTransformer loader resolves such a path locally and never contacts the hub. The pre-flight now mirrors the loader's `os.path.exists` guard and admits an existing local path (blank/degenerate names still fail closed), so a local model directory is no longer false-denied. ([#4924](https://github.com/LearningCircuit/local-deep-research/pull/4924))
+- **Benchmark YAML exports now quote and escape free-form string fields** (`model`, `model_provider`, `search_engine`, `dataset`, and the evaluator's `model`/`provider`/`endpoint_url`). They were previously interpolated raw, so a value containing a YAML-special character — a colon-space, `#`, quote, or newline — could produce malformed YAML (or inject an unintended key). These fields now go through the existing `yamlEscape()` helper, matching how the settings-snapshot block already serializes strings. Numeric fields and the controlled-format `date_tested`/`ldr_version` are left as-is.
+- Benchmark results now persist reliably instead of intermittently showing "Found 0 results" with no progress. The request-thread and worker-thread result syncs could both insert the same row (or a dataset repeating a question could produce a duplicate `query_hash`), tripping the `benchmark_results` unique constraint; because that failed at commit time it rolled back the whole pending batch, discarding results that had actually completed. Result persistence is now serialized and deduplicated, so concurrent syncs and duplicate questions are handled cleanly.
+- Cap persisted log messages at 5000 chars so long langgraph runs no longer accumulate 10 KB ``ResearchLog`` rows. Mirrors PR #4004's frontend truncation discipline, now applied to the database sink. Also promotes the duplicated ``/history/logs`` pagination-cap literals into a single shared constant (``HISTORY_LOGS_HARD_CAP`` / ``HISTORY_LOGS_DEFAULT_LIMIT``, exposed to the log panel as ``window.LDR_LOG_LIMITS``) so the route clamp and the panel's DOM cap no longer drift. Full diagnostics remain available in stderr/file sinks.
+- Collection indexing now behaves identically whether started via the streaming
+  (SSE) route or the background worker. The two paths had duplicated and drifted:
+  the background worker did not persist a collection's ``embedding_dimension`` (so
+  collections indexed in the background lost it), and the SSE route skipped the
+  force-reindex cleanup that clears old chunks and FAISS indices (so a streamed
+  force-reindex could leave stale, mixed-model vectors behind). The shared
+  embedding-metadata, force-reindex cleanup, and document-query logic is now
+  factored into single helpers used by both paths.
+- Converting research history into the searchable library now pages through reports in bounded batches instead of loading every report body into memory at once, preventing `MemoryError` on large histories.
+- Detailed-mode research reports now honor the ``report.enable_file_backup`` setting. The detailed completion path saved ``report_content`` via a raw ORM write that bypassed the report-storage abstraction, so a user who enabled file backup got on-disk ``.md``/``_metadata.json`` files for quick-mode research but silently never for detailed-mode research. Route the detailed save through ``get_report_storage().save_report()`` exactly like the quick and error paths already do (DRY-review finding H2).
+- Emojis in exported PDFs no longer render as empty "tofu" boxes. The default WeasyPrint stylesheet now lists ``"Noto Color Emoji"`` (with ``"Noto Emoji"`` as a monochrome fallback and ``"Segoe UI Emoji"`` / ``"Apple Color Emoji"`` for native installs) at the tail of both the body and ``code``/``pre`` font-family stacks, and the official Docker image now bundles ``fonts-noto-color-emoji`` alongside ``fonts-noto-cjk``. Non-Docker users on Linux without an emoji font installed still need to install one (e.g. ``apt install fonts-noto-color-emoji``) — see ``docs/faq.md``. This change also fixes a pre-existing flaw in the ``custom_css`` path: ``markdown_to_pdf(custom_css=...)`` previously *replaced* the default stylesheet entirely, silently dropping the CJK fallback from #4055; it now layers custom CSS on top of the default so the font fallbacks survive.
+- Fixed 5 `TestQuickSummary` API tests that broke after the egress fail-closed change (#4300): the `quick_summary` endpoint re-imported `get_user_db_session` locally, shadowing the module-level binding so `patch("...web.api.get_user_db_session")` never intercepted — the tests passed only because the old code swallowed the resulting DB error. Removed the redundant local import (one consistent patch target) and updated the settings-failure test to assert the new fail-closed `503` behaviour instead of the obsolete empty-snapshot fallback.
+- Fixed ScaleSerp silently returning zero results when the API includes a result with a `null` link: the eagerly evaluated `link[:50]` display fallback raised `TypeError`, which the outer handler swallowed along with the entire result set.
+- Follow-up relevance filtering now rejects negative and out-of-range source indices and deduplicates repeated ones, so a malformed LLM response can no longer select the wrong source or list the same source twice.
+- Harden the central `ProcessingLLMWrapper` think-tag stripping against non-string responses. `remove_think_tags` is text-only, so it now runs only when `response.content` is a `str`; non-string content (e.g. provider content-block lists such as Anthropic's, or `None`) is passed through unchanged instead of raising `TypeError`. This removes the wrapper-level crash that the per-site `str(...)` guards used to absorb before they were dropped. Note: a few direct `.content` consumers still assume string content, so robust list-content handling at those call sites is tracked separately.
+- Honor ``?limit`` on ``/api/research/<id>/logs`` (the route the log panel actually fetches). It previously ignored the parameter and returned every row, so a long langgraph run could still ship thousands of rows to the browser despite the panel requesting a bounded tail. The limit now returns the newest N rows (oldest-first, unchanged ordering) clamped to ``HISTORY_LOGS_HARD_CAP``; omitting ``?limit`` preserves the existing return-all contract for direct API callers.
+- Library document indexing, collection document listing, and filesystem sync no longer load every document's full text body into memory at once — the large `text_content` column is now deferred (and a document's "has text" flag is computed in SQL), preventing `MemoryError` on large libraries.
+- Library/collection uploads now actually accept ODT, DOCX, PPTX and XLSX/XLS files. These formats were advertised as supported but failed at runtime with a swallowed ``ModuleNotFoundError`` because their ``unstructured`` parser dependencies (``python-docx``, ``python-pptx``, ``openpyxl``, ``msoffcrypto-tool``, ``xlrd``) were never installed — only PDF worked. Those parsers are now declared dependencies, and the loader registry probes each format's real runtime dependency so it only advertises formats it can actually extract: missing deps now yield a clear "Unsupported format" instead of a silent failure. Legacy binary ``.doc``/``.ppt`` (which need a LibreOffice ``soffice`` binary) and image OCR (which needs ``pytesseract`` + ``tesseract``) are offered only when those tools are present.
+- Link analytics (`GET /api/link-analytics`) now projects only the columns it needs plus a SQL-level "has preview" flag, instead of loading full research-resource rows. This avoids materializing every resource's `content_preview` text on the whole-table scan that runs when no time filter is applied (#4560).
+- Make research-log ordering deterministic when rows share a timestamp. ``ResearchLog.timestamp`` is not unique, so ``order_by(timestamp.desc()).limit(N)`` left the rows surviving a shared-timestamp boundary SQL-undefined — under dense logging the newest-N tail (and the "latest milestone" lookup) could vary between requests. Add ``ResearchLog.id`` as a tie-break across all three log-ordering sites (the ``/api/research/<id>/logs`` route, the ``get_logs_for_research`` helper behind ``/history/logs``, and the latest-milestone query). Follow-up to #4645.
+- News subscriptions now treat the ``status`` column as the single source of truth for whether a subscription is active. Previously the scheduler decided what to run by filtering on the separate ``is_active`` boolean, which ``create_subscription`` never sets (it writes only ``status``), so a subscription created as *paused* kept ``is_active=True`` by default and was still run by the scheduler. The active/overdue query was also spelled three different ways across the scheduler and the news API, and one copy omitted the ``next_refresh is not None`` guard. All of these now go through ``NewsSubscription.active_filter`` / ``due_filter``. Manually running a subscription ("run now") also reads its saved model/provider/strategy/search-engine from the database (it previously read a trimmed dict that dropped those fields, so the manual run ignored the subscription's configured model) and advances the refresh schedule on success so an overdue subscription is not immediately re-run by the scheduler. The shared run payload and refresh-schedule arithmetic are extracted into ``news/subscription_runner.py``. The post-completion refresh-time update in the research service (which previously called a subscription-storage helper that referenced non-existent columns and so always failed silently) now works, and the subscriptions list API now returns each subscription's saved model/provider/strategy/search-engine instead of dropping them.
+- Parse ``LDR_TEST_MODE`` as a proper boolean when deciding whether to relax the SQLCipher KDF floor. Previously ``_get_min_kdf_iterations()`` used a bare truthiness check, so any non-empty value — including ``LDR_TEST_MODE=0`` or ``=false`` — enabled the relaxed test minimum, silently weakening database encryption for anyone setting the flag explicitly to disable it. ``PYTEST_CURRENT_TEST`` stays presence-based (pytest sets it to a non-boolean string).
+- REST `/api/v1/analyze_documents` now rejects unknown body parameters with a clear 400 listing the allowed ones (previously a typo like `max_result` surfaced as an opaque 500), and the `/api/v1/` docs endpoint documents the `allow_default_settings` flag for all three research endpoints.
+- Remove redundant per-site `<think>`-tag stripping now that `get_llm` LLMs are normalized centrally in `ProcessingLLMWrapper`. Direct `self.model.invoke(...).content` is already think-stripped, so 11 scattered stripping calls (6 `remove_think_tags(...)` and 5 `get_llm_response_text(...)`) across the constraint/evidence/knowledge components, the synthesis path, and the document-analysis API were dropped (relying on the single central strip point), keeping explicit handling only where it is load-bearing (agent/`bind_tools` paths, injected LLMs). Empty-answer fallbacks now log a warning so a silent degrade is visible.
+- The "download all" and "queue all undownloaded" library actions now project only the columns they use instead of loading full `ResearchResource` rows, avoiding materializing every resource's `content_preview` text on a whole-table scan for large libraries (#4560).
+- The `/api/history` endpoint now paginates its results (default 200, max 500) instead of loading every research row — and its `research_meta` JSON — into memory at once.
+- The library documents API now clamps the `limit`/`offset` query parameters, so a negative `limit` (which SQLite treats as "no limit") can no longer bypass pagination and load an entire collection into memory.
+- The library domain-filter scan (`get_unique_domains`) now streams document URLs in batches instead of loading every row at once, avoiding excessive memory use on very large libraries (#4560).
+- The rate-limiting reset (`POST /api/rate-limiting/engines/<engine>/reset`) and cleanup (`POST /api/rate-limiting/cleanup`) endpoints now operate on the persisted `RateLimitEstimate` table instead of the per-request `get_tracker()` singleton. Like the read endpoints fixed in #4721, the tracker's mutation path is gated on a research-session context that is absent in an analytics HTTP request, so these endpoints were silent no-ops that never cleared the estimates the `/status` and `/current` panels display. Reset now deletes the engine's stored estimate (so it re-learns), and cleanup deletes estimates not updated within the requested window.
+- The research-session filter dropdown on the library page no longer loads every research row unbounded; it is now capped to the most-recent sessions, avoiding excessive memory/DOM use on very large histories (#4560).
+- The three egress warning banners (public-egress, cloud LLM, cloud embeddings) now have separate dismiss flags. Previously they shared one, so dismissing the fresh-install public-egress notice also permanently hid the critical cloud-LLM and cloud-embeddings warnings.
+- When testing a notification webhook URL through the settings UI, the
+  "Test" button now surfaces the validator's reason instead of a generic
+  "Invalid notification service URL." message. For private/internal IP
+  rejections that the operator can unblock (loopback, RFC1918, CGNAT,
+  link-local, IPv6 private), the error message names
+  `LDR_NOTIFICATIONS_ALLOW_PRIVATE_IPS`; for NAT64-wrapped non-metadata
+  destinations on IPv6-only deployments (RFC 6052 well-known
+  `64:ff9b::/96` or RFC 8215 local-use `64:ff9b:1::/48`) it names
+  `LDR_SECURITY_ALLOW_NAT64` — the only flag that can unblock those, which
+  the hint probes for independently. Cloud-metadata IPs are always blocked and
+  the env-var hint is intentionally suppressed for them — neither flag
+  re-opens metadata, so naming them would mislead the user.
+
+  Also adds an "IPv6-only deployments (NAT64)" subsection to
+  `docs/SearXNG-Setup.md` so operators routing IPv4 through NAT64 know
+  about the opt-in.
+- `get_llm_response_text` now extracts the text from list-type LLM content blocks (Anthropic extended-thinking / tool-use responses, where `message.content` is a list of blocks) instead of stringifying the list to its Python `repr`. This fixes garbled entity, sub-query, and candidate parsing, and stops direct `.content` consumers from crashing on list content.
+
+### 🗑️ Removed
+
+- Dropped the orphaned `cache` and `search_cache` tables and removed their unused
+  `Cache`/`SearchCache` models (migration 0016). Neither table was ever populated
+  by any code path, so existing databases lose no data — the empty tables are
+  removed automatically on the next migration. ([#drop-orphaned-cache-tables](https://github.com/LearningCircuit/local-deep-research/pull/drop-orphaned-cache-tables))
+- Removed the `/api/v1/quick_summary_test` REST endpoint. It was a near-duplicate of `/quick_summary` with hardcoded test defaults; call `/quick_summary` with `search_tool`, `iterations`, and `temperature` set explicitly instead. ([#3661](https://github.com/LearningCircuit/local-deep-research/pull/3661))
+- Removed the benchmark result-reuse feature: new benchmark runs no longer silently import results from previous completed runs with matching search settings. The compatibility check covered only a handful of settings (ignoring evaluation config, dataset versions, and the rest of the configuration), so results produced under different conditions could contaminate a run's accuracy, and the reuse accounting could report more than 100% completion. Every run now researches its own sampled questions and reports only its own results. ([#4498](https://github.com/LearningCircuit/local-deep-research/pull/4498))
+- Removed the DOAJ Seal quality tier (score 8) and the Tier 4 "+1 Seal bonus": DOAJ retired the Seal in April 2025 and removed it from its metadata, so the tier could never be earned again and only ever fired on stale pre-2025 data. DOAJ-listed journals keep their score-5 floor. The `has_doaj_seal` column was dropped from the reference DB (schema version 4 — the DB rebuilds automatically on first use) and the Seal star/count no longer appear on the journal-quality dashboard.
+- Removed the experimental search strategies that were only reachable through the "Show All Strategies" toggle (rapid, parallel, iterative, recursive, adaptive, smart, standard, iterdrag, browsecomp, evidence, the constrained/dual-confidence family, modular, and others), along with the toggle itself. The strategy dropdown now offers source-based, focused-iteration, focused-iteration-standard, topic-organization, mcp, and langgraph-agent. The news feature's internal news-aggregation strategy is unaffected.
+- Removed the now-unused `PathValidator.confine_to_base` helper (and its tests). It was added only to confine the local-folder RAG indexing route, which has been removed, leaving it with no callers.
+- Removed the unused `GET /research/api/config` endpoint. Every field it returned (`version`, `llm_provider`, `search_tool`, `features.notifications`) was read from `current_app.config` keys that are never set, so it only ever returned hardcoded placeholder values. It had no frontend or production callers. Use `GET /research/api/settings/current-config` for live configuration instead.
+- Removed the unused `PricingCache._load_cache` and `PricingCache._save_cache` methods. They were left behind as deprecated no-op stubs (empty `pass` bodies) when the disk-backed pricing cache was replaced with an in-memory bounded `TTLCache`, and they had no callers anywhere in the codebase.
+- Removed the unused ``news.subscription_manager`` storage subsystem
+  (``SQLSubscriptionStorage``, ``SearchSubscription``, ``TopicSubscription``,
+  ``BaseSubscription`` and their factories), the abstract ``SubscriptionStorage``
+  interface, and ``StorageManager.get_user_subscriptions`` /
+  ``get_user_stats``. This code was never reached by any live path -- all real
+  subscription functionality goes through ``news.api`` and the scheduler -- and
+  was broken against the current ``NewsSubscription`` model (it referenced
+  columns such as ``user_id``/``refresh_count``/``results_count`` that do not
+  exist, so it raised ``AttributeError`` whenever called). The package-level
+  exports ``SearchSubscription`` and ``TopicSubscription`` from
+  ``local_deep_research.news`` are removed as part of this; nothing in the
+  codebase imported them.
+- Removed two orphaned RAG HTTP endpoints that had no UI and no other caller: `GET /library/api/rag/index-local` (local-folder indexing — superseded by collection-based indexing) and `POST /library/api/rag/index-research` (which called a method that no longer exists and always errored). The unused `index_local_file` service method and glob-pattern allowlist were removed with them.
+
+### 📝 Other Changes
+
+- The available-models endpoint no longer hand-rolls Ollama/OpenAI/Anthropic model listing that was immediately overwritten by provider auto-discovery — auto-discovery is now the single fetch path. This removes a redundant per-refresh network round-trip to each provider, and local-only model listing now goes through the provider classes (which validate the URL for SSRF and send auth headers, so an auth-protected local Ollama now lists its models). ([#4646](https://github.com/LearningCircuit/local-deep-research/pull/4646))
+- The two Ollama health-check endpoints (`/check/ollama_status`, `/check/ollama_model`) now share a single `/api/tags` probe helper instead of separately re-implementing the fetch + new/old API-format parsing + error classification, so "is Ollama up?" answers consistently across them. No change to the endpoints' responses. ([#4650](https://github.com/LearningCircuit/local-deep-research/pull/4650))
+- Add a comprehensive Puppeteer UI suite for the egress-policy feature: `tests/ui_tests/test_egress_policy_ui.js` (CI-safe, 32 assertions) covers every UX touchpoint — Privacy & Egress panel rendering, all four scopes painting their expected border/text colors (amber/teal/indigo palette verified via computed-style reads, not just DOM state), the STRICT-on-meta-picker guard reverting to "Both" with an explanatory hint, per-research overrides round-tripping through the settings DB, scope cue propagating to /history, /metrics and /chat via base.html's `body[data-scope]`, settings dashboard listing the policy keys, and the require-local toggles persisting across reloads. A second file, `tests/ui_tests/NO_CI_test_egress_policy_live_research.js`, drives end-to-end research runs against a real Ollama + SearXNG instance (defaults to a lab box; overridable via env) and verifies the run-start PEP at `research_routes.py:248` — STRICT+searxng succeeds, STRICT+auto returns 400 with the meta-picker incoherence message, PUBLIC_ONLY refuses a SearXNG whose instance URL resolves to a private IP (`scope_mismatch_public_only`), PRIVATE_ONLY accepts the same SearXNG (URL classifier overrides the static `is_public=True`), and a cloud LLM under `require_local_endpoint=true` is accepted at the precheck but reaches `failed` status as the strategy's `get_llm()` hits the PEP. Both files gate screenshots behind `!process.env.CI` matching the `tests/ui_tests/test_settings_page.js:57` convention; the live test's `NO_CI_` filename prefix means the runner skips it in CI (where the lab endpoints are unreachable). Screenshots land in `tests/ui_tests/screenshots/egress-policy{,-live}/` (already gitignored).
+- CI: set ``LDR_TEST_MODE=1`` on every test/scan workflow that starts a server or initialises the test database, so the intended fast SQLCipher KDF (``LDR_DB_CONFIG_KDF_ITERATIONS=1000``) is actually honoured. ``_get_min_kdf_iterations()`` deliberately ignores generic ``CI``/``TEST_ENV`` and only relaxes for ``PYTEST_CURRENT_TEST``/``LDR_TEST_MODE``; without the latter the requested 1000 was silently clamped to the production minimum (256000), making every DB open and the post-login background backup ~256× slower. Under contended CI runners that turned the backup into a multi-minute, GIL-holding stall that timed out release-gate UI shards (chat/settings navigations) — the failures seen on clean ``main`` while investigating #4430. Also renames the deprecated ``LDR_DB_KDF_ITERATIONS`` env var to its canonical ``LDR_DB_CONFIG_KDF_ITERATIONS`` across all six workflows (docker-tests, nuclei, owasp-zap-scan, playwright-webkit-tests, puppeteer-e2e-tests, docker-multiarch-test).
+- Consolidate the egress-policy code into a dedicated `security/egress/` subpackage (`policy.py` + `audit_hook.py`) with a README documenting the design, scope model, and the full map of enforcement points. No behavior change — module paths moved from `security.egress_policy` / `security.egress_audit_hook` to `security.egress.policy` / `security.egress.audit_hook`.
+- Extracted the benchmark YAML export helpers (`yamlEscape` / `formatSettingValue` / `formatSettingsSnapshot`) from the inline `benchmark_results.html` template into a unit-tested module (`static/js/utils/yaml_export.js`, loaded via `<script src>` like the other shared JS utils). No behavior change — it adds regression coverage (`tests/js/utils/yaml_export.test.js`) for the escaping that guards every benchmark YAML download, so a future change to `yamlEscape` can't silently corrupt exports.
+- Hardened the #4804 `<think>`-stripper fix: corrected the now-stale `bind_tools` comments in the LangGraph agent, made the async wrapper test actually assert stripping, and added a real `create_agent` integration regression test (sync + async).
+- Moved `ProcessingLLMWrapper` out of the `wrap_llm_without_think_tags` closure to module scope in `config/llm_config.py` — it's now importable and `isinstance`-checkable, and defined once instead of rebuilt on every call. Pure refactor; no behavior change.
+- Recolor the egress-scope visual cues to match the actual data-leak risk hierarchy. Previously BOTH (the default) wore a light-amber accent and PUBLIC_ONLY a deeper-amber one, which reads as "PUBLIC_ONLY is more dangerous than the default" — but BOTH is actually the riskiest scope: it permits local engines AND a cloud LLM in the same run, so local library/RAG chunks can land in OpenAI via the LLM call. PUBLIC_ONLY blocks local engines, so local data never enters that pipeline. New palette: BOTH gets no left-border accent at all (the existing "Public search egress enabled" banner does the nagging in text); PUBLIC_ONLY uses sky-blue (#0ea5e9 — informational "you chose public sources only"); PRIVATE_ONLY stays teal (#14a37f); STRICT moves from indigo to violet (#8b5cf6) so it's clearly distinct from the new PUBLIC_ONLY blue for color-blind users. The privacy panel itself defaults to a neutral slate when BOTH is active rather than amber. The risk-honest rationale is documented in the `base.html` style-block comment.
+- Removed dead and inert code from Chat Mode's context manager that shipped without a consumer: the unused ``build_prompt_context()`` and ``_get_recent_messages()`` helpers, the ``conversation_history`` and ``accumulated_sources`` keys in the research-context dict (nothing downstream read them), and the cross-turn source-count tracking (``_extract_sources_from_history``, the ``source_count`` field, and ``update_accumulated_context``'s ``source_count_delta`` parameter) — which never functioned because the producer was never called with sources. The ``chat.max_context_messages`` setting is removed with it, since it only fed the now-deleted recent-message inclusion path. ``chat.max_findings_to_include`` (which still controls how many prior findings carry into a follow-up) is unchanged.
+- Removed the redundant local re-import of `get_settings_manager` inside the `quick_summary` API endpoint (it was already bound at module level), eliminating the same import-shadowing anti-pattern fixed for `get_user_db_session` — so `patch("...web.api.get_settings_manager")` now reliably intercepts. Both helpers are now module-level for one consistent patch surface; only `quick_summary` itself remains a local import (it pulls in the research stack, which would cycle).
+- Tests: fix a GC-timing flake in ``test_base_downloader.py::test_context_manager_calls_close``. The test patched ``close`` on the ``BaseDownloader`` *class*, but ``BaseDownloader.__del__`` also calls ``self.close()`` — so whenever the garbage collector happened to finalize a downloader left over from an earlier test while the patch was live, the shared class-level mock counted an extra call and the assertion failed with "Expected 'close' to have been called once. Called 2 times." (seen failing an unrelated PR's CI, #4415). Patch on the instance instead — the same pattern the neighbouring ``__del__`` test already uses — which is immune to other instances' finalizers (mechanism reproduced deterministically both ways).
+- Tests: give the logpanel ``prunes the oldest entries when count exceeds MAX_LOG_ENTRIES`` vitest case an explicit 20s timeout. #4304 fixed the real root cause (501 ``setTimeout(autoscroll, 0)`` tasks piling on the real timer queue) by faking all timers, but the test's remaining cost is honest O(n²) DOM work — 501 inserts each running ``addLogEntryToPanel``'s full-container ``querySelectorAll`` scans, ~2.6–2.9s in happy-dom on a dev machine — leaving no headroom against the 5s vitest default under parallel CI load. The recurring timeout flake intermittently failed the shared "All Pytest Tests + Coverage" job on unrelated PRs (e.g. #4415). The test must fill to the real 500-entry cap to exercise the prune, so the work can't shrink; a bigger budget is now the right lever (unlike #4299, which proposed it while the timer bug was still live).
+- Tests: lock in the ``LDR_TEST_MODE`` boolean-parsing behaviour for the SQLCipher KDF floor. Adds direct ``_get_min_kdf_iterations()`` unit tests for falsey values (``0``/``false``/``no``/``off`` must NOT relax the floor — the #4564 regression) and the full truthy set (``on``/``enabled`` must relax it, guarding against a narrower parser). Also moves the integration test off the registry's ``min_value`` boundary so it exercises the KDF-floor clamp rather than range validation.
+- Tests: stop the ``QueueProcessorV2`` background thread between tests in the ``reset_singletons`` autouse fixture (it already did this for ``BackgroundJobScheduler``). The module-level ``queue_processor`` singleton was started by the first ``create_app()`` in an xdist worker and never stopped, so one test's processor thread ran for the whole worker — looping through ``SettingsManager`` → SQLCipher connection opens on the shared ``db_manager`` concurrently with every later test, and emitting logs to a closed pytest stderr sink at teardown (the same class of bug the scheduler handling fixes). Tests that exercise the queue patch the singleton, so stopping the real thread does not affect them.
+- The pre-commit CI job now retries only the hook-environment download (the network-flaky step), not the lint run itself. Previously the entire `pre-commit run` was wrapped in a 2-attempt retry, so an auto-fixing hook (e.g. ``ruff-format``) that rewrote a file would fail the first attempt, leave the now-fixed tree in place, and pass the retry — silently turning a formatting violation into a green check while the unformatted code stayed on the branch. Hook downloads are retried via ``pre-commit install-hooks``; the check now runs exactly once and fails honestly.
+- The release gate now imports the SIMD-heavy native dependencies (numpy, pandas, pyarrow, scikit-learn, faiss, torch) under qemu-emulated SandyBridge (AVX without AVX2) and Haswell (AVX2) CPUs. This catches dependency wheels that silently raise the x86-64 instruction-set baseline and would crash with SIGILL on older CPUs — the failure mode that shipped in v1.7.0 when faiss-cpu 1.14.2 executed an AVX2 instruction at import time on AVX-only hosts (#4480). Building the gate surfaced that AVX is already LDR's de-facto minimum CPU requirement: the pandas and scikit-learn wheels crash on pre-AVX CPUs (Nehalem/Westmere era, pre-2011), so those are not part of the gate. The check runs only in the release gate, not on every PR, since full CPU emulation is slow and the failure mode can only ship via dependency bumps.
@@ -0,0 +1,43 @@
+
+### 🔒 Security
+
+- Raw exception variables (`e`, `exc`, etc.) are no longer interpolated directly into `logger.exception()` messages across all LLM provider, embedding provider, and web search engine paths; each catch site now scrubs the message through `sanitize_error_message()`/`redact_secrets()` before logging, preventing error text from leaking API keys, credentials, or internal URL structure into production logs. Scrubbed exception log lines now include the exception class name so exceptions that stringify to an empty message still produce useful diagnostics. A new `check-sensitive-logging` pre-commit hook enforces this rule going forward, rejecting direct exception-variable interpolation in the secure-logging directories and flagging `exc_info=True` on `warning`/`error`/`critical` calls. ([#4888](https://github.com/LearningCircuit/local-deep-research/pull/4888))
+- `NewsAPIException.to_dict()` now redacts credential shapes from the fields it serializes to the client through the two `@errorhandler(NewsAPIException)` handlers: the human-readable `message` goes through the central `sanitize_error_for_client()`, and string leaves of `details` go through `sanitize_error_message()`. This is a defence-in-depth backstop at the response boundary — the primary defence stays at the `news/api.py` raise sites (curated generic messages, #4843) — so a future raise site, subclass, or external caller that lets a credential-bearing string into `message` or `details` (e.g. the caller-supplied search `query` forwarded in `details`) has it redacted before it ships. Redaction is credential-shape based (Bearer/`Authorization`, `?api_key=`-style params, known token prefixes, `http(s)` URL userinfo), so legitimate text is unchanged; `error_code` / `status_code` are left untouched. It does not attempt to catch DSN userinfo (`postgresql://user:pass@`), SQL, or filesystem paths — those remain the responsibility of the raise-site discipline. ([#4931](https://github.com/LearningCircuit/local-deep-research/pull/4931))
+- The central credential sanitizer (`sanitize_error_message` / `sanitize_error_for_client`) now redacts URL userinfo credentials in *any* scheme (case-insensitive), not just `http(s)`. Raw **URL-form** database connection errors — SQLAlchemy `dialect+driver` DSNs (`postgresql+psycopg2://user:pass@host`, `mysql+pymysql://…`, `mongodb+srv://…`), plain DB schemes, and password-only DSNs (`redis://:pass@host`) — previously leaked their password through any surfaced exception message; the password is now replaced with `[REDACTED]`. Credential-less DSNs with a port (`postgresql://host:5432/db`) and all existing `http(s)` behavior are unchanged. Key=value DSNs (e.g. pyodbc `Server=...;Pwd=...`) have no `://` and remain out of scope for a userinfo regex. This benefits every consumer of the central sanitizer (download service, fetch tool, agent strategies, search engines, and the news error-response backstop). ([#4933](https://github.com/LearningCircuit/local-deep-research/pull/4933))
+- `WebAPIException.to_dict()` now redacts credential shapes from the fields it serializes to the client (`message` via `sanitize_error_for_client()`, `details` string leaves via `sanitize_error_message()`), mirroring the `NewsAPIException` backstop. This is a defence-in-depth measure at the response boundary so that a future raise site, subclass, or external caller that lets a credential-bearing string into `message` or `details` has it redacted before it ships. Legitimate text is unchanged, and `status` / `error_code` / `status_code` are left untouched. ([#4937](https://github.com/LearningCircuit/local-deep-research/pull/4937))
+- The egress guardrail now enforces a two-axis (data-sensitivity × destination-exposure) admissibility rule at the start of a research run (ADR-0007): the run is refused when a sensitive source (a private collection/library/store) would reach an exposing sink (a public search engine or a cloud LLM/embeddings provider), in addition to the existing scope checks. Override by setting Egress Scope to **Unprotected**, marking the collection public, or trusting the destination.
+
+### ✨ New Features
+
+- **New Zotero integration: import a Zotero library or collection into your document library, with optional background auto-sync.** Add your Zotero API key, library type/ID and (optionally) a collection key under **Settings → Zotero**, then use the new **Library → Zotero** page to test the connection, list collections, and **Sync now**. Imported papers are stored, text-extracted and RAG-indexed like any other library document, so they become searchable during research via the collection search engine — no separate search engine to configure. Sync is incremental (new, changed and removed items are reconciled), and items without an attached PDF are imported as metadata/abstract text by default (toggle off to import PDFs only). Enable **Auto-Sync** to refresh in the background on a configurable interval. ([#4723](https://github.com/LearningCircuit/local-deep-research/pull/4723))
+- Zotero integration polish from first real-world testing: standalone PDF
+  attachments (PDFs added without a parent item) now import as documents;
+  the library ID is auto-detected from the API key (usernames resolve
+  too); manual syncs re-examine previously skipped items; imports appear
+  in the main Library view; a live progress bar tracks syncs; the config
+  page autosaves with essentials-first layout and runs the connection
+  test on saving a key; friendlier defaults (text-only PDF storage,
+  integration enabled once a key is set) and actionable error messages. ([#4960](https://github.com/LearningCircuit/local-deep-research/pull/4960))
+- Add a per-category entry counter next to each filter button in the Research Logs panel (All, Milestones, Info, Warning, Errors). Counts reflect entries currently rendered in the DOM and update on insert, prune, and batch load, so users can see at a glance which categories still have entries when the global DOM cap is hit.
+- Add an **Unprotected** egress scope — an explicit opt-out that disables the egress-scope restrictions for a run (the hard SSRF and cloud-metadata blocks still apply), surfaced with a light-red panel and a non-dismissible "protection disabled" banner. The rarely-used **Both** scope is retired: it is removed from the selector, and existing saved `both` values are migrated to **Adaptive** (a residual value is also coerced to Adaptive at read time). If you relied on `both` to run a private collection with a cloud model, mark that collection **public** or choose **Unprotected**.
+- Add per-destination trust to the egress model (ADR-0007): `policy.trusted_inference_providers` and `policy.trusted_search_engines` let you mark specific off-machine LLM/embeddings providers or search engines as trusted, so the two-axis classification treats them as *contained* (e.g. a zero-retention enterprise Anthropic endpoint, or a self-hosted Elasticsearch/Paperless on a public hostname). A banner surfaces active trust entries.
+
+### 🐛 Bug Fixes
+
+- Registration no longer leaves a bricked account behind: if creating the encrypted user database fails after the auth row is committed, both the orphaned auth row and any partial on-disk database files (including the per-database salt) are now cleaned up, so the username is genuinely reusable on retry. ([#4934](https://github.com/LearningCircuit/local-deep-research/pull/4934))
+- The settings write API now rejects malformed keys (a trailing dot like `local_search_chunk_size.`, a leading dot, an empty `..` segment, or blank/whitespace) with HTTP 400 instead of silently persisting them. Such rows made `get_setting()` return a `{"": value}` wrapper dict that rendered as `[object Object]` on settings pages (#4840). The manager (`set_setting`, `create_or_update_setting`) also refuses to create malformed keys, and `import_settings` skips them so a corrupted export can't reintroduce them. Complements the read-side fix in #4852. ([#4935](https://github.com/LearningCircuit/local-deep-research/pull/4935))
+- Registration now recovers from an orphaned per-database salt file (left with no matching database by a prior interrupted registration), so a username that was previously stuck as un-registerable can be registered again. ([#4942](https://github.com/LearningCircuit/local-deep-research/pull/4942))
+- Both settings prefix-read paths — `SettingsManager.get_setting` (DB) and `get_setting_from_snapshot` (the snapshot path used by research threads) — now filter out malformed rows (e.g. a legacy `foo..` or `foo. ` key), so a stray malformed row can no longer turn a leaf read into a `{".": value}` wrapper dict that renders as `[object Object]`. Completes the read side of #4840 beyond #4852's exact single-trailing-dot exclusion; existing malformed rows remain harmless and are not deleted. ([#4946](https://github.com/LearningCircuit/local-deep-research/pull/4946))
+- Fix the 'Info' filter button in the Research Logs panel being a silent no-op. Clicking it left the panel showing every entry (warnings, errors, milestones) because the visibility helper returned true for all log types when the active filter was 'info'. The filter now narrows to info entries only, matching the button's label.
+- Fixed a 30–50 second visible lag between clicking **Stop** and a focused-iteration research actually stopping. `FocusedIterationStrategy.analyze_topic` now calls `self.check_termination()` at the start of every iteration, after the parallel search (before optional verification searches), and just before the final synthesis LLM call, so a cancellation request is detected within the current iteration rather than only at the next iteration's progress emit.
+- Fixes a subtle data-loss bug in the Research Logs panel: identical warning, error, and milestone entries were being collapsed into a single `(N×)` counter, the same way repetitive info entries like "Status: OK" already were. Collapsing diagnostic entries like that strips the recency signal — after the second identical error you can no longer tell *when* the last failure happened, and repeated retries look like a single stuck event instead of forward progress.
+
+  The content-dedup scan now applies only to `info` entries (where collapsing repetitive noise is useful). Warning / error / milestone entries always render. The id-based dedup earlier in the pipeline still catches exact retransmits with the same id, so the bypass is strictly about *content-similar-but-distinct-events* getting the same timestamp.
+
+### 📝 Other Changes
+
+- Fixed broken copy-paste examples in the README and docs (benchmark CLI
+  module path, `rate_limiting reset --engine`, nonexistent `openclaw`
+  search engine) plus two dead external links, and added a pre-commit
+  check that keeps README links, anchors, and examples in sync with the
+  codebase. ([#4950](https://github.com/LearningCircuit/local-deep-research/pull/4950))
@@ -0,0 +1,30 @@
+
+### 🔒 Security
+
+- User-supplied strings that reach SQL `LIKE` queries — library search and domain filters, domain-classifier sampling, and news subscription filters — now escape `%` and `_` wildcards. Previously a value like `100%` or `a_b` was interpreted as a wildcard and matched unrelated rows; searches and filters now match literally, closing a wildcard-injection/enumeration vector within a user's own database. ([#3094](https://github.com/LearningCircuit/local-deep-research/pull/3094))
+- All LLM provider, embedding provider, and web search engine modules now log through the diagnose-gated `security.secure_logging` wrapper, so exception tracebacks in these paths only appear in logs when diagnose mode is explicitly enabled (`LDR_APP_DEBUG` + `LDR_LOGURU_DIAGNOSE`) — production logs no longer receive full tracebacks that could echo API keys or request internals from provider/engine failures. The `check-sensitive-logging` pre-commit hook now enforces this by construction: raw `loguru` imports and every known bypass route (`logger.opt()`, `@logger.catch`, private logger handles, `traceback`/`sys.exc_info()` interpolation, dynamic loguru/traceback imports, logger rebinding, and `getattr` dodges) are rejected in these directories, `bind()`/`patch()` chain messages get the same exception-scrubbing checks as direct logger calls, and exception detail reached through attribute or subscript access (`e.args`, `e.response.text`, `e.strerror`) is flagged like `str(e)` itself. ([#4976](https://github.com/LearningCircuit/local-deep-research/pull/4976))
+- Fixed a plaintext-secret disclosure in `GET /settings/api/bulk`: a namespace request (e.g. `keys[]=llm`) returned nested `*.api_key` values unredacted because the redaction check looked only at the outer requested key, and `keys[]=%` could dump the whole settings table via an unescaped `LIKE`. Redaction now recurses into setting subtrees and the key lookup escapes LIKE wildcards. ([#5028](https://github.com/LearningCircuit/local-deep-research/pull/5028))
+- `GET /settings/api/bulk` now redacts password-typed settings whose leaf name is a non-classic secret token (e.g. `client_secret`, `secret_key`, `bearer_token`, `api_secret`, `app_secret`). The bulk endpoint redacts by key name only, so these were previously shipped in the clear while the singular GET masked them. ([#5038](https://github.com/LearningCircuit/local-deep-research/pull/5038))
+- Exception-derived strings flowing through the `/api/v1/quick_summary` and `/api/v1/analyze_documents` JSON responses are now scrubbed at the HTTP boundary (and at the strategy layer in `SourceBasedSearchStrategy`) via `sanitize_error_for_client`, closing a CWE-209 information-exposure vector flagged by CodeQL (#8019): a caught exception interpolated into a strategy's error fields could otherwise reach the API client — as `summary` (the key `quick_summary()`/`analyze_documents()` actually return), `formatted_findings`, or per-finding `content` — with credentials or stack-trace text intact. Credentials are scrubbed and length is capped; the `"Error: "` prefix is preserved so error payloads stay recognizable to clients, and the strategy-layer scrub independently covers the web-UI SSE path (`research_service.py` → `ErrorReportGenerator`), which consumes strategy output directly. `/api/v1/generate_report` gets the same boundary scrub purely as a precaution — its current payload (`content`/`metadata`) carries no error fields.
+
+### ✨ New Features
+
+- **Sofya search engine** — Added [Sofya](https://sofya.co) as an optional hosted web-search engine. Sofya's `/v1/search` returns ranked results with SERP snippets and extracted page content (markdown) in a single call, fitting LDR's two-phase retrieval model without a separate fetch round trip. Supports `basic`/`snippets` depth, `general`/`news` topics, recency filtering, and domain include/exclude. Configure your API key under Settings → Search → Sofya (`search.engine.web.sofya.api_key`) or the `LDR_SEARCH_ENGINE_WEB_SOFYA_API_KEY` environment variable; GitHub sign-up grants 1000 free credits/month.
+
+### 🐛 Bug Fixes
+
+- Fixed the news subscription-history endpoint (`/news/api/subscriptions/<id>/history`) raising an internal error for any subscription that had research runs: `get_subscription_history` called `.isoformat()` on the `created_at`/`completed_at` values, which are stored as isoformat strings, so it now returns them directly. ([#3094](https://github.com/LearningCircuit/local-deep-research/pull/3094))
+- Fix collapsed-whitespace bug in SearXNG-sourced titles. The HTML parser was calling `BeautifulSoup.Tag.get_text(strip=True)` to read result titles, which strips leading/trailing whitespace *and* collapses every internal whitespace run to nothing — so multi-word titles like "Word One Word Two Word Three" rendered as "WordOneWordTwoWordThree" in the `## Sources` block. Swapped to `get_text(" ", strip=True)` (the documented BeautifulSoup idiom for "strip edges, replace internal runs with a single space") so word boundaries survive. The same fix was applied to result snippets. Title preservation flows through to the LLM relevance filter and `BaseCandidateExplorer` / `ProgressiveExplorer` candidate-phrase extraction, both of which previously saw single-token titles from SearXNG-sourced collections. ([#4970](https://github.com/LearningCircuit/local-deep-research/pull/4970))
+- Registration/login no longer leave a user silently logged in when post-authentication setup fails partway: the partial session is now rolled back, so a reported failure matches reality (you are not logged in). ([#5006](https://github.com/LearningCircuit/local-deep-research/pull/5006))
+- Fix citation labels rendering as empty `[]` for library/RAG sources. The `DOMAIN_HYPERLINKS`, `DOMAIN_ID_HYPERLINKS`, and `DOMAIN_ID_ALWAYS_HYPERLINKS` modes previously fed `urlparse(url).netloc` straight into the citation label, so any relative URL (most notably RAG / library hits that look like `/library/document/<uuid>`) produced an empty string. With a single relative-URL citation the label collapsed to `[[]](url)` and rendered as a `[]` link with no anchor text; with multiple relative-URL citations the suffix leaked out as `[[-1]](url)`, `[[-2]](url)`. The formatter now falls back to a slugified (lowercased, alphanumeric-only) version of the document title, and as a final fallback to the citation number itself, so the label is always non-empty and the link carries the document name. Real web URLs (arxiv.org, github.com, etc.) are unchanged — the domain still wins when `netloc` is non-empty.
+- Fixed `formatBytes` rendering sizes of 1 TB or larger as `"N undefined"` (e.g. a library blob total of 1 TB showed as `1 undefined`). The shared byte formatter only defined units up to GB; it now covers TB through EB and clamps the unit index at both ends, so very large values no longer index past the end of the unit list and sub-1-byte values no longer produce a negative index.
+- When the Research Logs panel hits its 500-entry DOM cap, the oldest entries are flushed first — which previously meant old warnings and errors got dropped even when the cap was blown by a flood of routine info entries. The prune now walks the cap-excess slots in priority order (info first, then milestones, then warnings, then errors), so the panel keeps its most diagnostic entries even on long research runs.
+
+### 📝 Other Changes
+
+- Removed the `importlib.util` disk-load workaround for `url_classifier` in `citation_formatter.py` (introduced in #4880). Now that `content_fetcher` no longer eagerly imports its heavy dependency tree (#5023), the formatter uses a normal `from ..content_fetcher.url_classifier import URLClassifier, URLType`. Also fixed two nits in the #5023 test file: an off-by-one project-root path in the subprocess tests and `check=True` calls that hid the subprocess stderr from failure reports. ([#4992](https://github.com/LearningCircuit/local-deep-research/pull/4992))
+- Fixed an order-dependent test flake: the security import-fallback tests replaced `local_deep_research.security*` modules with fresh copies, recreating enums like `Sensitivity` and breaking identity checks in tests that ran later on the same worker.
+- Source-tagged citation formatting no longer re-runs the URL-classifier label lookup for every citation occurrence — the pre-computed label cache is consulted lazily, so each source's label resolves once. (The disk-loading import this originally worked around was removed entirely in #4992.)
+- The Sofya egress-labels test now compares enum members by `.value` instead of by identity. Under the Docker test setup the package is simultaneously available from `/app/src` (PYTHONPATH) and the installed wheel, so `SofyaSearchEngine`'s class attribute and the test's `Sensitivity` can resolve to distinct class objects with identical members. Python's `Enum.__eq__` returns `NotImplemented` across distinct classes (so `==` fails just like `is`); only `.value` (or `.name`) survives the dual-class scenario.
+- The `check-pathlib-usage` pre-commit hook now catches `import os as <alias>` followed by `<alias>.path.*`. Previously the AST matcher only flagged the literal name `os`, so an aliased import silently bypassed the check — which is how a `_os.path.join(...)` slipped into #4880 and had to be caught by human review instead.
+- `content_fetcher.ContentFetcher` is now resolved lazily on first attribute access (PEP 562 module `__getattr__`) instead of being eagerly imported in the package `__init__`. `url_classifier` (and any other import that goes through the `content_fetcher` package) no longer drags in the fetcher's transitive deps (`requests`, `playwright`, `bs4`, `lxml`, etc.), so minimal test setups and the citation formatter can stop using the disk-load workaround introduced in #4880. Public API is unchanged.
@@ -0,0 +1,14 @@
+
+### ✨ New Features
+
+- Research steps in chat mode now expand to show the actual tool output (search results, fetched page content — capped at 4,000 chars) when clicked, and the agent's "selecting next action from …" heartbeat lists every enabled tool instead of sampling 3 with "+N more". ([#5051](https://github.com/LearningCircuit/local-deep-research/pull/5051))
+
+### 🐛 Bug Fixes
+
+- Fix the Language Model dropdown frequently coming up empty on the New Research page (requiring a manual click of the refresh button). Three root causes are addressed: the page fired two concurrent `/available-models` requests on every load, which contended against a cold Ollama; Ollama's model-list query used a too-tight 2-second timeout (raised to 5s) whose failure was silently turned into an empty list; and a transient empty result could linger in the client cache. The list now loads once per page and self-heals from a transient empty response.
+- Fixed numbers in exported PDFs rendering as wide, spaced-out emoji glyphs ("2 0 2 6" instead of "2026"). Digits 0-9 carry the Unicode Emoji property, so listing "Noto Color Emoji" in the PDF font stacks (#4730) made Pango draw every digit with the emoji font. Emoji families are removed from the stacks; emoji still render via fontconfig per-character fallback to the installed emoji font.
+- Validate that a language model is selected before starting a research from the New Research page. Submitting with an empty model now shows an inline "Please select or enter a model." error and focuses the model field instead of kicking off a run that would fail downstream.
+
+### 📝 Other Changes
+
+- Fixed cross-test state leaks from tests that reload or evict production modules: the rate-limiter tests restored a fake config, the history-routes fixture leaked auth-bypassed blueprints, the office-format tests could leak a blocked-dependency loader registry, the settings-logger tests leaked the log level, the security import-fallback helper re-created the whole security tree (stacking permanent audit hooks), and four sibling sites (thread-local-session eviction, auth-db reload, env-definitions re-import, loader-registry import guards) now restore the exact pre-test modules.