learningcircuit--local-deep-research/docs/release_notes/1.9.1.md

### 🔒 Security

- User-supplied strings that reach SQL `LIKE` queries — library search and domain filters, domain-classifier sampling, and news subscription filters — now escape `%` and `_` wildcards. Previously a value like `100%` or `a_b` was interpreted as a wildcard and matched unrelated rows; searches and filters now match literally, closing a wildcard-injection/enumeration vector within a user's own database. ([#3094](https://github.com/LearningCircuit/local-deep-research/pull/3094))
- All LLM provider, embedding provider, and web search engine modules now log through the diagnose-gated `security.secure_logging` wrapper, so exception tracebacks in these paths only appear in logs when diagnose mode is explicitly enabled (`LDR_APP_DEBUG` + `LDR_LOGURU_DIAGNOSE`) — production logs no longer receive full tracebacks that could echo API keys or request internals from provider/engine failures. The `check-sensitive-logging` pre-commit hook now enforces this by construction: raw `loguru` imports and every known bypass route (`logger.opt()`, `@logger.catch`, private logger handles, `traceback`/`sys.exc_info()` interpolation, dynamic loguru/traceback imports, logger rebinding, and `getattr` dodges) are rejected in these directories, `bind()`/`patch()` chain messages get the same exception-scrubbing checks as direct logger calls, and exception detail reached through attribute or subscript access (`e.args`, `e.response.text`, `e.strerror`) is flagged like `str(e)` itself. ([#4976](https://github.com/LearningCircuit/local-deep-research/pull/4976))
- Fixed a plaintext-secret disclosure in `GET /settings/api/bulk`: a namespace request (e.g. `keys[]=llm`) returned nested `*.api_key` values unredacted because the redaction check looked only at the outer requested key, and `keys[]=%` could dump the whole settings table via an unescaped `LIKE`. Redaction now recurses into setting subtrees and the key lookup escapes LIKE wildcards. ([#5028](https://github.com/LearningCircuit/local-deep-research/pull/5028))
- `GET /settings/api/bulk` now redacts password-typed settings whose leaf name is a non-classic secret token (e.g. `client_secret`, `secret_key`, `bearer_token`, `api_secret`, `app_secret`). The bulk endpoint redacts by key name only, so these were previously shipped in the clear while the singular GET masked them. ([#5038](https://github.com/LearningCircuit/local-deep-research/pull/5038))
- Exception-derived strings flowing through the `/api/v1/quick_summary` and `/api/v1/analyze_documents` JSON responses are now scrubbed at the HTTP boundary (and at the strategy layer in `SourceBasedSearchStrategy`) via `sanitize_error_for_client`, closing a CWE-209 information-exposure vector flagged by CodeQL (#8019): a caught exception interpolated into a strategy's error fields could otherwise reach the API client — as `summary` (the key `quick_summary()`/`analyze_documents()` actually return), `formatted_findings`, or per-finding `content` — with credentials or stack-trace text intact. Credentials are scrubbed and length is capped; the `"Error: "` prefix is preserved so error payloads stay recognizable to clients, and the strategy-layer scrub independently covers the web-UI SSE path (`research_service.py` → `ErrorReportGenerator`), which consumes strategy output directly. `/api/v1/generate_report` gets the same boundary scrub purely as a precaution — its current payload (`content`/`metadata`) carries no error fields.

### ✨ New Features

- **Sofya search engine** — Added [Sofya](https://sofya.co) as an optional hosted web-search engine. Sofya's `/v1/search` returns ranked results with SERP snippets and extracted page content (markdown) in a single call, fitting LDR's two-phase retrieval model without a separate fetch round trip. Supports `basic`/`snippets` depth, `general`/`news` topics, recency filtering, and domain include/exclude. Configure your API key under Settings → Search → Sofya (`search.engine.web.sofya.api_key`) or the `LDR_SEARCH_ENGINE_WEB_SOFYA_API_KEY` environment variable; GitHub sign-up grants 1000 free credits/month.

### 🐛 Bug Fixes

- Fixed the news subscription-history endpoint (`/news/api/subscriptions/<id>/history`) raising an internal error for any subscription that had research runs: `get_subscription_history` called `.isoformat()` on the `created_at`/`completed_at` values, which are stored as isoformat strings, so it now returns them directly. ([#3094](https://github.com/LearningCircuit/local-deep-research/pull/3094))
- Fix collapsed-whitespace bug in SearXNG-sourced titles. The HTML parser was calling `BeautifulSoup.Tag.get_text(strip=True)` to read result titles, which strips leading/trailing whitespace *and* collapses every internal whitespace run to nothing — so multi-word titles like "Word One Word Two Word Three" rendered as "WordOneWordTwoWordThree" in the `## Sources` block. Swapped to `get_text(" ", strip=True)` (the documented BeautifulSoup idiom for "strip edges, replace internal runs with a single space") so word boundaries survive. The same fix was applied to result snippets. Title preservation flows through to the LLM relevance filter and `BaseCandidateExplorer` / `ProgressiveExplorer` candidate-phrase extraction, both of which previously saw single-token titles from SearXNG-sourced collections. ([#4970](https://github.com/LearningCircuit/local-deep-research/pull/4970))
- Registration/login no longer leave a user silently logged in when post-authentication setup fails partway: the partial session is now rolled back, so a reported failure matches reality (you are not logged in). ([#5006](https://github.com/LearningCircuit/local-deep-research/pull/5006))
- Fix citation labels rendering as empty `[]` for library/RAG sources. The `DOMAIN_HYPERLINKS`, `DOMAIN_ID_HYPERLINKS`, and `DOMAIN_ID_ALWAYS_HYPERLINKS` modes previously fed `urlparse(url).netloc` straight into the citation label, so any relative URL (most notably RAG / library hits that look like `/library/document/<uuid>`) produced an empty string. With a single relative-URL citation the label collapsed to `[[]](url)` and rendered as a `[]` link with no anchor text; with multiple relative-URL citations the suffix leaked out as `[[-1]](url)`, `[[-2]](url)`. The formatter now falls back to a slugified (lowercased, alphanumeric-only) version of the document title, and as a final fallback to the citation number itself, so the label is always non-empty and the link carries the document name. Real web URLs (arxiv.org, github.com, etc.) are unchanged — the domain still wins when `netloc` is non-empty.
- Fixed `formatBytes` rendering sizes of 1 TB or larger as `"N undefined"` (e.g. a library blob total of 1 TB showed as `1 undefined`). The shared byte formatter only defined units up to GB; it now covers TB through EB and clamps the unit index at both ends, so very large values no longer index past the end of the unit list and sub-1-byte values no longer produce a negative index.
- When the Research Logs panel hits its 500-entry DOM cap, the oldest entries are flushed first — which previously meant old warnings and errors got dropped even when the cap was blown by a flood of routine info entries. The prune now walks the cap-excess slots in priority order (info first, then milestones, then warnings, then errors), so the panel keeps its most diagnostic entries even on long research runs.

### 📝 Other Changes

- Removed the `importlib.util` disk-load workaround for `url_classifier` in `citation_formatter.py` (introduced in #4880). Now that `content_fetcher` no longer eagerly imports its heavy dependency tree (#5023), the formatter uses a normal `from ..content_fetcher.url_classifier import URLClassifier, URLType`. Also fixed two nits in the #5023 test file: an off-by-one project-root path in the subprocess tests and `check=True` calls that hid the subprocess stderr from failure reports. ([#4992](https://github.com/LearningCircuit/local-deep-research/pull/4992))
- Fixed an order-dependent test flake: the security import-fallback tests replaced `local_deep_research.security*` modules with fresh copies, recreating enums like `Sensitivity` and breaking identity checks in tests that ran later on the same worker.
- Source-tagged citation formatting no longer re-runs the URL-classifier label lookup for every citation occurrence — the pre-computed label cache is consulted lazily, so each source's label resolves once. (The disk-loading import this originally worked around was removed entirely in #4992.)
- The Sofya egress-labels test now compares enum members by `.value` instead of by identity. Under the Docker test setup the package is simultaneously available from `/app/src` (PYTHONPATH) and the installed wheel, so `SofyaSearchEngine`'s class attribute and the test's `Sensitivity` can resolve to distinct class objects with identical members. Python's `Enum.__eq__` returns `NotImplemented` across distinct classes (so `==` fails just like `is`); only `.value` (or `.name`) survives the dual-class scenario.
- The `check-pathlib-usage` pre-commit hook now catches `import os as <alias>` followed by `<alias>.path.*`. Previously the AST matcher only flagged the literal name `os`, so an aliased import silently bypassed the check — which is how a `_os.path.join(...)` slipped into #4880 and had to be caught by human review instead.
- `content_fetcher.ContentFetcher` is now resolved lazily on first attribute access (PEP 562 module `__getattr__`) instead of being eagerly imported in the package `__init__`. `url_classifier` (and any other import that goes through the `content_fetcher` package) no longer drags in the fetcher's transitive deps (`requests`, `playwright`, `bs4`, `lxml`, etc.), so minimal test setups and the citation formatter can stop using the disk-load workaround introduced in #4880. Public API is unchanged.