learningcircuit--local-deep-research/docs/release_notes/1.9.1.md

🔒 Security

• User-supplied strings that reach SQL LIKE queries — library search and domain filters, domain-classifier sampling, and news subscription filters — now escape % and _ wildcards. Previously a value like 100% or a_b was interpreted as a wildcard and matched unrelated rows; searches and filters now match literally, closing a wildcard-injection/enumeration vector within a user's own database. (#3094)
• All LLM provider, embedding provider, and web search engine modules now log through the diagnose-gated security.secure_logging wrapper, so exception tracebacks in these paths only appear in logs when diagnose mode is explicitly enabled (LDR_APP_DEBUG + LDR_LOGURU_DIAGNOSE) — production logs no longer receive full tracebacks that could echo API keys or request internals from provider/engine failures. The check-sensitive-logging pre-commit hook now enforces this by construction: raw loguru imports and every known bypass route (logger.opt(), @logger.catch, private logger handles, traceback/sys.exc_info() interpolation, dynamic loguru/traceback imports, logger rebinding, and getattr dodges) are rejected in these directories, bind()/patch() chain messages get the same exception-scrubbing checks as direct logger calls, and exception detail reached through attribute or subscript access (e.args, e.response.text, e.strerror) is flagged like str(e) itself. (#4976)
• Fixed a plaintext-secret disclosure in GET /settings/api/bulk: a namespace request (e.g. keys[]=llm) returned nested *.api_key values unredacted because the redaction check looked only at the outer requested key, and keys[]=% could dump the whole settings table via an unescaped LIKE. Redaction now recurses into setting subtrees and the key lookup escapes LIKE wildcards. (#5028)
• GET /settings/api/bulk now redacts password-typed settings whose leaf name is a non-classic secret token (e.g. client_secret, secret_key, bearer_token, api_secret, app_secret). The bulk endpoint redacts by key name only, so these were previously shipped in the clear while the singular GET masked them. (#5038)
• Exception-derived strings flowing through the /api/v1/quick_summary and /api/v1/analyze_documents JSON responses are now scrubbed at the HTTP boundary (and at the strategy layer in SourceBasedSearchStrategy) via sanitize_error_for_client, closing a CWE-209 information-exposure vector flagged by CodeQL (#8019): a caught exception interpolated into a strategy's error fields could otherwise reach the API client — as summary (the key quick_summary()/analyze_documents() actually return), formatted_findings, or per-finding content — with credentials or stack-trace text intact. Credentials are scrubbed and length is capped; the "Error: " prefix is preserved so error payloads stay recognizable to clients, and the strategy-layer scrub independently covers the web-UI SSE path (research_service.py → ErrorReportGenerator), which consumes strategy output directly. /api/v1/generate_report gets the same boundary scrub purely as a precaution — its current payload (content/metadata) carries no error fields.

✨ New Features

• Sofya search engine — Added Sofya as an optional hosted web-search engine. Sofya's /v1/search returns ranked results with SERP snippets and extracted page content (markdown) in a single call, fitting LDR's two-phase retrieval model without a separate fetch round trip. Supports basic/snippets depth, general/news topics, recency filtering, and domain include/exclude. Configure your API key under Settings → Search → Sofya (search.engine.web.sofya.api_key) or the LDR_SEARCH_ENGINE_WEB_SOFYA_API_KEY environment variable; GitHub sign-up grants 1000 free credits/month.

🐛 Bug Fixes

• Fixed the news subscription-history endpoint (/news/api/subscriptions/<id>/history) raising an internal error for any subscription that had research runs: get_subscription_history called .isoformat() on the created_at/completed_at values, which are stored as isoformat strings, so it now returns them directly. (#3094)
• Fix collapsed-whitespace bug in SearXNG-sourced titles. The HTML parser was calling BeautifulSoup.Tag.get_text(strip=True) to read result titles, which strips leading/trailing whitespace and collapses every internal whitespace run to nothing — so multi-word titles like "Word One Word Two Word Three" rendered as "WordOneWordTwoWordThree" in the ## Sources block. Swapped to get_text(" ", strip=True) (the documented BeautifulSoup idiom for "strip edges, replace internal runs with a single space") so word boundaries survive. The same fix was applied to result snippets. Title preservation flows through to the LLM relevance filter and BaseCandidateExplorer / ProgressiveExplorer candidate-phrase extraction, both of which previously saw single-token titles from SearXNG-sourced collections. (#4970)
• Registration/login no longer leave a user silently logged in when post-authentication setup fails partway: the partial session is now rolled back, so a reported failure matches reality (you are not logged in). (#5006)
• Fix citation labels rendering as empty [] for library/RAG sources. The DOMAIN_HYPERLINKS, DOMAIN_ID_HYPERLINKS, and DOMAIN_ID_ALWAYS_HYPERLINKS modes previously fed urlparse(url).netloc straight into the citation label, so any relative URL (most notably RAG / library hits that look like /library/document/<uuid>) produced an empty string. With a single relative-URL citation the label collapsed to [[]](url) and rendered as a [] link with no anchor text; with multiple relative-URL citations the suffix leaked out as [[-1]](url), [[-2]](url). The formatter now falls back to a slugified (lowercased, alphanumeric-only) version of the document title, and as a final fallback to the citation number itself, so the label is always non-empty and the link carries the document name. Real web URLs (arxiv.org, github.com, etc.) are unchanged — the domain still wins when netloc is non-empty.
• Fixed formatBytes rendering sizes of 1 TB or larger as "N undefined" (e.g. a library blob total of 1 TB showed as 1 undefined). The shared byte formatter only defined units up to GB; it now covers TB through EB and clamps the unit index at both ends, so very large values no longer index past the end of the unit list and sub-1-byte values no longer produce a negative index.
• When the Research Logs panel hits its 500-entry DOM cap, the oldest entries are flushed first — which previously meant old warnings and errors got dropped even when the cap was blown by a flood of routine info entries. The prune now walks the cap-excess slots in priority order (info first, then milestones, then warnings, then errors), so the panel keeps its most diagnostic entries even on long research runs.

📝 Other Changes

• Removed the importlib.util disk-load workaround for url_classifier in citation_formatter.py (introduced in #4880). Now that content_fetcher no longer eagerly imports its heavy dependency tree (#5023), the formatter uses a normal from ..content_fetcher.url_classifier import URLClassifier, URLType. Also fixed two nits in the #5023 test file: an off-by-one project-root path in the subprocess tests and check=True calls that hid the subprocess stderr from failure reports. (#4992)
• Fixed an order-dependent test flake: the security import-fallback tests replaced local_deep_research.security* modules with fresh copies, recreating enums like Sensitivity and breaking identity checks in tests that ran later on the same worker.
• Source-tagged citation formatting no longer re-runs the URL-classifier label lookup for every citation occurrence — the pre-computed label cache is consulted lazily, so each source's label resolves once. (The disk-loading import this originally worked around was removed entirely in #4992.)
• The Sofya egress-labels test now compares enum members by .value instead of by identity. Under the Docker test setup the package is simultaneously available from /app/src (PYTHONPATH) and the installed wheel, so SofyaSearchEngine's class attribute and the test's Sensitivity can resolve to distinct class objects with identical members. Python's Enum.__eq__ returns NotImplemented across distinct classes (so == fails just like is); only .value (or .name) survives the dual-class scenario.
• The check-pathlib-usage pre-commit hook now catches import os as <alias> followed by <alias>.path.*. Previously the AST matcher only flagged the literal name os, so an aliased import silently bypassed the check — which is how a _os.path.join(...) slipped into #4880 and had to be caught by human review instead.
• content_fetcher.ContentFetcher is now resolved lazily on first attribute access (PEP 562 module __getattr__) instead of being eagerly imported in the package __init__. url_classifier (and any other import that goes through the content_fetcher package) no longer drags in the fetcher's transitive deps (requests, playwright, bs4, lxml, etc.), so minimal test setups and the citation formatter can stop using the disk-load workaround introduced in #4880. Public API is unchanged.