chore: import upstream snapshot with attribution

docs/developing/EXTENDING.md

@@ -0,0 +1,680 @@
+# Extension Guide
+
+This guide explains how to extend Local Deep Research with custom components.
+
+## Table of Contents
+
+- [Adding Custom Search Engines](#adding-custom-search-engines)
+- [Adding Custom Search Strategies](#adding-custom-search-strategies)
+- [Using LangChain Retrievers](#using-langchain-retrievers)
+- [Adding Custom LLM Providers](#adding-custom-llm-providers)
+- [Registering Custom LLMs](#registering-custom-llms)
+
+---
+
+## Adding Custom Search Engines
+
+Search engines are responsible for fetching results from external sources. All engines extend `BaseSearchEngine`.
+
+### Basic Search Engine
+
+Create a new file in `src/local_deep_research/web_search_engines/engines/`:
+
+```python
+# search_engine_custom.py
+from typing import Any, Dict, List, Optional
+
+from langchain_core.language_models import BaseLLM
+
+from ...security.secure_logging import logger
+
+from ..search_engine_base import BaseSearchEngine
+
+
+class CustomSearchEngine(BaseSearchEngine):
+    """Custom search engine implementation."""
+
+    # Classification flags - set appropriately for your engine
+    is_public = True       # Searches public internet
+    is_generic = False     # Specialized (vs general web search)
+    is_scientific = False  # Academic/scientific content
+    is_local = False       # Local document search
+    is_news = False        # News content
+    is_code = False        # Code repositories
+    is_lexical = False                  # Uses keyword/lexical search (informational)
+    needs_llm_relevance_filter = False  # Set True to auto-enable LLM relevance filtering
+
+    def __init__(
+        self,
+        max_results: int = 10,
+        credential: Optional[str] = None,
+        llm: Optional[BaseLLM] = None,
+        max_filtered_results: Optional[int] = None,
+        **kwargs,
+    ):
+        """
+        Initialize the search engine.
+
+        Args:
+            max_results: Maximum number of results to return
+            credential: API credential for the service (if required)
+            llm: Language model for relevance filtering
+            max_filtered_results: Max results after filtering
+            **kwargs: Additional parameters
+        """
+        super().__init__(
+            llm=llm,
+            max_filtered_results=max_filtered_results,
+            max_results=max_results,
+        )
+        self.credential = credential
+
+    def _get_previews(self, query: str) -> List[Dict[str, Any]]:
+        """
+        Get preview results (first phase of two-phase retrieval).
+
+        Args:
+            query: Search query
+
+        Returns:
+            List of preview dictionaries with keys:
+            - id: Unique identifier
+            - title: Result title
+            - snippet: Brief description/summary
+            - link: URL to the content
+            - source: Source name (e.g., "CustomEngine")
+        """
+        logger.info(f"Searching custom engine for: {query}")
+
+        # Apply rate limiting before request
+        self._last_wait_time = self.rate_tracker.apply_rate_limit(self.engine_type)
+
+        # Your search implementation here
+        results = self._call_api(query)
+
+        previews = []
+        for item in results:
+            previews.append({
+                "id": item["id"],
+                "title": item["title"],
+                "snippet": item["description"],
+                "link": item["url"],
+                "source": "CustomEngine",
+            })
+
+        return previews
+
+    def _get_full_content(
+        self, relevant_items: List[Dict[str, Any]]
+    ) -> List[Dict[str, Any]]:
+        """
+        Get full content for relevant items (second phase).
+
+        Args:
+            relevant_items: Items that passed relevance filtering
+
+        Returns:
+            Items enriched with full content
+        """
+        results = []
+        for item in relevant_items:
+            # Apply rate limiting
+            self._last_wait_time = self.rate_tracker.apply_rate_limit(self.engine_type)
+
+            # Fetch full content
+            full_content = self._fetch_content(item["link"])
+
+            result = item.copy()
+            result["content"] = full_content
+            result["full_content"] = full_content
+            results.append(result)
+
+        return results
+
+    def _call_api(self, query: str) -> List[Dict]:
+        """Your API implementation."""
+        # Implement your search logic here
+        pass
+
+    def _fetch_content(self, url: str) -> str:
+        """Fetch full content from URL."""
+        # Implement content fetching
+        pass
+```
+
+### Registering the Engine
+
+**Option 1: Register in engine_registry.py (Required)**
+
+Add the engine to `src/local_deep_research/web_search_engines/engine_registry.py` so the system knows how to load it. The registry maps engine names to their Python module and class:
+
+```python
+# In engine_registry.py — ENGINE_REGISTRY dict
+"custom_engine": EngineEntry(
+    module_path=".engines.search_engine_custom",
+    class_name="CustomSearchEngine",
+),
+```
+
+Module paths must be relative (starting with `.`) and listed in the security whitelist (`ALLOWED_MODULE_PATHS` in `module_whitelist.py`).
+
+**Option 1b: Configure user-facing settings (Optional)**
+
+After registering in the engine registry, you can expose user-configurable settings via the settings database:
+
+```python
+# Key: search.engine.web.custom_engine
+config = {
+    "requires_api_key": True,
+    "requires_llm": False,
+    "description": "Custom search engine for specific use case",
+    "strengths": ["Feature 1", "Feature 2"],
+    "weaknesses": ["Limitation 1"],
+    "reliability": 0.8,
+    "default_params": {
+        "max_results": 10
+    }
+}
+```
+
+**Option 2: Modify Factory (For Core Engines)**
+
+Add to `search_engine_factory.py`:
+
+```python
+def create_search_engine(engine_name: str, ...) -> BaseSearchEngine:
+    # ... existing code ...
+
+    if engine_name.lower() == "custom_engine":
+        from .engines.search_engine_custom import CustomSearchEngine
+        return CustomSearchEngine(
+            max_results=max_results,
+            api_key=api_key,
+            llm=llm,
+            **kwargs
+        )
+```
+
+### Search Engine Best Practices
+
+1. **Always apply rate limiting** before API calls:
+   ```python
+   self._last_wait_time = self.rate_tracker.apply_rate_limit(self.engine_type)
+   ```
+
+2. **Set classification flags** accurately - they affect engine selection. For keyword-based engines without ML ranking, set `is_lexical = True` and `needs_llm_relevance_filter = True` — the factory will auto-enable LLM relevance filtering
+
+3. **Handle errors gracefully** - return empty list on failure, don't crash
+
+4. **Use logging** for debugging — engine/provider dirs must import the
+   diagnose-gated `security.secure_logging` wrapper, never raw loguru
+   (a pre-commit hook enforces this), and error messages must be scrubbed
+   before logging because they are production-visible:
+   ```python
+   from ...security.secure_logging import logger
+
+   logger.info(f"Searching for: {query}")
+
+   # In except blocks: never interpolate the raw exception — scrub it first.
+   # (Engines inherit self._scrub_error() from BaseSearchEngine.)
+   except Exception as e:
+       safe_msg = self._scrub_error(e)
+       logger.exception(f"API error: {safe_msg}")
+   ```
+   The hook is syntactic, not data-flow analysis: reviewers should still
+   reject laundering raw exception text through temporary variables such as
+   `msg = str(e); logger.error(msg)` unless the variable came from a scrubber.
+
+5. **Support snippet-only mode** by checking the config:
+   ```python
+   from ...config import search_config
+   if search_config.SEARCH_SNIPPETS_ONLY:
+       return relevant_items  # Skip full content
+   ```
+
+---
+
+## Adding Custom Search Strategies
+
+Strategies define how research is conducted - question generation, iteration, and synthesis.
+
+### Basic Strategy
+
+Create a new file in `src/local_deep_research/advanced_search_system/strategies/`:
+
+```python
+# my_custom_strategy.py
+from typing import Dict, List, Optional
+from loguru import logger
+
+from .base_strategy import BaseSearchStrategy
+
+
+class MyCustomStrategy(BaseSearchStrategy):
+    """Custom search strategy implementation."""
+
+    def __init__(
+        self,
+        search=None,
+        model=None,
+        all_links_of_system=None,
+        settings_snapshot=None,
+        max_iterations: int = 3,
+        **kwargs,
+    ):
+        """
+        Initialize the strategy.
+
+        Args:
+            search: Search engine instance
+            model: LLM for question generation and synthesis
+            all_links_of_system: Shared list for discovered links
+            settings_snapshot: Configuration snapshot
+            max_iterations: Maximum research iterations
+            **kwargs: Additional parameters
+        """
+        super().__init__(
+            all_links_of_system=all_links_of_system,
+            settings_snapshot=settings_snapshot,
+        )
+        self.search = search
+        self.model = model
+        self.max_iterations = max_iterations
+
+    def analyze_topic(self, query: str) -> Dict:
+        """
+        Execute the research strategy.
+
+        Args:
+            query: Research query
+
+        Returns:
+            Dict with:
+            - findings: List of research findings
+            - iterations: Number of iterations completed
+            - questions: Dict of questions by iteration
+            - formatted_findings: Formatted output string
+            - current_knowledge: Accumulated knowledge dict
+            - error: Optional error message
+        """
+        logger.info(f"Starting custom strategy for: {query}")
+
+        findings = []
+        current_knowledge = {}
+
+        try:
+            for iteration in range(1, self.max_iterations + 1):
+                # Update progress
+                self._update_progress(
+                    f"Iteration {iteration}/{self.max_iterations}",
+                    progress_percent=int(iteration / self.max_iterations * 100),
+                    metadata={"iteration": iteration}
+                )
+
+                # Generate questions for this iteration
+                questions = self._generate_questions(query, current_knowledge)
+                self.questions_by_iteration[iteration] = questions
+
+                # Search for each question
+                for question in questions:
+                    results = self._search(question)
+                    findings.extend(results)
+
+                    # Track links
+                    for result in results:
+                        if result.get("link"):
+                            self.all_links_of_system.append(result["link"])
+
+                # Synthesize findings
+                current_knowledge = self._synthesize(findings)
+
+                # Check if we should stop early
+                if self._should_stop(current_knowledge):
+                    logger.info(f"Early stopping at iteration {iteration}")
+                    break
+
+            # Format final output
+            formatted = self._format_findings(findings, current_knowledge)
+
+            return {
+                "findings": findings,
+                "iterations": iteration,
+                "questions": self.questions_by_iteration,
+                "formatted_findings": formatted,
+                "current_knowledge": current_knowledge,
+            }
+
+        except Exception as e:
+            logger.error(f"Strategy error: {e}")
+            return {
+                "findings": findings,
+                "iterations": 0,
+                "questions": self.questions_by_iteration,
+                "formatted_findings": "",
+                "current_knowledge": current_knowledge,
+                "error": str(e),
+            }
+
+    def _generate_questions(self, query: str, knowledge: Dict) -> List[str]:
+        """Generate research questions using the LLM."""
+        prompt = f"""Given the query: {query}
+        And current knowledge: {knowledge}
+        Generate 3 specific research questions."""
+
+        response = self.model.invoke(prompt)
+        # Parse response into questions
+        return self._parse_questions(response.content)
+
+    def _search(self, question: str) -> List[Dict]:
+        """Execute search for a question."""
+        return self.search.run(question)
+
+    def _synthesize(self, findings: List[Dict]) -> Dict:
+        """Synthesize findings into knowledge."""
+        # Implement synthesis logic
+        return {"summary": "...", "key_points": [...]}
+
+    def _should_stop(self, knowledge: Dict) -> bool:
+        """Check if research should stop early."""
+        # Implement stopping criteria
+        return False
+
+    def _format_findings(self, findings: List[Dict], knowledge: Dict) -> str:
+        """Format findings as output string."""
+        # Implement formatting
+        return "Formatted research results..."
+
+    def _parse_questions(self, content: str) -> List[str]:
+        """Parse LLM response into question list."""
+        # Implement parsing
+        return content.strip().split("\n")
+```
+
+### Registering the Strategy
+
+Add to `search_system_factory.py`:
+
+```python
+def create_strategy(strategy_name: str, ...) -> BaseSearchStrategy:
+    strategy_name_lower = strategy_name.lower()
+
+    # ... existing strategies ...
+
+    elif strategy_name_lower in ["my-custom", "mycustom", "custom"]:
+        from .advanced_search_system.strategies.my_custom_strategy import (
+            MyCustomStrategy,
+        )
+        return MyCustomStrategy(
+            search=search,
+            model=model,
+            all_links_of_system=all_links_of_system,
+            settings_snapshot=settings_snapshot,
+            **kwargs
+        )
+```
+
+### Strategy Best Practices
+
+1. **Use progress callbacks** to update the UI:
+   ```python
+   self._update_progress("Searching...", progress_percent=50)
+   ```
+
+2. **Track all discovered links** in `self.all_links_of_system`
+
+3. **Store questions by iteration** in `self.questions_by_iteration`
+
+4. **Access settings** via the snapshot:
+   ```python
+   max_results = self.get_setting("search.max_results", default=10)
+   ```
+
+5. **Handle errors gracefully** - return partial results with error message
+
+---
+
+## Using LangChain Retrievers
+
+The easiest way to add custom search is through LangChain retrievers.
+
+### Registering a Retriever
+
+```python
+from langchain_community.vectorstores import FAISS
+from langchain_openai import OpenAIEmbeddings
+from local_deep_research.web_search_engines.retriever_registry import retriever_registry
+
+# Create your retriever
+embeddings = OpenAIEmbeddings()
+vectorstore = FAISS.from_documents(documents, embeddings)
+retriever = vectorstore.as_retriever(search_kwargs={"k": 10})
+
+# Register globally
+retriever_registry.register("my_documents", retriever)
+
+# Now use in research
+from local_deep_research.api import quick_summary
+
+result = quick_summary(
+    query="What does the documentation say about X?",
+    search_tool="my_documents",  # Use registered retriever
+    programmatic_mode=True
+)
+```
+
+### Passing Retrievers Directly
+
+```python
+from local_deep_research.api import quick_summary
+
+# Create retriever
+retriever = my_vectorstore.as_retriever()
+
+# Pass directly to API
+result = quick_summary(
+    query="Search my documents",
+    retrievers={"private_docs": retriever},
+    search_tool="private_docs",
+    programmatic_mode=True
+)
+```
+
+### Registry Methods
+
+```python
+from local_deep_research.web_search_engines.retriever_registry import retriever_registry
+
+# Register
+retriever_registry.register("name", retriever)
+retriever_registry.register_multiple({"a": ret1, "b": ret2})
+
+# Query
+retriever_registry.get("name")
+retriever_registry.is_registered("name")
+retriever_registry.list_registered()
+
+# Remove
+retriever_registry.unregister("name")
+retriever_registry.clear()
+```
+
+---
+
+## Adding Custom LLM Providers
+
+LLM providers wrap language model APIs for use in LDR.
+
+### Basic Provider
+
+Create in `src/local_deep_research/llm/providers/implementations/`:
+
+```python
+# my_provider.py
+from typing import Dict, Optional
+
+from langchain_core.language_models import BaseChatModel
+from langchain_openai import ChatOpenAI
+
+from ..openai_base import OpenAICompatibleProvider
+
+
+class MyProvider(OpenAICompatibleProvider):
+    """Custom LLM provider."""
+
+    provider_name = "My Provider"
+    api_key_setting = "llm.my_provider.api_key"
+    url_setting = "llm.my_provider.url"
+    default_base_url = "https://api.myprovider.com/v1"
+    default_model = "my-model-v1"
+    # Optional: set to True if missing key should fall back to a placeholder
+    # rather than raising ValueError.
+    api_key_optional = False
+
+    @classmethod
+    def create_llm(
+        cls,
+        model_name: Optional[str] = None,
+        temperature: float = 0.7,
+        settings_snapshot: Optional[Dict] = None,
+        **kwargs
+    ) -> BaseChatModel:
+        """
+        Create LLM instance.
+
+        Args:
+            model_name: Model to use
+            temperature: Sampling temperature
+            settings_snapshot: Configuration
+            **kwargs: Additional parameters
+
+        Returns:
+            LangChain chat model instance
+        """
+        from ....config.thread_settings import get_setting_from_snapshot
+
+        # Resolve API key via the base helper. Raises ValueError when
+        # required and missing, returns the unified placeholder when
+        # api_key_optional=True and the key is unset.
+        api_key = cls.resolve_api_key_or_placeholder(settings_snapshot)
+
+        # Get base URL
+        base_url = get_setting_from_snapshot(
+            cls.url_setting,
+            cls.default_base_url,
+            settings_snapshot=settings_snapshot,
+        )
+
+        return ChatOpenAI(
+            model=model_name or cls.default_model,
+            temperature=temperature,
+            api_key=api_key,
+            base_url=base_url,
+            **kwargs
+        )
+
+    @classmethod
+    def list_models(cls, settings_snapshot: Optional[Dict] = None) -> list[str]:
+        """List available models."""
+        return ["my-model-v1", "my-model-v2", "my-model-large"]
+```
+
+### Register in Auto-Discovery
+
+Drop the provider class file into
+`src/local_deep_research/llm/providers/implementations/`. Auto-discovery
+will scan that directory at import time and register every class whose
+name ends with `Provider`, subclasses `BaseLLMProvider`, and has
+`provider_name` set to a real value (i.e., overridden away from the
+``"unknown"`` default). Setting `provider_name = "unknown"` — or leaving
+it unset on the class — will cause the class to be **silently filtered
+out** of auto-discovery, which is a common gotcha when copying an
+existing provider as a template.
+
+Optional cloud-metadata registration in `auto_discovery.py`:
+
+```python
+PROVIDER_METADATA = {
+    # ... existing providers ...
+    "my_provider": ProviderMetadata(
+        provider_id="my_provider",
+        provider_name="My Provider",
+        company_name="My Company",
+        region="US",
+        country="United States",
+        data_location="US",
+        gdpr_compliant=False,
+        is_cloud=True,
+    ),
+}
+```
+
+---
+
+## Registering Custom LLMs
+
+For programmatic use, register LLMs directly:
+
+```python
+from langchain_openai import ChatOpenAI
+from local_deep_research.llm.llm_registry import register_llm, get_llm_from_registry
+
+# Create custom LLM
+custom_llm = ChatOpenAI(
+    model="gpt-4",
+    temperature=0.5,
+    api_key="...",
+)
+
+# Register it
+register_llm("my_gpt4", custom_llm)
+
+# Use in research
+from local_deep_research.api import quick_summary
+
+result = quick_summary(
+    query="Research topic",
+    llms={"my_gpt4": custom_llm},  # Or use registered name
+    provider_name="my_gpt4",
+    programmatic_mode=True
+)
+```
+
+### Factory Functions
+
+You can also register factory functions:
+
+```python
+def create_my_llm(temperature=0.7):
+    return ChatOpenAI(model="gpt-4", temperature=temperature)
+
+register_llm("my_factory", create_my_llm)
+
+# Will be called when needed
+llm = get_llm_from_registry("my_factory")
+```
+
+### Registry caveat
+
+The built-in providers (ollama, openai, anthropic, ...) live in the same
+registry, auto-registered at import time. `clear_llm_registry()` removes
+them too, and `get_llm()` has no other construction path — every provider
+will raise "was not registered by auto-discovery" until you restore them:
+
+```python
+from local_deep_research.llm.providers import discover_providers
+
+discover_providers(force_refresh=True)
+```
+
+Prefer `unregister_llm("<your name>")` over `clear_llm_registry()` to
+remove only your own registrations.
+
+---
+
+## See Also
+
+- [Architecture Overview](../architecture/OVERVIEW.md) - System architecture
+- [Database Schema](../architecture/DATABASE_SCHEMA.md) - Data models
+- [Full Configuration Reference](../CONFIGURATION.md) - All settings and environment variables
+- [Troubleshooting](../troubleshooting.md) - Common issues
+- [API Quickstart](../api-quickstart.md) - Using the API

docs/developing/credential-scrubbing.md

@@ -0,0 +1,61 @@
+# Credential scrubbing in error/log text
+
+`local_deep_research.security.log_sanitizer` scrubs credentials out of
+exception/error strings **before** they reach a client (HTTP/SSE/JSON
+responses) or the logs. It is a runtime, defense-in-depth *sanitizer* — not a
+git/CI secret *scanner* (use [gitleaks](https://github.com/gitleaks/gitleaks)
+for that, which this repo already runs in pre-commit/CI).
+
+## The two layers
+
+1. **`sanitize_error_message(text)`** — a regex first-pass over `_CREDENTIAL_PATTERNS`
+   for *common credential shapes* (API-key query params, `Authorization:`/
+   `x-api-key:` headers, `user:pass@host`, and well-known token prefixes:
+   `sk-`/`pk-`, Google `AIza`/`ya29.`, GitHub `ghp_`/`github_pat_`, AWS
+   `AKIA`/`ASIA`, Slack `xox*-`, JWTs). `sanitize_error_for_client()` composes
+   it with control-char stripping + length capping.
+2. **`redact_secrets(text, *known_literals)`** — the **backstop**. When you
+   hold the actual secret value (e.g. the configured API key), pass it here so
+   it is scrubbed regardless of shape. This is the real guarantee for
+   *known* secrets; the regex layer is best-effort for *unknown* ones.
+
+Always pair them ("dual-scrub") on any path that surfaces secret-adjacent
+text — via **`scrub_error(error, *known_literals)`** (`security.log_sanitizer`),
+which composes both passes plus the defensive guards (a raising `__str__`
+or a non-string secret cannot crash the `except` handler). Engine
+subclasses use `self._scrub_error(error)`, which resolves the engine's
+`_secret_attrs` and delegates to the same helper. Don't hand-inline the
+two calls: per-site copies are how scrub passes drift apart.
+
+## Design constraints (why it is curated, not exhaustive)
+
+This runs on **arbitrary, possibly attacker-influenced** strings at runtime, so:
+
+- **No ReDoS.** Every pattern must scale linearly — prefer prefix-anchored,
+  single-quantifier regexes; avoid nested quantifiers / ambiguous alternations.
+  Spot-check new patterns on a 200k-char adversarial input.
+- **Over-redaction is the safe failure; under-redaction (a leak) is not** —
+  but gratuitous over-redaction harms log readability, so patterns are
+  anchored/length-floored to avoid eating prose.
+- Keep the set **small and high-signal**. Matching gitleaks' full 100+ rules
+  at runtime is overkill and raises false positives; the `redact_secrets`
+  backstop covers the long tail for known secrets.
+
+## Keeping the patterns current
+
+The prefix regexes mirror the **canonical, actively-maintained gitleaks
+ruleset** — gitleaks' own upstream `config/gitleaks.toml`
+(<https://github.com/gitleaks/gitleaks>), **not** this repo's root
+`.gitleaks.toml` (which only configures the gitleaks *scan* run in
+pre-commit/CI). To refresh when a provider introduces a new token format:
+
+1. Find the rule in gitleaks' upstream `config/gitleaks.toml` (or detect-secrets).
+2. Adapt the regex to a prefix-anchored, ReDoS-safe form here; redact to
+   `[REDACTED_KEY]`.
+3. Add a positive (redacts) **and** a negative (prose not over-redacted) case
+   to `tests/security/test_log_sanitizer.py`.
+4. Run `pytest tests/security/` and a 200k-char ReDoS spot-check.
+
+A periodic (e.g. quarterly) glance at gitleaks' changelog for new
+widely-used token prefixes is enough — this layer only needs the formats that
+plausibly appear in *this app's* error/log text.

docs/developing/resource-cleanup.md

@@ -0,0 +1,971 @@
+# Resource cleanup in LDR
+
+This document captures how LDR manages process-level resources (DB
+connections, HTTP clients, file descriptors, threads) and the reasoning
+trail behind the current model. It exists because file-descriptor
+exhaustion has been a recurring class of bug in LDR, and the *journey*
+of fixing it — what's been tried, what worked, what was ruled out — is
+not reconstructable from `git log` alone.
+
+If you're contributing code that holds a network connection, a database
+session, an LLM client, or a thread, read this before adding `__del__`,
+`weakref.finalize`, or a context manager.
+
+---
+
+## Current model
+
+### Database connections
+
+- **One shared per-user `QueuePool`.** No per-thread engines. Pool
+  sizing: `pool_size=20`, `max_overflow=40`, with periodic `dispose()`
+  every 30 minutes.
+- **SQLCipher is decrypted once per connection-open.** `PRAGMA key`
+  takes ~0.2 ms; pool reuse keeps that off the hot path.
+- Engines are created at login, closed at logout (or process exit via
+  the registered `atexit` shutdown).
+- Background threads (research workers, metric writers, news scheduler
+  jobs) use the same per-user pool — they no longer maintain a separate
+  thread-engine system.
+
+See [ADR-0004](../decisions/0004-nullpool-for-sqlcipher.md) for the
+QueuePool-vs-NullPool decision and PR #3441 for the per-thread-engine
+removal.
+
+### LLM wrappers
+
+LDR wraps every LLM in `ProcessingLLMWrapper` (and optionally
+`RateLimitedLLMWrapper`) so that callers see a uniform interface and
+the project owns the close path:
+
+```
+caller -> ProcessingLLMWrapper.close()
+       -> _close_base_llm(base_llm) in utilities/llm_utils.py
+       -> for ChatOllama:
+            sync httpx client (ollama.Client._client) .close()
+            async httpx client (ollama.AsyncClient._client) .aclose()
+       -> for ChatOpenAI / ChatAnthropic:
+            no close (those use @lru_cache'd shared httpx clients)
+```
+
+Key invariants:
+
+- `ChatOllama` is the *only* provider where `_close_base_llm()` actually
+  closes anything. ChatOpenAI and ChatAnthropic share LRU-cached httpx
+  clients across instances; closing them would break other live LLMs.
+- Both `_client` (sync) **and** `_async_client` (async) are released —
+  the async side is exercised by every `ainvoke()` call (langgraph
+  agents, modular strategies). Closing only the sync side leaks the
+  async transport per call (root cause of #3816).
+- The function is idempotent via an `_ldr_closed` sentinel on the inner
+  httpx clients.
+- The async close uses `asyncio.run(client.aclose())` only when no
+  event loop is currently running. When called from inside async code
+  it skips and leaves the close to the loop's owner.
+
+### Search engines
+
+- `BaseSearchEngine.close()` is the single entry point and **cascades**
+  into `_preview_filters` and `_content_filters`. That cascade is what
+  releases per-engine LLMs (e.g., `JournalReputationFilter.model`),
+  SearXNG sessions, and other filter-held resources.
+- Search-engine cleanup happens at the per-research finally block in
+  `web/services/research_service.py:run_research_process()` and at the
+  programmatic API entry points in `api/research_functions.py`.
+- The `_owns_llm` flag pattern (introduced in #2712) tracks whether a
+  filter or engine constructed its own LLM (and thus owns it) versus
+  borrowed one from a caller (and must not close it).
+
+### Thread lifecycle
+
+- `@thread_cleanup` (decorator on `run_research_process` and similar
+  workers) ensures thread-local DB sessions are released even on
+  abnormal exits.
+- `cleanup_current_thread()` is called from Flask teardown, the queue
+  processor, the auth flow, and the RAG routes — six tier-1 paths in
+  total.
+- Background threads are daemon threads; the process exit handles any
+  thread that did not clean up gracefully.
+
+### Conventions
+
+- **Use `safe_close(resource, "human name")`** from
+  `utilities/resource_utils.py` for every cleanup. Never bare `.close()`
+  in a `finally` (it can mask the original exception).
+- **Prefer `try/finally` over `__del__`**. Python doesn't guarantee
+  finalization order at interpreter exit; `__del__` interacts subtly
+  with reference cycles and `weakref`.
+- **Track ownership explicitly with `_owns_llm` (or analogous flag)**
+  when a class accepts an injected resource that may or may not be its
+  own.
+- **News fragments (`changelog.d/<id>.bugfix.md`) are required for any
+  user-visible cleanup behavior change** — see `changelog.d/README.md`.
+
+---
+
+## How to close X correctly
+
+| You're holding | Do this |
+| --- | --- |
+| A `ChatOllama` (raw or wrapped) | Call `wrapper.close()` in a `finally`, or pass to `safe_close(wrapper, "...")`. The wrapper chain handles both sync and async httpx clients. |
+| A search engine you constructed | `safe_close(engine, "...")` in `finally`. The engine's `close()` cascades into preview/content filters. |
+| A holder class with an LLM | Add a `close()` method, gate the LLM close on `self._owns_llm`, document who calls it. Don't add `__del__`. |
+| A long-lived service holder (news scheduler, etc.) | Wrap construction in `try/finally` at the cycle boundary. Don't store the LLM if you can recreate it cheaply. |
+| A DB session | Use `with get_user_db_session(username) as session:`. Don't bypass via `get_settings_manager(username=...)` without `owns_session=False` (see #3023). |
+| An asyncio event loop | Use the existing one. If you genuinely need a new one (background thread fallback), call `loop.close()` in a `finally` — see `news_strategy.py` for the reference pattern (post-#3018). |
+
+---
+
+## Anti-patterns
+
+These look reasonable but break specific things in this codebase:
+
+- **Adding `__del__` to a class with `close()`.** At interpreter exit
+  the `logger`, `httpx`, and event-loop modules may already be torn
+  down. `__del__` can run after them and raise. Use explicit close in
+  a `finally` instead.
+- **Closing a shared httpx client.** ChatOpenAI / ChatAnthropic share
+  one httpx pool across instances via `@lru_cache`. Closing it kills
+  every other live LLM in the same process. The Ollama check in
+  `_close_base_llm` exists exactly to gate this.
+- **Truthy idempotency sentinels on Mock objects.** `Mock()` without a
+  `spec` auto-generates child Mocks for any attribute access, so
+  `getattr(client, "_ldr_closed", False)` returns a truthy Mock and
+  short-circuits the close. Always use `is True` / `is None` checks
+  for sentinels — see the pattern in `_close_base_llm`.
+- **Skipping `super().close()` in a search-engine subclass.**
+  `BaseSearchEngine.close()` is what cascades into preview/content
+  filters. Override it without calling super and you leak every
+  filter's resources (this was a Copilot finding on #3818).
+- **Treating `asyncio.run()` as safe inside an event loop.** It raises
+  `RuntimeError` if called from a thread that already has a running
+  loop. The pattern in `_close_base_llm` is: detect a running loop
+  with `get_running_loop()`, skip the async close in that branch (the
+  loop owner will close), only call `asyncio.run` in the no-loop case.
+
+---
+
+## History
+
+The FD-leak campaign spans roughly four months of iterative work. Each
+fix narrowed the remaining surface; each subsequent issue was found in
+a corner the previous wave hadn't touched.
+
+### Wave 1 — initial leak inventory (Jan 2026)
+
+- **#1832, #1849, #1856, #1860** — first comprehensive sweep. Identified
+  seven distinct leak sources: `auth_db` engine, `download_management`
+  DB, search cache, subprocess zombies, HTTP sessions in
+  `SemanticScholarSearchEngine` and `BaseDownloader`, Socket.IO threads.
+  Established context-manager + `try/finally` patterns. Added a
+  pre-commit hook to catch missing cleanup at commit time.
+
+### Wave 2 — thread-local engine accumulation (Mar 2026)
+
+- **#2495** — diagnosed that Flask's teardown only cleaned the
+  request-scoped `g.db_session` while a separate `_thread_engines` dict
+  accumulated NullPool engines per thread, leaking ~3 FDs per request.
+  Added `cleanup_current_thread()` across six tier-1 paths.
+- **#2591** — dead-thread engines (when threads crashed they left
+  engines behind) plus `stream=True` socket holds in the generic
+  downloader. Added a throttled dead-thread sweep, removed `stream=True`,
+  raised the Docker ulimit from 1024 to 65536.
+
+### Wave 3 — LLM wrapper lifecycle (Mar 2026)
+
+- **#2708** — diagnosed `ChatOllama` → `httpx.Client` chains with no
+  `__del__`. With the news scheduler triggering 50–300
+  `quick_summary()` calls per hour, a 1024-FD container exhausted in
+  3–4 hours. Wrapped four programmatic API entry points in
+  `try/finally` with explicit close.
+- **#2712** — extracted `close_llm()` to a shared utility. Added
+  `close()` and `_owns_llm` to `NewsAnalyzer`, `HeadlineGenerator`,
+  `TopicGenerator`, `JournalReputationFilter`, `DomainClassifier`,
+  `GitHubSearchEngine`, `IntegratedReportGenerator`,
+  `ElasticsearchSearchEngine`, and the benchmark graders.
+- **#2756** — wrapped bare `.close()` calls in `finally` blocks with
+  `safe_close()` to prevent masking the original exception.
+- **#2732** — moved `close()` into `ProcessingLLMWrapper` and
+  `RateLimitedLLMWrapper` directly; eliminated the standalone
+  `close_llm()` free function.
+
+### Wave 4 — DB session leaks + per-call patterns (late Mar / early Apr 2026)
+
+- **#3018** — `get_settings_manager(username=...)` was bypassing
+  `g.db_session` and creating QueuePool sessions per-thread; live
+  diagnostics showed 321 sockets allocated, only 66 in use.
+  `DownloadService.close()` leaked the inner `SettingsManager` session.
+  Also fixed `TopicBasedRecommender._create_recommendation_card()`
+  (per-call LLM with no cleanup) and an `asyncio.new_event_loop()` in
+  `news_strategy.py` that never closed.
+- **#3204** — test fixtures using `return` instead of `yield` left
+  engines un-disposed. Migrated 8 test files to `yield` +
+  `engine.dispose()`.
+
+### Wave 5 — DB pool architecture (Apr 2026)
+
+- **#3340** — kept QueuePool but minimized FDs (`pool_size=1`,
+  `max_overflow=2`, periodic `dispose()` every 30 min).
+- **#3337** (closed) — proposed switching SQLCipher engines to
+  NullPool for zero persistent FDs. Superseded by #3441.
+- **#3441** — removed per-thread NullPool engines entirely
+  (~2,100 lines of sweep logic deleted) and routed metrics through a
+  single shared per-user QueuePool with bounded sizing
+  (`pool_size=20`, `max_overflow=40`).
+- **#3477** — created [ADR-0004](../decisions/0004-nullpool-for-sqlcipher.md)
+  capturing the final pool model and updated stale FD calculations
+  across docs.
+
+### Wave 6 — async client close (May 2026)
+
+- **#3818** (open, declined for merge) — proposed session-pooling
+  around `safe_get`/`safe_post` to address #3816. The session refactor
+  is reasonable in isolation, but the lsof in #3816 showed ~72% of
+  leaked FDs as `a_inode [eventpoll]` selectors, not HTTP request
+  sockets — pointing at async-client transports rather than `safe_get`
+  callers (whose response bodies were already consumed). See
+  [the PR comment](https://github.com/LearningCircuit/local-deep-research/pull/3818#issuecomment-4402290677)
+  for the full reasoning.
+- **#3855** — extended `_close_base_llm()` to also close
+  `ChatOllama._async_client` (the actual gap the lsof pointed to).
+  Added the `IntegratedReportGenerator` close that was missing from the
+  per-research `finally` block. Idempotency via `_ldr_closed` sentinels
+  on the inner httpx clients.
+
+### Wave 7 — async close inside a running loop (May 2026)
+
+- **#4047** — `_close_base_llm`'s async branch had a documented "skip if
+  a loop is running; loop owner closes" path. **No loop-owner cleanup
+  code existed anywhere in the project**, so when the close was called
+  inside an active asyncio loop the inner `httpx.AsyncClient` (and its
+  `epoll_create` FD) was silently abandoned. Reproduced in production:
+  a v1.6.10 single-host Ollama container reached 1024 FDs with the
+  /proc histogram showing **929 `anon_inode:[eventpoll]` (91%)** — the
+  same FD class as #3816 but in a code path #3855's fix didn't cover.
+  The fix runs the async close in a brief daemon thread that owns its
+  own loop, so `asyncio.run(aclose())` works regardless of the caller's
+  loop state. A bounded 5-second `join` keeps the cleanup from blocking
+  shutdown when the Ollama server is unresponsive; on timeout
+  `_ldr_closed` is left unset so a later call retries, and a WARNING
+  surfaces so the situation is observable instead of silent.
+- **Healthcheck pidfd leak (same PR).** Dockerfile's
+  `HEALTHCHECK CMD python -c "... urllib.request.urlopen(...)"` had no
+  `timeout=` argument; Docker's 10s timeout SIGKILL'd the `sh -c`
+  parent but the python child was reparented to PID 1 and hung
+  forever, each surviving child holding a `pidfd` + TCP socket against
+  the app. Same /proc dump showed **64 `anon_inode:[pidfd]` (6%)** from
+  this. Adding `timeout=8` lets the child return/raise inside Docker's
+  budget so it exits cleanly and gets reaped.
+#### Audit ledger — what the broader sweep checked
+
+The PR included a wide audit (50+ parallel exploration agents across
+seven rounds plus direct `/proc` inspection) to catch any other latent
+FD leak. To save the next contributor from re-running the same checks,
+here is the full ledger:
+
+##### Checked and confirmed clean (no action needed)
+
+- **Non-Ollama LLM providers.** xAI, Google Gemini, OpenRouter, IONOS,
+  LM Studio, llama.cpp HTTP, DeepSeek, OpenAI-compatible endpoint, plus
+  OpenAI and Anthropic themselves. All extend `ChatOpenAI` or
+  `ChatAnthropic`, which use `@lru_cache`'d shared httpx clients.
+  `_close_base_llm`'s short-circuit on these classes is correct by
+  design — closing them would brick every other live LLM in the
+  process.
+- **HTTP session lifecycle.** Six instantiation sites checked
+  (`PricingFetcher` aiohttp, `LDRClient` SafeSession, `BaseDownloader`,
+  `SemanticScholarSearchEngine`, `MCPClient`, `CostCalculator`). All
+  context-managed via `with` or owned by a class with a paired
+  `close()` and `__exit__`.
+- **subprocess / pidfd.** Three call sites, all `subprocess.run()`
+  (blocking). No `subprocess.Popen` paths anywhere in `src/`. No
+  `ProcessPoolExecutor`. No FD leak surface beyond the healthcheck
+  child, already addressed by the Dockerfile `timeout=8` change.
+- **asyncio event loops.** Zero raw `asyncio.new_event_loop()`
+  outside safe `asyncio.run()` patterns. The historical leak in
+  `news_strategy.py` (#3018) is still fixed.
+- **File handles.** All 37 `open()` call sites are inside `with`.
+  Zero bare opens. `tempfile.NamedTemporaryFile` / `TemporaryDirectory`
+  all context-managed.
+- **SocketIO connect/disconnect.** Non-disconnect handlers
+  (`subscribe`, `unsubscribe`, `connect`) do not acquire DB sessions
+  (an early-round agent claim that they did was refuted on re-read).
+  The `__socket_subscriptions` dict is cleaned on disconnect. The
+  PID-1 FD breakdown showed only 3 sockets out of 1024 — socket
+  accumulation is not a contributor.
+
+##### Flagged by audit, then verified NOT a real FD leak
+
+- **OllamaEmbeddings httpx (historical — current state covered in
+  Wave 10 below).** At the time of this Wave-7 audit LDR imported the
+  **deprecated** `langchain_community.embeddings.OllamaEmbeddings`,
+  which used `requests.post()` per call — no persistent httpx client,
+  no `_client` / `_async_client` attribute. Direct introspection:
+  `[a for a in dir(e) if 'client' in a.lower()]` returned `[]`. Zero
+  FDs per call. An audit agent confused this class with `ChatOllama`,
+  which is a different class. The migration to
+  `langchain_ollama.OllamaEmbeddings` predicted in the next subsection
+  has since shipped (#4352/#4353) and the resulting FD-leak regression
+  has been fixed — see Wave 10.
+- **`auth_db` and `journal_quality` engines escaping
+  `shutdown_databases()`.** `auth_db` uses
+  `QueuePool(pool_size=10, max_overflow=20)` and `journal_quality`
+  uses `StaticPool` with `immutable=1`. Both are **bounded** and do
+  not grow at runtime. Live `/proc` on the affected container showed
+  only 21 SQLite-related FDs total on PID 1 — well below the ~91-FD
+  ceiling these unmanaged engines could theoretically reach. The
+  kernel reclaims FDs at process exit regardless of `engine.dispose()`,
+  and SQLite WAL files auto-checkpoint on next open. Missing dispose
+  at exit is hygiene, not a leak.
+- **`LibraryRAGService` in three RAG SSE endpoints.**
+  `rag_routes.py:693, 1054, 1827` do construct the service outside
+  the generator and never close it, **but** `LibraryRAGService.close()`
+  only sets references to `None` — it releases no FDs. FAISS uses
+  `pickle.load()` (not mmap); OllamaEmbeddings holds no FDs per the
+  item above; the SentenceTransformer model+tokenizer mmaps are
+  process-wide singletons. What gets delayed is ~50–200 MB of
+  embedding-model RAM until GC. A memory-pressure question, not the
+  eventpoll FD class this Wave addressed.
+- **Residual `pidfd` accumulation via Playwright fallback** —
+  identified in a Round-8 follow-up after the eventpoll fix landed.
+  Live `/proc` on the prerelease container showed ~29 pidfds steady
+  state, growing ~3.6/hour, all targeting `Pid: -1` (children that
+  had exited). Rate was stable during active benchmark execution,
+  ruling out a per-task source. Eight parallel agents converged on
+  the same chain: `_check_subscription` → `quick_summary` →
+  `FullSearchResults.batch_fetch_and_extract` → `AutoHTMLDownloader`
+  fallback to `PlaywrightHTMLDownloader._fetch_with_playwright`. Each
+  `sync_playwright().start()` invokes
+  `asyncio.create_subprocess_exec()` for the Node.js driver (opens a
+  pidfd via Linux's `PidfdChildWatcher`); the driver then fails
+  because Chromium is not installed in the production `ldr` Dockerfile
+  stage (only `ldr-test` runs `playwright install --with-deps
+  chromium`), and the asyncio child watcher does not promptly close
+  the pidfd on the failed-child exit. CPython 3.14 was confirmed to
+  not use pidfd in `subprocess.py` at all (`subprocess.run`/`Popen`
+  use `waitpid(WNOHANG)` polling), so subprocess-based hypotheses
+  were ruled out. **Fixed by PR #3971** (default
+  `web.enable_javascript_rendering=false`): the fallback short-circuits
+  before any subprocess is spawned, so no pidfd is opened. The PR was
+  motivated by issue #3826 (confusing tracebacks); the FD-leak
+  finding is the second motivation, surfaced here.
+
+##### Minor findings (not steady-state leaks; worth knowing)
+
+- **Daemon threads without explicit shutdown.**
+  `journal_reputation_filter.py` background fetcher, `log_utils.py`
+  queue processor. All daemonized — reaped by the OS at process exit.
+  Not steady-state leaks; no per-request growth.
+- **Abandoned-research thread on socket disconnect.** If a client
+  closes the tab mid-research, the socket subscription is removed but
+  the research thread keeps running until completion;
+  `_active_research[research_id]` is not cleared on disconnect. Not an
+  FD leak; potentially compute/memory waste if the user wanted the
+  research to stop. Out of scope for the FD-leak story.
+
+#### Future-proofing note — `langchain_ollama.OllamaEmbeddings` migration (resolved in Wave 10)
+
+Status: **resolved**. The migration this note predicted shipped in
+#4352/#4353; the FD-leak regression it predicted then surfaced and was
+fixed in Wave 10 (see below). Kept here as the source of the prediction
+that the next contributor's audit can cross-reference.
+
+`langchain_community.embeddings.OllamaEmbeddings` was deprecated ("will
+be removed in langchain 1.0.0", per the import warning). Its replacement,
+`langchain_ollama.OllamaEmbeddings`, **does** carry `_client` and
+`_async_client` attributes — same shape as `ChatOllama`. Verified by
+direct introspection at the time of writing:
+
+```
+langchain_ollama.OllamaEmbeddings client attrs:
+  ['_set_clients', 'async_client_kwargs', 'client_kwargs',
+   'sync_client_kwargs']
+Has _client?       True
+Has _async_client? True
+```
+
+The prediction was: once LDR migrates, the eventpoll FD leak class
+returns for embeddings unless `_close_base_llm` is called on embedding
+instances. The introspection turned out to be slightly different from
+expected — both clients are constructed *eagerly* by a Pydantic
+`@model_validator(mode="after")` in `langchain_ollama.embeddings.py`,
+so the leak fires per-instance regardless of whether the async path is
+exercised. Wave 10 contains the post-mortem and fix.
+
+### Wave 10 — embeddings FD leak after langchain_ollama migration (June 2026)
+
+The migration predicted above shipped without the matching close-path
+generalization, exactly as feared. Verified by four independent agents:
+`langchain_ollama.OllamaEmbeddings(...)` eagerly constructs both a sync
+`ollama.Client` (→ `httpx.Client`) and an async `ollama.AsyncClient`
+(→ `httpx.AsyncClient` → one `epoll_create` FD) inside its
+`@model_validator(mode="after")` at
+`.venv/.../langchain_ollama/embeddings.py:295-315`. No `close()`,
+`aclose()`, `__del__`, or `weakref.finalize` exists on the new class or
+the underlying `ollama` / `httpx` clients, so dropping the Python
+reference does not release the FDs.
+
+`_close_base_llm` already handled the shape — its module-prefix checks
+(`type(...).__module__.startswith("ollama")` at
+`src/local_deep_research/utilities/llm_utils.py:97,114`) match
+`ollama.Client` / `ollama.AsyncClient` regardless of which langchain
+wrapper holds them. The function just wasn't called on embeddings
+instances — `LocalEmbeddingManager.close()` and `LibraryRAGService.close()`
+only nulled their `_embeddings` / `embedding_manager` references,
+relying on GC that would never run the close.
+
+Fix: route the close call through the existing manager lifecycle.
+`LocalEmbeddingManager.close()` now calls `_close_base_llm(self._embeddings)`
+before nulling. `LibraryRAGService.close()` now calls
+`self.embedding_manager.close()` before nulling — guarded by an
+`_owns_embedding_manager` flag so a caller-supplied manager (test
+fixtures, multi-service callers) stays under caller control. The
+`_close_base_llm` docstring is updated to acknowledge it also handles
+`OllamaLLM` and `OllamaEmbeddings`; no behaviour change, only
+documentation. Regression coverage lives next to the existing
+ChatOllama tests in `tests/utilities/test_close_base_llm.py` —
+`TestCloseBaseLLMRealOllamaEmbeddings` is the canary that fires if a
+future migration breaks the close path again.
+
+A follow-up PR (PR-B) hardens the `rag_routes.py` call sites that
+construct `LibraryRAGService` without a `with` block: 4 simple
+synchronous sites get a `with` wrap; 3 SSE-streaming sites have the
+construction moved *inside* the `stream_with_context` generator (a
+`with` at request-handler scope would close the service before the
+stream runs). A safety-net PR (PR-C) registers a `weakref.finalize`
+inside `OllamaEmbeddingsProvider.create_embeddings()` so that callers
+that bypass the manager — for example the programmatic-API examples
+migrated in #4399 — still get eventual cleanup at GC time.
+
+### Round 9 — broader resource audit (May 2026)
+
+Once the FD-leak classes were closed, a follow-up audit looked for
+*other* slow-growth patterns that wouldn't trip the FD counters but
+could still degrade a long-running container: memory and cache growth,
+thread / asyncio Task / lock lifecycle, DB state hygiene beyond
+connections. Three parallel agents per round, two rounds (Round 1
+hypothesis generation, Round 2 fact-check), captured here in
+verified form so the next contributor doesn't re-derive the same
+conclusions.
+
+#### Refuted (false positives from Round 1, verified in Round 2)
+
+- **`@cache` on `get_available_providers`** (was in `config/llm_config.py`;
+  **removed in #4590**, so this no longer exists). Round 1 claimed unbounded
+  cache growth if the function were called with differing `settings_snapshot`
+  dicts. Round 2 verified: dicts are unhashable, so `@cache` would raise
+  `TypeError` on them, not silently grow. In practice the call sites passed
+  `settings_snapshot=None` (hashable, cardinality 1). Not a leak — and the
+  function (a dead duplicate of the provider auto-discovery path) has since
+  been deleted entirely. Kept here for the audit record.
+- **Thread-local Session identity-map growth**
+  (`database/thread_local_session.py`). Round 1 claimed long-running
+  research threads would accumulate ORM objects in the per-thread
+  Session's identity map. Round 2 verified: SQLAlchemy's default
+  `expire_on_commit=True` clears the identity map at every commit;
+  the codebase commits periodically. Bounded by typical query volume,
+  not unbounded by uptime.
+- **`token_usage` table unbounded growth.** Append-only per LLM call
+  with no TTL or retention job. Round 2 verified: **feature by
+  design**. Schema has compound time-series indexes
+  (`idx_token_research_timestamp`, etc.); `/api/context-overflow` and
+  `/metrics/api/metrics` explicitly query historical windows for cost
+  analysis. The table is a permanent audit trail by intent. Adding
+  retention would break the metrics dashboards.
+- **`search_calls` table unbounded growth.** Same shape and same
+  verdict — compound time-series indexes confirm intentional design
+  as a permanent search-analytics record.
+
+#### Fixed in this PR — three per-user lock dicts
+
+- **Three per-user lock dicts** — `_user_init_locks` and `_user_locks`
+  are module-level dicts in `database/library_init.py` and
+  `database/backup/backup_service.py` respectively; `_user_critical_locks`
+  is an instance attribute on the `QueueProcessorV2` singleton in
+  `web/queue/processor_v2.py`. Each stored one `threading.Lock` per
+  username with no removal hook. Bounded ceiling (~296 bytes/entry ×
+  3 dicts at 1000 users = ~900 KB), so not urgent — but easy to fix
+  cleanly. The two module-level dicts now expose
+  `pop_user_init_lock` / `pop_user_lock` functions; the queue
+  processor exposes the equivalent as an instance method
+  `queue_processor.pop_user_critical_lock`. A shared
+  `_pop_per_user_locks(username)` helper in `connection_cleanup.py`
+  calls all three with lazy imports and individual try/except
+  (WARNING-level so dict accumulation is observable, matching the
+  sibling scheduler-unregister error path). The helper is invoked
+  unconditionally — outside the `close_user_database` try/except so
+  it still runs when the DB close itself fails — in both the
+  idle-connection sweeper (`connection_cleanup.py:cleanup_idle_connections`)
+  and the logout / password-change paths (`web/auth/routes.py`).
+  Tests in `tests/web/auth/test_connection_cleanup.py::TestPopPerUserLocks`
+  cover the helper directly and through the idle-close path.
+
+#### Real but small (survives verification)
+
+- **`app_logs` (ResearchLog) table — no automatic retention.** Grows
+  by ~100s-1000s of rows per research. Cleaned only via cascade-delete
+  when the parent `Research` row is deleted manually. Unlike
+  `token_usage` / `search_calls`, this table has no UI dashboard or
+  time-series API consuming it — it's debug context for a specific
+  research session, not an analytics record. For users who keep all
+  research, logs accumulate indefinitely. See "Intentionally not done
+  (deferred)" for the retention design when a symptom report
+  justifies it.
+
+---
+
+## Debugging FD leaks — playbook for the next one
+
+When the next FD leak shows up (and there will be one, eventually), this
+section is the shortcut. It captures the actual diagnostic flow that
+worked across Waves 6 and 7 so a future contributor doesn't have to
+re-derive it from the symptom.
+
+### 0. Symptoms that mean "investigate this as an FD leak"
+
+- Tracebacks like `OSError: [Errno 24] Too many open files`, typically
+  from `selectors.DefaultSelector()` in werkzeug or `send_from_directory`
+  in Flask. These are usually the *first* visible failure.
+- Browser-side MIME-type errors on static assets (`text/html` instead of
+  `text/css` / `application/javascript`). These are downstream of FD
+  exhaustion — Flask can't open the static file, returns an HTML 500,
+  and the browser refuses to apply it because of
+  `X-Content-Type-Options: nosniff`.
+- `High FD count (N) — approaching system limit` warnings from
+  `web/auth/connection_cleanup.py` (fires at FD > 800 every 5-minute
+  cleanup tick).
+- Container health turns `unhealthy` because the healthcheck `urlopen`
+  hangs on a process that no longer has FDs to accept connections.
+
+### 0a. Rule out first — local UI-test "fresh-user churn" false positive
+
+Before treating climbing FDs as a leak, confirm you are measuring the
+**single-CI-user** condition. A very convincing *false* FD leak appears
+when reproducing UI tests locally:
+
+- The Puppeteer harness (`tests/ui_tests/auth_helper.js` →
+  `ensureAuthenticated`) logs in as the shared CI user `test_admin` when
+  `CI=true`. If that login fails, it **falls back to registering a fresh
+  `testuser_<timestamp>` per test**. The usual local trigger is
+  `test_admin` getting *failed-login lockout-locked* after a few
+  iterations.
+- Each fresh user opens its own per-user encrypted DB + engine. Those are
+  disposed only on logout or the ~300s connection-cleanup sweep, so within
+  one sub-300s shard run they accumulate and the server's FD count to
+  `encrypted_databases/*.db(-wal/-shm)` climbs ~linearly (e.g. 0→90 per
+  shard run, 0→533 over six runs). It looks identical to a real per-user
+  connection leak.
+- It is **not** a server bug. In real CI the one working `test_admin` is
+  reused → one engine → FDs bounded by the pool cap (pool_size 20 +
+  max_overflow 40 = 60). Confirm by grepping the server log for many
+  distinct `testuser_<ts>` engine opens, or by checking the username the
+  leaked FDs' DB files belong to.
+
+Concretely: the **chat UI shards** (`chat-core`, `chat-lifecycle`) failing
+in CI were investigated as a per-user DB FD leak and traced *twice* to
+this artifact. Both shards pass locally in faithful CI mode with bounded
+FDs; their CI failures are runner **contention** (60s navigation timeouts
+on a heavily-loaded Docker runner), not a connection leak. Cross-verify
+the user identity before committing to a leak hypothesis.
+
+### 1. Capture diagnostic state BEFORE restarting
+
+The single most important rule: **the snapshot does not survive a
+container restart**. Every minute spent on the live broken container is
+worth an hour of after-the-fact agent guessing. Save the diagnostic
+output to a host-side file first.
+
+#### One-shot host-side snapshot (works even when the container is
+FD-starved enough that `docker exec` can't fork)
+
+```bash
+# Run on the Docker host. No docker exec required.
+P=$(docker inspect -f '{{.State.Pid}}' <container-name>)
+sudo bash -c "
+  echo '=== Total FDs ==='
+  ls /proc/$P/fd | wc -l
+  echo '=== FD-type histogram (digits collapsed) ==='
+  ls -l /proc/$P/fd | awk '{print \$NF}' \
+    | sed -E 's/\[[0-9]+\]/[N]/g; s/[0-9]{4,}/NUM/g' \
+    | sort | uniq -c | sort -rn | head -30
+  echo '=== Counts by category ==='
+  printf 'socket:     %s\n' \$(find /proc/$P/fd -lname 'socket:*'     | wc -l)
+  printf 'pipe:       %s\n' \$(find /proc/$P/fd -lname 'pipe:*'       | wc -l)
+  printf 'eventpoll:  %s\n' \$(find /proc/$P/fd -lname '*eventpoll*'  | wc -l)
+  printf 'pidfd:      %s\n' \$(find /proc/$P/fd -lname '*pidfd*'      | wc -l)
+  printf 'WAL files:  %s\n' \$(find /proc/$P/fd -lname '*-wal'        | wc -l)
+  printf 'SHM files:  %s\n' \$(find /proc/$P/fd -lname '*-shm'        | wc -l)
+  printf '.db files:  %s\n' \$(find /proc/$P/fd -lname '*.db'         | wc -l)
+" | tee /tmp/ldr-fd-snapshot.txt
+```
+
+Why host-side: reading the container's PID 1 FDs from inside the
+container requires the same UID that started PID 1. The Dockerfile
+entrypoint runs as root then `setpriv`s to `ldruser`, so the
+`docker exec` shell (ldruser) cannot `readlink` PID 1's FDs even though
+it can count them. Host root via `sudo` sidesteps the UID check.
+
+#### Inside-container alternative (if the host is locked down)
+
+```bash
+docker exec --user 0 <container-name> sh -c '...same body...'
+```
+
+`--user 0` runs the exec'd shell as root inside the container,
+sidestepping the same UID restriction.
+
+### 2. The lookup table — FD type → likely source
+
+| Dominant FD type            | Likely source                                                                              | Diagnostic deep-dive                                                                                      |
+|-----------------------------|---------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------|
+| `anon_inode:[eventpoll]`    | `asyncio` event loop or `httpx.AsyncClient` selector. Each leaked async client = +1.        | Grep `asyncio.create_subprocess`, `httpx.AsyncClient`, `_async_client`, `ainvoke`. See Wave 6, Wave 7.   |
+| `anon_inode:[pidfd]`        | `asyncio.create_subprocess_*` or `multiprocessing.Process` (uses `pidfd_open` on Linux).    | Read `/proc/PID/fdinfo/N` for each pidfd; the `Pid:` line shows the target (`-1` = child already exited). |
+| `socket:*` (lots)           | HTTP keep-alive, SSE streams, SocketIO connections.                                         | Cross-reference with `/proc/PID/net/tcp` states; check Round 7 R7A8 patterns.                            |
+| `pipe:*` (lots)             | `subprocess.run`/`Popen` with `stdout=PIPE`, multiprocessing IPC, loguru queue.             | Check `subprocess.run` sites and APScheduler executor type.                                              |
+| `REG` `*-wal` / `*-shm`     | SQLCipher in WAL mode. Each pooled connection holds ~3 FDs.                                 | See ADR-0004. If growing without bound, the periodic `engine.dispose()` is silently failing.             |
+| `REG /data/*.db` (lots)     | Plain SQLite connections from an engine without bounded pool.                               | Audit `create_engine` sites (R7A6 caught two unmanaged ones).                                            |
+| `REG /home/...mmap...`      | Memory-mapped model weights or FAISS indexes — usually process-wide singletons (not leaks). | Check whether the count grows per request. If yes → real leak.                                           |
+
+### 3. Pinpointing the source for a specific FD type
+
+#### Eventpoll
+
+`anon_inode:[eventpoll]` always comes from `EpollSelector` — created
+by every asyncio loop and every `httpx.AsyncClient`. Grep:
+
+```
+grep -rn 'asyncio.create_subprocess\|httpx.AsyncClient\|_async_client' src/
+```
+
+Then check whether each site explicitly closes the client. The Wave 7
+fix to `_close_base_llm` is the reference pattern for "close async
+httpx even when called inside a running loop."
+
+#### Pidfd
+
+Pidfds expose their target PID via fdinfo:
+
+```bash
+# Run inside the container (or via docker exec --user 0):
+for fd in $(ls /proc/1/fd 2>/dev/null); do
+  link=$(readlink /proc/1/fd/$fd 2>/dev/null)
+  case "$link" in
+    *pidfd*)
+      tpid=$(awk '/^Pid:/ {print $2}' /proc/1/fdinfo/$fd 2>/dev/null)
+      if [ "$tpid" -gt 0 ] 2>/dev/null; then
+        cmd=$(tr '\0' ' ' < /proc/$tpid/cmdline 2>/dev/null | cut -c1-80)
+        echo "fd=$fd alive pid=$tpid : $cmd"
+      else
+        echo "fd=$fd ORPHAN (child exited; pidfd not closed)"
+      fi
+      ;;
+  esac
+done
+```
+
+A high "ORPHAN" count = something called `asyncio.create_subprocess_*`
+or `multiprocessing.Process`, the child exited, but the pidfd in the
+parent was never closed. Common in Round-8: Playwright's Node.js
+driver subprocess failing because Chromium isn't installed in the
+production image.
+
+**Note:** CPython 3.14's `subprocess.py` does not use pidfd at all
+(`waitpid(WNOHANG)` polling instead). So pidfds in a 3.14 process
+necessarily come from asyncio or multiprocessing, not from
+`subprocess.run` / `Popen`.
+
+#### Syscall-level pinpointing with bpftrace (mysterious cases)
+
+When the source isn't obvious from the FD type, `bpftrace` can record
+the Python stack of every relevant syscall on the live process. This
+would have caught the Playwright leak in seconds instead of two rounds
+of agent exploration. Requires kernel headers and `bpftrace` installed
+on the host (NOT the container — bpftrace runs in host kernel space
+and can target a host PID by number):
+
+```bash
+# Find host-side PID of container's PID 1
+P=$(docker inspect -f '{{.State.Pid}}' <container>)
+
+# Trace every pidfd_open syscall, grouped by user-stack:
+sudo bpftrace -e "tracepoint:syscalls:sys_enter_pidfd_open
+                  /pid == $P/ { @[ustack(perf)] = count(); }"
+
+# Same idea for epoll_create / epoll_create1 (eventpoll FDs):
+sudo bpftrace -e "tracepoint:syscalls:sys_enter_epoll_create1
+                  /pid == $P/ { @[ustack(perf)] = count(); }"
+```
+
+Let it run for a minute, then Ctrl-C; you get a histogram of every
+unique stack that triggered the syscall, ranked by frequency. The hot
+stacks are your culprits. Works for any syscall — useful future
+candidates: `socket`, `inotify_init1`, `timerfd_create`,
+`memfd_create`.
+
+#### WAL/SHM
+
+`engine.dispose()` is expected to release these. If the count climbs
+across the periodic 30-minute dispose cycles, the dispose is silently
+failing. The observability commit (f86c3f7af) elevates dispose
+failures to WARNING — check the logs for `Error disposing engine for
+<user>`.
+
+### 4. Existing instrumentation already in the codebase
+
+- **`_count_open_fds()`** in
+  `src/local_deep_research/web/auth/connection_cleanup.py` —
+  fast `/proc/self/fd`-based counter with macOS fallback. Reusable.
+- **`Resource monitor: open_fds=…`** debug log line in
+  `connection_cleanup.py`, fires every 5-minute cleanup tick.
+- **`High FD count (N)` WARNING** in `connection_cleanup.py`
+  when FDs exceed 800. The single most useful production signal.
+- **`GET /api/v1/health` resource diagnostics** (PR #4915) — for
+  *authenticated* callers the response carries a `resources` block
+  (`fd_count`, `fd_soft_limit`, `fd_hard_limit`, `fd_usage_percent`,
+  `thread_count`) and flips `status` to `"warning"` above 70% FD usage.
+  This is the live, queryable form of the `_count_open_fds()` log
+  signal — `curl` it during a leak hunt instead of grepping container
+  logs. It returns counts only (never fd targets), so no open file
+  paths or socket peers are exposed. Anonymous callers (the Docker
+  healthcheck) get only the basic `status`/`message`/`timestamp`.
+- **In-CI FD-growth canaries** in
+  `tests/utilities/test_close_base_llm.py`. These run on every PR:
+  - `TestCloseBaseLLMRealHttpxAsync::test_no_fd_growth_across_repeated_close_cycles`
+    — guards the eventpoll FD class against Wave-6-shaped regressions.
+  - `TestCloseBaseLLMRealHttpxAsync::test_no_fd_growth_when_closed_inside_running_loop`
+    — guards the Wave-7-shaped in-running-loop skip regression.
+  - `TestAsyncioSubprocessFDBaseline::test_no_fd_growth_across_asyncio_subprocess_cycles`
+    — guards the pidfd FD class against the child-watcher leak shape.
+  - `TestAsyncioSubprocessFDBaseline::test_no_fd_growth_when_subprocess_fails_to_exec`
+    — pins the *exact* Wave-7-pidfd shape (failed exec, child watcher
+    must still clean up). Catches platform-level regressions in
+    Python's asyncio child watcher.
+
+  All four use `_open_fd_count()` (also in that file) which reads
+  `/proc/self/fd` on Linux with an `RLIMIT_NOFILE` fallback on macOS.
+  Slack is +2 FDs across 5–10 iterations. A real per-cycle leak would
+  blow past that.
+
+### 4a. Development-time detection (catch leaks at test time)
+
+Production /proc inspection catches leaks **after** they ship. The
+cheapest catch is to make Python itself complain at test time. Three
+Python features cooperate to surface unclosed resources during a
+normal test run — none of them were on by default during Waves 6 and
+7, which is part of why those leaks made it to production.
+
+**`PYTHONASYNCIODEBUG=1` plus `-W default::ResourceWarning`.** When
+asyncio debug mode is on, unclosed transports/coroutines emit a
+`ResourceWarning` at GC time. The `-W` filter makes Python actually
+display them. Together they would have caught the Wave 7 in-running-loop
+skip: every leaked `httpx.AsyncClient` produces a visible warning the
+first time the GC sweeps after the test fixture exits. From
+[the asyncio dev docs](https://docs.python.org/3/library/asyncio-dev.html):
+
+> When a transport is no longer needed, call its `close()` method to
+> release resources. ... If a transport or an event loop is not closed
+> explicitly, a `ResourceWarning` warning will be emitted in its
+> destructor.
+
+To enable in `pyproject.toml` `[tool.pytest.ini_options]`:
+
+```toml
+filterwarnings = [
+    "default::ResourceWarning",
+]
+env = [
+    "PYTHONASYNCIODEBUG=1",
+]
+```
+
+Or in CI for a one-off check:
+
+```bash
+PYTHONASYNCIODEBUG=1 python -W default::ResourceWarning -m pytest tests/
+```
+
+For a CI gate that **fails** on any leak (more aggressive — use only
+on a targeted subset of tests, not the whole suite, because
+third-party libraries also emit ResourceWarning):
+
+```toml
+filterwarnings = [
+    "error::ResourceWarning",
+]
+```
+
+**`python -X dev`.** Enables Python's dev mode, which turns on a
+bundle of safety checks including ResourceWarning display, asyncio
+debug mode, and warnings as default. Cheap one-flag alternative for
+local development; not recommended in production (overhead).
+
+```bash
+python -X dev -m pytest tests/
+```
+
+**`psutil` for portable FD counting in tests.** Our in-codebase
+`_count_open_fds` uses `/proc/self/fd` (Linux-fast path, macOS
+fallback). `psutil` is the cross-platform alternative many other
+projects use:
+
+- `psutil.Process().num_fds()` — Linux/BSD only; same number as our
+  helper.
+- `psutil.Process().open_files()` — list of named files; gives the
+  paths for `REG`-type FDs (e.g., `/data/*.db-wal`).
+- `psutil.Process().connections(kind='all')` — sockets visible to the
+  process, with state and remote address.
+
+These are useful in unit tests when you want to assert "no new file
+of pattern X is open after the close path runs," and they work on the
+macOS dev environments without `/proc`.
+
+**For tracking which Python object holds a leaked FD: `tracemalloc`
++ `objgraph`.** Not FD tools per se, but when a leak is reproducible,
+take a `tracemalloc` snapshot before and after the suspect operation
+and diff — the new allocation is usually the wrapper holding the FD.
+`objgraph.show_backrefs([leaked_obj])` then renders the reference
+chain keeping it alive. Both are pure-Python and zero-dependency.
+
+### 5. Why we don't have an automated FD-growth test in CI
+
+Several reasons, weighed during Wave 6 and Wave 7:
+
+- **Per-request FD growth is hard to assert.** Many legitimate
+  request paths transiently open and close FDs; a noisy delta is the
+  norm. Distinguishing "leak" from "in-flight" requires a stable
+  quiescent state, which a CI test doesn't naturally provide.
+- **The CI environment spawns its own subprocesses.** pytest,
+  coverage, gunicorn workers (for some test variants), gh-runner
+  cleanups — all add their own FDs that pollute the count.
+- **PID-namespace differences between CI and prod.** Counts you
+  observe in a CI container's /proc are not directly comparable to a
+  production container's /proc; the subprocess sources differ.
+- **The actual leaks have been "slow drip" patterns** that need
+  hours of uptime to surface. Wave 6's eventpoll leak took multiple
+  hours of `ainvoke` calls to reach the 1024 cap. CI can't run for
+  hours per PR.
+
+What works instead:
+1. **Per-leak unit-level regression tests.** Each fix in Waves 1-7
+   landed with a targeted test that exercises the specific close path
+   (e.g. `tests/utilities/test_close_base_llm.py::test_no_fd_growth_when_closed_inside_running_loop`).
+   These are fast, deterministic, and run on every PR.
+2. **Opt-in manual smoke suite** (`RUN_MANUAL_SMOKE=1`) for the
+   end-to-end "run-the-cycle-N-times-and-count" pattern, used during
+   investigation but not on every CI run.
+3. **Production /proc inspection** when a leak is suspected — the
+   playbook above. Faster than CI for the long-drip patterns.
+
+If you want to add a long-run CI job, the right shape would be a
+**nightly** workflow (not per-PR) that:
+
+1. Builds the production Docker image.
+2. Starts it with a synthetic user account and ~5 news subscriptions.
+3. Lets it idle for 20-30 minutes.
+4. Runs the host-side snapshot script above.
+5. Asserts `total FDs < N` and `eventpoll < M` and `pidfd < K`,
+   where the thresholds are tuned for the steady-state ceilings the
+   codebase intentionally permits (auth_db pool, etc.).
+
+That would have caught Waves 6, 7 in a single nightly cycle instead
+of through a user crash report. The reason it doesn't exist yet is
+cost (a half-hour idle job per night per platform) and the lack of a
+clear baseline; the Round-8 finding is the moment to consider adding
+one if you want to invest the maintenance time.
+
+### 6. Lookup: which Wave fixed which leak class
+
+| FD class               | Wave / PR              | Root mechanism                                                            |
+|------------------------|------------------------|---------------------------------------------------------------------------|
+| `eventpoll`            | Wave 6 #3855 + Wave 7 #4047 | ChatOllama `_async_client` not closed (Wave 6) → also not closed when called inside a running loop (Wave 7). |
+| `pidfd` from healthcheck | Wave 7 #4047           | `urlopen` no `timeout=` → child hangs → reparented to PID 1 with pidfd held. |
+| `pidfd` from Playwright fallback | Round 8 / #3971  | Production image lacks Chromium binary; Playwright invocation opens pidfd then fails. |
+| WAL/SHM accumulation   | Wave 5 / ADR-0004      | SQLCipher+WAL leaks handles on out-of-order close; periodic `engine.dispose()` resets the pool.  |
+| Per-thread engine FDs  | Wave 5 #3441           | Removed per-thread `NullPool` engines entirely; shared per-user `QueuePool`. |
+| HTTP session sockets   | Wave 1 / Wave 3        | `SafeSession` / `BaseDownloader` close-in-`finally` discipline.            |
+| `asyncio.new_event_loop` | Wave 4 #3018         | Replaced manual loop creation with `asyncio.run()` in `news_strategy.py`.   |
+
+Use this table to skip the rediscovery step the next time a specific
+FD type dominates a snapshot.
+
+---
+
+## Intentionally not done (deferred)
+
+These showed up during planning and were deliberately *not* done. If
+they get rediscovered as "missing work" by future contributors, please
+reference this section first.
+
+- **`weakref.finalize` defense-in-depth on the LLM wrappers.** Designed
+  and verified safe (no `__del__` conflicts, `__getattr__` doesn't
+  intercept `_finalizer`, no reference cycles). Deferred until a
+  fourth wave of "missed close" leaks justifies adding a new pattern
+  that future contributors must understand. Current explicit-close
+  discipline has held since #2712 / #2732 / #3018.
+- **LLM caching in `get_llm()`.** Bounding total `ChatOllama` instances
+  to N=distinct configs would make leak shapes architecturally
+  impossible. Orthogonal optimization, deferred — adds complexity
+  around settings invalidation and multi-tenant isolation.
+- **Pre-commit hook flagging `get_llm()` callers without `close()`.**
+  Useful in principle, deferred — high false-positive risk
+  (caller-passed LLMs, lazy-init holders, factory-returned LLMs all
+  legitimately don't close). Needs a careful design.
+- **Per-FD-type/inode breakdown on the health endpoint.** The basic
+  version — aggregate FD count, limits, and usage percent on
+  `GET /api/v1/health` — shipped in PR #4915 (see section 4). The two
+  earlier attempts were closed rather than merged: PR #3033 (superseded
+  by #4915, a clean reapplication onto current main) and PR #3036 (a
+  `utilities/fd_monitor.py` FD circuit breaker — closed because its
+  premise, a retry-driven "death spiral," did not match the real
+  WAL/SHM-handle root cause already handled by the periodic pool
+  disposal above; `fd_monitor.py` was never merged and does not exist).
+  A type/inode breakdown (eventpoll vs pidfd vs WAL — the histogram the
+  section-1 `/proc` snapshot produces) is feasible but deferred until an
+  active leak hunt actually needs it.
+- **Automated reproduction of #3816's eventpoll-FD leak in a test
+  suite.** Explored in closed PR #3930 — a single-thread
+  `asyncio.run(ainvoke)` loop against real Ollama does *not* reproduce
+  eventpoll accumulation, because `asyncio.run` deterministically closes
+  its loop's selector each call. Reliable reproduction would need
+  sustained concurrent load (multi-worker harness over a shared loop).
+  In-CI mock + no-network real `ChatOllama` tests in
+  `tests/utilities/test_close_base_llm.py` already cover the close-chain
+  introspection regressions; a load-shape reproduction is deferred
+  until a future leak justifies the maintenance burden.
+- **`app_logs` (ResearchLog) retention setting + scheduled cleanup
+  job.** Identified in Round 9; the only audit finding that wasn't
+  refuted but also isn't impactful enough today. *Trigger to do this
+  work:* a user reports the SQLCipher DB growing >100 MB and
+  complains about query slowdown, OR a self-hosted instance keeping
+  research logs for >1 year sees DB bloat, OR the metrics dashboard
+  starts noting research-detail page load slowdown traced to
+  `app_logs` joins. *Implementation sketch:* add
+  `logs.research_log_retention_days` to
+  `defaults/default_settings.json` (default `0` = disabled, preserves
+  current behavior; e.g. `30` to keep last 30 days). Extend the
+  existing `BackgroundJobScheduler` in `scheduler/background.py`
+  (which already runs `cleanup_inactive_users` hourly and
+  `_reload_config` every 30 min) with a daily `_cleanup_old_research_logs`
+  job that deletes `ResearchLog` rows older than the retention
+  window. Skip rows belonging to favorited / starred researches if a
+  flag exists. ~30 LOC + a regression test that inserts old rows,
+  triggers the job, asserts old rows are deleted and recent ones
+  survive. Add `changelog.d/<id>.feature.md`.
+---
+
+## Glossary
+
+- **`_owns_llm`** — instance flag set in `__init__` to `True` when the
+  class fetched its own LLM via `get_llm()`, `False` when an LLM was
+  injected by the caller. Gates whether `close()` actually closes the
+  LLM.
+- **`safe_close(resource, name)`** — helper in `utilities/resource_utils.py`
+  that calls `resource.close()` inside a try/except, logging on failure.
+  Never raises. Used in every `finally` block.
+- **`_ldr_closed`** — sentinel attribute set on inner httpx clients by
+  `_close_base_llm` to make the function idempotent. Checked with
+  `is True` (not truthy) so Mock objects without a `spec` don't trip
+  the guard.
+- **eventpoll FD** — Linux `a_inode` file descriptor type for
+  `epoll_create`'d kernel objects. Each asyncio event loop registers
+  one. Leaked AsyncClients hold them via the loop's selector.