learningcircuit--local-deep-research/docs/developing/resource-cleanup.md

# Resource cleanup in LDR

This document captures how LDR manages process-level resources (DB
connections, HTTP clients, file descriptors, threads) and the reasoning
trail behind the current model. It exists because file-descriptor
exhaustion has been a recurring class of bug in LDR, and the *journey*
of fixing it — what's been tried, what worked, what was ruled out — is
not reconstructable from `git log` alone.

If you're contributing code that holds a network connection, a database
session, an LLM client, or a thread, read this before adding `__del__`,
`weakref.finalize`, or a context manager.

---

## Current model

### Database connections

- **One shared per-user `QueuePool`.** No per-thread engines. Pool
  sizing: `pool_size=20`, `max_overflow=40`, with periodic `dispose()`
  every 30 minutes.
- **SQLCipher is decrypted once per connection-open.** `PRAGMA key`
  takes ~0.2 ms; pool reuse keeps that off the hot path.
- Engines are created at login, closed at logout (or process exit via
  the registered `atexit` shutdown).
- Background threads (research workers, metric writers, news scheduler
  jobs) use the same per-user pool — they no longer maintain a separate
  thread-engine system.

See [ADR-0004](../decisions/0004-nullpool-for-sqlcipher.md) for the
QueuePool-vs-NullPool decision and PR #3441 for the per-thread-engine
removal.

### LLM wrappers

LDR wraps every LLM in `ProcessingLLMWrapper` (and optionally
`RateLimitedLLMWrapper`) so that callers see a uniform interface and
the project owns the close path:

```
caller -> ProcessingLLMWrapper.close()
       -> _close_base_llm(base_llm) in utilities/llm_utils.py
       -> for ChatOllama:
            sync httpx client (ollama.Client._client) .close()
            async httpx client (ollama.AsyncClient._client) .aclose()
       -> for ChatOpenAI / ChatAnthropic:
            no close (those use @lru_cache'd shared httpx clients)
```

Key invariants:

- `ChatOllama` is the *only* provider where `_close_base_llm()` actually
  closes anything. ChatOpenAI and ChatAnthropic share LRU-cached httpx
  clients across instances; closing them would break other live LLMs.
- Both `_client` (sync) **and** `_async_client` (async) are released —
  the async side is exercised by every `ainvoke()` call (langgraph
  agents, modular strategies). Closing only the sync side leaks the
  async transport per call (root cause of #3816).
- The function is idempotent via an `_ldr_closed` sentinel on the inner
  httpx clients.
- The async close uses `asyncio.run(client.aclose())` only when no
  event loop is currently running. When called from inside async code
  it skips and leaves the close to the loop's owner.

### Search engines

- `BaseSearchEngine.close()` is the single entry point and **cascades**
  into `_preview_filters` and `_content_filters`. That cascade is what
  releases per-engine LLMs (e.g., `JournalReputationFilter.model`),
  SearXNG sessions, and other filter-held resources.
- Search-engine cleanup happens at the per-research finally block in
  `web/services/research_service.py:run_research_process()` and at the
  programmatic API entry points in `api/research_functions.py`.
- The `_owns_llm` flag pattern (introduced in #2712) tracks whether a
  filter or engine constructed its own LLM (and thus owns it) versus
  borrowed one from a caller (and must not close it).

### Thread lifecycle

- `@thread_cleanup` (decorator on `run_research_process` and similar
  workers) ensures thread-local DB sessions are released even on
  abnormal exits.
- `cleanup_current_thread()` is called from Flask teardown, the queue
  processor, the auth flow, and the RAG routes — six tier-1 paths in
  total.
- Background threads are daemon threads; the process exit handles any
  thread that did not clean up gracefully.

### Conventions

- **Use `safe_close(resource, "human name")`** from
  `utilities/resource_utils.py` for every cleanup. Never bare `.close()`
  in a `finally` (it can mask the original exception).
- **Prefer `try/finally` over `__del__`**. Python doesn't guarantee
  finalization order at interpreter exit; `__del__` interacts subtly
  with reference cycles and `weakref`.
- **Track ownership explicitly with `_owns_llm` (or analogous flag)**
  when a class accepts an injected resource that may or may not be its
  own.
- **News fragments (`changelog.d/<id>.bugfix.md`) are required for any
  user-visible cleanup behavior change** — see `changelog.d/README.md`.

---

## How to close X correctly

| You're holding | Do this |
| --- | --- |
| A `ChatOllama` (raw or wrapped) | Call `wrapper.close()` in a `finally`, or pass to `safe_close(wrapper, "...")`. The wrapper chain handles both sync and async httpx clients. |
| A search engine you constructed | `safe_close(engine, "...")` in `finally`. The engine's `close()` cascades into preview/content filters. |
| A holder class with an LLM | Add a `close()` method, gate the LLM close on `self._owns_llm`, document who calls it. Don't add `__del__`. |
| A long-lived service holder (news scheduler, etc.) | Wrap construction in `try/finally` at the cycle boundary. Don't store the LLM if you can recreate it cheaply. |
| A DB session | Use `with get_user_db_session(username) as session:`. Don't bypass via `get_settings_manager(username=...)` without `owns_session=False` (see #3023). |
| An asyncio event loop | Use the existing one. If you genuinely need a new one (background thread fallback), call `loop.close()` in a `finally` — see `news_strategy.py` for the reference pattern (post-#3018). |

---

## Anti-patterns

These look reasonable but break specific things in this codebase:

- **Adding `__del__` to a class with `close()`.** At interpreter exit
  the `logger`, `httpx`, and event-loop modules may already be torn
  down. `__del__` can run after them and raise. Use explicit close in
  a `finally` instead.
- **Closing a shared httpx client.** ChatOpenAI / ChatAnthropic share
  one httpx pool across instances via `@lru_cache`. Closing it kills
  every other live LLM in the same process. The Ollama check in
  `_close_base_llm` exists exactly to gate this.
- **Truthy idempotency sentinels on Mock objects.** `Mock()` without a
  `spec` auto-generates child Mocks for any attribute access, so
  `getattr(client, "_ldr_closed", False)` returns a truthy Mock and
  short-circuits the close. Always use `is True` / `is None` checks
  for sentinels — see the pattern in `_close_base_llm`.
- **Skipping `super().close()` in a search-engine subclass.**
  `BaseSearchEngine.close()` is what cascades into preview/content
  filters. Override it without calling super and you leak every
  filter's resources (this was a Copilot finding on #3818).
- **Treating `asyncio.run()` as safe inside an event loop.** It raises
  `RuntimeError` if called from a thread that already has a running
  loop. The pattern in `_close_base_llm` is: detect a running loop
  with `get_running_loop()`, skip the async close in that branch (the
  loop owner will close), only call `asyncio.run` in the no-loop case.

---

## History

The FD-leak campaign spans roughly four months of iterative work. Each
fix narrowed the remaining surface; each subsequent issue was found in
a corner the previous wave hadn't touched.

### Wave 1 — initial leak inventory (Jan 2026)

- **#1832, #1849, #1856, #1860** — first comprehensive sweep. Identified
  seven distinct leak sources: `auth_db` engine, `download_management`
  DB, search cache, subprocess zombies, HTTP sessions in
  `SemanticScholarSearchEngine` and `BaseDownloader`, Socket.IO threads.
  Established context-manager + `try/finally` patterns. Added a
  pre-commit hook to catch missing cleanup at commit time.

### Wave 2 — thread-local engine accumulation (Mar 2026)

- **#2495** — diagnosed that Flask's teardown only cleaned the
  request-scoped `g.db_session` while a separate `_thread_engines` dict
  accumulated NullPool engines per thread, leaking ~3 FDs per request.
  Added `cleanup_current_thread()` across six tier-1 paths.
- **#2591** — dead-thread engines (when threads crashed they left
  engines behind) plus `stream=True` socket holds in the generic
  downloader. Added a throttled dead-thread sweep, removed `stream=True`,
  raised the Docker ulimit from 1024 to 65536.

### Wave 3 — LLM wrapper lifecycle (Mar 2026)

- **#2708** — diagnosed `ChatOllama` → `httpx.Client` chains with no
  `__del__`. With the news scheduler triggering 50–300
  `quick_summary()` calls per hour, a 1024-FD container exhausted in
  3–4 hours. Wrapped four programmatic API entry points in
  `try/finally` with explicit close.
- **#2712** — extracted `close_llm()` to a shared utility. Added
  `close()` and `_owns_llm` to `NewsAnalyzer`, `HeadlineGenerator`,
  `TopicGenerator`, `JournalReputationFilter`, `DomainClassifier`,
  `GitHubSearchEngine`, `IntegratedReportGenerator`,
  `ElasticsearchSearchEngine`, and the benchmark graders.
- **#2756** — wrapped bare `.close()` calls in `finally` blocks with
  `safe_close()` to prevent masking the original exception.
- **#2732** — moved `close()` into `ProcessingLLMWrapper` and
  `RateLimitedLLMWrapper` directly; eliminated the standalone
  `close_llm()` free function.

### Wave 4 — DB session leaks + per-call patterns (late Mar / early Apr 2026)

- **#3018** — `get_settings_manager(username=...)` was bypassing
  `g.db_session` and creating QueuePool sessions per-thread; live
  diagnostics showed 321 sockets allocated, only 66 in use.
  `DownloadService.close()` leaked the inner `SettingsManager` session.
  Also fixed `TopicBasedRecommender._create_recommendation_card()`
  (per-call LLM with no cleanup) and an `asyncio.new_event_loop()` in
  `news_strategy.py` that never closed.
- **#3204** — test fixtures using `return` instead of `yield` left
  engines un-disposed. Migrated 8 test files to `yield` +
  `engine.dispose()`.

### Wave 5 — DB pool architecture (Apr 2026)

- **#3340** — kept QueuePool but minimized FDs (`pool_size=1`,
  `max_overflow=2`, periodic `dispose()` every 30 min).
- **#3337** (closed) — proposed switching SQLCipher engines to
  NullPool for zero persistent FDs. Superseded by #3441.
- **#3441** — removed per-thread NullPool engines entirely
  (~2,100 lines of sweep logic deleted) and routed metrics through a
  single shared per-user QueuePool with bounded sizing
  (`pool_size=20`, `max_overflow=40`).
- **#3477** — created [ADR-0004](../decisions/0004-nullpool-for-sqlcipher.md)
  capturing the final pool model and updated stale FD calculations
  across docs.

### Wave 6 — async client close (May 2026)

- **#3818** (open, declined for merge) — proposed session-pooling
  around `safe_get`/`safe_post` to address #3816. The session refactor
  is reasonable in isolation, but the lsof in #3816 showed ~72% of
  leaked FDs as `a_inode [eventpoll]` selectors, not HTTP request
  sockets — pointing at async-client transports rather than `safe_get`
  callers (whose response bodies were already consumed). See
  [the PR comment](https://github.com/LearningCircuit/local-deep-research/pull/3818#issuecomment-4402290677)
  for the full reasoning.
- **#3855** — extended `_close_base_llm()` to also close
  `ChatOllama._async_client` (the actual gap the lsof pointed to).
  Added the `IntegratedReportGenerator` close that was missing from the
  per-research `finally` block. Idempotency via `_ldr_closed` sentinels
  on the inner httpx clients.

### Wave 7 — async close inside a running loop (May 2026)

- **#4047** — `_close_base_llm`'s async branch had a documented "skip if
  a loop is running; loop owner closes" path. **No loop-owner cleanup
  code existed anywhere in the project**, so when the close was called
  inside an active asyncio loop the inner `httpx.AsyncClient` (and its
  `epoll_create` FD) was silently abandoned. Reproduced in production:
  a v1.6.10 single-host Ollama container reached 1024 FDs with the
  /proc histogram showing **929 `anon_inode:[eventpoll]` (91%)** — the
  same FD class as #3816 but in a code path #3855's fix didn't cover.
  The fix runs the async close in a brief daemon thread that owns its
  own loop, so `asyncio.run(aclose())` works regardless of the caller's
  loop state. A bounded 5-second `join` keeps the cleanup from blocking
  shutdown when the Ollama server is unresponsive; on timeout
  `_ldr_closed` is left unset so a later call retries, and a WARNING
  surfaces so the situation is observable instead of silent.
- **Healthcheck pidfd leak (same PR).** Dockerfile's
  `HEALTHCHECK CMD python -c "... urllib.request.urlopen(...)"` had no
  `timeout=` argument; Docker's 10s timeout SIGKILL'd the `sh -c`
  parent but the python child was reparented to PID 1 and hung
  forever, each surviving child holding a `pidfd` + TCP socket against
  the app. Same /proc dump showed **64 `anon_inode:[pidfd]` (6%)** from
  this. Adding `timeout=8` lets the child return/raise inside Docker's
  budget so it exits cleanly and gets reaped.
#### Audit ledger — what the broader sweep checked

The PR included a wide audit (50+ parallel exploration agents across
seven rounds plus direct `/proc` inspection) to catch any other latent
FD leak. To save the next contributor from re-running the same checks,
here is the full ledger:

##### Checked and confirmed clean (no action needed)

- **Non-Ollama LLM providers.** xAI, Google Gemini, OpenRouter, IONOS,
  LM Studio, llama.cpp HTTP, DeepSeek, OpenAI-compatible endpoint, plus
  OpenAI and Anthropic themselves. All extend `ChatOpenAI` or
  `ChatAnthropic`, which use `@lru_cache`'d shared httpx clients.
  `_close_base_llm`'s short-circuit on these classes is correct by
  design — closing them would brick every other live LLM in the
  process.
- **HTTP session lifecycle.** Six instantiation sites checked
  (`PricingFetcher` aiohttp, `LDRClient` SafeSession, `BaseDownloader`,
  `SemanticScholarSearchEngine`, `MCPClient`, `CostCalculator`). All
  context-managed via `with` or owned by a class with a paired
  `close()` and `__exit__`.
- **subprocess / pidfd.** Three call sites, all `subprocess.run()`
  (blocking). No `subprocess.Popen` paths anywhere in `src/`. No
  `ProcessPoolExecutor`. No FD leak surface beyond the healthcheck
  child, already addressed by the Dockerfile `timeout=8` change.
- **asyncio event loops.** Zero raw `asyncio.new_event_loop()`
  outside safe `asyncio.run()` patterns. The historical leak in
  `news_strategy.py` (#3018) is still fixed.
- **File handles.** All 37 `open()` call sites are inside `with`.
  Zero bare opens. `tempfile.NamedTemporaryFile` / `TemporaryDirectory`
  all context-managed.
- **SocketIO connect/disconnect.** Non-disconnect handlers
  (`subscribe`, `unsubscribe`, `connect`) do not acquire DB sessions
  (an early-round agent claim that they did was refuted on re-read).
  The `__socket_subscriptions` dict is cleaned on disconnect. The
  PID-1 FD breakdown showed only 3 sockets out of 1024 — socket
  accumulation is not a contributor.

##### Flagged by audit, then verified NOT a real FD leak

- **OllamaEmbeddings httpx (historical — current state covered in
  Wave 10 below).** At the time of this Wave-7 audit LDR imported the
  **deprecated** `langchain_community.embeddings.OllamaEmbeddings`,
  which used `requests.post()` per call — no persistent httpx client,
  no `_client` / `_async_client` attribute. Direct introspection:
  `[a for a in dir(e) if 'client' in a.lower()]` returned `[]`. Zero
  FDs per call. An audit agent confused this class with `ChatOllama`,
  which is a different class. The migration to
  `langchain_ollama.OllamaEmbeddings` predicted in the next subsection
  has since shipped (#4352/#4353) and the resulting FD-leak regression
  has been fixed — see Wave 10.
- **`auth_db` and `journal_quality` engines escaping
  `shutdown_databases()`.** `auth_db` uses
  `QueuePool(pool_size=10, max_overflow=20)` and `journal_quality`
  uses `StaticPool` with `immutable=1`. Both are **bounded** and do
  not grow at runtime. Live `/proc` on the affected container showed
  only 21 SQLite-related FDs total on PID 1 — well below the ~91-FD
  ceiling these unmanaged engines could theoretically reach. The
  kernel reclaims FDs at process exit regardless of `engine.dispose()`,
  and SQLite WAL files auto-checkpoint on next open. Missing dispose
  at exit is hygiene, not a leak.
- **`LibraryRAGService` in three RAG SSE endpoints.**
  `rag_routes.py:693, 1054, 1827` do construct the service outside
  the generator and never close it, **but** `LibraryRAGService.close()`
  only sets references to `None` — it releases no FDs. FAISS uses
  `pickle.load()` (not mmap); OllamaEmbeddings holds no FDs per the
  item above; the SentenceTransformer model+tokenizer mmaps are
  process-wide singletons. What gets delayed is ~50–200 MB of
  embedding-model RAM until GC. A memory-pressure question, not the
  eventpoll FD class this Wave addressed.
- **Residual `pidfd` accumulation via Playwright fallback** —
  identified in a Round-8 follow-up after the eventpoll fix landed.
  Live `/proc` on the prerelease container showed ~29 pidfds steady
  state, growing ~3.6/hour, all targeting `Pid: -1` (children that
  had exited). Rate was stable during active benchmark execution,
  ruling out a per-task source. Eight parallel agents converged on
  the same chain: `_check_subscription` → `quick_summary` →
  `FullSearchResults.batch_fetch_and_extract` → `AutoHTMLDownloader`
  fallback to `PlaywrightHTMLDownloader._fetch_with_playwright`. Each
  `sync_playwright().start()` invokes
  `asyncio.create_subprocess_exec()` for the Node.js driver (opens a
  pidfd via Linux's `PidfdChildWatcher`); the driver then fails
  because Chromium is not installed in the production `ldr` Dockerfile
  stage (only `ldr-test` runs `playwright install --with-deps
  chromium`), and the asyncio child watcher does not promptly close
  the pidfd on the failed-child exit. CPython 3.14 was confirmed to
  not use pidfd in `subprocess.py` at all (`subprocess.run`/`Popen`
  use `waitpid(WNOHANG)` polling), so subprocess-based hypotheses
  were ruled out. **Fixed by PR #3971** (default
  `web.enable_javascript_rendering=false`): the fallback short-circuits
  before any subprocess is spawned, so no pidfd is opened. The PR was
  motivated by issue #3826 (confusing tracebacks); the FD-leak
  finding is the second motivation, surfaced here.

##### Minor findings (not steady-state leaks; worth knowing)

- **Daemon threads without explicit shutdown.**
  `journal_reputation_filter.py` background fetcher, `log_utils.py`
  queue processor. All daemonized — reaped by the OS at process exit.
  Not steady-state leaks; no per-request growth.
- **Abandoned-research thread on socket disconnect.** If a client
  closes the tab mid-research, the socket subscription is removed but
  the research thread keeps running until completion;
  `_active_research[research_id]` is not cleared on disconnect. Not an
  FD leak; potentially compute/memory waste if the user wanted the
  research to stop. Out of scope for the FD-leak story.

#### Future-proofing note — `langchain_ollama.OllamaEmbeddings` migration (resolved in Wave 10)

Status: **resolved**. The migration this note predicted shipped in
#4352/#4353; the FD-leak regression it predicted then surfaced and was
fixed in Wave 10 (see below). Kept here as the source of the prediction
that the next contributor's audit can cross-reference.

`langchain_community.embeddings.OllamaEmbeddings` was deprecated ("will
be removed in langchain 1.0.0", per the import warning). Its replacement,
`langchain_ollama.OllamaEmbeddings`, **does** carry `_client` and
`_async_client` attributes — same shape as `ChatOllama`. Verified by
direct introspection at the time of writing:

```
langchain_ollama.OllamaEmbeddings client attrs:
  ['_set_clients', 'async_client_kwargs', 'client_kwargs',
   'sync_client_kwargs']
Has _client?       True
Has _async_client? True
```

The prediction was: once LDR migrates, the eventpoll FD leak class
returns for embeddings unless `_close_base_llm` is called on embedding
instances. The introspection turned out to be slightly different from
expected — both clients are constructed *eagerly* by a Pydantic
`@model_validator(mode="after")` in `langchain_ollama.embeddings.py`,
so the leak fires per-instance regardless of whether the async path is
exercised. Wave 10 contains the post-mortem and fix.

### Wave 10 — embeddings FD leak after langchain_ollama migration (June 2026)

The migration predicted above shipped without the matching close-path
generalization, exactly as feared. Verified by four independent agents:
`langchain_ollama.OllamaEmbeddings(...)` eagerly constructs both a sync
`ollama.Client` (→ `httpx.Client`) and an async `ollama.AsyncClient`
(→ `httpx.AsyncClient` → one `epoll_create` FD) inside its
`@model_validator(mode="after")` at
`.venv/.../langchain_ollama/embeddings.py:295-315`. No `close()`,
`aclose()`, `__del__`, or `weakref.finalize` exists on the new class or
the underlying `ollama` / `httpx` clients, so dropping the Python
reference does not release the FDs.

`_close_base_llm` already handled the shape — its module-prefix checks
(`type(...).__module__.startswith("ollama")` at
`src/local_deep_research/utilities/llm_utils.py:97,114`) match
`ollama.Client` / `ollama.AsyncClient` regardless of which langchain
wrapper holds them. The function just wasn't called on embeddings
instances — `LocalEmbeddingManager.close()` and `LibraryRAGService.close()`
only nulled their `_embeddings` / `embedding_manager` references,
relying on GC that would never run the close.

Fix: route the close call through the existing manager lifecycle.
`LocalEmbeddingManager.close()` now calls `_close_base_llm(self._embeddings)`
before nulling. `LibraryRAGService.close()` now calls
`self.embedding_manager.close()` before nulling — guarded by an
`_owns_embedding_manager` flag so a caller-supplied manager (test
fixtures, multi-service callers) stays under caller control. The
`_close_base_llm` docstring is updated to acknowledge it also handles
`OllamaLLM` and `OllamaEmbeddings`; no behaviour change, only
documentation. Regression coverage lives next to the existing
ChatOllama tests in `tests/utilities/test_close_base_llm.py` —
`TestCloseBaseLLMRealOllamaEmbeddings` is the canary that fires if a
future migration breaks the close path again.

A follow-up PR (PR-B) hardens the `rag_routes.py` call sites that
construct `LibraryRAGService` without a `with` block: 4 simple
synchronous sites get a `with` wrap; 3 SSE-streaming sites have the
construction moved *inside* the `stream_with_context` generator (a
`with` at request-handler scope would close the service before the
stream runs). A safety-net PR (PR-C) registers a `weakref.finalize`
inside `OllamaEmbeddingsProvider.create_embeddings()` so that callers
that bypass the manager — for example the programmatic-API examples
migrated in #4399 — still get eventual cleanup at GC time.

### Round 9 — broader resource audit (May 2026)

Once the FD-leak classes were closed, a follow-up audit looked for
*other* slow-growth patterns that wouldn't trip the FD counters but
could still degrade a long-running container: memory and cache growth,
thread / asyncio Task / lock lifecycle, DB state hygiene beyond
connections. Three parallel agents per round, two rounds (Round 1
hypothesis generation, Round 2 fact-check), captured here in
verified form so the next contributor doesn't re-derive the same
conclusions.

#### Refuted (false positives from Round 1, verified in Round 2)

- **`@cache` on `get_available_providers`** (was in `config/llm_config.py`;
  **removed in #4590**, so this no longer exists). Round 1 claimed unbounded
  cache growth if the function were called with differing `settings_snapshot`
  dicts. Round 2 verified: dicts are unhashable, so `@cache` would raise
  `TypeError` on them, not silently grow. In practice the call sites passed
  `settings_snapshot=None` (hashable, cardinality 1). Not a leak — and the
  function (a dead duplicate of the provider auto-discovery path) has since
  been deleted entirely. Kept here for the audit record.
- **Thread-local Session identity-map growth**
  (`database/thread_local_session.py`). Round 1 claimed long-running
  research threads would accumulate ORM objects in the per-thread
  Session's identity map. Round 2 verified: SQLAlchemy's default
  `expire_on_commit=True` clears the identity map at every commit;
  the codebase commits periodically. Bounded by typical query volume,
  not unbounded by uptime.
- **`token_usage` table unbounded growth.** Append-only per LLM call
  with no TTL or retention job. Round 2 verified: **feature by
  design**. Schema has compound time-series indexes
  (`idx_token_research_timestamp`, etc.); `/api/context-overflow` and
  `/metrics/api/metrics` explicitly query historical windows for cost
  analysis. The table is a permanent audit trail by intent. Adding
  retention would break the metrics dashboards.
- **`search_calls` table unbounded growth.** Same shape and same
  verdict — compound time-series indexes confirm intentional design
  as a permanent search-analytics record.

#### Fixed in this PR — three per-user lock dicts

- **Three per-user lock dicts** — `_user_init_locks` and `_user_locks`
  are module-level dicts in `database/library_init.py` and
  `database/backup/backup_service.py` respectively; `_user_critical_locks`
  is an instance attribute on the `QueueProcessorV2` singleton in
  `web/queue/processor_v2.py`. Each stored one `threading.Lock` per
  username with no removal hook. Bounded ceiling (~296 bytes/entry ×
  3 dicts at 1000 users = ~900 KB), so not urgent — but easy to fix
  cleanly. The two module-level dicts now expose
  `pop_user_init_lock` / `pop_user_lock` functions; the queue
  processor exposes the equivalent as an instance method
  `queue_processor.pop_user_critical_lock`. A shared
  `_pop_per_user_locks(username)` helper in `connection_cleanup.py`
  calls all three with lazy imports and individual try/except
  (WARNING-level so dict accumulation is observable, matching the
  sibling scheduler-unregister error path). The helper is invoked
  unconditionally — outside the `close_user_database` try/except so
  it still runs when the DB close itself fails — in both the
  idle-connection sweeper (`connection_cleanup.py:cleanup_idle_connections`)
  and the logout / password-change paths (`web/auth/routes.py`).
  Tests in `tests/web/auth/test_connection_cleanup.py::TestPopPerUserLocks`
  cover the helper directly and through the idle-close path.

#### Real but small (survives verification)

- **`app_logs` (ResearchLog) table — no automatic retention.** Grows
  by ~100s-1000s of rows per research. Cleaned only via cascade-delete
  when the parent `Research` row is deleted manually. Unlike
  `token_usage` / `search_calls`, this table has no UI dashboard or
  time-series API consuming it — it's debug context for a specific
  research session, not an analytics record. For users who keep all
  research, logs accumulate indefinitely. See "Intentionally not done
  (deferred)" for the retention design when a symptom report
  justifies it.

---

## Debugging FD leaks — playbook for the next one

When the next FD leak shows up (and there will be one, eventually), this
section is the shortcut. It captures the actual diagnostic flow that
worked across Waves 6 and 7 so a future contributor doesn't have to
re-derive it from the symptom.

### 0. Symptoms that mean "investigate this as an FD leak"

- Tracebacks like `OSError: [Errno 24] Too many open files`, typically
  from `selectors.DefaultSelector()` in werkzeug or `send_from_directory`
  in Flask. These are usually the *first* visible failure.
- Browser-side MIME-type errors on static assets (`text/html` instead of
  `text/css` / `application/javascript`). These are downstream of FD
  exhaustion — Flask can't open the static file, returns an HTML 500,
  and the browser refuses to apply it because of
  `X-Content-Type-Options: nosniff`.
- `High FD count (N) — approaching system limit` warnings from
  `web/auth/connection_cleanup.py` (fires at FD > 800 every 5-minute
  cleanup tick).
- Container health turns `unhealthy` because the healthcheck `urlopen`
  hangs on a process that no longer has FDs to accept connections.

### 0a. Rule out first — local UI-test "fresh-user churn" false positive

Before treating climbing FDs as a leak, confirm you are measuring the
**single-CI-user** condition. A very convincing *false* FD leak appears
when reproducing UI tests locally:

- The Puppeteer harness (`tests/ui_tests/auth_helper.js` →
  `ensureAuthenticated`) logs in as the shared CI user `test_admin` when
  `CI=true`. If that login fails, it **falls back to registering a fresh
  `testuser_<timestamp>` per test**. The usual local trigger is
  `test_admin` getting *failed-login lockout-locked* after a few
  iterations.
- Each fresh user opens its own per-user encrypted DB + engine. Those are
  disposed only on logout or the ~300s connection-cleanup sweep, so within
  one sub-300s shard run they accumulate and the server's FD count to
  `encrypted_databases/*.db(-wal/-shm)` climbs ~linearly (e.g. 0→90 per
  shard run, 0→533 over six runs). It looks identical to a real per-user
  connection leak.
- It is **not** a server bug. In real CI the one working `test_admin` is
  reused → one engine → FDs bounded by the pool cap (pool_size 20 +
  max_overflow 40 = 60). Confirm by grepping the server log for many
  distinct `testuser_<ts>` engine opens, or by checking the username the
  leaked FDs' DB files belong to.

Concretely: the **chat UI shards** (`chat-core`, `chat-lifecycle`) failing
in CI were investigated as a per-user DB FD leak and traced *twice* to
this artifact. Both shards pass locally in faithful CI mode with bounded
FDs; their CI failures are runner **contention** (60s navigation timeouts
on a heavily-loaded Docker runner), not a connection leak. Cross-verify
the user identity before committing to a leak hypothesis.

### 1. Capture diagnostic state BEFORE restarting

The single most important rule: **the snapshot does not survive a
container restart**. Every minute spent on the live broken container is
worth an hour of after-the-fact agent guessing. Save the diagnostic
output to a host-side file first.

#### One-shot host-side snapshot (works even when the container is
FD-starved enough that `docker exec` can't fork)

```bash
# Run on the Docker host. No docker exec required.
P=$(docker inspect -f '{{.State.Pid}}' <container-name>)
sudo bash -c "
  echo '=== Total FDs ==='
  ls /proc/$P/fd | wc -l
  echo '=== FD-type histogram (digits collapsed) ==='
  ls -l /proc/$P/fd | awk '{print \$NF}' \
    | sed -E 's/\[[0-9]+\]/[N]/g; s/[0-9]{4,}/NUM/g' \
    | sort | uniq -c | sort -rn | head -30
  echo '=== Counts by category ==='
  printf 'socket:     %s\n' \$(find /proc/$P/fd -lname 'socket:*'     | wc -l)
  printf 'pipe:       %s\n' \$(find /proc/$P/fd -lname 'pipe:*'       | wc -l)
  printf 'eventpoll:  %s\n' \$(find /proc/$P/fd -lname '*eventpoll*'  | wc -l)
  printf 'pidfd:      %s\n' \$(find /proc/$P/fd -lname '*pidfd*'      | wc -l)
  printf 'WAL files:  %s\n' \$(find /proc/$P/fd -lname '*-wal'        | wc -l)
  printf 'SHM files:  %s\n' \$(find /proc/$P/fd -lname '*-shm'        | wc -l)
  printf '.db files:  %s\n' \$(find /proc/$P/fd -lname '*.db'         | wc -l)
" | tee /tmp/ldr-fd-snapshot.txt
```

Why host-side: reading the container's PID 1 FDs from inside the
container requires the same UID that started PID 1. The Dockerfile
entrypoint runs as root then `setpriv`s to `ldruser`, so the
`docker exec` shell (ldruser) cannot `readlink` PID 1's FDs even though
it can count them. Host root via `sudo` sidesteps the UID check.

#### Inside-container alternative (if the host is locked down)

```bash
docker exec --user 0 <container-name> sh -c '...same body...'
```

`--user 0` runs the exec'd shell as root inside the container,
sidestepping the same UID restriction.

### 2. The lookup table — FD type → likely source

| Dominant FD type            | Likely source                                                                              | Diagnostic deep-dive                                                                                      |
|-----------------------------|---------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------|
| `anon_inode:[eventpoll]`    | `asyncio` event loop or `httpx.AsyncClient` selector. Each leaked async client = +1.        | Grep `asyncio.create_subprocess`, `httpx.AsyncClient`, `_async_client`, `ainvoke`. See Wave 6, Wave 7.   |
| `anon_inode:[pidfd]`        | `asyncio.create_subprocess_*` or `multiprocessing.Process` (uses `pidfd_open` on Linux).    | Read `/proc/PID/fdinfo/N` for each pidfd; the `Pid:` line shows the target (`-1` = child already exited). |
| `socket:*` (lots)           | HTTP keep-alive, SSE streams, SocketIO connections.                                         | Cross-reference with `/proc/PID/net/tcp` states; check Round 7 R7A8 patterns.                            |
| `pipe:*` (lots)             | `subprocess.run`/`Popen` with `stdout=PIPE`, multiprocessing IPC, loguru queue.             | Check `subprocess.run` sites and APScheduler executor type.                                              |
| `REG` `*-wal` / `*-shm`     | SQLCipher in WAL mode. Each pooled connection holds ~3 FDs.                                 | See ADR-0004. If growing without bound, the periodic `engine.dispose()` is silently failing.             |
| `REG /data/*.db` (lots)     | Plain SQLite connections from an engine without bounded pool.                               | Audit `create_engine` sites (R7A6 caught two unmanaged ones).                                            |
| `REG /home/...mmap...`      | Memory-mapped model weights or FAISS indexes — usually process-wide singletons (not leaks). | Check whether the count grows per request. If yes → real leak.                                           |

### 3. Pinpointing the source for a specific FD type

#### Eventpoll

`anon_inode:[eventpoll]` always comes from `EpollSelector` — created
by every asyncio loop and every `httpx.AsyncClient`. Grep:

```
grep -rn 'asyncio.create_subprocess\|httpx.AsyncClient\|_async_client' src/
```

Then check whether each site explicitly closes the client. The Wave 7
fix to `_close_base_llm` is the reference pattern for "close async
httpx even when called inside a running loop."

#### Pidfd

Pidfds expose their target PID via fdinfo:

```bash
# Run inside the container (or via docker exec --user 0):
for fd in $(ls /proc/1/fd 2>/dev/null); do
  link=$(readlink /proc/1/fd/$fd 2>/dev/null)
  case "$link" in
    *pidfd*)
      tpid=$(awk '/^Pid:/ {print $2}' /proc/1/fdinfo/$fd 2>/dev/null)
      if [ "$tpid" -gt 0 ] 2>/dev/null; then
        cmd=$(tr '\0' ' ' < /proc/$tpid/cmdline 2>/dev/null | cut -c1-80)
        echo "fd=$fd alive pid=$tpid : $cmd"
      else
        echo "fd=$fd ORPHAN (child exited; pidfd not closed)"
      fi
      ;;
  esac
done
```

A high "ORPHAN" count = something called `asyncio.create_subprocess_*`
or `multiprocessing.Process`, the child exited, but the pidfd in the
parent was never closed. Common in Round-8: Playwright's Node.js
driver subprocess failing because Chromium isn't installed in the
production image.

**Note:** CPython 3.14's `subprocess.py` does not use pidfd at all
(`waitpid(WNOHANG)` polling instead). So pidfds in a 3.14 process
necessarily come from asyncio or multiprocessing, not from
`subprocess.run` / `Popen`.

#### Syscall-level pinpointing with bpftrace (mysterious cases)

When the source isn't obvious from the FD type, `bpftrace` can record
the Python stack of every relevant syscall on the live process. This
would have caught the Playwright leak in seconds instead of two rounds
of agent exploration. Requires kernel headers and `bpftrace` installed
on the host (NOT the container — bpftrace runs in host kernel space
and can target a host PID by number):

```bash
# Find host-side PID of container's PID 1
P=$(docker inspect -f '{{.State.Pid}}' <container>)

# Trace every pidfd_open syscall, grouped by user-stack:
sudo bpftrace -e "tracepoint:syscalls:sys_enter_pidfd_open
                  /pid == $P/ { @[ustack(perf)] = count(); }"

# Same idea for epoll_create / epoll_create1 (eventpoll FDs):
sudo bpftrace -e "tracepoint:syscalls:sys_enter_epoll_create1
                  /pid == $P/ { @[ustack(perf)] = count(); }"
```

Let it run for a minute, then Ctrl-C; you get a histogram of every
unique stack that triggered the syscall, ranked by frequency. The hot
stacks are your culprits. Works for any syscall — useful future
candidates: `socket`, `inotify_init1`, `timerfd_create`,
`memfd_create`.

#### WAL/SHM

`engine.dispose()` is expected to release these. If the count climbs
across the periodic 30-minute dispose cycles, the dispose is silently
failing. The observability commit (f86c3f7af) elevates dispose
failures to WARNING — check the logs for `Error disposing engine for
<user>`.

### 4. Existing instrumentation already in the codebase

- **`_count_open_fds()`** in
  `src/local_deep_research/web/auth/connection_cleanup.py` —
  fast `/proc/self/fd`-based counter with macOS fallback. Reusable.
- **`Resource monitor: open_fds=…`** debug log line in
  `connection_cleanup.py`, fires every 5-minute cleanup tick.
- **`High FD count (N)` WARNING** in `connection_cleanup.py`
  when FDs exceed 800. The single most useful production signal.
- **`GET /api/v1/health` resource diagnostics** (PR #4915) — for
  *authenticated* callers the response carries a `resources` block
  (`fd_count`, `fd_soft_limit`, `fd_hard_limit`, `fd_usage_percent`,
  `thread_count`) and flips `status` to `"warning"` above 70% FD usage.
  This is the live, queryable form of the `_count_open_fds()` log
  signal — `curl` it during a leak hunt instead of grepping container
  logs. It returns counts only (never fd targets), so no open file
  paths or socket peers are exposed. Anonymous callers (the Docker
  healthcheck) get only the basic `status`/`message`/`timestamp`.
- **In-CI FD-growth canaries** in
  `tests/utilities/test_close_base_llm.py`. These run on every PR:
  - `TestCloseBaseLLMRealHttpxAsync::test_no_fd_growth_across_repeated_close_cycles`
    — guards the eventpoll FD class against Wave-6-shaped regressions.
  - `TestCloseBaseLLMRealHttpxAsync::test_no_fd_growth_when_closed_inside_running_loop`
    — guards the Wave-7-shaped in-running-loop skip regression.
  - `TestAsyncioSubprocessFDBaseline::test_no_fd_growth_across_asyncio_subprocess_cycles`
    — guards the pidfd FD class against the child-watcher leak shape.
  - `TestAsyncioSubprocessFDBaseline::test_no_fd_growth_when_subprocess_fails_to_exec`
    — pins the *exact* Wave-7-pidfd shape (failed exec, child watcher
    must still clean up). Catches platform-level regressions in
    Python's asyncio child watcher.

  All four use `_open_fd_count()` (also in that file) which reads
  `/proc/self/fd` on Linux with an `RLIMIT_NOFILE` fallback on macOS.
  Slack is +2 FDs across 5–10 iterations. A real per-cycle leak would
  blow past that.

### 4a. Development-time detection (catch leaks at test time)

Production /proc inspection catches leaks **after** they ship. The
cheapest catch is to make Python itself complain at test time. Three
Python features cooperate to surface unclosed resources during a
normal test run — none of them were on by default during Waves 6 and
7, which is part of why those leaks made it to production.

**`PYTHONASYNCIODEBUG=1` plus `-W default::ResourceWarning`.** When
asyncio debug mode is on, unclosed transports/coroutines emit a
`ResourceWarning` at GC time. The `-W` filter makes Python actually
display them. Together they would have caught the Wave 7 in-running-loop
skip: every leaked `httpx.AsyncClient` produces a visible warning the
first time the GC sweeps after the test fixture exits. From
[the asyncio dev docs](https://docs.python.org/3/library/asyncio-dev.html):

> When a transport is no longer needed, call its `close()` method to
> release resources. ... If a transport or an event loop is not closed
> explicitly, a `ResourceWarning` warning will be emitted in its
> destructor.

To enable in `pyproject.toml` `[tool.pytest.ini_options]`:

```toml
filterwarnings = [
    "default::ResourceWarning",
]
env = [
    "PYTHONASYNCIODEBUG=1",
]
```

Or in CI for a one-off check:

```bash
PYTHONASYNCIODEBUG=1 python -W default::ResourceWarning -m pytest tests/
```

For a CI gate that **fails** on any leak (more aggressive — use only
on a targeted subset of tests, not the whole suite, because
third-party libraries also emit ResourceWarning):

```toml
filterwarnings = [
    "error::ResourceWarning",
]
```

**`python -X dev`.** Enables Python's dev mode, which turns on a
bundle of safety checks including ResourceWarning display, asyncio
debug mode, and warnings as default. Cheap one-flag alternative for
local development; not recommended in production (overhead).

```bash
python -X dev -m pytest tests/
```

**`psutil` for portable FD counting in tests.** Our in-codebase
`_count_open_fds` uses `/proc/self/fd` (Linux-fast path, macOS
fallback). `psutil` is the cross-platform alternative many other
projects use:

- `psutil.Process().num_fds()` — Linux/BSD only; same number as our
  helper.
- `psutil.Process().open_files()` — list of named files; gives the
  paths for `REG`-type FDs (e.g., `/data/*.db-wal`).
- `psutil.Process().connections(kind='all')` — sockets visible to the
  process, with state and remote address.

These are useful in unit tests when you want to assert "no new file
of pattern X is open after the close path runs," and they work on the
macOS dev environments without `/proc`.

**For tracking which Python object holds a leaked FD: `tracemalloc`
+ `objgraph`.** Not FD tools per se, but when a leak is reproducible,
take a `tracemalloc` snapshot before and after the suspect operation
and diff — the new allocation is usually the wrapper holding the FD.
`objgraph.show_backrefs([leaked_obj])` then renders the reference
chain keeping it alive. Both are pure-Python and zero-dependency.

### 5. Why we don't have an automated FD-growth test in CI

Several reasons, weighed during Wave 6 and Wave 7:

- **Per-request FD growth is hard to assert.** Many legitimate
  request paths transiently open and close FDs; a noisy delta is the
  norm. Distinguishing "leak" from "in-flight" requires a stable
  quiescent state, which a CI test doesn't naturally provide.
- **The CI environment spawns its own subprocesses.** pytest,
  coverage, gunicorn workers (for some test variants), gh-runner
  cleanups — all add their own FDs that pollute the count.
- **PID-namespace differences between CI and prod.** Counts you
  observe in a CI container's /proc are not directly comparable to a
  production container's /proc; the subprocess sources differ.
- **The actual leaks have been "slow drip" patterns** that need
  hours of uptime to surface. Wave 6's eventpoll leak took multiple
  hours of `ainvoke` calls to reach the 1024 cap. CI can't run for
  hours per PR.

What works instead:
1. **Per-leak unit-level regression tests.** Each fix in Waves 1-7
   landed with a targeted test that exercises the specific close path
   (e.g. `tests/utilities/test_close_base_llm.py::test_no_fd_growth_when_closed_inside_running_loop`).
   These are fast, deterministic, and run on every PR.
2. **Opt-in manual smoke suite** (`RUN_MANUAL_SMOKE=1`) for the
   end-to-end "run-the-cycle-N-times-and-count" pattern, used during
   investigation but not on every CI run.
3. **Production /proc inspection** when a leak is suspected — the
   playbook above. Faster than CI for the long-drip patterns.

If you want to add a long-run CI job, the right shape would be a
**nightly** workflow (not per-PR) that:

1. Builds the production Docker image.
2. Starts it with a synthetic user account and ~5 news subscriptions.
3. Lets it idle for 20-30 minutes.
4. Runs the host-side snapshot script above.
5. Asserts `total FDs < N` and `eventpoll < M` and `pidfd < K`,
   where the thresholds are tuned for the steady-state ceilings the
   codebase intentionally permits (auth_db pool, etc.).

That would have caught Waves 6, 7 in a single nightly cycle instead
of through a user crash report. The reason it doesn't exist yet is
cost (a half-hour idle job per night per platform) and the lack of a
clear baseline; the Round-8 finding is the moment to consider adding
one if you want to invest the maintenance time.

### 6. Lookup: which Wave fixed which leak class

| FD class               | Wave / PR              | Root mechanism                                                            |
|------------------------|------------------------|---------------------------------------------------------------------------|
| `eventpoll`            | Wave 6 #3855 + Wave 7 #4047 | ChatOllama `_async_client` not closed (Wave 6) → also not closed when called inside a running loop (Wave 7). |
| `pidfd` from healthcheck | Wave 7 #4047           | `urlopen` no `timeout=` → child hangs → reparented to PID 1 with pidfd held. |
| `pidfd` from Playwright fallback | Round 8 / #3971  | Production image lacks Chromium binary; Playwright invocation opens pidfd then fails. |
| WAL/SHM accumulation   | Wave 5 / ADR-0004      | SQLCipher+WAL leaks handles on out-of-order close; periodic `engine.dispose()` resets the pool.  |
| Per-thread engine FDs  | Wave 5 #3441           | Removed per-thread `NullPool` engines entirely; shared per-user `QueuePool`. |
| HTTP session sockets   | Wave 1 / Wave 3        | `SafeSession` / `BaseDownloader` close-in-`finally` discipline.            |
| `asyncio.new_event_loop` | Wave 4 #3018         | Replaced manual loop creation with `asyncio.run()` in `news_strategy.py`.   |

Use this table to skip the rediscovery step the next time a specific
FD type dominates a snapshot.

---

## Intentionally not done (deferred)

These showed up during planning and were deliberately *not* done. If
they get rediscovered as "missing work" by future contributors, please
reference this section first.

- **`weakref.finalize` defense-in-depth on the LLM wrappers.** Designed
  and verified safe (no `__del__` conflicts, `__getattr__` doesn't
  intercept `_finalizer`, no reference cycles). Deferred until a
  fourth wave of "missed close" leaks justifies adding a new pattern
  that future contributors must understand. Current explicit-close
  discipline has held since #2712 / #2732 / #3018.
- **LLM caching in `get_llm()`.** Bounding total `ChatOllama` instances
  to N=distinct configs would make leak shapes architecturally
  impossible. Orthogonal optimization, deferred — adds complexity
  around settings invalidation and multi-tenant isolation.
- **Pre-commit hook flagging `get_llm()` callers without `close()`.**
  Useful in principle, deferred — high false-positive risk
  (caller-passed LLMs, lazy-init holders, factory-returned LLMs all
  legitimately don't close). Needs a careful design.
- **Per-FD-type/inode breakdown on the health endpoint.** The basic
  version — aggregate FD count, limits, and usage percent on
  `GET /api/v1/health` — shipped in PR #4915 (see section 4). The two
  earlier attempts were closed rather than merged: PR #3033 (superseded
  by #4915, a clean reapplication onto current main) and PR #3036 (a
  `utilities/fd_monitor.py` FD circuit breaker — closed because its
  premise, a retry-driven "death spiral," did not match the real
  WAL/SHM-handle root cause already handled by the periodic pool
  disposal above; `fd_monitor.py` was never merged and does not exist).
  A type/inode breakdown (eventpoll vs pidfd vs WAL — the histogram the
  section-1 `/proc` snapshot produces) is feasible but deferred until an
  active leak hunt actually needs it.
- **Automated reproduction of #3816's eventpoll-FD leak in a test
  suite.** Explored in closed PR #3930 — a single-thread
  `asyncio.run(ainvoke)` loop against real Ollama does *not* reproduce
  eventpoll accumulation, because `asyncio.run` deterministically closes
  its loop's selector each call. Reliable reproduction would need
  sustained concurrent load (multi-worker harness over a shared loop).
  In-CI mock + no-network real `ChatOllama` tests in
  `tests/utilities/test_close_base_llm.py` already cover the close-chain
  introspection regressions; a load-shape reproduction is deferred
  until a future leak justifies the maintenance burden.
- **`app_logs` (ResearchLog) retention setting + scheduled cleanup
  job.** Identified in Round 9; the only audit finding that wasn't
  refuted but also isn't impactful enough today. *Trigger to do this
  work:* a user reports the SQLCipher DB growing >100 MB and
  complains about query slowdown, OR a self-hosted instance keeping
  research logs for >1 year sees DB bloat, OR the metrics dashboard
  starts noting research-detail page load slowdown traced to
  `app_logs` joins. *Implementation sketch:* add
  `logs.research_log_retention_days` to
  `defaults/default_settings.json` (default `0` = disabled, preserves
  current behavior; e.g. `30` to keep last 30 days). Extend the
  existing `BackgroundJobScheduler` in `scheduler/background.py`
  (which already runs `cleanup_inactive_users` hourly and
  `_reload_config` every 30 min) with a daily `_cleanup_old_research_logs`
  job that deletes `ResearchLog` rows older than the retention
  window. Skip rows belonging to favorited / starred researches if a
  flag exists. ~30 LOC + a regression test that inserts old rows,
  triggers the job, asserts old rows are deleted and recent ones
  survive. Add `changelog.d/<id>.feature.md`.
---

## Glossary

- **`_owns_llm`** — instance flag set in `__init__` to `True` when the
  class fetched its own LLM via `get_llm()`, `False` when an LLM was
  injected by the caller. Gates whether `close()` actually closes the
  LLM.
- **`safe_close(resource, name)`** — helper in `utilities/resource_utils.py`
  that calls `resource.close()` inside a try/except, logging on failure.
  Never raises. Used in every `finally` block.
- **`_ldr_closed`** — sentinel attribute set on inner httpx clients by
  `_close_base_llm` to make the function idempotent. Checked with
  `is True` (not truthy) so Mock objects without a `spec` don't trip
  the guard.
- **eventpoll FD** — Linux `a_inode` file descriptor type for
  `epoll_create`'d kernel objects. Each asyncio event loop registers
  one. Leaked AsyncClients hold them via the loop's selector.