# Resource cleanup in LDR This document captures how LDR manages process-level resources (DB connections, HTTP clients, file descriptors, threads) and the reasoning trail behind the current model. It exists because file-descriptor exhaustion has been a recurring class of bug in LDR, and the *journey* of fixing it — what's been tried, what worked, what was ruled out — is not reconstructable from `git log` alone. If you're contributing code that holds a network connection, a database session, an LLM client, or a thread, read this before adding `__del__`, `weakref.finalize`, or a context manager. --- ## Current model ### Database connections - **One shared per-user `QueuePool`.** No per-thread engines. Pool sizing: `pool_size=20`, `max_overflow=40`, with periodic `dispose()` every 30 minutes. - **SQLCipher is decrypted once per connection-open.** `PRAGMA key` takes ~0.2 ms; pool reuse keeps that off the hot path. - Engines are created at login, closed at logout (or process exit via the registered `atexit` shutdown). - Background threads (research workers, metric writers, news scheduler jobs) use the same per-user pool — they no longer maintain a separate thread-engine system. See [ADR-0004](../decisions/0004-nullpool-for-sqlcipher.md) for the QueuePool-vs-NullPool decision and PR #3441 for the per-thread-engine removal. ### LLM wrappers LDR wraps every LLM in `ProcessingLLMWrapper` (and optionally `RateLimitedLLMWrapper`) so that callers see a uniform interface and the project owns the close path: ``` caller -> ProcessingLLMWrapper.close() -> _close_base_llm(base_llm) in utilities/llm_utils.py -> for ChatOllama: sync httpx client (ollama.Client._client) .close() async httpx client (ollama.AsyncClient._client) .aclose() -> for ChatOpenAI / ChatAnthropic: no close (those use @lru_cache'd shared httpx clients) ``` Key invariants: - `ChatOllama` is the *only* provider where `_close_base_llm()` actually closes anything. ChatOpenAI and ChatAnthropic share LRU-cached httpx clients across instances; closing them would break other live LLMs. - Both `_client` (sync) **and** `_async_client` (async) are released — the async side is exercised by every `ainvoke()` call (langgraph agents, modular strategies). Closing only the sync side leaks the async transport per call (root cause of #3816). - The function is idempotent via an `_ldr_closed` sentinel on the inner httpx clients. - The async close uses `asyncio.run(client.aclose())` only when no event loop is currently running. When called from inside async code it skips and leaves the close to the loop's owner. ### Search engines - `BaseSearchEngine.close()` is the single entry point and **cascades** into `_preview_filters` and `_content_filters`. That cascade is what releases per-engine LLMs (e.g., `JournalReputationFilter.model`), SearXNG sessions, and other filter-held resources. - Search-engine cleanup happens at the per-research finally block in `web/services/research_service.py:run_research_process()` and at the programmatic API entry points in `api/research_functions.py`. - The `_owns_llm` flag pattern (introduced in #2712) tracks whether a filter or engine constructed its own LLM (and thus owns it) versus borrowed one from a caller (and must not close it). ### Thread lifecycle - `@thread_cleanup` (decorator on `run_research_process` and similar workers) ensures thread-local DB sessions are released even on abnormal exits. - `cleanup_current_thread()` is called from Flask teardown, the queue processor, the auth flow, and the RAG routes — six tier-1 paths in total. - Background threads are daemon threads; the process exit handles any thread that did not clean up gracefully. ### Conventions - **Use `safe_close(resource, "human name")`** from `utilities/resource_utils.py` for every cleanup. Never bare `.close()` in a `finally` (it can mask the original exception). - **Prefer `try/finally` over `__del__`**. Python doesn't guarantee finalization order at interpreter exit; `__del__` interacts subtly with reference cycles and `weakref`. - **Track ownership explicitly with `_owns_llm` (or analogous flag)** when a class accepts an injected resource that may or may not be its own. - **News fragments (`changelog.d/.bugfix.md`) are required for any user-visible cleanup behavior change** — see `changelog.d/README.md`. --- ## How to close X correctly | You're holding | Do this | | --- | --- | | A `ChatOllama` (raw or wrapped) | Call `wrapper.close()` in a `finally`, or pass to `safe_close(wrapper, "...")`. The wrapper chain handles both sync and async httpx clients. | | A search engine you constructed | `safe_close(engine, "...")` in `finally`. The engine's `close()` cascades into preview/content filters. | | A holder class with an LLM | Add a `close()` method, gate the LLM close on `self._owns_llm`, document who calls it. Don't add `__del__`. | | A long-lived service holder (news scheduler, etc.) | Wrap construction in `try/finally` at the cycle boundary. Don't store the LLM if you can recreate it cheaply. | | A DB session | Use `with get_user_db_session(username) as session:`. Don't bypass via `get_settings_manager(username=...)` without `owns_session=False` (see #3023). | | An asyncio event loop | Use the existing one. If you genuinely need a new one (background thread fallback), call `loop.close()` in a `finally` — see `news_strategy.py` for the reference pattern (post-#3018). | --- ## Anti-patterns These look reasonable but break specific things in this codebase: - **Adding `__del__` to a class with `close()`.** At interpreter exit the `logger`, `httpx`, and event-loop modules may already be torn down. `__del__` can run after them and raise. Use explicit close in a `finally` instead. - **Closing a shared httpx client.** ChatOpenAI / ChatAnthropic share one httpx pool across instances via `@lru_cache`. Closing it kills every other live LLM in the same process. The Ollama check in `_close_base_llm` exists exactly to gate this. - **Truthy idempotency sentinels on Mock objects.** `Mock()` without a `spec` auto-generates child Mocks for any attribute access, so `getattr(client, "_ldr_closed", False)` returns a truthy Mock and short-circuits the close. Always use `is True` / `is None` checks for sentinels — see the pattern in `_close_base_llm`. - **Skipping `super().close()` in a search-engine subclass.** `BaseSearchEngine.close()` is what cascades into preview/content filters. Override it without calling super and you leak every filter's resources (this was a Copilot finding on #3818). - **Treating `asyncio.run()` as safe inside an event loop.** It raises `RuntimeError` if called from a thread that already has a running loop. The pattern in `_close_base_llm` is: detect a running loop with `get_running_loop()`, skip the async close in that branch (the loop owner will close), only call `asyncio.run` in the no-loop case. --- ## History The FD-leak campaign spans roughly four months of iterative work. Each fix narrowed the remaining surface; each subsequent issue was found in a corner the previous wave hadn't touched. ### Wave 1 — initial leak inventory (Jan 2026) - **#1832, #1849, #1856, #1860** — first comprehensive sweep. Identified seven distinct leak sources: `auth_db` engine, `download_management` DB, search cache, subprocess zombies, HTTP sessions in `SemanticScholarSearchEngine` and `BaseDownloader`, Socket.IO threads. Established context-manager + `try/finally` patterns. Added a pre-commit hook to catch missing cleanup at commit time. ### Wave 2 — thread-local engine accumulation (Mar 2026) - **#2495** — diagnosed that Flask's teardown only cleaned the request-scoped `g.db_session` while a separate `_thread_engines` dict accumulated NullPool engines per thread, leaking ~3 FDs per request. Added `cleanup_current_thread()` across six tier-1 paths. - **#2591** — dead-thread engines (when threads crashed they left engines behind) plus `stream=True` socket holds in the generic downloader. Added a throttled dead-thread sweep, removed `stream=True`, raised the Docker ulimit from 1024 to 65536. ### Wave 3 — LLM wrapper lifecycle (Mar 2026) - **#2708** — diagnosed `ChatOllama` → `httpx.Client` chains with no `__del__`. With the news scheduler triggering 50–300 `quick_summary()` calls per hour, a 1024-FD container exhausted in 3–4 hours. Wrapped four programmatic API entry points in `try/finally` with explicit close. - **#2712** — extracted `close_llm()` to a shared utility. Added `close()` and `_owns_llm` to `NewsAnalyzer`, `HeadlineGenerator`, `TopicGenerator`, `JournalReputationFilter`, `DomainClassifier`, `GitHubSearchEngine`, `IntegratedReportGenerator`, `ElasticsearchSearchEngine`, and the benchmark graders. - **#2756** — wrapped bare `.close()` calls in `finally` blocks with `safe_close()` to prevent masking the original exception. - **#2732** — moved `close()` into `ProcessingLLMWrapper` and `RateLimitedLLMWrapper` directly; eliminated the standalone `close_llm()` free function. ### Wave 4 — DB session leaks + per-call patterns (late Mar / early Apr 2026) - **#3018** — `get_settings_manager(username=...)` was bypassing `g.db_session` and creating QueuePool sessions per-thread; live diagnostics showed 321 sockets allocated, only 66 in use. `DownloadService.close()` leaked the inner `SettingsManager` session. Also fixed `TopicBasedRecommender._create_recommendation_card()` (per-call LLM with no cleanup) and an `asyncio.new_event_loop()` in `news_strategy.py` that never closed. - **#3204** — test fixtures using `return` instead of `yield` left engines un-disposed. Migrated 8 test files to `yield` + `engine.dispose()`. ### Wave 5 — DB pool architecture (Apr 2026) - **#3340** — kept QueuePool but minimized FDs (`pool_size=1`, `max_overflow=2`, periodic `dispose()` every 30 min). - **#3337** (closed) — proposed switching SQLCipher engines to NullPool for zero persistent FDs. Superseded by #3441. - **#3441** — removed per-thread NullPool engines entirely (~2,100 lines of sweep logic deleted) and routed metrics through a single shared per-user QueuePool with bounded sizing (`pool_size=20`, `max_overflow=40`). - **#3477** — created [ADR-0004](../decisions/0004-nullpool-for-sqlcipher.md) capturing the final pool model and updated stale FD calculations across docs. ### Wave 6 — async client close (May 2026) - **#3818** (open, declined for merge) — proposed session-pooling around `safe_get`/`safe_post` to address #3816. The session refactor is reasonable in isolation, but the lsof in #3816 showed ~72% of leaked FDs as `a_inode [eventpoll]` selectors, not HTTP request sockets — pointing at async-client transports rather than `safe_get` callers (whose response bodies were already consumed). See [the PR comment](https://github.com/LearningCircuit/local-deep-research/pull/3818#issuecomment-4402290677) for the full reasoning. - **#3855** — extended `_close_base_llm()` to also close `ChatOllama._async_client` (the actual gap the lsof pointed to). Added the `IntegratedReportGenerator` close that was missing from the per-research `finally` block. Idempotency via `_ldr_closed` sentinels on the inner httpx clients. ### Wave 7 — async close inside a running loop (May 2026) - **#4047** — `_close_base_llm`'s async branch had a documented "skip if a loop is running; loop owner closes" path. **No loop-owner cleanup code existed anywhere in the project**, so when the close was called inside an active asyncio loop the inner `httpx.AsyncClient` (and its `epoll_create` FD) was silently abandoned. Reproduced in production: a v1.6.10 single-host Ollama container reached 1024 FDs with the /proc histogram showing **929 `anon_inode:[eventpoll]` (91%)** — the same FD class as #3816 but in a code path #3855's fix didn't cover. The fix runs the async close in a brief daemon thread that owns its own loop, so `asyncio.run(aclose())` works regardless of the caller's loop state. A bounded 5-second `join` keeps the cleanup from blocking shutdown when the Ollama server is unresponsive; on timeout `_ldr_closed` is left unset so a later call retries, and a WARNING surfaces so the situation is observable instead of silent. - **Healthcheck pidfd leak (same PR).** Dockerfile's `HEALTHCHECK CMD python -c "... urllib.request.urlopen(...)"` had no `timeout=` argument; Docker's 10s timeout SIGKILL'd the `sh -c` parent but the python child was reparented to PID 1 and hung forever, each surviving child holding a `pidfd` + TCP socket against the app. Same /proc dump showed **64 `anon_inode:[pidfd]` (6%)** from this. Adding `timeout=8` lets the child return/raise inside Docker's budget so it exits cleanly and gets reaped. #### Audit ledger — what the broader sweep checked The PR included a wide audit (50+ parallel exploration agents across seven rounds plus direct `/proc` inspection) to catch any other latent FD leak. To save the next contributor from re-running the same checks, here is the full ledger: ##### Checked and confirmed clean (no action needed) - **Non-Ollama LLM providers.** xAI, Google Gemini, OpenRouter, IONOS, LM Studio, llama.cpp HTTP, DeepSeek, OpenAI-compatible endpoint, plus OpenAI and Anthropic themselves. All extend `ChatOpenAI` or `ChatAnthropic`, which use `@lru_cache`'d shared httpx clients. `_close_base_llm`'s short-circuit on these classes is correct by design — closing them would brick every other live LLM in the process. - **HTTP session lifecycle.** Six instantiation sites checked (`PricingFetcher` aiohttp, `LDRClient` SafeSession, `BaseDownloader`, `SemanticScholarSearchEngine`, `MCPClient`, `CostCalculator`). All context-managed via `with` or owned by a class with a paired `close()` and `__exit__`. - **subprocess / pidfd.** Three call sites, all `subprocess.run()` (blocking). No `subprocess.Popen` paths anywhere in `src/`. No `ProcessPoolExecutor`. No FD leak surface beyond the healthcheck child, already addressed by the Dockerfile `timeout=8` change. - **asyncio event loops.** Zero raw `asyncio.new_event_loop()` outside safe `asyncio.run()` patterns. The historical leak in `news_strategy.py` (#3018) is still fixed. - **File handles.** All 37 `open()` call sites are inside `with`. Zero bare opens. `tempfile.NamedTemporaryFile` / `TemporaryDirectory` all context-managed. - **SocketIO connect/disconnect.** Non-disconnect handlers (`subscribe`, `unsubscribe`, `connect`) do not acquire DB sessions (an early-round agent claim that they did was refuted on re-read). The `__socket_subscriptions` dict is cleaned on disconnect. The PID-1 FD breakdown showed only 3 sockets out of 1024 — socket accumulation is not a contributor. ##### Flagged by audit, then verified NOT a real FD leak - **OllamaEmbeddings httpx (historical — current state covered in Wave 10 below).** At the time of this Wave-7 audit LDR imported the **deprecated** `langchain_community.embeddings.OllamaEmbeddings`, which used `requests.post()` per call — no persistent httpx client, no `_client` / `_async_client` attribute. Direct introspection: `[a for a in dir(e) if 'client' in a.lower()]` returned `[]`. Zero FDs per call. An audit agent confused this class with `ChatOllama`, which is a different class. The migration to `langchain_ollama.OllamaEmbeddings` predicted in the next subsection has since shipped (#4352/#4353) and the resulting FD-leak regression has been fixed — see Wave 10. - **`auth_db` and `journal_quality` engines escaping `shutdown_databases()`.** `auth_db` uses `QueuePool(pool_size=10, max_overflow=20)` and `journal_quality` uses `StaticPool` with `immutable=1`. Both are **bounded** and do not grow at runtime. Live `/proc` on the affected container showed only 21 SQLite-related FDs total on PID 1 — well below the ~91-FD ceiling these unmanaged engines could theoretically reach. The kernel reclaims FDs at process exit regardless of `engine.dispose()`, and SQLite WAL files auto-checkpoint on next open. Missing dispose at exit is hygiene, not a leak. - **`LibraryRAGService` in three RAG SSE endpoints.** `rag_routes.py:693, 1054, 1827` do construct the service outside the generator and never close it, **but** `LibraryRAGService.close()` only sets references to `None` — it releases no FDs. FAISS uses `pickle.load()` (not mmap); OllamaEmbeddings holds no FDs per the item above; the SentenceTransformer model+tokenizer mmaps are process-wide singletons. What gets delayed is ~50–200 MB of embedding-model RAM until GC. A memory-pressure question, not the eventpoll FD class this Wave addressed. - **Residual `pidfd` accumulation via Playwright fallback** — identified in a Round-8 follow-up after the eventpoll fix landed. Live `/proc` on the prerelease container showed ~29 pidfds steady state, growing ~3.6/hour, all targeting `Pid: -1` (children that had exited). Rate was stable during active benchmark execution, ruling out a per-task source. Eight parallel agents converged on the same chain: `_check_subscription` → `quick_summary` → `FullSearchResults.batch_fetch_and_extract` → `AutoHTMLDownloader` fallback to `PlaywrightHTMLDownloader._fetch_with_playwright`. Each `sync_playwright().start()` invokes `asyncio.create_subprocess_exec()` for the Node.js driver (opens a pidfd via Linux's `PidfdChildWatcher`); the driver then fails because Chromium is not installed in the production `ldr` Dockerfile stage (only `ldr-test` runs `playwright install --with-deps chromium`), and the asyncio child watcher does not promptly close the pidfd on the failed-child exit. CPython 3.14 was confirmed to not use pidfd in `subprocess.py` at all (`subprocess.run`/`Popen` use `waitpid(WNOHANG)` polling), so subprocess-based hypotheses were ruled out. **Fixed by PR #3971** (default `web.enable_javascript_rendering=false`): the fallback short-circuits before any subprocess is spawned, so no pidfd is opened. The PR was motivated by issue #3826 (confusing tracebacks); the FD-leak finding is the second motivation, surfaced here. ##### Minor findings (not steady-state leaks; worth knowing) - **Daemon threads without explicit shutdown.** `journal_reputation_filter.py` background fetcher, `log_utils.py` queue processor. All daemonized — reaped by the OS at process exit. Not steady-state leaks; no per-request growth. - **Abandoned-research thread on socket disconnect.** If a client closes the tab mid-research, the socket subscription is removed but the research thread keeps running until completion; `_active_research[research_id]` is not cleared on disconnect. Not an FD leak; potentially compute/memory waste if the user wanted the research to stop. Out of scope for the FD-leak story. #### Future-proofing note — `langchain_ollama.OllamaEmbeddings` migration (resolved in Wave 10) Status: **resolved**. The migration this note predicted shipped in #4352/#4353; the FD-leak regression it predicted then surfaced and was fixed in Wave 10 (see below). Kept here as the source of the prediction that the next contributor's audit can cross-reference. `langchain_community.embeddings.OllamaEmbeddings` was deprecated ("will be removed in langchain 1.0.0", per the import warning). Its replacement, `langchain_ollama.OllamaEmbeddings`, **does** carry `_client` and `_async_client` attributes — same shape as `ChatOllama`. Verified by direct introspection at the time of writing: ``` langchain_ollama.OllamaEmbeddings client attrs: ['_set_clients', 'async_client_kwargs', 'client_kwargs', 'sync_client_kwargs'] Has _client? True Has _async_client? True ``` The prediction was: once LDR migrates, the eventpoll FD leak class returns for embeddings unless `_close_base_llm` is called on embedding instances. The introspection turned out to be slightly different from expected — both clients are constructed *eagerly* by a Pydantic `@model_validator(mode="after")` in `langchain_ollama.embeddings.py`, so the leak fires per-instance regardless of whether the async path is exercised. Wave 10 contains the post-mortem and fix. ### Wave 10 — embeddings FD leak after langchain_ollama migration (June 2026) The migration predicted above shipped without the matching close-path generalization, exactly as feared. Verified by four independent agents: `langchain_ollama.OllamaEmbeddings(...)` eagerly constructs both a sync `ollama.Client` (→ `httpx.Client`) and an async `ollama.AsyncClient` (→ `httpx.AsyncClient` → one `epoll_create` FD) inside its `@model_validator(mode="after")` at `.venv/.../langchain_ollama/embeddings.py:295-315`. No `close()`, `aclose()`, `__del__`, or `weakref.finalize` exists on the new class or the underlying `ollama` / `httpx` clients, so dropping the Python reference does not release the FDs. `_close_base_llm` already handled the shape — its module-prefix checks (`type(...).__module__.startswith("ollama")` at `src/local_deep_research/utilities/llm_utils.py:97,114`) match `ollama.Client` / `ollama.AsyncClient` regardless of which langchain wrapper holds them. The function just wasn't called on embeddings instances — `LocalEmbeddingManager.close()` and `LibraryRAGService.close()` only nulled their `_embeddings` / `embedding_manager` references, relying on GC that would never run the close. Fix: route the close call through the existing manager lifecycle. `LocalEmbeddingManager.close()` now calls `_close_base_llm(self._embeddings)` before nulling. `LibraryRAGService.close()` now calls `self.embedding_manager.close()` before nulling — guarded by an `_owns_embedding_manager` flag so a caller-supplied manager (test fixtures, multi-service callers) stays under caller control. The `_close_base_llm` docstring is updated to acknowledge it also handles `OllamaLLM` and `OllamaEmbeddings`; no behaviour change, only documentation. Regression coverage lives next to the existing ChatOllama tests in `tests/utilities/test_close_base_llm.py` — `TestCloseBaseLLMRealOllamaEmbeddings` is the canary that fires if a future migration breaks the close path again. A follow-up PR (PR-B) hardens the `rag_routes.py` call sites that construct `LibraryRAGService` without a `with` block: 4 simple synchronous sites get a `with` wrap; 3 SSE-streaming sites have the construction moved *inside* the `stream_with_context` generator (a `with` at request-handler scope would close the service before the stream runs). A safety-net PR (PR-C) registers a `weakref.finalize` inside `OllamaEmbeddingsProvider.create_embeddings()` so that callers that bypass the manager — for example the programmatic-API examples migrated in #4399 — still get eventual cleanup at GC time. ### Round 9 — broader resource audit (May 2026) Once the FD-leak classes were closed, a follow-up audit looked for *other* slow-growth patterns that wouldn't trip the FD counters but could still degrade a long-running container: memory and cache growth, thread / asyncio Task / lock lifecycle, DB state hygiene beyond connections. Three parallel agents per round, two rounds (Round 1 hypothesis generation, Round 2 fact-check), captured here in verified form so the next contributor doesn't re-derive the same conclusions. #### Refuted (false positives from Round 1, verified in Round 2) - **`@cache` on `get_available_providers`** (was in `config/llm_config.py`; **removed in #4590**, so this no longer exists). Round 1 claimed unbounded cache growth if the function were called with differing `settings_snapshot` dicts. Round 2 verified: dicts are unhashable, so `@cache` would raise `TypeError` on them, not silently grow. In practice the call sites passed `settings_snapshot=None` (hashable, cardinality 1). Not a leak — and the function (a dead duplicate of the provider auto-discovery path) has since been deleted entirely. Kept here for the audit record. - **Thread-local Session identity-map growth** (`database/thread_local_session.py`). Round 1 claimed long-running research threads would accumulate ORM objects in the per-thread Session's identity map. Round 2 verified: SQLAlchemy's default `expire_on_commit=True` clears the identity map at every commit; the codebase commits periodically. Bounded by typical query volume, not unbounded by uptime. - **`token_usage` table unbounded growth.** Append-only per LLM call with no TTL or retention job. Round 2 verified: **feature by design**. Schema has compound time-series indexes (`idx_token_research_timestamp`, etc.); `/api/context-overflow` and `/metrics/api/metrics` explicitly query historical windows for cost analysis. The table is a permanent audit trail by intent. Adding retention would break the metrics dashboards. - **`search_calls` table unbounded growth.** Same shape and same verdict — compound time-series indexes confirm intentional design as a permanent search-analytics record. #### Fixed in this PR — three per-user lock dicts - **Three per-user lock dicts** — `_user_init_locks` and `_user_locks` are module-level dicts in `database/library_init.py` and `database/backup/backup_service.py` respectively; `_user_critical_locks` is an instance attribute on the `QueueProcessorV2` singleton in `web/queue/processor_v2.py`. Each stored one `threading.Lock` per username with no removal hook. Bounded ceiling (~296 bytes/entry × 3 dicts at 1000 users = ~900 KB), so not urgent — but easy to fix cleanly. The two module-level dicts now expose `pop_user_init_lock` / `pop_user_lock` functions; the queue processor exposes the equivalent as an instance method `queue_processor.pop_user_critical_lock`. A shared `_pop_per_user_locks(username)` helper in `connection_cleanup.py` calls all three with lazy imports and individual try/except (WARNING-level so dict accumulation is observable, matching the sibling scheduler-unregister error path). The helper is invoked unconditionally — outside the `close_user_database` try/except so it still runs when the DB close itself fails — in both the idle-connection sweeper (`connection_cleanup.py:cleanup_idle_connections`) and the logout / password-change paths (`web/auth/routes.py`). Tests in `tests/web/auth/test_connection_cleanup.py::TestPopPerUserLocks` cover the helper directly and through the idle-close path. #### Real but small (survives verification) - **`app_logs` (ResearchLog) table — no automatic retention.** Grows by ~100s-1000s of rows per research. Cleaned only via cascade-delete when the parent `Research` row is deleted manually. Unlike `token_usage` / `search_calls`, this table has no UI dashboard or time-series API consuming it — it's debug context for a specific research session, not an analytics record. For users who keep all research, logs accumulate indefinitely. See "Intentionally not done (deferred)" for the retention design when a symptom report justifies it. --- ## Debugging FD leaks — playbook for the next one When the next FD leak shows up (and there will be one, eventually), this section is the shortcut. It captures the actual diagnostic flow that worked across Waves 6 and 7 so a future contributor doesn't have to re-derive it from the symptom. ### 0. Symptoms that mean "investigate this as an FD leak" - Tracebacks like `OSError: [Errno 24] Too many open files`, typically from `selectors.DefaultSelector()` in werkzeug or `send_from_directory` in Flask. These are usually the *first* visible failure. - Browser-side MIME-type errors on static assets (`text/html` instead of `text/css` / `application/javascript`). These are downstream of FD exhaustion — Flask can't open the static file, returns an HTML 500, and the browser refuses to apply it because of `X-Content-Type-Options: nosniff`. - `High FD count (N) — approaching system limit` warnings from `web/auth/connection_cleanup.py` (fires at FD > 800 every 5-minute cleanup tick). - Container health turns `unhealthy` because the healthcheck `urlopen` hangs on a process that no longer has FDs to accept connections. ### 0a. Rule out first — local UI-test "fresh-user churn" false positive Before treating climbing FDs as a leak, confirm you are measuring the **single-CI-user** condition. A very convincing *false* FD leak appears when reproducing UI tests locally: - The Puppeteer harness (`tests/ui_tests/auth_helper.js` → `ensureAuthenticated`) logs in as the shared CI user `test_admin` when `CI=true`. If that login fails, it **falls back to registering a fresh `testuser_` per test**. The usual local trigger is `test_admin` getting *failed-login lockout-locked* after a few iterations. - Each fresh user opens its own per-user encrypted DB + engine. Those are disposed only on logout or the ~300s connection-cleanup sweep, so within one sub-300s shard run they accumulate and the server's FD count to `encrypted_databases/*.db(-wal/-shm)` climbs ~linearly (e.g. 0→90 per shard run, 0→533 over six runs). It looks identical to a real per-user connection leak. - It is **not** a server bug. In real CI the one working `test_admin` is reused → one engine → FDs bounded by the pool cap (pool_size 20 + max_overflow 40 = 60). Confirm by grepping the server log for many distinct `testuser_` engine opens, or by checking the username the leaked FDs' DB files belong to. Concretely: the **chat UI shards** (`chat-core`, `chat-lifecycle`) failing in CI were investigated as a per-user DB FD leak and traced *twice* to this artifact. Both shards pass locally in faithful CI mode with bounded FDs; their CI failures are runner **contention** (60s navigation timeouts on a heavily-loaded Docker runner), not a connection leak. Cross-verify the user identity before committing to a leak hypothesis. ### 1. Capture diagnostic state BEFORE restarting The single most important rule: **the snapshot does not survive a container restart**. Every minute spent on the live broken container is worth an hour of after-the-fact agent guessing. Save the diagnostic output to a host-side file first. #### One-shot host-side snapshot (works even when the container is FD-starved enough that `docker exec` can't fork) ```bash # Run on the Docker host. No docker exec required. P=$(docker inspect -f '{{.State.Pid}}' ) sudo bash -c " echo '=== Total FDs ===' ls /proc/$P/fd | wc -l echo '=== FD-type histogram (digits collapsed) ===' ls -l /proc/$P/fd | awk '{print \$NF}' \ | sed -E 's/\[[0-9]+\]/[N]/g; s/[0-9]{4,}/NUM/g' \ | sort | uniq -c | sort -rn | head -30 echo '=== Counts by category ===' printf 'socket: %s\n' \$(find /proc/$P/fd -lname 'socket:*' | wc -l) printf 'pipe: %s\n' \$(find /proc/$P/fd -lname 'pipe:*' | wc -l) printf 'eventpoll: %s\n' \$(find /proc/$P/fd -lname '*eventpoll*' | wc -l) printf 'pidfd: %s\n' \$(find /proc/$P/fd -lname '*pidfd*' | wc -l) printf 'WAL files: %s\n' \$(find /proc/$P/fd -lname '*-wal' | wc -l) printf 'SHM files: %s\n' \$(find /proc/$P/fd -lname '*-shm' | wc -l) printf '.db files: %s\n' \$(find /proc/$P/fd -lname '*.db' | wc -l) " | tee /tmp/ldr-fd-snapshot.txt ``` Why host-side: reading the container's PID 1 FDs from inside the container requires the same UID that started PID 1. The Dockerfile entrypoint runs as root then `setpriv`s to `ldruser`, so the `docker exec` shell (ldruser) cannot `readlink` PID 1's FDs even though it can count them. Host root via `sudo` sidesteps the UID check. #### Inside-container alternative (if the host is locked down) ```bash docker exec --user 0 sh -c '...same body...' ``` `--user 0` runs the exec'd shell as root inside the container, sidestepping the same UID restriction. ### 2. The lookup table — FD type → likely source | Dominant FD type | Likely source | Diagnostic deep-dive | |-----------------------------|---------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------| | `anon_inode:[eventpoll]` | `asyncio` event loop or `httpx.AsyncClient` selector. Each leaked async client = +1. | Grep `asyncio.create_subprocess`, `httpx.AsyncClient`, `_async_client`, `ainvoke`. See Wave 6, Wave 7. | | `anon_inode:[pidfd]` | `asyncio.create_subprocess_*` or `multiprocessing.Process` (uses `pidfd_open` on Linux). | Read `/proc/PID/fdinfo/N` for each pidfd; the `Pid:` line shows the target (`-1` = child already exited). | | `socket:*` (lots) | HTTP keep-alive, SSE streams, SocketIO connections. | Cross-reference with `/proc/PID/net/tcp` states; check Round 7 R7A8 patterns. | | `pipe:*` (lots) | `subprocess.run`/`Popen` with `stdout=PIPE`, multiprocessing IPC, loguru queue. | Check `subprocess.run` sites and APScheduler executor type. | | `REG` `*-wal` / `*-shm` | SQLCipher in WAL mode. Each pooled connection holds ~3 FDs. | See ADR-0004. If growing without bound, the periodic `engine.dispose()` is silently failing. | | `REG /data/*.db` (lots) | Plain SQLite connections from an engine without bounded pool. | Audit `create_engine` sites (R7A6 caught two unmanaged ones). | | `REG /home/...mmap...` | Memory-mapped model weights or FAISS indexes — usually process-wide singletons (not leaks). | Check whether the count grows per request. If yes → real leak. | ### 3. Pinpointing the source for a specific FD type #### Eventpoll `anon_inode:[eventpoll]` always comes from `EpollSelector` — created by every asyncio loop and every `httpx.AsyncClient`. Grep: ``` grep -rn 'asyncio.create_subprocess\|httpx.AsyncClient\|_async_client' src/ ``` Then check whether each site explicitly closes the client. The Wave 7 fix to `_close_base_llm` is the reference pattern for "close async httpx even when called inside a running loop." #### Pidfd Pidfds expose their target PID via fdinfo: ```bash # Run inside the container (or via docker exec --user 0): for fd in $(ls /proc/1/fd 2>/dev/null); do link=$(readlink /proc/1/fd/$fd 2>/dev/null) case "$link" in *pidfd*) tpid=$(awk '/^Pid:/ {print $2}' /proc/1/fdinfo/$fd 2>/dev/null) if [ "$tpid" -gt 0 ] 2>/dev/null; then cmd=$(tr '\0' ' ' < /proc/$tpid/cmdline 2>/dev/null | cut -c1-80) echo "fd=$fd alive pid=$tpid : $cmd" else echo "fd=$fd ORPHAN (child exited; pidfd not closed)" fi ;; esac done ``` A high "ORPHAN" count = something called `asyncio.create_subprocess_*` or `multiprocessing.Process`, the child exited, but the pidfd in the parent was never closed. Common in Round-8: Playwright's Node.js driver subprocess failing because Chromium isn't installed in the production image. **Note:** CPython 3.14's `subprocess.py` does not use pidfd at all (`waitpid(WNOHANG)` polling instead). So pidfds in a 3.14 process necessarily come from asyncio or multiprocessing, not from `subprocess.run` / `Popen`. #### Syscall-level pinpointing with bpftrace (mysterious cases) When the source isn't obvious from the FD type, `bpftrace` can record the Python stack of every relevant syscall on the live process. This would have caught the Playwright leak in seconds instead of two rounds of agent exploration. Requires kernel headers and `bpftrace` installed on the host (NOT the container — bpftrace runs in host kernel space and can target a host PID by number): ```bash # Find host-side PID of container's PID 1 P=$(docker inspect -f '{{.State.Pid}}' ) # Trace every pidfd_open syscall, grouped by user-stack: sudo bpftrace -e "tracepoint:syscalls:sys_enter_pidfd_open /pid == $P/ { @[ustack(perf)] = count(); }" # Same idea for epoll_create / epoll_create1 (eventpoll FDs): sudo bpftrace -e "tracepoint:syscalls:sys_enter_epoll_create1 /pid == $P/ { @[ustack(perf)] = count(); }" ``` Let it run for a minute, then Ctrl-C; you get a histogram of every unique stack that triggered the syscall, ranked by frequency. The hot stacks are your culprits. Works for any syscall — useful future candidates: `socket`, `inotify_init1`, `timerfd_create`, `memfd_create`. #### WAL/SHM `engine.dispose()` is expected to release these. If the count climbs across the periodic 30-minute dispose cycles, the dispose is silently failing. The observability commit (f86c3f7af) elevates dispose failures to WARNING — check the logs for `Error disposing engine for `. ### 4. Existing instrumentation already in the codebase - **`_count_open_fds()`** in `src/local_deep_research/web/auth/connection_cleanup.py` — fast `/proc/self/fd`-based counter with macOS fallback. Reusable. - **`Resource monitor: open_fds=…`** debug log line in `connection_cleanup.py`, fires every 5-minute cleanup tick. - **`High FD count (N)` WARNING** in `connection_cleanup.py` when FDs exceed 800. The single most useful production signal. - **`GET /api/v1/health` resource diagnostics** (PR #4915) — for *authenticated* callers the response carries a `resources` block (`fd_count`, `fd_soft_limit`, `fd_hard_limit`, `fd_usage_percent`, `thread_count`) and flips `status` to `"warning"` above 70% FD usage. This is the live, queryable form of the `_count_open_fds()` log signal — `curl` it during a leak hunt instead of grepping container logs. It returns counts only (never fd targets), so no open file paths or socket peers are exposed. Anonymous callers (the Docker healthcheck) get only the basic `status`/`message`/`timestamp`. - **In-CI FD-growth canaries** in `tests/utilities/test_close_base_llm.py`. These run on every PR: - `TestCloseBaseLLMRealHttpxAsync::test_no_fd_growth_across_repeated_close_cycles` — guards the eventpoll FD class against Wave-6-shaped regressions. - `TestCloseBaseLLMRealHttpxAsync::test_no_fd_growth_when_closed_inside_running_loop` — guards the Wave-7-shaped in-running-loop skip regression. - `TestAsyncioSubprocessFDBaseline::test_no_fd_growth_across_asyncio_subprocess_cycles` — guards the pidfd FD class against the child-watcher leak shape. - `TestAsyncioSubprocessFDBaseline::test_no_fd_growth_when_subprocess_fails_to_exec` — pins the *exact* Wave-7-pidfd shape (failed exec, child watcher must still clean up). Catches platform-level regressions in Python's asyncio child watcher. All four use `_open_fd_count()` (also in that file) which reads `/proc/self/fd` on Linux with an `RLIMIT_NOFILE` fallback on macOS. Slack is +2 FDs across 5–10 iterations. A real per-cycle leak would blow past that. ### 4a. Development-time detection (catch leaks at test time) Production /proc inspection catches leaks **after** they ship. The cheapest catch is to make Python itself complain at test time. Three Python features cooperate to surface unclosed resources during a normal test run — none of them were on by default during Waves 6 and 7, which is part of why those leaks made it to production. **`PYTHONASYNCIODEBUG=1` plus `-W default::ResourceWarning`.** When asyncio debug mode is on, unclosed transports/coroutines emit a `ResourceWarning` at GC time. The `-W` filter makes Python actually display them. Together they would have caught the Wave 7 in-running-loop skip: every leaked `httpx.AsyncClient` produces a visible warning the first time the GC sweeps after the test fixture exits. From [the asyncio dev docs](https://docs.python.org/3/library/asyncio-dev.html): > When a transport is no longer needed, call its `close()` method to > release resources. ... If a transport or an event loop is not closed > explicitly, a `ResourceWarning` warning will be emitted in its > destructor. To enable in `pyproject.toml` `[tool.pytest.ini_options]`: ```toml filterwarnings = [ "default::ResourceWarning", ] env = [ "PYTHONASYNCIODEBUG=1", ] ``` Or in CI for a one-off check: ```bash PYTHONASYNCIODEBUG=1 python -W default::ResourceWarning -m pytest tests/ ``` For a CI gate that **fails** on any leak (more aggressive — use only on a targeted subset of tests, not the whole suite, because third-party libraries also emit ResourceWarning): ```toml filterwarnings = [ "error::ResourceWarning", ] ``` **`python -X dev`.** Enables Python's dev mode, which turns on a bundle of safety checks including ResourceWarning display, asyncio debug mode, and warnings as default. Cheap one-flag alternative for local development; not recommended in production (overhead). ```bash python -X dev -m pytest tests/ ``` **`psutil` for portable FD counting in tests.** Our in-codebase `_count_open_fds` uses `/proc/self/fd` (Linux-fast path, macOS fallback). `psutil` is the cross-platform alternative many other projects use: - `psutil.Process().num_fds()` — Linux/BSD only; same number as our helper. - `psutil.Process().open_files()` — list of named files; gives the paths for `REG`-type FDs (e.g., `/data/*.db-wal`). - `psutil.Process().connections(kind='all')` — sockets visible to the process, with state and remote address. These are useful in unit tests when you want to assert "no new file of pattern X is open after the close path runs," and they work on the macOS dev environments without `/proc`. **For tracking which Python object holds a leaked FD: `tracemalloc` + `objgraph`.** Not FD tools per se, but when a leak is reproducible, take a `tracemalloc` snapshot before and after the suspect operation and diff — the new allocation is usually the wrapper holding the FD. `objgraph.show_backrefs([leaked_obj])` then renders the reference chain keeping it alive. Both are pure-Python and zero-dependency. ### 5. Why we don't have an automated FD-growth test in CI Several reasons, weighed during Wave 6 and Wave 7: - **Per-request FD growth is hard to assert.** Many legitimate request paths transiently open and close FDs; a noisy delta is the norm. Distinguishing "leak" from "in-flight" requires a stable quiescent state, which a CI test doesn't naturally provide. - **The CI environment spawns its own subprocesses.** pytest, coverage, gunicorn workers (for some test variants), gh-runner cleanups — all add their own FDs that pollute the count. - **PID-namespace differences between CI and prod.** Counts you observe in a CI container's /proc are not directly comparable to a production container's /proc; the subprocess sources differ. - **The actual leaks have been "slow drip" patterns** that need hours of uptime to surface. Wave 6's eventpoll leak took multiple hours of `ainvoke` calls to reach the 1024 cap. CI can't run for hours per PR. What works instead: 1. **Per-leak unit-level regression tests.** Each fix in Waves 1-7 landed with a targeted test that exercises the specific close path (e.g. `tests/utilities/test_close_base_llm.py::test_no_fd_growth_when_closed_inside_running_loop`). These are fast, deterministic, and run on every PR. 2. **Opt-in manual smoke suite** (`RUN_MANUAL_SMOKE=1`) for the end-to-end "run-the-cycle-N-times-and-count" pattern, used during investigation but not on every CI run. 3. **Production /proc inspection** when a leak is suspected — the playbook above. Faster than CI for the long-drip patterns. If you want to add a long-run CI job, the right shape would be a **nightly** workflow (not per-PR) that: 1. Builds the production Docker image. 2. Starts it with a synthetic user account and ~5 news subscriptions. 3. Lets it idle for 20-30 minutes. 4. Runs the host-side snapshot script above. 5. Asserts `total FDs < N` and `eventpoll < M` and `pidfd < K`, where the thresholds are tuned for the steady-state ceilings the codebase intentionally permits (auth_db pool, etc.). That would have caught Waves 6, 7 in a single nightly cycle instead of through a user crash report. The reason it doesn't exist yet is cost (a half-hour idle job per night per platform) and the lack of a clear baseline; the Round-8 finding is the moment to consider adding one if you want to invest the maintenance time. ### 6. Lookup: which Wave fixed which leak class | FD class | Wave / PR | Root mechanism | |------------------------|------------------------|---------------------------------------------------------------------------| | `eventpoll` | Wave 6 #3855 + Wave 7 #4047 | ChatOllama `_async_client` not closed (Wave 6) → also not closed when called inside a running loop (Wave 7). | | `pidfd` from healthcheck | Wave 7 #4047 | `urlopen` no `timeout=` → child hangs → reparented to PID 1 with pidfd held. | | `pidfd` from Playwright fallback | Round 8 / #3971 | Production image lacks Chromium binary; Playwright invocation opens pidfd then fails. | | WAL/SHM accumulation | Wave 5 / ADR-0004 | SQLCipher+WAL leaks handles on out-of-order close; periodic `engine.dispose()` resets the pool. | | Per-thread engine FDs | Wave 5 #3441 | Removed per-thread `NullPool` engines entirely; shared per-user `QueuePool`. | | HTTP session sockets | Wave 1 / Wave 3 | `SafeSession` / `BaseDownloader` close-in-`finally` discipline. | | `asyncio.new_event_loop` | Wave 4 #3018 | Replaced manual loop creation with `asyncio.run()` in `news_strategy.py`. | Use this table to skip the rediscovery step the next time a specific FD type dominates a snapshot. --- ## Intentionally not done (deferred) These showed up during planning and were deliberately *not* done. If they get rediscovered as "missing work" by future contributors, please reference this section first. - **`weakref.finalize` defense-in-depth on the LLM wrappers.** Designed and verified safe (no `__del__` conflicts, `__getattr__` doesn't intercept `_finalizer`, no reference cycles). Deferred until a fourth wave of "missed close" leaks justifies adding a new pattern that future contributors must understand. Current explicit-close discipline has held since #2712 / #2732 / #3018. - **LLM caching in `get_llm()`.** Bounding total `ChatOllama` instances to N=distinct configs would make leak shapes architecturally impossible. Orthogonal optimization, deferred — adds complexity around settings invalidation and multi-tenant isolation. - **Pre-commit hook flagging `get_llm()` callers without `close()`.** Useful in principle, deferred — high false-positive risk (caller-passed LLMs, lazy-init holders, factory-returned LLMs all legitimately don't close). Needs a careful design. - **Per-FD-type/inode breakdown on the health endpoint.** The basic version — aggregate FD count, limits, and usage percent on `GET /api/v1/health` — shipped in PR #4915 (see section 4). The two earlier attempts were closed rather than merged: PR #3033 (superseded by #4915, a clean reapplication onto current main) and PR #3036 (a `utilities/fd_monitor.py` FD circuit breaker — closed because its premise, a retry-driven "death spiral," did not match the real WAL/SHM-handle root cause already handled by the periodic pool disposal above; `fd_monitor.py` was never merged and does not exist). A type/inode breakdown (eventpoll vs pidfd vs WAL — the histogram the section-1 `/proc` snapshot produces) is feasible but deferred until an active leak hunt actually needs it. - **Automated reproduction of #3816's eventpoll-FD leak in a test suite.** Explored in closed PR #3930 — a single-thread `asyncio.run(ainvoke)` loop against real Ollama does *not* reproduce eventpoll accumulation, because `asyncio.run` deterministically closes its loop's selector each call. Reliable reproduction would need sustained concurrent load (multi-worker harness over a shared loop). In-CI mock + no-network real `ChatOllama` tests in `tests/utilities/test_close_base_llm.py` already cover the close-chain introspection regressions; a load-shape reproduction is deferred until a future leak justifies the maintenance burden. - **`app_logs` (ResearchLog) retention setting + scheduled cleanup job.** Identified in Round 9; the only audit finding that wasn't refuted but also isn't impactful enough today. *Trigger to do this work:* a user reports the SQLCipher DB growing >100 MB and complains about query slowdown, OR a self-hosted instance keeping research logs for >1 year sees DB bloat, OR the metrics dashboard starts noting research-detail page load slowdown traced to `app_logs` joins. *Implementation sketch:* add `logs.research_log_retention_days` to `defaults/default_settings.json` (default `0` = disabled, preserves current behavior; e.g. `30` to keep last 30 days). Extend the existing `BackgroundJobScheduler` in `scheduler/background.py` (which already runs `cleanup_inactive_users` hourly and `_reload_config` every 30 min) with a daily `_cleanup_old_research_logs` job that deletes `ResearchLog` rows older than the retention window. Skip rows belonging to favorited / starred researches if a flag exists. ~30 LOC + a regression test that inserts old rows, triggers the job, asserts old rows are deleted and recent ones survive. Add `changelog.d/.feature.md`. --- ## Glossary - **`_owns_llm`** — instance flag set in `__init__` to `True` when the class fetched its own LLM via `get_llm()`, `False` when an LLM was injected by the caller. Gates whether `close()` actually closes the LLM. - **`safe_close(resource, name)`** — helper in `utilities/resource_utils.py` that calls `resource.close()` inside a try/except, logging on failure. Never raises. Used in every `finally` block. - **`_ldr_closed`** — sentinel attribute set on inner httpx clients by `_close_base_llm` to make the function idempotent. Checked with `is True` (not truthy) so Mock objects without a `spec` don't trip the guard. - **eventpoll FD** — Linux `a_inode` file descriptor type for `epoll_create`'d kernel objects. Each asyncio event loop registers one. Leaked AsyncClients hold them via the loop's selector.