# Main-thread stall budget — extraction & resolution follow-up

**Status: IMPLEMENTED** (same branch as the #1212 tail fix — attribution runs
promoted "suspects" to proven culprits fast enough to justify shipping
together). What landed, per suspect:

- **Post-index maintenance — the proven killer, not on the original suspect
  list.** The first full kernel `init` on the FIXED tail completed every
  synthesis pass (cFnPtr alone ran 433s at default heap, yielding throughout)
  and was then SIGKILLed by the default-window watchdog at
  `db.runMaintenance()`: `PRAGMA optimize` + `wal_checkpoint(PASSIVE)` over a
  4.2GB DB with a 593MB WAL is minutes of synchronous IO on 2 cores.
  `runMaintenance` now runs on a worker thread with its own connection
  (checkpointing from a second connection is standard; `PRAGMA optimize`
  persists stats in sqlite_stat tables), with a bounded in-line fallback that
  skips the checkpoint (close() checkpoints after the CLI disarms the
  watchdog).
- **Per-file store commits:** `storeExtractionResult` chunks its node/edge/ref
  inserts (2,000 rows) with time-budgeted yields between; the ordered-commit
  pump serializes async stores on a promise chain (preserving the #1015
  file-order determinism invariant) and its backpressure now also waits on the
  commit chain so the parse buffer stays bounded.
- **Resolver warm-up:** `warmCachesYielding` streams the DISTINCT name set
  with yields (the sync `warmCaches` stays for non-async callers). The 28.2s
  `sync` stall dropped to ≤4s total across the whole sync.
- **Resolution batch-tail:** edge inserts and keyed deletes run in 1,000-row
  sub-transactions with yields between (crash semantics unchanged — the batch
  was already several transactions, and #1187's sweep re-resolves leftovers).
- **Scan:** attributed (phase timings now in the code, `[phase-timing]` on
  `CODEGRAPH_SYNTH_TIMINGS`) — it is the synchronous git enumeration
  (`getGitVisibleFiles`/`collectGitFiles`), NOT a hash loop. See "Accepted
  residuals" below for why it was left synchronous.

**Verification:** full-graph parity (every node id + edge, sorted dump diff)
byte-identical on fresh redis and vim indexes, baseline vs fixed; full test
suite green; kernel `sync` worst stall 28.2s → ~4s; ES synthesis tail worst
stall ≤2.7s.

**Acceptance gate PASSED:** fresh full kernel `init` (70,129 indexed files,
2,048,673 nodes / 6,402,391 edges) completed in **27m 8s** on the 2-core/6GB
container at Node's default heap with the default 60s watchdog — `EXIT 0`,
identical node/edge counts to the pre-fix partial runs, maintenance 48.8s
off-thread with the WAL fully checkpointed (0 bytes). v1.3.0 could not finish
this repo at all (OOM at default heap; watchdog kill at the maintenance step
even with the tail fixed). Post-run, the one genuine synchronous span the run
exposed — the merged synthesized-edge insert (~275k rows, 20.2s in one
transaction) — was chunked (2k rows + yield) like the rest; redis parity
re-verified byte-identical after.

## Accepted residuals (measured, documented, deliberately not fixed)

- **Git enumeration (scan): 2.2–10.5s** single sync span on ~95k-file repos.
  Fixing it means async-ifying `collectGitFiles`' recursive gitlink/submodule
  logic (#1038/#1065) or forking sync/async variants — high regression risk
  for a CPU-bound span ~6× under the watchdog window even on a 2-core
  container (its cost does not get the Windows/Defender per-file-IO
  multiplier; it scales with CPU only).
- **End-of-sync aggregates: ~2.7s** (count recompute / vocab backfill on a
  4.2GB DB).
- **Warm-up first chunk: ~2.6s** — the DISTINCT name scan's initial sort
  chunk before the first cursor row arrives; the rest of the scan yields.
- **Worker-contention timer lag on tiny containers** — with 2 cpuset cores,
  the off-thread checkpoint (and the parse pool early in the run) can delay
  main-loop timers 15–20s even though the main thread executes nothing. The
  stall monitor and the watchdog heartbeat both measure timer latency, so on
  a ~1-core box a long checkpoint could still starve heartbeats; if that ever
  reproduces, the mitigations are a niced worker or heartbeat-side allowance,
  not more yields.

If any of these ever shows up in a real watchdog kill, the async-refactor
shape for the scan is: thread a `MaybeYield` through `collectGitFiles`'
per-line loop and make `getGitVisibleFiles` async, keeping `scanDirectory`
(sync) on the walk fallback only.

---

*Original plan below, kept for the record.*

## Context

The #850 liveness watchdog SIGKILLs the indexer when its event loop stalls past
the window (default 60s). #1091 → #1122/#1137 → #1212 each moved the fix deeper:
per-batch yields, per-ref yields, then (with #1212) yields + streamed queries +
language gates across the entire dynamic-edge synthesis tail, which eliminated
the 14–57s single-pass stalls and the two whole-graph OOMs.

While validating #1212 with an event-loop stall monitor over *full* `init` runs
(Linux kernel, 70k indexed files / 2.05M nodes, 2-core 6GB container; and
llvm-project, 180k tracked files, macOS), the phases **before** the synthesis
tail showed recurring single stalls that nothing currently yields through:

| Run | Phase | Observed single stalls |
|---|---|---|
| kernel (2 cores) | initial scan (t+14s, t+22s) | 5.1s, 10.5s |
| kernel (2 cores) | extraction (t+980–1080s) | 3.0–3.3s |
| kernel (2 cores) | extraction→resolution boundary (t+1354s) | 8.5s |
| llvm (mac, fast) | extraction / early resolution (t+1000–1320s) | 5–14s, recurring |
| kernel (2 cores) | `codegraph sync` on the same DB (110 files) | **28.2s** (single stall) |

None of these approaches 60s on the tested hardware, and none are regressions —
they pre-date #1212. But the #1212 pattern (Windows NTFS + Defender, small VMs)
multiplies per-file and per-transaction costs several-fold, and 14s × a few-fold
is a watchdog kill. These are the spans that will produce the *fourth* iteration
of this bug class if left unmeasured.

## Suspects (with code locations)

1. **Per-file store commits on the main thread** —
   `ExtractionOrchestrator.storeExtractionResult` (`src/extraction/index.ts:2065`)
   runs one synchronous transaction per file (`insertNodes` + `insertEdges` +
   unresolved-ref batch + FTS triggers). A giant generated file (llvm has
   many multi-MB generated `.inc`/`.cpp`) inserts tens of thousands of nodes in
   one unyielding span. The parse pool (#1015) moved *parsing* off-thread; the
   *commit* is still a single main-thread block per file.
2. **Resolver cache warm-up** — `warmCaches` (`src/resolution/index.ts:319`)
   calls `getAllNodeNames()` (`src/db/queries.ts:1879`, `SELECT DISTINCT name`
   over the whole node table) plus `getAllFilePaths()` synchronously. On the
   kernel's 2M-row table the DISTINCT alone is seconds; it is the prime suspect
   for the 8.5s boundary stall and the 28.2s `sync` stall (sync also enters
   resolution via the orphan sweep, #1191).
3. **Resolution batch-tail DB ops** — between the per-ref yields,
   `resolveAndPersistBatched` (`src/resolution/index.ts`) runs per-5000-ref
   synchronous spans: `insertEdges(batch)`,
   `deleteSpecificResolvedReferences` × 2 (a 5000-statement transaction), and
   `getUnresolvedReferencesCount()`. On a multi-GB DB each is a solid block.
4. **Initial scan** (kernel t+14/22s) — file enumeration + content hashing
   before extraction starts. Unattributed; measure before assuming.

## Diagnosis plan (before any fix)

Extend the env-gated timing that located #1212 (`CODEGRAPH_SYNTH_TIMINGS`) to
the suspects — or add a sibling `CODEGRAPH_PHASE_TIMINGS` — so each suspect
logs spans >250ms with a label:

- wrap `storeExtractionResult` (log file path + node count when slow — this
  also identifies the offending generated files),
- wrap `warmCaches` (split `getAllNodeNames` vs `getAllFilePaths`),
- wrap the three batch-tail ops in `resolveAndPersistBatched`,
- wrap the scan phase.

Re-run the stall monitor + timings on the two existing indexes (assets below).
Attribution first: the fix for each suspect is different, and #1180 showed the
first guess is often wrong.

## Fix sketches (per suspect, once confirmed)

1. **Chunked per-file commits:** split a file's node/edge/ref inserts into
   bounded sub-transactions (e.g. 2–5k rows) with `maybeYield()` between chunks.
   **Invariant to preserve:** files must still commit in scan order, whole-file
   at a time from the resolver's perspective (#1015 — resolution disambiguates
   same-named candidates by insertion order; chunking *within* one file keeps
   the order stable). The existing index-completeness marker (`index_state`)
   already covers a mid-file kill.
2. **Yielding warm-up:** stream `SELECT DISTINCT name` with a cursor
   (`stmt.iterate()`), building the Set with a periodic `maybeYield()` — an
   async `warmCachesYielding()` used from the async entry points
   (`resolveAndPersistBatched`, the sync path), leaving the sync `warmCaches()`
   for callers that can't await. Memory is unchanged (the Set already exists).
3. **Chunked batch-tail ops:** split the keyed-delete transaction and the edge
   insert into sub-transactions with yields between, same pattern as (1).
   `getUnresolvedReferencesCount` is an indexed aggregate; leave it unless
   timing says otherwise.
4. **Scan:** measure first; likely chunk the hash loop with yields.

## Acceptance criteria

- Instrumented full `init` on the kernel index (2-core/6GB container) and
  llvm-project shows **no single event-loop stall > ~2s** in any phase.
- `codegraph sync` on the kernel DB shows the same bound (kills the 28.2s span).
- Graph parity: byte-identical node/edge sets on a re-index of at least
  elasticsearch + redis (the #1212 parity harness in the session scratchpad
  automates the synthesized-edge half; extraction parity = compare
  `getNodeAndEdgeCount` + a sorted node-id dump).
- No end-to-end throughput regression beyond noise (< ~5%) on the same runs —
  chunked transactions can slow bulk inserts; measure, don't assume.

## Repro assets (from the #1212 investigation, 2026-07-08)

- Docker container `cg1212` (2 cores / 6GB, node:22-bookworm) with the Linux
  kernel cloned at `/work/linux` and its 4.2GB index.
- llvm-project (180,074 files) + elasticsearch (45k) + redis + vim clones with
  indexes in the session scratchpad.
- `stall-monitor.cjs` (preload; logs event-loop gaps >1s with timestamps),
  `synth-only.mjs` / `synth-watchdog.mjs` (drive resolution+synthesis directly
  against an existing index — ~2 min iteration instead of a 40-min re-index),
  `parity.mjs` (synthesized-edge set differ).
- The #1091 methodology note applies: a real CLI run at a lowered
  `CODEGRAPH_WATCHDOG_TIMEOUT_MS` is the authoritative kill/no-kill test.