kenn-io--agentsview/docs/internal/background-sync-efficiency.md

# Background Sync Efficiency

This document records the runtime and cost-model contracts that keep active
session writers from turning into unbounded background work. The implementation
lives primarily in `internal/sync/watcher.go`,
`internal/parser/codex_cursor.go`, and the provider incremental path in
`internal/sync/engine.go`.

## Watcher runtime contract

- Production watcher events use a 500 ms first-event batching window. Later
  events join the pending set without postponing its deadline.
- Watcher callback start times are at least five seconds apart. One worker runs
  callbacks serially while the fsnotify loop continues draining events and
  errors, so a long sync cannot block event intake or overlap another sync.
- An idle watcher has no running timer or ticker. The first relevant event
  creates the next one-shot timer.
- Each pending or in-flight batch retains at most 8,192 unique paths and 2 MiB
  of path-string bytes. At most one batch is in flight while one more
  accumulates. Entry count separately bounds map and slice overhead.
- Exceeding either batch limit replaces its individual paths with one explicit
  full-sync marker. The worker clears event-sensitive freshness caches and
  force-verifies every discovered file under the same serialization,
  dispatch-floor, cancellation, and shutdown rules, so overflow bounds memory
  without losing same-stat source changes.
- Shutdown discards pending paths and waits only for an already-running
  callback. Normal discovery on the next startup recovers discarded changes.

## Codex append cursor contract

The Codex provider factory owns one in-memory cursor cache shared by its
per-source provider instances. Its lifetime is the sync engine's lifetime; it is
not persisted across daemon restarts.

Each cursor is keyed by the cleaned physical path, exact safe byte offset, and
inode/device identity where the platform exposes them. The cache is an LRU
bounded to 256 entries and 2 MiB of estimated retained data. It contains compact
continuation state only: never parsed messages, raw JSON lines, complete prompt
bodies, file contents, or open file descriptors.

The database's committed source offset is the cursor commit token. A parse may
stage old- and new-offset entries, but only an exact offset from the next
database request is eligible. A failed database write therefore retries from the
old cursor; an unreachable staged entry is eventually evicted.

Every nonzero resume offset must immediately follow a newline. Incremental
parsing commits only complete, valid, newline-terminated JSONL records. Partial
records and valid JSON at a newline-less EOF are retried or force a full parse;
they are never published as safe cursor boundaries.

Truncation, known file-identity replacement, manual or project refreshes,
`session_index.jsonl` title changes, and records that retroactively update
stored messages all fall back to an authoritative full replacement. Safe
incremental writes preserve the index-folded mtime and lifecycle-derived
termination status alongside message and token aggregates.

## Append-only limitation

Cursor correctness assumes that growth is append-only. A same-inode file can
grow after bytes inside its already-committed prefix have been rewritten. Size,
identity, and boundary checks do not detect that case, and the current
full-source fingerprint is not compared with a separately verified stored prefix
before incremental parsing. Closing this gap would require rolling hash state or
explicit prefix verification and remains deferred.

## Cost model and regression evidence

A warm Codex cursor makes continuation-state parsing scale with appended records
rather than transcript history. End-to-end append sync is still O(file): the
provider's `Fingerprint` hashes the complete source and the engine's
`ComputeFileHashPrefix` hashes through the newly committed offset.

- `BenchmarkCodexIncrementalCursor` in `internal/parser` compares cold prefix
  reconstruction with the exact warm cursor. It is diagnostic because
  `internal/parser` is not in `BENCH_GATE_PACKAGES`.
- `BenchmarkCodexIncrementalSyncReads` in `internal/sync` measures the warm tail
  between the two remaining linear reads. It is PR-gated because
  `internal/sync` is in `BENCH_GATE_PACKAGES`.

The maintained behavioral gate inventory is in
[Performance Gates](performance-gates.md).