everyinc--compound-engineering-plugin/skills/ce-sweep/references/state-schema.md

Sweep state schema (v1)

This is the canonical, versioned contract for the ce-sweep state file. The deterministic state engine (scripts/sweep-state.py) is the only writer; every peer agent (source connectors, the analyzer, the orchestrator) reads the file and mutates it exclusively through the engine's subcommands so the rules below are enforced in one place. Read this before touching state.

Top-level shape

schema_version: 1
lease:
  writer: "sweep-2026-07-02-cron"
  timestamp: "2026-07-02T12:00:00+00:00"
  ttl_minutes: 60
sources:
  "slack:C42":
    cursor: "1699999999.000100"
    sensitive: true          # optional; config-derived
items:
  "slack:C42:1699999999.000100":
    source: "slack:C42"
    status: "acknowledged"
    # ...arbitrary connector fields...
last_run:
  timestamp: "2026-07-02T12:05:00+00:00"
  outcome: "completed"
  writer: "sweep-2026-07-02-cron"
  counts: {"ingested": 5, "closed": 1}

key	type	meaning
schema_version	int	Contract version. Currently 1. A file missing this key is treated as corrupt.
lease	map	Single-writer mutex (see Lease). Absent when no writer holds it.
sources	map keyed by source id	Per-source resume cursor and optional flags.
items	map keyed by <source-id>:<item-id>	Per-item lifecycle record. The key is source-scoped so a source-native id (a Slack ts, a GitHub issue number) never collides with the same id from another source. Personas pass a source-native id plus --source; the engine composes the storage key and records both source and id on the item so it stays self-describing.
last_run	map	Bookkeeping for the most recent sweep (see run-record).

Compatibility rule (forward/backward)

The engine is deliberately additive-safe so a newer writer and an older reader can share a file:

situation	engine behavior
Unknown top-level key	Preserved on every write-back. Never dropped.
Unknown field on an item or source	Preserved on write-back. Never dropped.
Unknown status value	Preserved and passed through. Skip-never-drop — the closed enum below is not a whitelist.
File parses but has no schema_version	Treated as CORRUPT; the engine refuses to write over it.
schema_version greater than the engine knows	Still read/written field-preservingly; the engine only adds rules per version, never removes fields.

Bump schema_version only for a change that a v1 reader could misinterpret; purely additive fields do not require a bump.

Status enum

Closed set of known lifecycle states. Unknown values are preserved, never dropped, so a future state can roll out writer-first.

status	meaning
ingested	Captured from a source; not yet triaged.
ack_deferred	Receipt noted, but triage deferred to a later pass.
acknowledged	Triaged and accepted into the sweep pipeline.
needs_download	Referenced media/attachment must be fetched before analysis.
needs_analysis	Content present, awaiting analysis.
manual_stuck	Blocked; needs manual intervention to proceed.
analyzed	Analysis complete; findings recorded on the item.
in_plan	Folded into a plan or tracked work item.
fix_pending	A fix is underway or awaiting verification. Also the downgrade target for an under-evidenced closed.
closed	Resolved and verified. REQUIRES all three evidence fields (see below).
source_gone	The originating source or message no longer exists.

Evidence fields and the validate downgrade rule

A closed item is a claim that work shipped and was verified, so the engine holds it to proof. An item may only remain closed if it carries all three:

field	meaning
fix_ref	Reference to the fix (PR/commit/issue link).
verified_merge_sha	The merge commit SHA the fix landed on.
verified_at	ISO timestamp the fix was verified.

validate scans every item and downgrades any closed item missing (or with a falsy value for) any of these back to fix_pending, then rewrites the file and returns the list of downgraded ids. This self-heals a state left inconsistent by a crashed run. validate is lease-agnostic (it is a repair, run at sweep start).

sensitive semantics

The primary sensitivity mechanism is per-item: the orchestrator reads each source's config sensitive flag and includes "sensitive": true in the item JSON on every upsert-item for that source (SKILL.md phase 2d). A sensitive: true on a source entry in state is a defensive fallback the engine also honors, but nothing seeds it today — the per-item flag is what enforces R28, so sensitivity works even though source entries carry only a cursor. On any upsert-item where either the item or its source entry is sensitive, the engine drops body and quote before writing — redacted content never reaches disk. All other fields (title, url, status, ids) are retained. Redaction happens at write time, so flipping a source to sensitive protects only items written after the flag is set; re-ingest to redact prior items.

id-keyed merge rule

Writers own keys, not the whole file. upsert-item performs an id-keyed merge: it loads the existing item, replaces only the keys present in the incoming JSON, and preserves every other field already on that item. source is always (re)set from --source. No subcommand semantically rewrites the whole file — each mutates only the keys it owns and preserves the rest — even though the physical write re-emits the file atomically. This lets independent connectors and the analyzer touch the same item across passes without clobbering each other's fields.

Lease (single-writer mutex)

field	meaning
writer	Unique id of the writer holding the lease.
timestamp	ISO time the lease was last stamped — on acquire, and re-stamped on every owned mutating write.
ttl_minutes	Minutes after which an un-refreshed lease is reclaimable (default 60).

Rules the engine enforces:

• lease-acquire succeeds (OK) when the lease is free or already held by the same writer (re-entrant, re-stamps). It returns LOCKED when a live lease is held by another writer, or STALE-RECLAIMED (with previous_writer / previous_timestamp) when it takes over a lease older than its TTL.
• Every mutating call (upsert-item, cursor-advance) re-checks ownership before writing and returns LEASE-LOST (no write) if the caller is not the current holder; on success it re-stamps the lease timestamp so a long sweep keeps the lease alive.
• lease-release clears the caller's own lease (OK, also OK if none is held); releasing another writer's lease returns LEASE-LOST and does not write.
• Staleness is only asserted when it can be proven from parseable timestamps; an unparseable lease timestamp is treated as live (never stomped).

Topology scope

The lease's guarantee depends on where the state file lives:

topology	lease scope	protocol
local-commit mode (default)	Single writer per checkout.	The lease serializes overlapping sweeps in the same working tree (e.g. a cron sweep and a manual one). The file is written in-tree (and may be committed locally). No cross-machine guarantee.
pushed-shared-branch	One writer per repo.	The state file lives on a shared branch multiple checkouts push to. lease-acquire must be committed, pushed, and confirmed (fetch back and verify our writer won) before any source-side write. This makes the lease a repo-wide mutex across machines.

TTL-based reclaim (STALE-RECLAIMED) is what lets a crashed or killed writer's lease be taken over after ttl_minutes without manual cleanup.

run-record

Records the outcome of a sweep run under last_run.

field	source	meaning
timestamp	--timestamp (required)	Caller-supplied ISO run time. The engine never invents it.
outcome	--outcome	One of completed, aborted-locked, partial, failed.
writer	--writer	The writer id that recorded the run.
counts	--counts (JSON object)	Free-form tallies (per status, per source, etc.).

run-record is intentionally lease-agnostic: a run that aborted precisely because the lease was LOCKED (outcome: aborted-locked) must still be able to record that fact — but that write happens while the lease holder is mid-sweep. To keep it from clobbering the holder's concurrent upserts, every mutating subcommand holds an OS advisory lock (flock on <state>.lock) across its whole load-modify-write, so two concurrent invocations serialize their writes regardless of lease ownership. The lease decides who owns the sweep; the file lock decides who is writing the file right now. The .lock file is ephemeral and never committed (the skill's commit step adds only the state file and the plan, never -A).

Engine status words

Every subcommand prints one status word on line 1, then an optional JSON payload on line 2. Operational conditions exit 0 (never a traceback); only CLI misuse exits non-zero.

word	when	payload
OK	success	command-specific JSON (or none)
NO-STATE	read on a file that does not exist yet	—
CORRUPT	file exists but does not parse as this schema	—
LOCKED	lease-acquire: a live lease is held by another writer	—
STALE-RECLAIMED	lease-acquire: an expired lease was taken over	{previous_writer, previous_timestamp}
LEASE-LOST	mutating call by a non-owner, or releasing another's lease (no write)	—
REFUSED	cursor-advance: unknown past-item, or a cursor that would regress	—
ERROR	unexpected internal error (defensive; still exit 0)	—

YAML subset

The state file is genuine YAML restricted to a small, deterministic subset so a stdlib serializer/parser round-trips it exactly. Any YAML parser can read it; only the engine writes it.

construct	rule
Indentation	2 spaces per level, no tabs.
Keys	Bare when they match ^[A-Za-z_][A-Za-z0-9_.-]*$; otherwise a JSON double-quoted string (so ids with : are quoted).
Scalars	null / true / false / integers / floats are bare. Strings are always JSON double-quoted on a single line (fully escaped) — never block scalars or multiline.
Non-empty maps	Emitted as block mappings, recursing to any depth.
Lists and empty maps	Emitted as inline JSON flow on one line (e.g. ["a", "b"], {}) — itself valid YAML.
Key order	Deterministic: a preferred order for known keys, then remaining keys sorted, so diffs stay stable.
Comments / blank lines	Ignored on read. The engine does not emit comments.

A file that fails to parse under these rules, or that parses without a schema_version, is CORRUPT: the engine reports it and refuses to overwrite, so a hand-mangled file is never silently clobbered.