Product Management · Incident Retro Deck

Turn an incident retro into
a learning deck, like a Platform VP

The daemon-restart data bug: what broke, why, the fix on the table, and the one decision engineering, SRE, and leadership need to co-sign this week.

DECISION NEEDEDShip the data-root guardrail before the next namespace migration — not another point patch.

Retro Arc · Decision → Measure

STEP 01

Decision needed

Which fix ships this sprint: patch the call sites, or remove the fallback entirely

STEP 02

User pain

41 projects looked "deleted" for 12 minutes after a routine restart

STEP 03 · CORE

Options & recommendation

Compare the narrow patch against the guardrail that closes the whole class

STEP 04

Measure

MTTR, recurrence count, and CI catch-rate we'll track post-ship

Chapter One · Decision Needed

Ship the guardrail —
before the next restart

On June 14, a routine daemon restart under a stale namespace let 41 projects' SQLite files resolve against a cwd-relative fallback instead of the daemon's actual data root.
The only question left this sprint is which fix we ship — a patch, or a guardrail.

User Pain · Options

12 minutes of "my projects vanished" × 41 teams

Option A

Patch the call sites

Fix the 9 known openDatabase(projectRoot) calls to pass the resolved data root explicitly. Ships in two days — but the next engineer who adds a call site can reintroduce the same escape.

Option B · Recommended

Remove the fallback entirely

Delete the cwd-relative default from openDatabase itself, so a missing data root fails loudly at startup instead of silently writing elsewhere. One migration, closes the whole bug class.

Fast, narrow coverage · Option A

Slower, closes the class · Option B

—— data path ┈┈ silent fallback (the bug)

Tradeoffs · The Fix

Why "patch the sites" loses to "kill the fallback"

A per-site patch depends on every future engineer remembering the rule. Removing the fallback makes the rule impossible to violate — the daemon simply refuses to start with an unresolved data root.

# apps/daemon/src/db.ts — before vs after
function openDatabase(projectRoot?) {
  const root = projectRoot ?? cwd-relative default // silent escape
  return sqlite.open(root)
}

# after: fallback deleted, root is required
function openDatabase(root = RUNTIME_DATA_DIR) {
  if (!root) throw "missing data root — refuse to start"
  return sqlite.open(root)
}

Recommendation · System Diagram

One resolved root, three enforced checks

Measure

What we track after ship

0

Recurrences targeted across next 2 restart drills

-92%

MTTR for data-path incidents once the CI guardrail lands (14m → 70s)

9 → 0

Legacy fallback call sites remaining after the migration

3 signals we watch

Restart-drill pass rate, CI guardrail catches on PRs, and support tickets tagged "missing project."

Sign-off owners

Platform infra and SRE co-sign the guardrail PR; engineering leadership signs off on the quarterly restart-drill cadence.

Next Step · The Ask

Approve the guardrail PR this week

One irreversible step: merge the fallback removal plus the CI check before the next namespace migration ships.

TODAY

Merge the guardrail PR

Removes the cwd-relative fallback from openDatabase

THIS WEEK

Run the restart drill

Two concurrent namespaces, verify no silent fallback writes

NEXT RELEASE · CORE

Ship the CI check

Blocks any new call site that recomputes a data root instead of receiving one

ONGOING

Quarterly drill review

SRE reports pass rate to leadership every release cycle

Engineering · SRE · Leadership

sign · off

Cross-functional agreement closes this retro: Platform, SRE, and Leadership all back the guardrail, not just the patch.

Turn an incident retro intoa learning deck, like a Platform VP

Ship the guardrail —before the next restart