wehub-resource-sync
216 lines
12 KiB
Markdown
216 lines
12 KiB
Markdown
# win-update-e2e — packaged NSIS update proof harness
|
|
|
|
**Windows only.** Given two Orca Windows installers (version N and N+1), this
|
|
harness performs a real silent update and proves, with machine-checkable
|
|
assertions, what happens to the terminal **daemon** and its **sessions** across
|
|
the update — and whether any console/terminal window flashes.
|
|
|
|
It is the Phase 0 "proof harness" deliverable from
|
|
[`docs/windows-terminal-update-survival-plan.md`](../../docs/windows-terminal-update-survival-plan.md).
|
|
It exists specifically because the July 2026 attempt shipped four broken RCs
|
|
without ever installing the packaged artifact (see
|
|
[`docs/windows-terminal-update-survival-postmortem.md`](../../docs/windows-terminal-update-survival-postmortem.md),
|
|
"Why verification missed every one of these"). Its design refuses to repeat
|
|
those verification failures:
|
|
|
|
- **Window visibility is measured by window enumeration + owner/canary
|
|
attribution — never by conhost command-line heuristics.** The post-mortem
|
|
proved conhost flags invert with parent console state and `MainWindowHandle`
|
|
is `0` for Windows-Terminal-hosted consoles. See `window-enum.ps1`.
|
|
- **Interactivity is proven by execution, not by result-shape.** Typed commands
|
|
write sentinel **files**; the harness checks the files. A command that "runs
|
|
and returns correct output" but flashes a window is still caught, because the
|
|
window watch runs independently.
|
|
- **The daemon is identified by command-line marker, never by exe name.** With
|
|
`ELECTRON_RUN_AS_NODE` the daemon image is `Orca.exe`; a relocated Phase 1
|
|
host may be a differently-named copied binary. See `daemon-processes.mjs`.
|
|
- **Each run uses an isolated userData dir** so its daemon's socket/token path
|
|
is unique and never collides with the many other daemons a dev box or CI
|
|
runner can host.
|
|
|
|
## Usage
|
|
|
|
```
|
|
pnpm win-update-e2e --from <setup.exe> --to <setup.exe> --expect <profile>
|
|
# or download release assets via gh (one call each):
|
|
pnpm win-update-e2e --from-release v1.4.124-rc.9 --to-release v1.4.125-rc.1 --expect cold-restore
|
|
```
|
|
|
|
Or directly: `node tools/win-update-e2e/run.mjs --from ... --to ... --expect ...`
|
|
|
|
### Flags
|
|
|
|
| Flag | Meaning |
|
|
| --------------------------------------------- | ----------------------------------------------------------- |
|
|
| `--from <path>` / `--to <path>` | Local `orca-windows-setup.exe` for base (N) / update (N+1) |
|
|
| `--from-release <tag>` / `--to-release <tag>` | Download the setup asset from a GitHub release tag via `gh` |
|
|
| `--expect cold-restore \| survival` | Assertion profile (required) |
|
|
| `--install-dir <path>` | Isolated-install mode (see below) — install into `<path>` |
|
|
| `--asset-pattern <glob>` | gh asset glob (default `*windows-setup.exe`) |
|
|
| `--soak-seconds <n>` | Post-relaunch window-watch soak (default `180`) |
|
|
| `--keep-install` | Skip teardown/uninstall for debugging (ignored in isolated) |
|
|
|
|
### Profiles
|
|
|
|
- **`cold-restore`** — **today's** behavior and the baseline that must keep
|
|
passing against current `main`. The installer's path sweep kills the in-dir
|
|
daemon, so: old daemon PID is **dead**, a **fresh** daemon exists, scrollback
|
|
is cold-restored (best-effort), a new terminal is interactive, and **zero**
|
|
unexpected console/terminal windows appear.
|
|
- **`survival`** — the Phase 1 target. Daemon PID **unchanged** across the
|
|
update, marker process **still alive**, the pre-update session still
|
|
interactive (typed input echoes, Ctrl+C interrupts), and **zero** unexpected
|
|
windows.
|
|
|
|
## Safety
|
|
|
|
This harness installs, overwrites, and can uninstall a real app. Two guards
|
|
protect a developer's machine; a clean CI/VM is unaffected by either:
|
|
|
|
- **Pre-existing app process → hard refusal.** If an Orca _app_ process (not a
|
|
daemon) is already running, the run aborts and prints the offending PIDs. The
|
|
harness never kills a process it did not start.
|
|
- **Pre-existing install → refusal unless `--allow-existing-install`.** If an
|
|
Orca install already exists under `%LOCALAPPDATA%\Programs`, the run refuses,
|
|
because installing N then N+1 would silently overwrite that build and leave
|
|
the `--to` version behind. Pass `--allow-existing-install` to proceed anyway.
|
|
|
|
Uninstall behavior at teardown follows ownership:
|
|
|
|
- **No pre-existing install** (harness fully owns it): teardown silently
|
|
uninstalls, unless `--keep-install`.
|
|
- **`--allow-existing-install` was used** (an install existed first): teardown
|
|
does **not** uninstall — removing a build the harness did not place would be
|
|
wrong. It prints a prominent note that the machine now has the `--to` version
|
|
and the prior build was not restored.
|
|
|
|
## Isolated install mode (developer machines)
|
|
|
|
On a clean CI/VM the harness installs into the default per-user location
|
|
(`%LOCALAPPDATA%\Programs\Orca`). A developer's box already has a real Orca there,
|
|
and the safety guards above would (correctly) refuse to run. **Isolated mode**
|
|
(`--install-dir <path>`) lets the harness run on that box without disturbing the
|
|
real install.
|
|
|
|
**The /D mechanism.** electron-builder's NSIS honors the standard NSIS `/D=<path>`
|
|
override for the install *directory* (`node_modules/app-builder-lib/templates/nsis/multiUser.nsh`).
|
|
`/D` is special: it must be the **last** argument and **cannot be quoted**, so the
|
|
path must be absolute and **spaces-free** (validated by `validateInstallDir`). The
|
|
installer's kill-sweep only matches processes under its own `$INSTDIR`, so a
|
|
separate directory never touches the real install's app or daemon processes.
|
|
|
|
**Why registry/shortcut backup-restore exists.** `/D` relocates *files only*.
|
|
Regardless of `/D`, the installer writes `InstallLocation` + the uninstall entry to
|
|
the **same per-user HKCU keys** as the real install
|
|
(`HKCU\Software\<APP_GUID>` and
|
|
`HKCU\Software\Microsoft\Windows\CurrentVersion\Uninstall\<key>`,
|
|
`node_modules/app-builder-lib/templates/nsis/include/installer.nsh`) and rewrites the
|
|
Start Menu / Desktop shortcuts. Left hijacked, the user's **next real update would
|
|
install into the test directory**. So isolated mode, before installing:
|
|
|
|
1. **Snapshots** the shared state (`registry-shortcut-backup.mjs`): `reg export`s
|
|
each existing key to `.reg` files, copies the Orca `*.lnk` shortcuts, and records
|
|
a manifest (which keys/shortcuts existed, the pre-run `InstallLocation`).
|
|
2. Runs the full install → update → assert proof against the isolated directory.
|
|
3. **Always restores** at teardown (a `try/finally` wraps everything after the
|
|
snapshot): `reg import`s keys that pre-existed, `reg delete`s keys the test
|
|
created, copies shortcuts back / deletes test-created ones, then **re-reads
|
|
`InstallLocation` and verifies** it matches the snapshot. On mismatch it prints a
|
|
loud block with the exact manual `reg import` command to recover. Isolated
|
|
teardown **always** uninstalls the test install (the harness owns the directory)
|
|
and removes the directory if empty — `--keep-install` is ignored.
|
|
|
|
**Residual risk.** The backup/restore covers `InstallLocation`, the uninstall entry,
|
|
and the Orca shortcuts — the state that steers a future update and the user-visible
|
|
launchers. It does **not** attempt to snapshot auto-update state files under the real
|
|
install's `userData` (the harness uses an isolated `userData` throughout, so it never
|
|
writes there), and it cannot restore state if the machine loses power mid-teardown
|
|
(re-run with a valid `--install-dir` to let restore complete, or run the printed
|
|
`reg import` by hand). The `.reg` backups live under the run's temp dir until a
|
|
successful teardown removes it.
|
|
|
|
**Example.**
|
|
|
|
```
|
|
pnpm win-update-e2e \
|
|
--from-release v1.4.124-rc.9 --to-release v1.4.125-rc.1 \
|
|
--expect cold-restore --install-dir C:\OrcaE2E
|
|
```
|
|
|
|
Read-only, touches nothing — print what isolated mode would snapshot on this machine:
|
|
|
|
```
|
|
node tools/win-update-e2e/registry-shortcut-backup.mjs
|
|
```
|
|
|
|
## What it does
|
|
|
|
1. **Preflight** — assert win32; warn if elevated; **refuse** to run if a
|
|
pre-existing Orca _app_ process (not a daemon) is running that the harness
|
|
did not start (it is printed and the run aborts — the harness never kills a
|
|
user's processes); snapshot the baseline set of visible top-level windows.
|
|
2. **Install N** silently (`<setup.exe> /S`) and locate `Orca.exe`.
|
|
3. **Launch** the installed app (Playwright `_electron`, isolated userData),
|
|
create ≥2 terminals, start a **marker** in one: a `powershell` loop that sets
|
|
a unique window-title **canary**, records its PID, and heartbeats a file.
|
|
4. **Record** the daemon PID (scoped pid-file + live-process scan), marker PID,
|
|
and session tab ids.
|
|
5. **Close** the app normally; verify the detached daemon is still alive.
|
|
6. **Start the window watch** — a background PowerShell loop polling visible
|
|
top-level windows every 500ms, diffing against baseline, recording every new
|
|
window (and title change) to a JSONL log through the update and soak.
|
|
7. **Install N+1** silently (the update).
|
|
8. **Relaunch** the app.
|
|
9. **Assert** per profile, then print a PASS/FAIL/INFO evidence table.
|
|
10. **Teardown** (unless `--keep-install`) — close app, kill only harness-created
|
|
processes, silent-uninstall.
|
|
|
|
Exit code is `0` when every non-informational assertion passes, else `1`
|
|
(`2` for a CLI usage error).
|
|
|
|
## Standalone instrument self-tests (no installers needed)
|
|
|
|
Each probe module runs on its own so the harness's own instruments are testable:
|
|
|
|
```
|
|
# Opens a real transient console window and asserts the watch catches it:
|
|
node tools/win-update-e2e/window-watch.mjs --selftest
|
|
|
|
# Read-only: list daemon processes + PID files on this machine:
|
|
node tools/win-update-e2e/daemon-processes.mjs [--user-data <dir>] [--scope <substr>]
|
|
|
|
# Emit the current visible-window snapshot as JSON:
|
|
powershell -File tools/win-update-e2e/window-enum.ps1
|
|
```
|
|
|
|
## Files
|
|
|
|
| File | Responsibility |
|
|
| -------------------------- | ----------------------------------------------------------------------------- |
|
|
| `run.mjs` | Orchestrator + CLI entry |
|
|
| `cli-args.mjs` | Argument parsing / validation |
|
|
| `preflight.mjs` | win32/elevation checks, pre-existing-app refusal, baseline snapshot |
|
|
| `installer-steps.mjs` | Silent install/update/uninstall, exe discovery, gh download |
|
|
| `registry-shortcut-backup.mjs` | Isolated mode: snapshot/restore the shared HKCU keys + Orca shortcuts |
|
|
| `app-driver.mjs` | Playwright Electron launch + terminal driving (production-safe DOM selectors) |
|
|
| `interactivity-probes.mjs` | Sentinel-file echo / heartbeat / Ctrl+C probes |
|
|
| `daemon-processes.mjs` | Daemon PID discovery (command-line marker + pid file), scoped |
|
|
| `window-enum.ps1` | Shared visible-top-level-window enumerator (P/Invoke `EnumWindows`) |
|
|
| `window-watch.ps1` | Background baseline-diff watch loop → JSONL |
|
|
| `window-watch.mjs` | Node wrapper: start/stop watch, `--selftest`, baseline capture |
|
|
| `assertions.mjs` | Window-event classification + profile PASS/FAIL table |
|
|
| `platform-guard.mjs` | `assertWin32`, elevation detection |
|
|
| `powershell-runner.mjs` | Windows PowerShell 5.1 spawn helpers |
|
|
|
|
## Known limitations
|
|
|
|
- **Scrollback fidelity is best-effort.** A production build renders the
|
|
terminal with WebGL, so xterm text is not reliably in the DOM and the e2e
|
|
`SerializeAddon` is not exposed. When text cannot be read the check reports
|
|
`INFO` (unknown), never a false `FAIL`.
|
|
- **Daemon file log** does not exist yet in packaged builds (the fork's stdio is
|
|
suppressed). The "daemon log free of ERROR lines" assertion is `INFO` until
|
|
Phase 0 daemon logging lands, then it reads `<userData>/logs/daemon.log`.
|
|
- The harness assumes the packaged main honors `ORCA_E2E_USER_DATA_DIR` to
|
|
relocate userData; verify this against a real packaged build.
|