chore: import upstream snapshot with attribution

showcase/bin/README.md

@@ -0,0 +1,164 @@
+# `bin/railway`
+
+Tagline: Ruby CLI for showcase Railway day-to-day ops — snapshot, restore,
+rollback, promote, pin, env-diff, resolve-digest, lint-prod. Mutating
+subcommands require typed `production` confirmation. For fleet auto-update
+config / new-service provisioning, see [`../RAILWAY.md`](../RAILWAY.md).
+
+Single-file Ruby tooling for showcase Railway operations.
+
+## Why
+
+The Showcase platform lives on Railway across two environments (staging and
+production). Day-to-day operations — promoting staging to production, pinning
+services to immutable image digests, rolling a bad deploy back, auditing drift
+between envs — used to require ad-hoc shell + GraphQL recipes. `bin/railway`
+makes those operations first-class CLI subcommands with consistent flags,
+exit codes, and production protection.
+
+## Install
+
+None. Requires system Ruby 3.x (stdlib only — no Bundler, no Gemfile).
+
+```sh
+showcase/bin/railway --help
+```
+
+## Auth
+
+The tool reads a Railway API token from (in order):
+
+1. `RAILWAY_TOKEN` environment variable
+2. `~/.railway/config.json` (the `token` field, or `user.token`)
+
+It never invokes `railway login`, `railway logout`, or `op`. If neither source
+yields a token, it exits with code 2 and a clear error.
+
+For GHCR digest resolution (`resolve-digest`, `pin`), set `GHCR_TOKEN` if you
+need to read private packages; public packages work anonymously via the GHCR
+`/token` endpoint.
+
+## Subcommands
+
+| Subcommand        | Purpose                                                                                                                                          |
+| ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------ |
+| `snapshot`        | Capture an env's services + config into a YAML snapshot.                                                                                         |
+| `restore`         | Restore an env to a snapshot (force-redeploy each service).                                                                                      |
+| `rollback`        | Roll a single service back one deploy (or to a specific deployment id with `--to`).                                                              |
+| `rollback-commit` | Restore an env to the snapshot committed at a given git SHA.                                                                                     |
+| `promote`         | Promote staging digests to production with prechecks.                                                                                            |
+| `pin`             | Pin a service to a specific image digest.                                                                                                        |
+| `env-diff`        | Diff two envs; exits 1 on drift.                                                                                                                 |
+| `resolve-digest`  | Resolve an image tag (e.g. `:latest`) to its `sha256:` digest.                                                                                   |
+| `lint-prod`       | CI gate (advisory): warn if any prod service is not digest-pinned. `--exit-zero` for advisory mode, `--format json` for machine-readable output. |
+
+Run any subcommand with `--help` for full flag list.
+
+## Production protection
+
+Every subcommand that mutates state requires both:
+
+- `--yes` flag, **and**
+- typed confirmation of the literal string `production` on stdin
+
+…before any production mutation runs. `--non-interactive` skips the prompt
+but still requires `--yes`. There is no way to mutate production without an
+explicit acknowledgement.
+
+## Exit codes
+
+| Code | Meaning                                                                  |
+| ---- | ------------------------------------------------------------------------ |
+| 0    | Clean / success                                                          |
+| 1    | Drift detected, findings reported, or promote refused for policy reasons |
+| 2    | Error (auth, network, GraphQL schema, refused confirmation, etc.)        |
+
+## Worked example: promote staging → production
+
+```sh
+# 1. Audit drift first (read-only).
+showcase/bin/railway env-diff staging production
+# DRIFT: 3 finding(s)
+#   service showcase-shell: digest sha256:abc != sha256:def
+#   ...
+
+# 2. Lint prod to confirm baseline is pinned.
+showcase/bin/railway lint-prod
+# OK: all production services digest-pinned.
+
+# 3. Capture a "before" snapshot in case we need to roll back.
+showcase/bin/railway snapshot --env production --output before-promote.yaml
+
+# 4. Run the promote with prechecks. Production confirmation prompt fires here.
+showcase/bin/railway promote --yes
+# Type 'production' to confirm promote: production
+# promoted showcase-shell -> ghcr.io/copilotkit/showcase-shell@sha256:def...
+# ...
+
+# If anything goes sideways:
+showcase/bin/railway restore --env production --snapshot before-promote.yaml --yes
+```
+
+## CI integration
+
+`.github/workflows/showcase_lint_prod.yml` runs `bin/railway lint-prod` on
+every PR that touches `showcase/**`.
+
+**Currently advisory** — the workflow passes `--exit-zero`, so findings print
+to the job log but do not fail the PR. This lets us soak the check against
+real production state before turning it into a hard gate. Once we've built
+confidence the findings are clean, remove `--exit-zero` from the workflow to
+flip the check to enforcing (exit 1 on drift).
+
+Long-term contract: every production service must be pinned to an immutable
+`ghcr.io/...@sha256:...` digest, and the lint job will fail any PR that
+drifts away from that.
+
+### Visibility surfaces
+
+Two human-facing surfaces render the audit result every run:
+
+1. **Workflow step summary** — the workflow writes a structured markdown
+   block to `$GITHUB_STEP_SUMMARY` so the audit shows at the top of every
+   run page (every event: `pull_request`, `push`, `workflow_dispatch`).
+2. **Sticky PR comment** — on `pull_request` events, the workflow posts (or
+   updates) a single comment per PR. The comment is keyed by the HTML marker
+   `<!-- lint-prod-sticky-comment -->` so re-runs update the same comment
+   instead of creating duplicates.
+
+Both surfaces show the same content: a one-line status, a table of the
+unpinned services (only — `pinned` services are not enumerated), and a
+Pacific-time run timestamp with the finding count.
+
+### Machine-readable output
+
+`lint-prod --format json` emits:
+
+```json
+{
+  "services": [
+    { "name": "...", "source": "...", "status": "pinned|mutable-tag" }
+  ],
+  "findings": 3,
+  "timestamp": "2026-05-27T18:00:00Z"
+}
+```
+
+The CI workflow uses this shape to render the step summary and PR comment.
+The `findings` count is also written to `$GITHUB_OUTPUT` so downstream jobs
+(e.g. a future Slack alert step) can compare against prior runs.
+
+## Tests
+
+```sh
+ruby showcase/bin/spec/all_tests.rb
+```
+
+Tests are minitest (stdlib). They cover:
+
+- argv parsing per subcommand
+- snapshot YAML round-trip
+- GHCR digest-resolution decision tree (mocked HTTP)
+- production-protection prompt behavior
+
+No Railway / GHCR network calls are made during tests.

showcase/bin/showcase

@@ -0,0 +1,218 @@
+#!/usr/bin/env bash
+# showcase — unified CLI for the CopilotKit showcase platform.
+#
+# Dispatches to built-in compose commands (up, down, build, ps, ports, logs)
+# and to plugin commands defined in scripts/cli/cmd-*.sh files.
+
+set -euo pipefail
+
+# ── Bootstrap ────────────────────────────────────────────────────────────────
+
+BIN_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+SHOWCASE_ROOT="$(cd "$BIN_DIR/.." && pwd)"
+export SHOWCASE_ROOT
+
+# shellcheck source=../scripts/cli/_common.sh
+source "$SHOWCASE_ROOT/scripts/cli/_common.sh"
+
+# ── Auto-discover plugin commands (cmd-*.sh) ─────────────────────────────────
+
+# Plugin names and their descriptions, stored as parallel arrays for bash 3
+# compatibility (no associative arrays on macOS default bash).
+_plugin_names=()
+_plugin_descs=()
+
+for cmd_file in "$SHOWCASE_ROOT"/scripts/cli/cmd-*.sh; do
+  [ -f "$cmd_file" ] || continue
+  # shellcheck disable=SC1090
+  source "$cmd_file"
+  # Extract command name: cmd-foo-bar.sh → foo-bar
+  _name="$(basename "$cmd_file" .sh)"
+  _name="${_name#cmd-}"
+  _plugin_names+=("$_name")
+  # Each cmd file should define CMD_<NAME>_DESC (dashes→underscores, uppercased)
+  # e.g. cmd-aimock-rebuild.sh defines CMD_AIMOCK_REBUILD_DESC
+  _desc_var="CMD_${_name//-/_}"
+  _desc_var="$(echo "$_desc_var" | tr '[:lower:]' '[:upper:]')_DESC"
+  _plugin_descs+=("${!_desc_var:-}")
+done
+
+# ── Built-in commands ────────────────────────────────────────────────────────
+
+cmd_up() {
+  require_env
+  trap restore_symlinks EXIT
+  stage_shared
+
+  # Fleet topology: bring up the SAME shape prod runs — the control-plane
+  # container + worker container(s) + PocketBase + demos + aimock — NOT an
+  # in-process pool. The harness control-plane and pool worker(s) live in the
+  # `infra` profile (alongside aimock/pocketbase/dashboard), so always include
+  # `--profile infra`.
+  #
+  # Demo slugs (bare args like `langgraph-python`) select that demo's compose
+  # PROFILE — they must become `--profile <slug>` flags, NOT positional service
+  # names. A positional service arg restricts `up` to only that service + its
+  # deps, which (a) leaves the demo's own profile disabled and (b) makes
+  # `--scale harness-pool-worker=N` fail with "no such service:
+  # harness-pool-worker: disabled" because the worker is no longer in the active
+  # set. Raw compose flags (anything starting with `-`, e.g. `--no-build`) are
+  # passed through verbatim.
+  #
+  # HARNESS_POOL_COUNT drives the worker fleet size. Local default is 1 (a
+  # single worker container); staging/prod export HARNESS_POOL_COUNT=2. The
+  # compose `harness-pool-worker` service is scaled to that count via
+  # `--scale` so the same compose file scales from local (1) to prod (N)
+  # purely by the env var — no per-environment compose edits.
+  local pool_count="${HARNESS_POOL_COUNT:-1}"
+  export HARNESS_POOL_COUNT="$pool_count"
+
+  # Split args: bare slugs → `--profile <slug>` + build target; flags (-*) →
+  # pass through. `--dev` is intercepted (not passed to compose): it layers
+  # the dev overlay (docker-compose.dev.yml) which bind-mounts integration
+  # source + overrides the run command to a hot-reload entrypoint — fast
+  # iteration without an image rebuild. The built-image mode (no --dev) stays
+  # the faithful default.
+  local profile_args=()
+  local build_targets=()
+  local passthrough_args=()
+  local dev_mode=false
+  local arg
+  for arg in "$@"; do
+    if [[ "$arg" == "--dev" ]]; then
+      dev_mode=true
+    elif [[ "$arg" == -* ]]; then
+      passthrough_args+=("$arg")
+    else
+      profile_args+=(--profile "$arg")
+      build_targets+=("$arg")
+    fi
+  done
+
+  local compose_cmd="$COMPOSE_CMD"
+  if $dev_mode; then
+    compose_cmd="$compose_cmd -f $SHOWCASE_ROOT/docker-compose.dev.yml"
+    info "DEV MODE: bind-mounting integration source + hot reload (NOT the faithful built-image path)"
+  fi
+
+  info "Launching fleet topology: control-plane + ${pool_count} worker(s) + PocketBase + demos + aimock (HARNESS_POOL_COUNT=${pool_count})"
+  # Two-call strategy to preserve A21's BuildKit-contention fix (target-only
+  # rebuild) WITHOUT regressing infra startup that A21 inadvertently dropped
+  # (A21b, issue #5495).
+  #
+  # docker compose semantics: positional service names after `up` restrict
+  # WHICH services start to the named ones + their `depends_on` chain — not
+  # just which ones get `--build`-rebuilt. So a single
+  # `up -d --build <slug>` only brings up the slug + its depends_on (aimock),
+  # leaving the rest of the infra profile (pocketbase/dashboard/harness/
+  # harness-pool-worker) down. Under `--isolate` with a sibling stack on the
+  # same host ports, health checks then cross onto foreign containers and
+  # cells silently misroute → 0.0s red.
+  #
+  # Fix: when slugs are present, emit TWO compose calls:
+  #   1. <profiles> up -d (no --build, no positional services) — start all
+  #      services in the active profiles using cached images.
+  #   2. <profiles> up -d --build <slug...> — force-rebuild ONLY the named
+  #      services and ensure they're up. Other services from call (1) are
+  #      no-ops.
+  # When no slugs are present (infra-only bring-up), keep the single blanket
+  # `--build` call so first-time bootstrap still builds missing images.
+  #
+  # bash 3.2 (macOS default) errors on `${arr[@]}` for an EMPTY array under
+  # `set -u`; the `${arr[@]+...}` guard expands to nothing when unset/empty.
+  if [[ ${#build_targets[@]} -gt 0 ]]; then
+    # Call 1: bring up ALL services in active profiles, no build.
+    $compose_cmd --profile infra ${profile_args[@]+"${profile_args[@]}"} up -d \
+      --scale harness-pool-worker="$pool_count" ${passthrough_args[@]+"${passthrough_args[@]}"}
+    # Call 2: rebuild ONLY the targeted slug(s) and ensure they're up.
+    $compose_cmd --profile infra ${profile_args[@]+"${profile_args[@]}"} up -d --build \
+      --scale harness-pool-worker="$pool_count" ${passthrough_args[@]+"${passthrough_args[@]}"} \
+      "${build_targets[@]}"
+  else
+    # Infra-only: single call with blanket --build for first-time bootstrap.
+    $compose_cmd --profile infra up -d --build \
+      --scale harness-pool-worker="$pool_count" ${passthrough_args[@]+"${passthrough_args[@]}"}
+  fi
+}
+
+cmd_down() {
+  $COMPOSE_CMD down "$@"
+}
+
+cmd_build() {
+  trap restore_symlinks EXIT
+  stage_shared
+  $COMPOSE_CMD build "$@"
+}
+
+cmd_ps() {
+  $COMPOSE_CMD ps "$@"
+}
+
+cmd_ports() {
+  if command -v jq &>/dev/null; then
+    jq -r 'to_entries[] | "\(.key)\t→ localhost:\(.value)"' "$PORTS_FILE"
+  else
+    cat "$PORTS_FILE"
+  fi
+}
+
+# ── Usage ────────────────────────────────────────────────────────────────────
+
+usage() {
+  cat <<'HEADER'
+Usage: showcase <command> [options]
+
+Core commands:
+  up [slug...]      Start containers (rebuilds if source changed)
+                    --dev  bind-mount integration source + hot reload
+                           (fast iteration; NOT the faithful built-image path)
+  down [slug...]    Stop containers
+  build [slug...]   Build Docker images
+  ps                Show running containers
+  ports             Print slug → host port mapping
+HEADER
+
+  # Print plugin commands if any are loaded
+  if [ ${#_plugin_names[@]} -gt 0 ]; then
+    echo ""
+    echo "Plugin commands:"
+    local i
+    for i in "${!_plugin_names[@]}"; do
+      printf "  %-17s %s\n" "${_plugin_names[$i]}" "${_plugin_descs[$i]}"
+    done
+  fi
+
+  cat <<'FOOTER'
+
+Run 'showcase <command> --help' for details on a specific command.
+FOOTER
+}
+
+# ── Dispatch ─────────────────────────────────────────────────────────────────
+
+subcmd="${1:-}"
+shift || true
+
+case "$subcmd" in
+  ""|"-h"|"--help"|"help")
+    usage
+    [ -z "$subcmd" ] && exit 1
+    exit 0
+    ;;
+  eval)
+    exec npx tsx "$SHOWCASE_ROOT/harness/src/cli.ts" eval "$@"
+    ;;
+  *)
+    # Convert dashes to underscores for function lookup: foo-bar → cmd_foo_bar
+    func_name="cmd_${subcmd//-/_}"
+    if type "$func_name" 2>/dev/null | head -1 | grep -q 'function'; then
+      "$func_name" "$@"
+    else
+      echo "Unknown command: $subcmd" >&2
+      echo ""
+      usage
+      exit 1
+    fi
+    ;;
+esac

showcase/bin/spec/all_tests.rb

@@ -0,0 +1,12 @@
+#!/usr/bin/env ruby
+# frozen_string_literal: true
+
+# Entry point for bin/railway minitest suite. Discovers and runs every test_*.rb
+# in this directory.
+
+$LOAD_PATH.unshift(File.expand_path("..", __dir__))
+$LOAD_PATH.unshift(__dir__)
+
+require "minitest/autorun"
+
+Dir.glob(File.join(__dir__, "test_*.rb")).sort.each { |f| require f }

showcase/bin/spec/spec_helper.rb

@@ -0,0 +1,11 @@
+# frozen_string_literal: true
+
+# Helper that loads bin/railway as a library (so we can access Railway:: classes
+# without invoking the CLI).
+
+require "minitest/autorun"
+
+# Stub $PROGRAM_NAME so the bottom-of-file invocation guard is skipped.
+unless defined?(::Railway)
+    load File.expand_path("../railway", __dir__)
+end

showcase/bin/spec/test_cli_parsing.rb

@@ -0,0 +1,112 @@
+# frozen_string_literal: true
+
+require_relative "spec_helper"
+require "stringio"
+
+class CLIParsingTest < Minitest::Test
+    def test_usage_lists_all_nine_subcommands
+        out = Railway.usage
+        %w[snapshot restore rollback rollback-commit promote pin env-diff
+            resolve-digest lint-prod].each do |sub|
+            assert_includes out, sub, "usage missing subcommand: #{sub}"
+        end
+    end
+
+    def test_help_returns_zero
+        rc = nil
+        silence_io { rc = Railway.run(["--help"]) }
+        assert_equal 0, rc
+    end
+
+    def test_version_returns_zero
+        rc = nil
+        silence_io { rc = Railway.run(["--version"]) }
+        assert_equal 0, rc
+    end
+
+    def test_unknown_subcommand_returns_2
+        rc = nil
+        silence_io { rc = Railway.run(["definitely-not-a-cmd"]) }
+        assert_equal 2, rc
+    end
+
+    def test_snapshot_command_parses_env_and_output
+        c = Railway::SnapshotCommand.new(["--env", "staging", "--output", "/tmp/x.yaml"])
+        c.parser.parse!(c.argv)
+        assert_equal "staging", c.options[:env]
+        assert_equal "/tmp/x.yaml", c.options[:output]
+    end
+
+    def test_restore_command_requires_env_and_snapshot
+        c = Railway::RestoreCommand.new([])
+        ex = nil
+        silence_io { ex = assert_raises(SystemExit) { c.run } }
+        assert_equal 2, ex.status
+    end
+
+    def test_rollback_command_parses_to_flag
+        c = Railway::RollbackCommand.new(["--env", "staging", "--service", "showcase-shell", "--to", "dep-123"])
+        c.parser.parse!(c.argv)
+        assert_equal "dep-123", c.options[:to]
+    end
+
+    def test_envdiff_requires_two_args
+        c = Railway::EnvDiffCommand.new(["staging"])
+        ex = nil
+        silence_io { ex = assert_raises(SystemExit) { c.run } }
+        assert_equal 2, ex.status
+    end
+
+    def test_promote_flags_parse
+        c = Railway::PromoteCommand.new(["--confirm-divergence", "--yes", "--dry-run"])
+        c.parser.parse!(c.argv)
+        assert c.options[:confirm_divergence]
+        assert c.options[:yes]
+        assert c.options[:dry_run]
+    end
+
+    def test_resolve_digest_requires_arg
+        c = Railway::ResolveDigestCommand.new([])
+        ex = nil
+        silence_io { ex = assert_raises(SystemExit) { c.run } }
+        assert_equal 2, ex.status
+    end
+
+    def test_lint_prod_parses_format_and_exit_zero
+        c = Railway::LintProdCommand.new(["--exit-zero", "--format", "json"])
+        c.parser.parse!(c.argv)
+        assert_equal true, c.instance_variable_get(:@exit_zero)
+        assert_equal "json", c.instance_variable_get(:@format)
+    end
+
+    def test_lint_prod_rejects_invalid_format
+        c = Railway::LintProdCommand.new(["--format", "yaml"])
+        assert_raises(OptionParser::InvalidArgument) { c.parser.parse!(c.argv) }
+    end
+
+    def test_lint_prod_defaults_format_to_text
+        c = Railway::LintProdCommand.new([])
+        c.parser.parse!(c.argv)
+        assert_equal "text", c.instance_variable_get(:@format)
+        assert_equal false, c.instance_variable_get(:@exit_zero)
+    end
+
+    def test_env_id_for_resolves_aliases
+        assert_equal Railway::PRODUCTION_ENV_ID, Railway.env_id_for("production")
+        assert_equal Railway::PRODUCTION_ENV_ID, Railway.env_id_for("prod")
+        assert_equal Railway::STAGING_ENV_ID, Railway.env_id_for("staging")
+        assert_equal Railway::STAGING_ENV_ID, Railway.env_id_for("stage")
+    end
+
+    private
+
+    def silence_io
+        orig_stdout, orig_stderr = $stdout, $stderr
+        $stdout = StringIO.new
+        $stderr = StringIO.new
+        yield
+    ensure
+        $stdout = orig_stdout
+        $stderr = orig_stderr
+    end
+end

showcase/bin/spec/test_env_diff.rb

@@ -0,0 +1,172 @@
+# frozen_string_literal: true
+
+require_relative "spec_helper"
+
+# EnvDiffCommand compares two env snapshots and reports drift. Historically it
+# diffed digest, startCommand, and env-var KEY sets — but the banner also
+# advertised "custom domains", which the diff loop never actually compared.
+# These tests pin the custom-domain comparison so the banner is honest:
+#
+#   1. Identical custom_domains across both envs => no drift line.
+#   2. Differing custom_domains => a drift line that names the service and the
+#      offending domains, in both directions (missing-in-a / missing-in-b).
+#
+# We exercise the pure diff helper (diff_services) directly so the test does
+# not need to stand up Railway GraphQL — the helper takes two already-built
+# snapshots and returns the drift array.
+class EnvDiffTest < Minitest::Test
+    def snapshot_with(domains_a: [], domains_b: [])
+        # Two single-service snapshots that agree on everything EXCEPT the
+        # custom_domains set, so any drift surfaced is attributable to domains.
+        snap_a = {
+            "services" => [
+                {
+                    "name"           => "showcase-shell",
+                    "digest"         => "sha256:abc",
+                    "start_command"  => nil,
+                    "env_keys"       => %w[PORT NODE_ENV],
+                    "custom_domains" => domains_a,
+                },
+            ],
+        }
+        snap_b = {
+            "services" => [
+                {
+                    "name"           => "showcase-shell",
+                    "digest"         => "sha256:abc",
+                    "start_command"  => nil,
+                    "env_keys"       => %w[PORT NODE_ENV],
+                    "custom_domains" => domains_b,
+                },
+            ],
+        }
+        [snap_a, snap_b]
+    end
+
+    def test_identical_custom_domains_no_drift
+        snap_a, snap_b = snapshot_with(
+            domains_a: ["shell.copilotkit.ai"],
+            domains_b: ["shell.copilotkit.ai"],
+        )
+        cmd = Railway::EnvDiffCommand.new(%w[staging production])
+        drift = cmd.diff_services(snap_a, snap_b, "staging", "production")
+        assert_empty drift
+    end
+
+    def test_differing_custom_domains_reported_both_directions
+        snap_a, snap_b = snapshot_with(
+            domains_a: ["only-in-staging.copilotkit.ai"],
+            domains_b: ["only-in-prod.copilotkit.ai"],
+        )
+        cmd = Railway::EnvDiffCommand.new(%w[staging production])
+        drift = cmd.diff_services(snap_a, snap_b, "staging", "production")
+
+        # One finding per direction, each naming the offending domain.
+        missing_in_b = drift.find { |l| l.include?("custom domains missing in production") }
+        missing_in_a = drift.find { |l| l.include?("custom domains missing in staging") }
+        assert missing_in_b, "expected a 'missing in production' finding, got: #{drift.inspect}"
+        assert missing_in_a, "expected a 'missing in staging' finding, got: #{drift.inspect}"
+        assert_includes missing_in_b, "only-in-staging.copilotkit.ai"
+        assert_includes missing_in_a, "only-in-prod.copilotkit.ai"
+    end
+
+    # A service that exists in one snapshot but not the other must surface a
+    # "missing in <env>" drift line. This also exercises the missing-service
+    # branch that was previously untested.
+    def test_service_missing_in_one_env_reported
+        snap_a = {
+            "services" => [
+                { "name" => "showcase-shell", "digest" => "sha256:abc",
+                  "start_command" => nil, "env_keys" => %w[PORT], "custom_domains" => [] },
+                { "name" => "extra-svc", "digest" => "sha256:def",
+                  "start_command" => nil, "env_keys" => %w[PORT], "custom_domains" => [] },
+            ],
+        }
+        snap_b = {
+            "services" => [
+                { "name" => "showcase-shell", "digest" => "sha256:abc",
+                  "start_command" => nil, "env_keys" => %w[PORT], "custom_domains" => [] },
+            ],
+        }
+        cmd = Railway::EnvDiffCommand.new(%w[staging production])
+        drift = cmd.diff_services(snap_a, snap_b, "staging", "production")
+        missing = drift.find { |l| l.include?("extra-svc") && l.include?("missing in production") }
+        assert missing, "expected 'extra-svc missing in production', got: #{drift.inspect}"
+    end
+
+    # A snapshot lacking a "services" key must not raise NoMethodError. NOTE:
+    # because the missing "services" key makes find_service return nil for the
+    # one service that exists only in snap_b, this case exercises ONLY the
+    # top-level `["services"] || []` guard and the missing-service branch — it
+    # `next`s before ever reaching the env_keys / custom_domains comparison.
+    # The env_keys/custom_domains accessors are pinned separately below.
+    def test_snapshot_without_services_key_does_not_crash
+        snap_a = {} # no "services" key at all
+        snap_b = {
+            "services" => [
+                { "name" => "showcase-shell", "digest" => "sha256:abc",
+                  "start_command" => nil, "env_keys" => %w[PORT], "custom_domains" => [] },
+            ],
+        }
+        cmd = Railway::EnvDiffCommand.new(%w[staging production])
+        drift = cmd.diff_services(snap_a, snap_b, "staging", "production")
+        missing = drift.find { |l| l.include?("showcase-shell") && l.include?("missing in staging") }
+        assert missing, "expected 'showcase-shell missing in staging', got: #{drift.inspect}"
+    end
+
+    # A service present in BOTH snapshots where one side OMITS "env_keys"
+    # (e.g. a v1/partial/hand-edited snapshot read from a git SHA) must not
+    # raise NoMethodError. This genuinely reaches the env_keys comparison
+    # branch (find_service returns the service on both sides, so the method
+    # does NOT `next`). Without an `|| []` guard, `sa["env_keys"] - sb[...]`
+    # raises `undefined method '-' for nil`.
+    def test_service_without_env_keys_does_not_crash
+        snap_a = {
+            "services" => [
+                # No "env_keys" key at all on this side.
+                { "name" => "showcase-shell", "digest" => "sha256:abc",
+                  "start_command" => nil, "custom_domains" => [] },
+            ],
+        }
+        snap_b = {
+            "services" => [
+                { "name" => "showcase-shell", "digest" => "sha256:abc",
+                  "start_command" => nil, "env_keys" => %w[PORT NODE_ENV],
+                  "custom_domains" => [] },
+            ],
+        }
+        cmd = Railway::EnvDiffCommand.new(%w[staging production])
+        drift = cmd.diff_services(snap_a, snap_b, "staging", "production")
+        # snap_a is missing both keys present in snap_b, so they must be
+        # reported as "missing in staging" (the a-side env).
+        missing_in_a = drift.find { |l| l.include?("env keys missing in staging") }
+        assert missing_in_a, "expected an 'env keys missing in staging' finding, got: #{drift.inspect}"
+        assert_includes missing_in_a, "PORT"
+        assert_includes missing_in_a, "NODE_ENV"
+    end
+
+    # Symmetric to the env_keys case: a service present in BOTH snapshots where
+    # one side OMITS "custom_domains" must not raise NoMethodError when the
+    # other side has domains to diff against. Pins the custom_domains guard at
+    # the comparison branch (not just the missing-service branch).
+    def test_service_without_custom_domains_does_not_crash
+        snap_a = {
+            "services" => [
+                { "name" => "showcase-shell", "digest" => "sha256:abc",
+                  "start_command" => nil, "env_keys" => %w[PORT] },
+            ],
+        }
+        snap_b = {
+            "services" => [
+                { "name" => "showcase-shell", "digest" => "sha256:abc",
+                  "start_command" => nil, "env_keys" => %w[PORT],
+                  "custom_domains" => ["shell.copilotkit.ai"] },
+            ],
+        }
+        cmd = Railway::EnvDiffCommand.new(%w[staging production])
+        drift = cmd.diff_services(snap_a, snap_b, "staging", "production")
+        missing_in_a = drift.find { |l| l.include?("custom domains missing in staging") }
+        assert missing_in_a, "expected a 'custom domains missing in staging' finding, got: #{drift.inspect}"
+        assert_includes missing_in_a, "shell.copilotkit.ai"
+    end
+end

showcase/bin/spec/test_expected_domains_parity.rb

@@ -0,0 +1,53 @@
+# frozen_string_literal: true
+
+require_relative "spec_helper"
+require "json"
+
+# Parity test: Ruby's Railway::EXPECTED_DOMAINS (derived at load time from
+# showcase/scripts/railway-envs.generated.json) MUST equal the public-host
+# subset computed directly from the same TS SSOT artifact. If this test fails,
+# either the JSON artifact is stale or bin/railway's derivation logic drifted
+# from the TS SSOT shape.
+class ExpectedDomainsParityTest < Minitest::Test
+    SSOT_JSON = File.expand_path("../../scripts/railway-envs.generated.json", __dir__)
+
+    def test_generated_json_exists
+        assert File.exist?(SSOT_JSON),
+               "expected SSOT artifact at #{SSOT_JSON} — run " \
+               "`npx tsx showcase/scripts/emit-railway-envs-json.ts`"
+    end
+
+    def test_ruby_expected_domains_matches_ts_ssot_public_hosts
+        data = JSON.parse(File.read(SSOT_JSON))
+        prod_env_id = data.fetch("envIds").fetch("prod")
+        staging_env_id = data.fetch("envIds").fetch("staging")
+
+        expected_prod = data.fetch("services")
+                            .map { |s| s.fetch("domains").fetch("prod") }
+                            .reject { |h| h.end_with?(".up.railway.app") }
+                            .sort
+                            .uniq
+        expected_staging = data.fetch("services")
+                               .map { |s| s.fetch("domains").fetch("staging") }
+                               .reject { |h| h.end_with?(".up.railway.app") }
+                               .sort
+                               .uniq
+
+        # Sanity: the env-id constants in the Ruby file must match the SSOT.
+        assert_equal prod_env_id, Railway::PRODUCTION_ENV_ID,
+                     "PRODUCTION_ENV_ID drifted from SSOT envIds.prod"
+        assert_equal staging_env_id, Railway::STAGING_ENV_ID,
+                     "STAGING_ENV_ID drifted from SSOT envIds.staging"
+
+        actual = Railway::EXPECTED_DOMAINS
+        assert_equal expected_prod, actual.fetch(prod_env_id).sort,
+                     "Ruby EXPECTED_DOMAINS[prod] != TS SSOT public prod hosts"
+        assert_equal expected_staging, actual.fetch(staging_env_id).sort,
+                     "Ruby EXPECTED_DOMAINS[staging] != TS SSOT public staging hosts"
+    end
+
+    def test_expected_domains_keys_are_only_prod_and_staging_env_ids
+        keys = Railway::EXPECTED_DOMAINS.keys.sort
+        assert_equal [Railway::PRODUCTION_ENV_ID, Railway::STAGING_ENV_ID].sort, keys
+    end
+end

showcase/bin/spec/test_ghcr_bearer.rb

@@ -0,0 +1,123 @@
+# frozen_string_literal: true
+
+require_relative "spec_helper"
+
+# Proves bearer_for ALWAYS performs the GHCR /token exchange instead of
+# returning a raw GitHub token. GHCR's OCI manifest endpoint rejects a raw
+# GitHub Actions token with HTTP 403 — only a bearer minted via the /token
+# exchange is accepted. When a token is present the exchange MUST authenticate
+# with Basic auth (base64("x-access-token:<token>")); for public packages the
+# exchange also succeeds anonymously.
+class GHCRBearerTest < Minitest::Test
+    # Recording fake HTTP layer: captures every (method, url, headers) call so
+    # tests can assert what was actually sent on the wire.
+    class RecordingHTTP
+        attr_reader :calls
+
+        def initialize(responses)
+            @responses = responses
+            @calls = []
+        end
+
+        def call(method:, url:, headers: {})
+            @calls << { method: method, url: url, headers: headers }
+            r = @responses[[method, url]] || @responses[url]
+            raise "no fake response for #{[method, url].inspect}" unless r
+            r
+        end
+    end
+
+    DIGEST      = "sha256:cafef00dcafef00dcafef00dcafef00dcafef00dcafef00dcafef00dcafef00d"
+    REF         = "ghcr.io/copilotkit/showcase-shell@#{DIGEST}"
+    MANIFEST    = "https://ghcr.io/v2/copilotkit/showcase-shell/manifests/#{DIGEST}"
+    TOKEN_URL   = "https://ghcr.io/token?service=ghcr.io&scope=repository:copilotkit/showcase-shell:pull"
+    RAW_TOKEN   = "ghs_rawGitHubActionsToken"
+    MINTED      = "minted-bearer-from-exchange"
+
+    def fakes(extra = {})
+        RecordingHTTP.new({
+            TOKEN_URL => { status: 200, headers: {}, body: %({"token":"#{MINTED}"}) },
+            MANIFEST  => { status: 200, headers: {}, body: "" },
+        }.merge(extra))
+    end
+
+    def test_token_present_performs_basic_auth_exchange
+        http = fakes
+        g = Railway::GHCR.new(token: RAW_TOKEN, http: http)
+        assert_equal :exists, g.manifest_exists(REF)
+
+        token_call = http.calls.find { |c| c[:method] == :get && c[:url] == TOKEN_URL }
+        refute_nil token_call, "bearer_for must hit the GHCR /token exchange even when a token is present"
+
+        expected_basic = "Basic " + ["x-access-token:#{RAW_TOKEN}"].pack("m0")
+        auth = token_call[:headers]["Authorization"]
+        assert_equal expected_basic, auth,
+            "token exchange must authenticate with Basic base64(x-access-token:<token>)"
+    end
+
+    def test_manifest_read_uses_minted_bearer_not_raw_token
+        http = fakes
+        g = Railway::GHCR.new(token: RAW_TOKEN, http: http)
+        g.manifest_exists(REF)
+
+        manifest_call = http.calls.find { |c| c[:method] == :head && c[:url] == MANIFEST }
+        refute_nil manifest_call
+        assert_equal "Bearer #{MINTED}", manifest_call[:headers]["Authorization"],
+            "manifest read must use the minted bearer, never the raw GitHub token"
+        refute_equal "Bearer #{RAW_TOKEN}", manifest_call[:headers]["Authorization"],
+            "sending the raw GitHub token as a Bearer is exactly the bug that 403s"
+    end
+
+    def test_anonymous_exchange_still_works_for_public_packages
+        # No token: the exchange must still run, anonymously (no Authorization
+        # header on the /token request), and the minted token used downstream.
+        http = fakes
+        g = Railway::GHCR.new(token: nil, http: http)
+        assert_equal :exists, g.manifest_exists(REF)
+
+        token_call = http.calls.find { |c| c[:method] == :get && c[:url] == TOKEN_URL }
+        refute_nil token_call, "anonymous path must still mint a bearer via /token"
+        assert_nil token_call[:headers]["Authorization"],
+            "anonymous exchange must not send an Authorization header"
+
+        manifest_call = http.calls.find { |c| c[:method] == :head && c[:url] == MANIFEST }
+        assert_equal "Bearer #{MINTED}", manifest_call[:headers]["Authorization"]
+    end
+
+    # When a token WAS supplied and the /token exchange returns a non-2xx,
+    # that is a real failure (bad/insufficient token), NOT a license to fall
+    # back to an anonymous manifest read. bearer_for must RAISE GHCR::Error so
+    # manifest_exists's documented raise-on-failure contract holds.
+    def test_token_present_exchange_401_raises
+        http = fakes(TOKEN_URL => { status: 401, headers: {}, body: "unauthorized" })
+        g = Railway::GHCR.new(token: RAW_TOKEN, http: http)
+
+        err = assert_raises(Railway::GHCR::Error) { g.manifest_exists(REF) }
+        assert_match(/token exchange failed/i, err.message)
+    end
+
+    # A 200 with a body that is not parseable JSON is a broken exchange, not a
+    # usable bearer. bearer_for must RAISE rather than swallow JSON::ParserError.
+    def test_token_present_malformed_body_raises
+        http = fakes(TOKEN_URL => { status: 200, headers: {}, body: "<html>not json</html>" })
+        g = Railway::GHCR.new(token: RAW_TOKEN, http: http)
+
+        err = assert_raises(Railway::GHCR::Error) { g.manifest_exists(REF) }
+        assert_match(/unparseable/i, err.message)
+    end
+
+    # Regression guard: when a token was supplied and the exchange failed, the
+    # manifest HEAD must NEVER be sent — and certainly never anonymously. The
+    # old swallow-to-nil behavior issued an anonymous HEAD, conflating "no
+    # token" with "supplied token failed to exchange".
+    def test_token_present_exchange_failure_never_does_anonymous_manifest_read
+        http = fakes(TOKEN_URL => { status: 403, headers: {}, body: "forbidden" })
+        g = Railway::GHCR.new(token: RAW_TOKEN, http: http)
+
+        assert_raises(Railway::GHCR::Error) { g.manifest_exists(REF) }
+
+        manifest_call = http.calls.find { |c| c[:method] == :head && c[:url] == MANIFEST }
+        assert_nil manifest_call,
+            "no manifest HEAD must be issued when a supplied token fails the /token exchange"
+    end
+end

showcase/bin/spec/test_ghcr_digest.rb

@@ -0,0 +1,76 @@
+# frozen_string_literal: true
+
+require_relative "spec_helper"
+
+class GHCRDigestTest < Minitest::Test
+    # Fake HTTP layer for the GHCR client.
+    class FakeHTTP
+        def initialize(responses)
+            @responses = responses
+        end
+
+        def call(method:, url:, headers: {})
+            key = [method, url]
+            r = @responses[key] || @responses[url]
+            raise "no fake response for #{key.inspect}" unless r
+            r
+        end
+    end
+
+    def test_parse_image_ref_handles_all_shapes
+        g = Railway::GHCR.new
+        p1 = g.parse_image_ref("ghcr.io/copilotkit/showcase-shell:latest")
+        assert_equal "ghcr.io", p1[:registry]
+        assert_equal "copilotkit", p1[:org]
+        assert_equal "showcase-shell", p1[:name]
+        assert_equal "latest", p1[:tag]
+        assert_nil p1[:digest]
+
+        p2 = g.parse_image_ref("ghcr.io/copilotkit/showcase-shell@sha256:abc")
+        assert_equal "sha256:abc", p2[:digest]
+
+        p3 = g.parse_image_ref("ghcr.io/copilotkit/showcase-shell:latest@sha256:def")
+        assert_equal "latest", p3[:tag]
+        assert_equal "sha256:def", p3[:digest]
+    end
+
+    # bearer_for ALWAYS performs the /token exchange now, so every fake that
+    # reaches a manifest HEAD must also model a successful exchange.
+    TOKEN_URL = "https://ghcr.io/token?service=ghcr.io&scope=repository:copilotkit/showcase-shell:pull"
+
+    def token_ok
+        { TOKEN_URL => { status: 200, headers: {}, body: %({"token":"minted"}) } }
+    end
+
+    def test_resolve_digest_returns_digest_from_header
+        url = "https://ghcr.io/v2/copilotkit/showcase-shell/manifests/latest"
+        fake = FakeHTTP.new(token_ok.merge(
+            url => { status: 200, headers: { "docker-content-digest" => "sha256:beefcafe" }, body: "" },
+        ))
+        g = Railway::GHCR.new(token: "x", http: fake)
+        assert_equal "sha256:beefcafe", g.resolve_digest("ghcr.io/copilotkit/showcase-shell:latest")
+    end
+
+    def test_resolve_digest_returns_existing_digest_immediately
+        # When the ref already has @sha256:..., we don't hit the network at all.
+        g = Railway::GHCR.new(token: "x", http: nil)
+        assert_equal "sha256:abc",
+            g.resolve_digest("ghcr.io/copilotkit/showcase-shell@sha256:abc")
+    end
+
+    def test_resolve_digest_returns_nil_on_404
+        url = "https://ghcr.io/v2/copilotkit/showcase-shell/manifests/nope"
+        fake = FakeHTTP.new(token_ok.merge(url => { status: 404, headers: {}, body: "" }))
+        g = Railway::GHCR.new(token: "x", http: fake)
+        assert_nil g.resolve_digest("ghcr.io/copilotkit/showcase-shell:nope")
+    end
+
+    def test_resolve_digest_raises_on_5xx
+        url = "https://ghcr.io/v2/copilotkit/showcase-shell/manifests/latest"
+        fake = FakeHTTP.new(token_ok.merge(url => { status: 500, headers: {}, body: "boom" }))
+        g = Railway::GHCR.new(token: "x", http: fake)
+        assert_raises(Railway::GHCR::Error) do
+            g.resolve_digest("ghcr.io/copilotkit/showcase-shell:latest")
+        end
+    end
+end

showcase/bin/spec/test_ghcr_manifest_exists.rb

@@ -0,0 +1,63 @@
+# frozen_string_literal: true
+
+require_relative "spec_helper"
+
+class GHCRManifestExistsTest < Minitest::Test
+    class FakeHTTP
+        def initialize(responses)
+            @responses = responses
+        end
+
+        def call(method:, url:, headers: {})
+            key = [method, url]
+            r = @responses[key] || @responses[url]
+            raise "no fake response for #{key.inspect}" unless r
+            r
+        end
+    end
+
+    DIGEST    = "sha256:cafef00dcafef00dcafef00dcafef00dcafef00dcafef00dcafef00dcafef00d"
+    REF       = "ghcr.io/copilotkit/showcase-shell@#{DIGEST}"
+    URL       = "https://ghcr.io/v2/copilotkit/showcase-shell/manifests/#{DIGEST}"
+    TOKEN_URL = "https://ghcr.io/token?service=ghcr.io&scope=repository:copilotkit/showcase-shell:pull"
+
+    # bearer_for ALWAYS performs the /token exchange now, so every fake that
+    # reaches a manifest HEAD must also model a successful exchange.
+    def token_ok
+        { TOKEN_URL => { status: 200, headers: {}, body: %({"token":"minted"}) } }
+    end
+
+    def test_manifest_exists_returns_exists_on_200
+        fake = FakeHTTP.new(token_ok.merge(URL => { status: 200, headers: {}, body: "" }))
+        g = Railway::GHCR.new(token: "x", http: fake)
+        assert_equal :exists, g.manifest_exists(REF)
+    end
+
+    def test_manifest_exists_returns_missing_on_404
+        fake = FakeHTTP.new(token_ok.merge(URL => { status: 404, headers: {}, body: "" }))
+        g = Railway::GHCR.new(token: "x", http: fake)
+        assert_equal :missing, g.manifest_exists(REF)
+    end
+
+    def test_manifest_exists_returns_auth_failed_on_401_403
+        fake401 = FakeHTTP.new(token_ok.merge(URL => { status: 401, headers: {}, body: "" }))
+        fake403 = FakeHTTP.new(token_ok.merge(URL => { status: 403, headers: {}, body: "" }))
+        assert_equal :auth_failed, Railway::GHCR.new(token: "x", http: fake401).manifest_exists(REF)
+        assert_equal :auth_failed, Railway::GHCR.new(token: "x", http: fake403).manifest_exists(REF)
+    end
+
+    def test_manifest_exists_raises_on_5xx
+        fake = FakeHTTP.new(token_ok.merge(URL => { status: 500, headers: {}, body: "boom" }))
+        g = Railway::GHCR.new(token: "x", http: fake)
+        assert_raises(Railway::GHCR::Error) { g.manifest_exists(REF) }
+    end
+
+    def test_manifest_exists_requires_pinned_digest
+        # An unpinned tag is a programmer error here — we are verifying the
+        # CONCRETE digest we are about to promote, not resolving a tag.
+        g = Railway::GHCR.new(token: "x")
+        assert_raises(ArgumentError) do
+            g.manifest_exists("ghcr.io/copilotkit/showcase-shell:latest")
+        end
+    end
+end

showcase/bin/spec/test_ghcr_token.rb

@@ -0,0 +1,45 @@
+# frozen_string_literal: true
+
+require_relative "spec_helper"
+
+class GHCRTokenTest < Minitest::Test
+    def setup
+        @prior_github  = ENV.delete("GITHUB_TOKEN")
+        @prior_ghcr    = ENV.delete("GHCR_TOKEN")
+        @prior_railway = ENV.delete("RAILWAY_TOKEN")
+    end
+
+    def teardown
+        # Unconditionally delete any test-set values so they don't leak across
+        # tests, THEN re-set the saved priors if those were present.
+        ENV.delete("GITHUB_TOKEN")
+        ENV.delete("GHCR_TOKEN")
+        ENV.delete("RAILWAY_TOKEN")
+        ENV["GITHUB_TOKEN"]  = @prior_github  if @prior_github
+        ENV["GHCR_TOKEN"]    = @prior_ghcr    if @prior_ghcr
+        ENV["RAILWAY_TOKEN"] = @prior_railway if @prior_railway
+    end
+
+    def test_prefers_explicit_ghcr_token
+        ENV["GITHUB_TOKEN"] = "ci-token"
+        ENV["GHCR_TOKEN"]   = "explicit-pat"
+        assert_equal "explicit-pat", Railway::Auth.ghcr_token
+    end
+
+    def test_falls_back_to_github_token_in_ci
+        ENV["GITHUB_TOKEN"] = "ci-token"
+        assert_equal "ci-token", Railway::Auth.ghcr_token
+    end
+
+    def test_returns_nil_when_no_token_available
+        # No GH_AUTH_TOKEN shim, no env vars: nil (caller decides to refuse).
+        assert_nil Railway::Auth.ghcr_token
+    end
+
+    def test_does_not_return_railway_token
+        ENV["RAILWAY_TOKEN"] = "railway-bearer"
+        assert_nil Railway::Auth.ghcr_token
+    ensure
+        ENV.delete("RAILWAY_TOKEN")
+    end
+end

showcase/bin/spec/test_image_ref_colon_split.rb

@@ -0,0 +1,99 @@
+# frozen_string_literal: true
+
+# Port-safe image-ref colon splitting.
+#
+# An image ref may carry a registry PORT (`host:PORT/org/img:tag`). Stripping
+# the tag with a FIRST-colon split (`split(":", 2)`) cuts at the PORT colon and
+# corrupts the ref — `localhost:5000/copilotkit/img:latest` collapses to base
+# `localhost`. The canonical `ghcr.io/...:tag` refs (no port) are unaffected,
+# but the LAST-colon helper (`String#rsplit_colon`, also used by
+# `GHCR#parse_image_ref`) is correct for BOTH shapes. These tests pin that the
+# tag-stripping call sites (`PromoteCommand#image_shape`, `PinCommand#run`'s
+# digest rewrite) use the last-colon semantics consistently.
+
+require_relative "spec_helper"
+
+class ImageRefColonSplitTest < Minitest::Test
+    PORT_REF = "localhost:5000/copilotkit/showcase-shell:latest"
+    PORT_REF_BASE = "localhost:5000/copilotkit/showcase-shell"
+    CANONICAL_REF = "ghcr.io/copilotkit/showcase-shell:latest"
+    CANONICAL_BASE = "ghcr.io/copilotkit/showcase-shell"
+
+    # rsplit_colon is the shared helper — sanity-pin its last-colon semantics
+    # for the port-bearing ref both fixes rely on.
+    def test_rsplit_colon_splits_on_last_colon_for_port_ref
+        base, tag = PORT_REF.rsplit_colon
+        assert_equal PORT_REF_BASE, base
+        assert_equal "latest", tag
+    end
+
+    # ── image_shape: a port-bearing tag ref must classify as :tag AND the tag
+    # detection must read the LAST-colon segment. A first-colon split returns
+    # the port-and-rest segment ("5000/copilotkit/showcase-shell:latest"),
+    # which happens to still classify :tag here, so we additionally assert the
+    # trailing-colon (blank-tag) port ref is NOT misclassified as :tag — that
+    # case is GREEN only with last-colon semantics.
+    def test_image_shape_tags_port_ref
+        c = Railway::PromoteCommand.new(["--non-interactive", "--yes"])
+        assert_equal :tag, c.send(:image_shape, PORT_REF)
+        assert_equal :tag, c.send(:image_shape, CANONICAL_REF)
+    end
+
+    # A port ref with an EMPTY tag (trailing colon) has no real tag. With a
+    # first-colon split the tail is "5000/.../img:" (non-blank) → wrongly :tag.
+    # Last-colon split yields a blank tail → correctly :other. RED before the
+    # fix, GREEN after.
+    def test_image_shape_does_not_tag_port_ref_with_empty_tag
+        c = Railway::PromoteCommand.new(["--non-interactive", "--yes"])
+        assert_equal :other, c.send(:image_shape, "localhost:5000/copilotkit/showcase-shell:")
+    end
+
+    # ── PinCommand#run: when given a tag ref, it resolves the digest then
+    # rewrites the ref as "<base>@<digest>". The <base> MUST retain the full
+    # registry:port/org/img — a first-colon split drops everything after the
+    # port colon, producing the corrupt pin "localhost@sha256:...". RED before
+    # the fix, GREEN after.
+    def test_pin_preserves_port_base_when_rewriting_to_digest
+        out = run_pin_dry_run(PORT_REF)
+        assert_match(/#{Regexp.escape("#{PORT_REF_BASE}@sha256:deadbeef")}/, out,
+            "pin must preserve the full registry:port/org/img base when " \
+            "rewriting a tag ref to a digest; got:\n#{out}")
+        refute_match(/localhost@sha256:/, out,
+            "first-colon split corrupts the base to just the registry host")
+    end
+
+    # Canonical (no-port) refs must keep working identically through the fix.
+    def test_pin_preserves_canonical_base_when_rewriting_to_digest
+        out = run_pin_dry_run(CANONICAL_REF)
+        assert_match(/#{Regexp.escape("#{CANONICAL_BASE}@sha256:deadbeef")}/, out,
+            "canonical ref base must be unchanged by the fix; got:\n#{out}")
+    end
+
+    private
+
+    # Drive PinCommand#run through its tag→digest rewrite under --dry-run while
+    # keeping the test HERMETIC. PinCommand#run resolves the service id via a
+    # fresh `RollbackCommand.new([]).resolve_service_id(...)` BEFORE the dry-run
+    # early-return, which would otherwise issue a real GraphQL call (and `die!`
+    # → exit on a tokenless CI runner, aborting the whole suite). Stub GHCR
+    # digest resolution on the command instance, and stub service-id resolution
+    # at the class level so no network is touched.
+    def run_pin_dry_run(image_ref)
+        cmd = Railway::PinCommand.new(
+            ["--env", "production", "--service", "shell",
+             "--image", image_ref, "--non-interactive", "--yes", "--dry-run"],
+        )
+        cmd.instance_variable_set(:@ghcr, Object.new.tap do |o|
+            def o.resolve_digest(_ref); "sha256:deadbeef"; end
+        end)
+
+        original = Railway::RollbackCommand.instance_method(:resolve_service_id)
+        Railway::RollbackCommand.define_method(:resolve_service_id) { |_env_id, _name| "svc-stub" }
+        begin
+            out, = capture_io { cmd.run }
+            out
+        ensure
+            Railway::RollbackCommand.define_method(:resolve_service_id, original)
+        end
+    end
+end

showcase/bin/spec/test_lint_prod_covers_starters.rb

@@ -0,0 +1,145 @@
+# frozen_string_literal: true
+
+# bin/railway lint-prod — the starter-* fleet is UNDER the gate (S2).
+#
+# Background: S1 folded the 12 starter-<slug> services into the railway-envs
+# SSOT but kept them gate-INERT (gateIgnore:true / ciBuilt:false / probeDriver
+# "shell"), and verify-railway-image-refs.ts + this CLI's promote parity scope
+# carved them out as "decoupled / staging-only". S2 reverses that fence: the
+# starters are now full dual-env, gateValidated, ciBuilt SSOT entries that must
+# receive the SAME pinned-prod treatment as a showcase-* agent.
+#
+# lint-prod asserts every PROD service is pinned to an immutable @sha256
+# digest. It reads the FULL live prod snapshot with NO per-service skip, so its
+# coverage is exactly "every service live in prod". The point of S2 is that the
+# 12 starters — which ARE live in prod — are within that coverage: a starter
+# floating on a mutable :latest tag in prod is DRIFT and lint-prod must flag it.
+#
+# Red-green for THIS change:
+#   - The 12 starter names are derived from the SSOT (railway-envs.generated.json
+#     STARTER_PROD_SERVICES below) — under S1 the starters were modeled as
+#     staging-only / gate-exempt, so a "lint-prod covers starters" assertion had
+#     no SSOT basis (the prod set excluded them). After S2 the SSOT places all
+#     12 in prod, and lint-prod flags each mutable-tag starter as drift.
+#   - Regression: a digest-PINNED starter in prod is NOT flagged (the gate
+#     accepts the canonical shape), exactly like any showcase-* service.
+
+require_relative "spec_helper"
+require "stringio"
+
+class LintProdCoversStartersTest < Minitest::Test
+    # The 12 starter Railway service names, derived from the SSOT
+    # (railway-envs.generated.json) rather than re-hardcoded here, so this test
+    # moves with the SSOT and can never silently drift from it.
+    STARTER_PROD_SERVICES = Railway::SSOT_DATA
+        .fetch("services")
+        .select { |s| s.fetch("name").start_with?("starter-") }
+        .map { |s| s.fetch("name") }
+        .sort
+        .freeze
+
+    # A couple of showcase services kept in the fixture so the starter coverage
+    # is exercised ALONGSIDE the existing fleet (not in isolation).
+    SHOWCASE_SAMPLE = %w[showcase-mastra aimock].freeze
+
+    # Install a fake prod snapshot onto SnapshotCommand#build_snapshot for the
+    # duration of the block. LintProdCommand#run constructs its OWN
+    # SnapshotCommand internally, so we stub at the class level (the only seam),
+    # restoring the original method in `ensure`.
+    def with_prod_snapshot(services)
+        snapshot = { "services" => services }
+        original = Railway::SnapshotCommand.instance_method(:build_snapshot)
+        Railway::SnapshotCommand.send(:define_method, :build_snapshot) do |_env_id|
+            snapshot
+        end
+        yield
+    ensure
+        Railway::SnapshotCommand.send(:define_method, :build_snapshot, original)
+    end
+
+    def starter_svc(name, image:)
+        { "name" => name, "service_id" => "prod-#{name}", "image" => image }
+    end
+
+    # Red-green anchor (S2 source change, as Ruby sees it): the generated.json
+    # that bin/railway reads via SSOT_DATA now marks all 12 starters
+    # ciBuilt:true + gateValidated:true. Under S1 these were false (gate-inert);
+    # S2 flips them — this is the exact byte-level change lint-prod's "starters
+    # are gate-covered" guarantee rests on. RED before S2 (false), GREEN after.
+    def test_ssot_marks_all_starters_ci_built_and_gate_validated
+        starters = Railway::SSOT_DATA.fetch("services")
+            .select { |s| s.fetch("name").start_with?("starter-") }
+        assert_equal 12, starters.length
+
+        starters.each do |s|
+            name = s.fetch("name")
+            assert_equal true, s["ciBuilt"],
+                "#{name} must be ciBuilt (built by showcase_build.yml build-starters)"
+            assert_equal true, s["gateValidated"],
+                "#{name} must be gateValidated (image-ref gate validates its shape)"
+            assert_equal name, s["dispatchName"],
+                "#{name} dispatchName must equal its SSOT key (starter dispatch value)"
+            # No repoNameOverride: service name === GHCR repo name.
+            assert_nil s["repoNameOverride"],
+                "#{name} must carry no repoNameOverride (name === GHCR repo)"
+            assert_equal "starter", s.dig("probe", "driver"),
+                "#{name} probe driver must be the S3 'starter' axis contract"
+        end
+    end
+
+    # GREEN (post-S2): the 12 starters sit in prod on a mutable :latest tag.
+    # lint-prod must COVER them — i.e. flag every one as not-digest-pinned drift.
+    def test_lint_prod_flags_mutable_tag_starters_in_prod
+        # Sanity: the SSOT actually carries the full 12-starter prod fleet.
+        assert_equal 12, STARTER_PROD_SERVICES.length,
+            "expected 12 starter-* prod services in the SSOT; got " \
+            "#{STARTER_PROD_SERVICES.inspect}"
+
+        services =
+            SHOWCASE_SAMPLE.map { |n| starter_svc(n, image: "ghcr.io/copilotkit/#{n}@sha256:abc") } +
+            STARTER_PROD_SERVICES.map { |n| starter_svc(n, image: "ghcr.io/copilotkit/#{n}:latest") }
+
+        rc = nil
+        out = nil
+        with_prod_snapshot(services) do
+            cmd = Railway::LintProdCommand.new([])
+            out, _ = capture_io { rc = cmd.run }
+        end
+
+        refute_equal 0, rc,
+            "lint-prod must FAIL when prod starters float on :latest (they are " \
+            "now gate-covered); got rc=#{rc.inspect}\nout=#{out}"
+
+        # Every one of the 12 starters must be named in the drift report —
+        # proving lint-prod's coverage INCLUDES the starter fleet, not just the
+        # showcase services.
+        STARTER_PROD_SERVICES.each do |name|
+            assert_match(/#{Regexp.escape(name)}: not digest-pinned/, out,
+                "lint-prod must flag mutable-tag starter #{name} as drift")
+        end
+        assert_match(/DRIFT: 12 production service\(s\) not digest-pinned/, out,
+            "exactly the 12 starters should be flagged (showcase sample is pinned)")
+    end
+
+    # Regression: a digest-PINNED starter in prod is accepted, exactly like any
+    # showcase-* service. (Confirms the coverage is the CANONICAL shape check,
+    # not a blanket starter REFUSE.)
+    def test_lint_prod_accepts_digest_pinned_starters
+        services =
+            (SHOWCASE_SAMPLE + STARTER_PROD_SERVICES).map do |n|
+                starter_svc(n, image: "ghcr.io/copilotkit/#{n}@sha256:#{'a' * 64}")
+            end
+
+        rc = nil
+        out = nil
+        with_prod_snapshot(services) do
+            cmd = Railway::LintProdCommand.new([])
+            out, _ = capture_io { rc = cmd.run }
+        end
+
+        assert_equal 0, rc,
+            "lint-prod must PASS when every prod service (starters included) is " \
+            "digest-pinned; got rc=#{rc.inspect}\nout=#{out}"
+        assert_match(/OK: all production services digest-pinned/, out)
+    end
+end

showcase/bin/spec/test_production_protection.rb

@@ -0,0 +1,76 @@
+# frozen_string_literal: true
+
+require_relative "spec_helper"
+require "stringio"
+
+class ProductionProtectionTest < Minitest::Test
+    def test_staging_does_not_require_confirmation
+        # staging is never a production env, so this returns true without prompting.
+        assert_equal true, Railway.confirm_destructive!(
+            env_label: "staging", action: "restore", yes: false, non_interactive: true,
+        )
+    end
+
+    def test_production_without_yes_aborts
+        ex = nil
+        capture_stderr do
+            ex = assert_raises(SystemExit) do
+                Railway.confirm_destructive!(env_label: "production", action: "restore",
+                                              yes: false, non_interactive: true)
+            end
+        end
+        assert_equal 2, ex.status
+    end
+
+    def test_production_with_yes_and_non_interactive_proceeds
+        # --yes + --non-interactive proceeds without prompting.
+        result = capture_stderr do
+            assert_equal true, Railway.confirm_destructive!(
+                env_label: "production", action: "restore",
+                yes: true, non_interactive: true,
+            )
+        end
+        assert_includes result, "non-interactive"
+    end
+
+    def test_production_with_yes_prompts_and_accepts_typed_phrase
+        # Simulate the user typing 'production' on stdin.
+        original_stdin = $stdin
+        $stdin = StringIO.new("production\n")
+        capture_stderr do
+            assert_equal true, Railway.confirm_destructive!(
+                env_label: "production", action: "restore",
+                yes: true, non_interactive: false,
+            )
+        end
+    ensure
+        $stdin = original_stdin
+    end
+
+    def test_production_with_yes_rejects_wrong_phrase
+        original_stdin = $stdin
+        $stdin = StringIO.new("yes\n")
+        ex = nil
+        capture_stderr do
+            ex = assert_raises(SystemExit) do
+                Railway.confirm_destructive!(env_label: "production",
+                                              action: "restore",
+                                              yes: true, non_interactive: false)
+            end
+        end
+        assert_equal 2, ex.status
+    ensure
+        $stdin = original_stdin
+    end
+
+    private
+
+    def capture_stderr
+        original = $stderr
+        $stderr = StringIO.new
+        yield
+        $stderr.string
+    ensure
+        $stderr = original
+    end
+end

showcase/bin/spec/test_promote_cr_fixes.rb

@@ -0,0 +1,422 @@
+# frozen_string_literal: true
+
+require_relative "spec_helper"
+
+# Covers nine CR fixes against PromoteCommand:
+#   FIX-1: @promote_refs is RESET per check_p1_ghcr_digests run (not memoized
+#          across A→B preflight invocations), and execute_promotion HARD-GUARDS
+#          against a nil @promote_refs.
+#   FIX-2: P2 in-flight race compares deployed_digest vs the digest portion of
+#          @promote_refs[name] (not the snapshot's tag-form nil `digest`).
+#          Also: JSON-string `meta` is parsed (not WARN-skipped); and
+#          fetch_latest_staging_deployments sorts createdAt-desc so `.first`
+#          is genuinely latest.
+#   FIX-3: execute_promotion pre-validates ALL prod-matched services have a
+#          digest-shaped @promote_refs entry BEFORE pinning anything.
+#   FIX-4: execute_promotion rescue broadens to MutationError + GraphQL::Error
+#          + StandardError, retains PARTIAL-PROMOTION report, and drops the
+#          duplicate `warn e.message`.
+#   FIX-5: check_p1_ghcr_digests emits REFUSE: P1 ... "no image" when staging
+#          service has nil/empty image (instead of silent skip).
+#   FIX-6: P1 per-service rescue broadens to StandardError so non-GHCR errors
+#          don't bypass the loop.
+#   FIX-7: pin_and_verify raises ArgumentError immediately when called with a
+#          tag-form (non-digest) image.
+#   FIX-8: pin_and_verify timestamp gate is non-vacuous: a nil observed
+#          updatedAt does NOT declare success, even when pre_update_ts is nil.
+#   FIX-9: run_staging_probe rescues launch failures (Errno::ENOENT, etc.)
+#          and returns ok:false with a descriptive summary.
+class PromoteCRFixesTest < Minitest::Test
+    # ----- shared fakes -----
+
+    class NullGQL
+        def query(*); {}; end
+    end
+
+    # GQL that records calls and returns canned responses for P2/preflight only
+    # (no mutations).
+    class P2GQL
+        attr_reader :calls
+        def initialize(deployments_by_svc)
+            @deployments_by_svc = deployments_by_svc
+            @calls = []
+        end
+        def query(q, vars = {})
+            @calls << [q, vars]
+            if q.include?("query Deployments")
+                edges = (@deployments_by_svc[vars[:serviceId]] || []).map { |n| { "node" => n } }
+                return { "deployments" => { "edges" => edges } }
+            end
+            {}
+        end
+    end
+
+    # GHCR fake that always reports :exists and resolves any :latest -> a digest.
+    class PassGHCR
+        def initialize(resolve_map: {}); @resolve_map = resolve_map; end
+        def resolve_digest(ref)
+            return ref.split("@", 2).last if ref.include?("@sha256:")
+            @resolve_map[ref] || "sha256:resolved_for_#{ref}"
+        end
+        def manifest_exists(_); :exists; end
+        def parse_image_ref(ref); Railway::GHCR.allocate.parse_image_ref(ref); end
+    end
+
+    def make_svc(name, image:)
+        {
+            "name" => name, "service_id" => "svc-stg-#{name}",
+            "image" => image,
+            "env_keys" => [],
+            "start_command" => "node server.js", "healthcheck_path" => "/health",
+            "region" => "us-west", "replicas" => 1, "restart_policy" => "ON_FAILURE",
+        }
+    end
+
+    # =================== FIX-1: @promote_refs is reset, not memoized ===================
+
+    def test_fix1_promote_refs_resets_between_p1_runs
+        # Reuse a single PromoteCommand instance across two distinct snapshots
+        # (snapshot A and snapshot B). After the second P1 run, @promote_refs
+        # must reflect ONLY snapshot B's services — no stale A entries.
+        cmd = Railway::PromoteCommand.new([])
+        cmd.instance_variable_set(:@ghcr, PassGHCR.new)
+        # resolved_prod_image now pins staging's RUNNING digest (meta.imageDigest
+        # from the latest SUCCESS deployment), not resolve_digest(:latest). Stub
+        # the deployment lookup so a digest is resolvable for each tag-form svc.
+        cmd.define_singleton_method(:fetch_latest_staging_deployments) do |svc_id|
+            name = svc_id.sub("svc-stg-", "")
+            [{ "id" => "d", "status" => "SUCCESS",
+               "meta" => { "image" => "ghcr.io/copilotkit/#{name}:latest",
+                           "imageDigest" => "sha256:running_#{name}" } }]
+        end
+
+        snapshot_a = { "services" => [make_svc("alpha", image: "ghcr.io/copilotkit/alpha:latest")] }
+        snapshot_b = { "services" => [make_svc("beta",  image: "ghcr.io/copilotkit/beta:latest")] }
+
+        cmd.send(:check_p1_ghcr_digests, snapshot_a)
+        refs_after_a = cmd.instance_variable_get(:@promote_refs).keys.sort
+        assert_equal ["alpha"], refs_after_a
+
+        cmd.send(:check_p1_ghcr_digests, snapshot_b)
+        refs_after_b = cmd.instance_variable_get(:@promote_refs).keys.sort
+        assert_equal ["beta"], refs_after_b,
+            "stale entries from snapshot A must be cleared; got #{refs_after_b.inspect}"
+    end
+
+    def test_fix1_execute_promotion_raises_when_promote_refs_nil
+        # execute_promotion called WITHOUT a prior preflight must raise an
+        # internal-error exception (not silently treat refs as empty).
+        cmd = Railway::PromoteCommand.new([])
+        cmd.instance_variable_set(:@ghcr, PassGHCR.new)
+        # Deliberately do NOT call check_p1_ghcr_digests; @promote_refs stays nil.
+        staging = { "services" => [make_svc("x", image: "ghcr.io/copilotkit/x:latest")] }
+        prod    = { "services" => [{ "name" => "x", "service_id" => "svc-prod-x",
+                                     "image" => "ghcr.io/copilotkit/x@sha256:OLD" }] }
+        err = assert_raises(RuntimeError) do
+            cmd.send(:execute_promotion, staging, prod)
+        end
+        assert_match(/internal error.*execute_promotion.*preflight/i, err.message)
+    end
+
+    # =================== FIX-2: P2 race-check is alive ===================
+
+    def test_fix2_p2_race_check_uses_promote_refs_not_snapshot_digest
+        # Staging service is tag-form (so svc["digest"] is nil — pre-fix the
+        # race-check was DEAD CODE). After fix: P2 compares against the digest
+        # captured in @promote_refs[name].
+        cmd = Railway::PromoteCommand.new([])
+        cmd.instance_variable_set(:@promote_refs, {
+            "x" => "ghcr.io/copilotkit/x@sha256:YYY",
+        })
+        cmd.define_singleton_method(:fetch_latest_staging_deployments) do |_svc_id|
+            [{ "id" => "d1", "status" => "SUCCESS",
+               "meta" => { "image" => "ghcr.io/copilotkit/x@sha256:XXX" },
+               "createdAt" => "2026-05-28T01:00:00Z" }]
+        end
+        staging = { "services" => [{
+            "name" => "x", "service_id" => "svc-1",
+            "image" => "ghcr.io/copilotkit/x:latest",  # tag-form; no "digest"
+            "env_keys" => [],
+        }] }
+        findings = cmd.send(:check_p2_staging_deployments, staging)
+        assert(findings.any? { |f| f =~ /REFUSE: P2 \(x\).*in-flight.*sha256:XXX.*sha256:YYY/ },
+            "expected REFUSE: P2 comparing deployed XXX vs P1-resolved YYY; got: #{findings.inspect}")
+    end
+
+    def test_fix2_p2_parses_meta_when_it_is_a_json_string
+        # Some Railway responses deserialize Deployment.meta as a JSON String
+        # (not a Hash). P2 must parse it before falling back to the WARN branch.
+        cmd = Railway::PromoteCommand.new([])
+        cmd.instance_variable_set(:@promote_refs, {
+            "x" => "ghcr.io/copilotkit/x@sha256:abc",
+        })
+        cmd.define_singleton_method(:fetch_latest_staging_deployments) do |_svc_id|
+            [{ "id" => "d1", "status" => "SUCCESS",
+               "meta" => '{"image":"ghcr.io/copilotkit/x@sha256:abc"}',
+               "createdAt" => "2026-05-28T01:00:00Z" }]
+        end
+        staging = { "services" => [{
+            "name" => "x", "service_id" => "svc-1",
+            "image" => "ghcr.io/copilotkit/x:latest",
+            "env_keys" => [],
+        }] }
+        findings = cmd.send(:check_p2_staging_deployments, staging)
+        refute(findings.any? { |f| f =~ /WARN: P2 \(x\)/ },
+            "JSON-string meta must be parsed (not WARN-skipped); got: #{findings.inspect}")
+        refute(findings.any? { |f| f =~ /REFUSE: P2/ },
+            "matching digest in parsed meta must not REFUSE; got: #{findings.inspect}")
+    end
+
+    def test_fix2_fetch_latest_staging_deployments_sorts_newest_first
+        # Stub gql.query to return deployments in OLDEST-first order — the
+        # helper must sort by createdAt DESC so `.first` is the newest.
+        cmd = Railway::PromoteCommand.new([])
+        nodes = [
+            { "id" => "d-old", "status" => "SUCCESS", "createdAt" => "2026-05-01T00:00:00Z" },
+            { "id" => "d-new", "status" => "SUCCESS", "createdAt" => "2026-05-28T00:00:00Z" },
+            { "id" => "d-mid", "status" => "FAILED",  "createdAt" => "2026-05-15T00:00:00Z" },
+        ]
+        edges = nodes.map { |n| { "node" => n } }
+        fake_gql = Object.new
+        fake_gql.define_singleton_method(:query) do |_q, _vars = {}|
+            { "deployments" => { "edges" => edges } }
+        end
+        cmd.instance_variable_set(:@gql, fake_gql)
+        deployments = cmd.send(:fetch_latest_staging_deployments, "svc-1")
+        assert_equal "d-new", deployments.first["id"],
+            "fetch_latest_staging_deployments must sort newest-first; got: #{deployments.map { |d| d['id'] }.inspect}"
+    end
+
+    # =================== FIX-3: execute_promotion pre-validation ===================
+
+    def test_fix3_execute_promotion_pre_validates_all_refs_before_any_pin
+        # Two prod-matched services; ONE missing from @promote_refs. The
+        # pre-validation must REFUSE+return 1 BEFORE any serviceInstanceUpdate
+        # mutation is issued.
+        cmd = Railway::PromoteCommand.new([])
+        recorded = []
+        fake_gql = Object.new
+        fake_gql.define_singleton_method(:query) do |q, vars = {}|
+            recorded << [q, vars]
+            { "serviceInstanceUpdate" => true, "serviceInstanceDeployV2" => "dep-new" }
+        end
+        cmd.instance_variable_set(:@gql, fake_gql)
+        cmd.instance_variable_set(:@promote_refs, {
+            "a" => "ghcr.io/copilotkit/a@sha256:aaa",
+            # "b" is MISSING
+        })
+        staging = { "services" => [make_svc("a", image: "ghcr.io/copilotkit/a:latest"),
+                                   make_svc("b", image: "ghcr.io/copilotkit/b:latest")] }
+        prod = { "services" => [
+            { "name" => "a", "service_id" => "svc-prod-a", "image" => "ghcr.io/copilotkit/a@sha256:OLD" },
+            { "name" => "b", "service_id" => "svc-prod-b", "image" => "ghcr.io/copilotkit/b@sha256:OLD" },
+        ] }
+        _out, _err = capture_io { @rc = cmd.send(:execute_promotion, staging, prod) }
+        assert_equal 1, @rc, "execute_promotion must return 1 when a ref is missing"
+        assert(recorded.none? { |q, _| q.include?("serviceInstanceUpdate") },
+            "no serviceInstanceUpdate mutations should be issued; got: #{recorded.map { |q, _| q[0, 30] }.inspect}")
+    end
+
+    # =================== FIX-4: broadened rescue + partial-promotion report ===================
+
+    def test_fix4_execute_promotion_rescues_graphql_error_with_partial_report
+        # First service pins successfully; second raises Railway::GraphQL::Error
+        # on its serviceInstanceUpdate mutation. The broadened rescue must
+        # catch it and still emit the PARTIAL-PROMOTION report.
+        cmd = Railway::PromoteCommand.new([])
+        call_count = 0
+        # FakeGQL that records calls and raises GraphQL::Error on the SECOND
+        # serviceInstanceUpdate mutation.
+        fake_gql = Object.new
+        @pinned = nil
+        @pre_ts = "2026-05-28T00:00:00Z"
+        pinned_ref = nil
+        fake_gql.define_singleton_method(:query) do |q, vars = {}|
+            if q.include?("serviceInstanceUpdate")
+                call_count += 1
+                raise Railway::GraphQL::Error, "boom on update #2" if call_count == 2
+                pinned_ref = vars.dig(:input, :source, :image)
+                { "serviceInstanceUpdate" => true }
+            elsif q.include?("serviceInstanceDeployV2")
+                { "serviceInstanceDeployV2" => "dep-new" }
+            elsif q.include?("ServiceInstanceRecheck")
+                if pinned_ref
+                    { "serviceInstance" => { "id" => "i",
+                                              "source" => { "image" => pinned_ref },
+                                              "updatedAt" => "2026-05-29T00:00:01Z",
+                                              "latestDeployment" => {
+                                                  "id" => "dep-new", "status" => "SUCCESS",
+                                                  "meta" => { "imageDigest" => (pinned_ref.include?("@") ? pinned_ref.split("@", 2).last : nil) },
+                                              } } }
+                else
+                    { "serviceInstance" => { "id" => "i",
+                                              "source" => { "image" => "ghcr.io/copilotkit/x@sha256:OLD" },
+                                              "updatedAt" => "2026-05-28T00:00:00Z" } }
+                end
+            else
+                {}
+            end
+        end
+        cmd.instance_variable_set(:@gql, fake_gql)
+        cmd.instance_variable_set(:@promote_refs, {
+            "a" => "ghcr.io/copilotkit/a@sha256:aaa",
+            "b" => "ghcr.io/copilotkit/b@sha256:bbb",
+        })
+        # Silence pin_and_verify retries.
+        original = Railway::PromoteCommand.const_get(:RETRY_DELAY_SEC)
+        Railway::PromoteCommand.send(:remove_const, :RETRY_DELAY_SEC)
+        Railway::PromoteCommand.const_set(:RETRY_DELAY_SEC, 0)
+        begin
+            staging = { "services" => [make_svc("a", image: "ghcr.io/copilotkit/a:latest"),
+                                       make_svc("b", image: "ghcr.io/copilotkit/b:latest")] }
+            prod = { "services" => [
+                { "name" => "a", "service_id" => "svc-prod-a", "image" => "ghcr.io/copilotkit/a@sha256:OLD" },
+                { "name" => "b", "service_id" => "svc-prod-b", "image" => "ghcr.io/copilotkit/b@sha256:OLD" },
+            ] }
+            out, err = capture_io { @rc = cmd.send(:execute_promotion, staging, prod) }
+            combined = out + err
+            assert_equal 1, @rc
+            assert_match(/PARTIAL PROMOTION/i, combined,
+                "must emit partial-promotion report on GraphQL::Error; combined=#{combined}")
+            assert_match(/already pinned.*\ba\b/m, combined,
+                "report must name 'a' as already-pinned; combined=#{combined}")
+            assert_match(/FAILED on b/, combined,
+                "report must name 'b' as failed; combined=#{combined}")
+            # Dedup check: the inner e.message should appear ONLY inside the
+            # composed PARTIAL-PROMOTION line, not on its own line as well.
+            assert_equal 1, combined.scan(/boom on update #2/).size,
+                "duplicate `warn e.message` line must be removed; combined=#{combined}"
+        ensure
+            Railway::PromoteCommand.send(:remove_const, :RETRY_DELAY_SEC)
+            Railway::PromoteCommand.const_set(:RETRY_DELAY_SEC, original)
+        end
+    end
+
+    # =================== FIX-5: P1 REFUSE on imageless service ===================
+
+    def test_fix5_p1_refuses_when_staging_service_has_no_image
+        cmd = Railway::PromoteCommand.new([])
+        cmd.instance_variable_set(:@ghcr, PassGHCR.new)
+        staging = { "services" => [
+            { "name" => "x", "service_id" => "svc-1", "image" => nil,
+              "env_keys" => [] },
+        ] }
+        findings = cmd.send(:check_p1_ghcr_digests, staging)
+        assert(findings.any? { |f| f =~ /REFUSE: P1 \(x\).*no image/i },
+            "expected REFUSE: P1 (x) about missing image; got: #{findings.inspect}")
+    end
+
+    # =================== FIX-6: P1 per-service rescue broadens to StandardError ===================
+
+    class ArgumentErrorGHCR
+        def resolve_digest(_); "sha256:fake"; end
+        def manifest_exists(_); raise ArgumentError, "non-ghcr error"; end
+        def parse_image_ref(ref); Railway::GHCR.allocate.parse_image_ref(ref); end
+    end
+
+    def test_fix6_p1_rescue_catches_non_ghcr_errors
+        # ArgumentError raised inside manifest_exists must be caught by the
+        # per-service rescue (broadened to StandardError) — the loop must
+        # continue and the service must get a per-service REFUSE.
+        cmd = Railway::PromoteCommand.new([])
+        cmd.instance_variable_set(:@ghcr, ArgumentErrorGHCR.new)
+        # Stub the running-digest lookup so resolved_prod_image succeeds and the
+        # flow reaches manifest_exists (which raises the ArgumentError under test).
+        cmd.define_singleton_method(:fetch_latest_staging_deployments) do |svc_id|
+            name = svc_id.sub("svc-stg-", "")
+            [{ "id" => "d", "status" => "SUCCESS",
+               "meta" => { "image" => "ghcr.io/copilotkit/#{name}:latest",
+                           "imageDigest" => "sha256:running_#{name}" } }]
+        end
+        staging = { "services" => [
+            make_svc("a", image: "ghcr.io/copilotkit/a:latest"),
+            make_svc("b", image: "ghcr.io/copilotkit/b:latest"),
+        ] }
+        findings = cmd.send(:check_p1_ghcr_digests, staging)
+        # Two services, each one should fail with ArgumentError-bearing REFUSE.
+        assert(findings.any? { |f| f =~ /REFUSE: P1 \(a\).*ArgumentError.*non-ghcr error/ },
+            "service 'a' must record a per-service REFUSE for ArgumentError; got: #{findings.inspect}")
+        assert(findings.any? { |f| f =~ /REFUSE: P1 \(b\).*ArgumentError.*non-ghcr error/ },
+            "service 'b' must record a per-service REFUSE for ArgumentError; got: #{findings.inspect}")
+    end
+
+    # =================== FIX-7: pin_and_verify upfront digest guard ===================
+
+    def test_fix7_pin_and_verify_raises_arg_error_on_tag_form_image
+        # Pre-fix: pin_and_verify would dutifully attempt N retries before
+        # raising a misleading MutationError. After fix: ArgumentError fires
+        # immediately, no retries.
+        gql = Object.new
+        gql.define_singleton_method(:query) { |*| raise "no query should be issued" }
+        err = assert_raises(ArgumentError) do
+            Railway::PromoteCommand.pin_and_verify(gql,
+                service_id: "svc-x", env_id: "env-prod",
+                image: "ghcr.io/copilotkit/x:latest",
+                sleeper: ->(_) {})
+        end
+        assert_match(/pin_and_verify.*@sha256.*pinned/i, err.message,
+            "ArgumentError message must explain the digest requirement; got: #{err.message}")
+    end
+
+    # =================== FIX-8: pin_and_verify ts gate is non-vacuous ===================
+
+    def test_fix8_pin_and_verify_requires_non_nil_updated_at_even_when_pre_ts_nil
+        # pre_update_ts is nil (new prod instance). Recheck returns matching
+        # digest but a nil updatedAt. Pre-fix: ts_ok was vacuously true → success.
+        # After fix: ts_ok requires actual_ts to be non-nil → kept retrying →
+        # MutationError after RETRY_COUNT attempts.
+        gql = Object.new
+        pinned = nil
+        gql.define_singleton_method(:query) do |q, vars = {}|
+            if q.include?("serviceInstanceUpdate")
+                pinned = vars.dig(:input, :source, :image)
+                { "serviceInstanceUpdate" => true }
+            elsif q.include?("serviceInstanceDeployV2")
+                { "serviceInstanceDeployV2" => "dep-new" }
+            elsif q.include?("ServiceInstanceRecheck")
+                if pinned.nil?
+                    # Pre-update: brand-new instance — both fields are nil.
+                    { "serviceInstance" => nil }
+                else
+                    { "serviceInstance" => { "id" => "i",
+                                              "source" => { "image" => pinned },
+                                              "updatedAt" => nil } }
+                end
+            else
+                {}
+            end
+        end
+        err = assert_raises(Railway::PromoteCommand::MutationError) do
+            Railway::PromoteCommand.pin_and_verify(gql,
+                service_id: "svc-x", env_id: "env-prod",
+                image: "ghcr.io/copilotkit/x@sha256:abc",
+                sleeper: ->(_) {})
+        end
+        assert_match(/did not observe image advance/i, err.message,
+            "expected timeout-style MutationError; got: #{err.message}")
+    end
+
+    # =================== FIX-9: run_staging_probe rescue on launch failure ===================
+
+    def test_fix9_run_staging_probe_returns_clean_failure_on_io_popen_error
+        cmd = Railway::PromoteCommand.new([])
+        # The probe binary must APPEAR present so we reach the IO.popen call.
+        # (Skip the File.exist? early-return.)
+        original_exist = File.method(:exist?)
+        File.define_singleton_method(:exist?) { |_path| true }
+        # Stub IO.popen to raise Errno::ENOENT (npx missing on PATH).
+        original_popen = IO.method(:popen)
+        IO.define_singleton_method(:popen) do |*_args, **_kw, &_blk|
+            raise Errno::ENOENT, "npx"
+        end
+        begin
+            result = cmd.send(:run_staging_probe, services: ["x"])
+            assert_equal false, result[:ok], "must return ok:false on launch failure"
+            assert_match(/staging probe failed to launch.*ENOENT/i, result[:summary],
+                "summary must describe the launch failure; got: #{result[:summary].inspect}")
+        ensure
+            IO.define_singleton_method(:popen, &original_popen)
+            File.define_singleton_method(:exist?, &original_exist)
+        end
+    end
+end

showcase/bin/spec/test_promote_execute.rb

@@ -0,0 +1,195 @@
+# frozen_string_literal: true
+
+require_relative "spec_helper"
+
+# execute_promotion must resolve any staging tag (e.g. ":latest") to its
+# concrete GHCR digest and pin THAT to prod — never a mutable tag. This is
+# the core invariant of the showcase deploy model (P6 enforces shape).
+class PromoteExecuteTest < Minitest::Test
+    # A FakeGQL that returns the most-recently-pinned image on recheck,
+    # so pin_and_verify sees the advance. Records all calls for assertions.
+    class RecordingGQL
+        def initialize(pre_ts: "2026-05-28T00:00:00Z")
+            @calls = []
+            @pinned_image = nil
+            @pre_ts = pre_ts
+        end
+        attr_reader :calls
+
+        def query(q, vars = {})
+            @calls << [q, vars]
+            if q.include?("serviceInstanceUpdate")
+                @pinned_image = vars.dig(:input, :source, :image)
+                { "serviceInstanceUpdate" => true }
+            elsif q.include?("serviceInstanceDeployV2")
+                # New deployment spawned; returns the new deployment id.
+                { "serviceInstanceDeployV2" => "dep-new" }
+            elsif q.include?("ServiceInstanceRecheck")
+                if @pinned_image.nil?
+                    # Pre-update snapshot.
+                    {
+                        "serviceInstance" => {
+                            "id" => "i",
+                            "source" => { "image" => "ghcr.io/copilotkit/x@sha256:OLD" },
+                            "updatedAt" => @pre_ts,
+                        },
+                    }
+                else
+                    # Post-update: config advanced AND the new deployment
+                    # (dep-new) has SUCCEEDED serving the pinned digest, so both
+                    # the config recheck and the bug-#2 serving gate pass.
+                    pinned_digest = @pinned_image.include?("@") ? @pinned_image.split("@", 2).last : nil
+                    {
+                        "serviceInstance" => {
+                            "id" => "i",
+                            "source" => { "image" => @pinned_image },
+                            "updatedAt" => "2026-05-29T00:00:01Z",
+                            "latestDeployment" => {
+                                "id" => "dep-new", "status" => "SUCCESS",
+                                "meta" => { "imageDigest" => pinned_digest },
+                            },
+                        },
+                    }
+                end
+            else
+                {}
+            end
+        end
+
+        # Find the `image:` arg passed to the serviceInstanceUpdate mutation.
+        def pinned_image
+            row = @calls.find { |q, _| q.include?("serviceInstanceUpdate") }
+            row && row[1].dig(:input, :source, :image)
+        end
+    end
+
+    # FakeGHCR that maps tag-form refs to a digest, and reports :exists for
+    # the corresponding digest-pinned ref so P1 passes.
+    class FakeGHCR
+        # `resolve_map` is { "ghcr.io/org/name:tag" => "sha256:abc..." } or nil for unresolvable.
+        # `exists_set` is the set of digest-pinned refs that report :exists.
+        def initialize(resolve_map: {}, exists_set: nil)
+            @resolve_map = resolve_map
+            @exists_set = exists_set
+        end
+
+        def resolve_digest(ref)
+            # Pass-through for already-pinned refs.
+            return ref.split("@", 2).last if ref.include?("@sha256:")
+            @resolve_map[ref]
+        end
+
+        def manifest_exists(ref)
+            return :missing if @exists_set && !@exists_set.include?(ref)
+            :exists
+        end
+
+        # Delegate to the real GHCR parser — pure function, no I/O.
+        def parse_image_ref(ref)
+            Railway::GHCR.allocate.parse_image_ref(ref)
+        end
+    end
+
+    # Build a command with staging-tag service, snapshot the prod target, and
+    # inject fakes. Returns [cmd, gql].
+    def build_cmd(staging_image:, resolve_map:, exists_set: nil)
+        # --confirm-divergence: retained as a harmless no-op (the only finding,
+        # missing EXPECTED_DOMAINS, is now ADVISORY and never blocks); we are
+        # testing pin behavior. All CRITICAL_ENV_KEYS present so the (now
+        # unconditional) critical env-key presence assertion does not fire.
+        cmd = Railway::PromoteCommand.new(["--non-interactive", "--yes", "--confirm-divergence"])
+        cmd.parser.parse!(cmd.argv)
+        cmd.instance_variable_set(:@staging_snapshot, {
+            "services" => [{
+                "name" => "x", "service_id" => "svc-staging",
+                "image" => staging_image,
+                "env_keys" => Railway::CRITICAL_ENV_KEYS.dup,
+                "start_command" => "node server.js", "healthcheck_path" => "/health",
+                "region" => "us-west", "replicas" => 1, "restart_policy" => "ON_FAILURE",
+            }],
+        })
+        cmd.instance_variable_set(:@prod_snapshot, {
+            "services" => [{
+                "name" => "x", "service_id" => "svc-prod",
+                "image" => "ghcr.io/copilotkit/x@sha256:OLD",
+                "env_keys" => Railway::CRITICAL_ENV_KEYS.dup,
+                "start_command" => "node server.js", "healthcheck_path" => "/health",
+                "region" => "us-west", "replicas" => 1, "restart_policy" => "ON_FAILURE",
+            }],
+        })
+        gql = RecordingGQL.new
+        cmd.instance_variable_set(:@gql, gql)
+        cmd.instance_variable_set(:@ghcr, FakeGHCR.new(resolve_map: resolve_map, exists_set: exists_set))
+        # resolved_prod_image pins staging's RUNNING digest, sourced from the
+        # latest SUCCESS deployment's meta.imageDigest (image-drift.ts mechanism).
+        # Map the staging RUNNING digest from resolve_map (the test's notion of
+        # the resolvable digest) or, for an already-digest-pinned staging image,
+        # the embedded digest. When resolve_map is EMPTY *and* the image is
+        # tag-only, the deployment carries NO imageDigest — modelling the
+        # "no running digest resolvable" REFUSE path (replaces the old
+        # "GHCR :latest unresolvable" REFUSE).
+        running_digest = resolve_map.values.first ||
+            (staging_image.include?("@") ? staging_image.split("@", 2).last : nil)
+        cmd.define_singleton_method(:fetch_latest_staging_deployments) do |_svc_id|
+            meta = { "image" => "ghcr.io/copilotkit/x:latest" }
+            meta["imageDigest"] = running_digest if running_digest
+            [{ "id" => "d", "status" => "SUCCESS", "meta" => meta }]
+        end
+        cmd.define_singleton_method(:run_staging_probe) { |services:| { ok: true, summary: "" } }
+        [cmd, gql]
+    end
+
+    # Silence pin_and_verify's 10s-per-retry waits. RETRY_DELAY_SEC is a
+    # constant on PromoteCommand; remap to 0 around the test body and restore.
+    def with_fast_sleeper
+        original = Railway::PromoteCommand.const_get(:RETRY_DELAY_SEC)
+        Railway::PromoteCommand.send(:remove_const, :RETRY_DELAY_SEC)
+        Railway::PromoteCommand.const_set(:RETRY_DELAY_SEC, 0)
+        yield
+    ensure
+        Railway::PromoteCommand.send(:remove_const, :RETRY_DELAY_SEC)
+        Railway::PromoteCommand.const_set(:RETRY_DELAY_SEC, original)
+    end
+
+    def test_resolves_staging_tag_to_digest_and_pins_digest_to_prod
+        # (a) staging image is `:latest`; resolve_map maps it to a digest.
+        cmd, gql = build_cmd(
+            staging_image: "ghcr.io/copilotkit/x:latest",
+            resolve_map:   { "ghcr.io/copilotkit/x:latest" => "sha256:abc123" },
+        )
+        out, _ = with_fast_sleeper { capture_io { @rc = cmd.run_with_preflight_only } }
+        assert_equal 0, @rc, "promote should succeed when staging tag resolves cleanly; got out=#{out}"
+        pinned = gql.pinned_image
+        assert_equal "ghcr.io/copilotkit/x@sha256:abc123", pinned,
+            "must pin the DIGEST-form ref, not the :latest tag; pinned=#{pinned.inspect}"
+        refute_includes pinned.to_s, ":latest", "must not pin a mutable tag"
+        assert_match(/promoted x -> ghcr\.io\/copilotkit\/x@sha256:abc123/, out)
+    end
+
+    def test_refuses_when_staging_tag_cannot_be_resolved_to_digest
+        # (b) staging :latest that GHCR cannot resolve (resolve_digest returns nil)
+        # → REFUSE; serviceInstanceUpdate is NEVER called.
+        cmd, gql = build_cmd(
+            staging_image: "ghcr.io/copilotkit/x:latest",
+            resolve_map:   {}, # unresolvable
+        )
+        out, _ = with_fast_sleeper { capture_io { @rc = cmd.run_with_preflight_only } }
+        assert_equal 1, @rc, "must refuse when staging tag is unresolvable"
+        assert_match(/cannot resolve .*:latest.* GHCR digest/i, out)
+        refuses_update = gql.calls.any? { |q, _| q.include?("serviceInstanceUpdate") }
+        refute refuses_update, "must NOT call serviceInstanceUpdate when refusing on unresolvable tag"
+    end
+
+    def test_already_digest_pinned_staging_image_is_used_as_is
+        # (c) Unit test of the resolved-prod-image helper. A staging service
+        # whose image is already digest-pinned must be passed through unchanged
+        # (no GHCR tag lookup needed, no rewrite). (This shape is irregular for
+        # showcase staging — P6 would normally REFUSE staging != :tag — but the
+        # helper itself must be safe and idempotent.)
+        cmd = Railway::PromoteCommand.new([])
+        cmd.instance_variable_set(:@ghcr, FakeGHCR.new(resolve_map: {}))
+        svc = { "name" => "x", "image" => "ghcr.io/copilotkit/x@sha256:def456" }
+        assert_equal "ghcr.io/copilotkit/x@sha256:def456",
+            cmd.send(:resolved_prod_image, svc)
+    end
+end

showcase/bin/spec/test_promote_fleet_target_invariant.rb

@@ -0,0 +1,286 @@
+# frozen_string_literal: true
+
+# bin/railway promote — fleet_*/target_* accessor invariant regression.
+#
+# This is the PINNING test for the convention enforced by the
+# fleet_staging/fleet_prod/target_staging/target_prod accessors. The
+# accessors only exist because TWO prior bugs were caused by a
+# fleet-scoped invariant reading the post-narrowing snapshot ivar:
+#
+#   (1) check_expected_prod_domains read the narrowed prod → fleet-wide
+#       public hosts looked "missing" on every single-service promote →
+#       spurious WARN → workflow refused without --confirm-divergence.
+#   (2) check_service_set_parity diffed narrowed staging vs narrowed prod
+#       (always equal post-narrow) → the invariant became a tautology
+#       and the "target-absent-from-prod" silent-skip path was no longer
+#       gated by a REFUSE.
+#
+# The fix replaced raw ivar reads in run_with_preflight_only with calls
+# to fleet_*/target_* accessors. This test pins the contract that those
+# accessors must point at DIFFERENT views when a single-service narrow
+# has been applied — i.e. flipping a fleet-scoped read to the target
+# view (or vice versa) must produce a regression we can see.
+#
+# Concretely: with a healthy fleet narrowed to a single domain-less
+# service, the promote must succeed silently; but if we monkey-patch
+# fleet_prod to return target_prod (the WRONG view), the same fixture
+# must produce the historic spurious "WARN: production missing expected
+# custom domains" finding. Symmetrically, swapping fleet_staging to
+# target_staging makes the service-set-parity check tautological — we
+# pin that the un-swapped version still catches a real fleet-shape
+# divergence (target-absent-from-prod surfaces as REFUSE).
+#
+# Sister spec: test_promote_single_service_fleet_invariants.rb pins the
+# behavioral CONTRACT (rc=0 on healthy fleet, REFUSE on absent target).
+# This spec pins that the accessor INDIRECTION is what enforces that
+# contract, so a future "simplify back to raw ivars" rewrite trips a
+# red light.
+
+require_relative "spec_helper"
+require "stringio"
+
+class PromoteFleetTargetInvariantTest < Minitest::Test
+    # Reuse the inline fakes pattern from
+    # test_promote_single_service_fleet_invariants.rb — kept inline so
+    # this spec is self-contained and unaffected by changes in sibling
+    # fixtures.
+    class FakeGQLBenign
+        attr_reader :calls
+        def initialize
+            @calls = []
+            @pinned_by_service = {}
+            @ts_counter = 0
+        end
+        def query(q, vars = {})
+            @calls << [q, vars]
+            sid = vars[:serviceId]
+            if q.include?("serviceInstanceUpdate")
+                @pinned_by_service[sid] = vars.dig(:input, :source, :image)
+                { "serviceInstanceUpdate" => true }
+            elsif q.include?("serviceInstanceDeployV2")
+                { "serviceInstanceDeployV2" => "dep-#{sid}" }
+            elsif q.include?("ServiceInstanceRecheck")
+                pinned = @pinned_by_service[sid]
+                if pinned.nil?
+                    {
+                        "serviceInstance" => {
+                            "id" => "i",
+                            "source" => { "image" => "ghcr.io/copilotkit/old@sha256:OLD" },
+                            "updatedAt" => "2026-05-28T00:00:00Z",
+                        },
+                    }
+                else
+                    @ts_counter += 1
+                    pinned_digest = pinned.include?("@") ? pinned.split("@", 2).last : nil
+                    {
+                        "serviceInstance" => {
+                            "id" => "i",
+                            "source" => { "image" => pinned },
+                            "updatedAt" => "2026-05-29T00:00:#{format('%02d', @ts_counter)}Z",
+                            "latestDeployment" => {
+                                "id" => "dep-#{sid}", "status" => "SUCCESS",
+                                "meta" => { "imageDigest" => pinned_digest },
+                            },
+                        },
+                    }
+                end
+            else
+                { "deployments" => { "edges" => [] } }
+            end
+        end
+
+        def pinned_services
+            @calls.select { |q, _| q.include?("serviceInstanceUpdate") }
+                  .map { |_, vars| [vars[:serviceId], vars.dig(:input, :source, :image)] }
+        end
+    end
+
+    class FakeGHCR
+        def initialize(resolve_map: {})
+            @resolve_map = resolve_map
+        end
+        def resolve_digest(ref)
+            return ref.split("@", 2).last if ref.include?("@sha256:")
+            @resolve_map[ref] || "sha256:default_digest_for_#{ref.sub(/[^a-z0-9]/i, '_')[0, 16]}"
+        end
+        def manifest_exists(_ref); :exists; end
+        def parse_image_ref(ref); Railway::GHCR.allocate.parse_image_ref(ref); end
+    end
+
+    def make_staging_service(name)
+        {
+            "name" => name, "service_id" => "svc-#{name}",
+            "image" => "ghcr.io/copilotkit/#{name}:latest",
+            # All CRITICAL_ENV_KEYS present so the (now unconditional) critical
+            # env-key presence assertion does not fire — this spec isolates the
+            # fleet/target accessor invariant, not env-key parity.
+            "env_keys" => Railway::CRITICAL_ENV_KEYS.dup,
+            "start_command" => "node server.js", "healthcheck_path" => "/health",
+            "region" => "us-west", "replicas" => 1, "restart_policy" => "ON_FAILURE",
+        }
+    end
+
+    def make_prod_service(name, custom_domains: [])
+        {
+            "name" => name, "service_id" => "prod-#{name}",
+            "image" => "ghcr.io/copilotkit/#{name}@sha256:OLD#{name.gsub(/[^a-z0-9]/i, '')}",
+            "env_keys" => Railway::CRITICAL_ENV_KEYS.dup,
+            "start_command" => "node server.js", "healthcheck_path" => "/health",
+            "region" => "us-west", "replicas" => 1, "restart_policy" => "ON_FAILURE",
+            "custom_domains" => custom_domains,
+        }
+    end
+
+    # A healthy fleet whose UNION of prod custom_domains covers the
+    # SSOT-published EXPECTED_DOMAINS[PRODUCTION_ENV_ID] set. The
+    # targeted service ("aimock") is intentionally domain-less so that
+    # the NARROWED prod view (target_prod) does NOT carry the public
+    # hosts — the post-narrowing view is precisely the broken view a
+    # fleet-scoped check must NOT see.
+    def install_fleet_fixture(cmd, gql, ghcr, target: "aimock")
+        domain_services_prod = [
+            make_prod_service("dashboard", custom_domains: ["dashboard.showcase.copilotkit.ai"]),
+            make_prod_service("docs",      custom_domains: ["docs.copilotkit.ai"]),
+            make_prod_service("dojo",      custom_domains: ["dojo.showcase.copilotkit.ai"]),
+            make_prod_service("webhooks",  custom_domains: ["hooks.showcase.copilotkit.ai"]),
+            make_prod_service("shell",     custom_domains: ["showcase.copilotkit.ai"]),
+            make_prod_service(target),
+            make_prod_service("harness"),
+        ]
+        staging_services = (domain_services_prod.map { |s| s["name"] }).uniq.map { |n| make_staging_service(n) }
+
+        cmd.instance_variable_set(:@staging_snapshot, { "services" => staging_services })
+        cmd.instance_variable_set(:@prod_snapshot,    { "services" => domain_services_prod })
+        cmd.instance_variable_set(:@gql, gql)
+        cmd.instance_variable_set(:@ghcr, ghcr)
+
+        cmd.define_singleton_method(:fetch_latest_staging_deployments) do |service_id|
+            name = service_id.sub(/^svc-/, "")
+            ghcr_obj = instance_variable_get(:@ghcr)
+            digest = ghcr_obj.resolve_digest("ghcr.io/copilotkit/#{name}:latest")
+            [{ "id" => "d", "status" => "SUCCESS", "meta" => { "image" => "ghcr.io/copilotkit/#{name}@#{digest}" } }]
+        end
+        cmd.define_singleton_method(:run_staging_probe) { |services:| { ok: true, summary: "" } }
+    end
+
+    def build_cmd(argv)
+        # CRITICAL: mirror real workflow — no --confirm-divergence.
+        Railway::PromoteCommand.new(argv + ["--non-interactive", "--yes"])
+    end
+
+    def with_fast_sleeper
+        original = Railway::PromoteCommand.const_get(:RETRY_DELAY_SEC)
+        Railway::PromoteCommand.send(:remove_const, :RETRY_DELAY_SEC)
+        Railway::PromoteCommand.const_set(:RETRY_DELAY_SEC, 0)
+        yield
+    ensure
+        Railway::PromoteCommand.send(:remove_const, :RETRY_DELAY_SEC)
+        Railway::PromoteCommand.const_set(:RETRY_DELAY_SEC, original)
+    end
+
+    # ── Positive: with the CORRECT accessor wiring, a healthy
+    # single-service promote produces neither spurious domain WARN nor
+    # spurious set-parity REFUSE. This is the "green" half of the gate.
+    def test_correct_accessors_produce_no_spurious_fleet_findings
+        gql  = FakeGQLBenign.new
+        ghcr = FakeGHCR.new(resolve_map: { "ghcr.io/copilotkit/aimock:latest" => "sha256:NEW_AIMOCK" })
+        cmd = build_cmd(["aimock"])
+        install_fleet_fixture(cmd, gql, ghcr, target: "aimock")
+
+        rc = nil
+        out, _ = with_fast_sleeper { capture_io { rc = cmd.run } }
+
+        assert_equal 0, rc,
+            "healthy fleet, single-service promote, correct accessors → rc=0. " \
+            "Got rc=#{rc.inspect}; out=\n#{out}"
+        refute_match(/WARN: production missing expected custom domains/, out,
+            "fleet_prod must see the FULL prod fleet — no spurious 'missing " \
+            "fleet domains' WARN should fire when the fleet legitimately " \
+            "carries every EXPECTED_DOMAINS host")
+        refute_match(/REFUSE: services in (?:staging|prod) not in (?:prod|staging)/, out,
+            "fleet_staging/fleet_prod must see the FULL fleet — set-parity " \
+            "must not surface a spurious REFUSE when the fleet is in sync")
+    end
+
+    # ── Gate: if a future refactor flips check_expected_prod_domains to
+    # read the NARROWED view (i.e. target_prod instead of fleet_prod),
+    # the same fixture must produce the historic spurious WARN. We
+    # simulate that flip by monkey-patching fleet_prod on the instance
+    # to return target_prod (the wrong view), then assert the WARN
+    # reappears. This makes the accessor distinction load-bearing.
+    def test_swapping_fleet_prod_to_target_prod_recreates_spurious_domain_finding
+        gql  = FakeGQLBenign.new
+        ghcr = FakeGHCR.new(resolve_map: { "ghcr.io/copilotkit/aimock:latest" => "sha256:NEW_AIMOCK" })
+        cmd = build_cmd(["aimock"])
+        install_fleet_fixture(cmd, gql, ghcr, target: "aimock")
+
+        # Flip fleet_prod → target_prod (the broken pre-fix wiring).
+        # Use a singleton method that delegates to target_prod so the
+        # narrowing applied by `run` still drives the result.
+        #
+        # We also flip fleet_staging in lockstep so the SET-PARITY
+        # check stays clean (full-fleet staging vs narrowed prod would
+        # mismatch on every other service name and surface as a REFUSE
+        # that short-circuits the finding). Isolating the BUG #1 symptom
+        # requires both reads to be uniformly broken — which is exactly
+        # the regression we're pinning against (a sweeping refactor
+        # that rewires both accessors at once).
+        cmd.define_singleton_method(:fleet_prod)    { send(:target_prod) }
+        cmd.define_singleton_method(:fleet_staging) { send(:target_staging) }
+
+        rc = nil
+        out, _ = with_fast_sleeper { capture_io { rc = cmd.run } }
+
+        # Per the 2026-06-22 prod↔staging comparison policy, missing expected
+        # prod domains is now an ADVISORY (report-only) finding, not a blocking
+        # WARN. The accessor distinction is STILL load-bearing: reading the
+        # narrowed view surfaces the spurious "missing domains" finding that the
+        # full-fleet view would not. We pin that the finding REAPPEARS (proving
+        # the accessor wiring matters) — but because it is advisory it no longer
+        # blocks the promote.
+        assert_match(/ADVISORY: production missing expected custom domains/, out,
+            "swapping fleet_prod to target_prod must reproduce the original " \
+            "BUG #1 symptom (spurious 'missing fleet domains' finding on a " \
+            "healthy fleet narrowed to a domain-less service). If this " \
+            "assertion fails, the accessor distinction is no longer " \
+            "load-bearing — check_expected_prod_domains may have been " \
+            "moved or its argument source changed."
+        )
+        assert_equal 0, rc,
+            "the spurious domain finding is now ADVISORY, so it must NOT " \
+            "block the promote (the historic blocking WARN was demoted)"
+    end
+
+    # ── Symmetric gate: if a future refactor flips
+    # check_service_set_parity to read the narrowed staging/prod, the
+    # invariant becomes tautological and the target-absent-from-prod
+    # case stops surfacing a REFUSE. Pin that the un-swapped version
+    # catches a real fleet-shape divergence (target only in staging).
+    def test_correct_accessors_catch_target_absent_from_prod
+        gql  = FakeGQLBenign.new
+        ghcr = FakeGHCR.new(resolve_map: { "ghcr.io/copilotkit/aimock:latest" => "sha256:NEW_AIMOCK" })
+        cmd = build_cmd(["aimock"])
+        install_fleet_fixture(cmd, gql, ghcr, target: "aimock")
+
+        # Surgically remove "aimock" from prod ONLY (full fleet keeps
+        # all other services + all expected domains). Staging keeps
+        # "aimock". After narrowing both snapshots to "aimock", a check
+        # reading the narrowed view would see [aimock] vs [] — which
+        # superficially seems to also catch the divergence — but the
+        # POINT of using fleet_* is so that the SAME check fires
+        # regardless of whether the run is full-fleet or single-service.
+        full_prod = cmd.instance_variable_get(:@prod_snapshot)
+        full_prod = full_prod.merge("services" => full_prod["services"].reject { |s| s["name"] == "aimock" })
+        cmd.instance_variable_set(:@prod_snapshot, full_prod)
+
+        rc = nil
+        out, _ = with_fast_sleeper { capture_io { rc = cmd.run } }
+
+        refute_equal 0, rc,
+            "target absent from prod must FAIL LOUD (nonzero rc). Got " \
+            "rc=#{rc.inspect}; out=\n#{out}"
+        assert_match(/REFUSE: services in staging not in prod/, out,
+            "fleet_staging vs fleet_prod must surface the staging-only " \
+            "target as a clear REFUSE")
+    end
+end

showcase/bin/spec/test_promote_healthcheck_reassert.rb

@@ -0,0 +1,107 @@
+# frozen_string_literal: true
+
+require_relative "spec_helper"
+
+# Re-assertion of the SSOT healthcheckPath on the promote pin (the durable fix
+# for the aimock silent-null incident). pin_and_verify must:
+#   - INCLUDE healthcheckPath in the serviceInstanceUpdate input when the SSOT
+#     declares a path for the service+env (e.g. aimock -> /health), and
+#   - OMIT it entirely (never send `healthcheckPath: null`) when the SSOT tracks
+#     none (a live-null service like docs), so a null-live service is never
+#     accidentally cleared OR set to a wrong path.
+class PromoteHealthcheckReassertTest < Minitest::Test
+    class FakeGQL
+        def initialize(plan); @plan = plan; @calls = []; end
+        attr_reader :calls
+
+        def query(q, vars = {})
+            @calls << [q, vars]
+            step = @plan.shift
+            raise "fake exhausted at call ##{@calls.size}: #{q[0, 40]}" unless step
+            raise step[:raise] if step[:raise]
+            step[:data]
+        end
+    end
+
+    NEW_DEPLOY_ID = "dep-new"
+
+    def pre(ts) = { data: { "serviceInstance" => { "id" => "i", "source" => { "image" => "ghcr.io/copilotkit/x@sha256:OLD" }, "updatedAt" => ts } } }
+    def post(image:, ts:) = { data: { "serviceInstance" => { "id" => "i", "source" => { "image" => image }, "updatedAt" => ts } } }
+    def deploy_ok(id = NEW_DEPLOY_ID) = { data: { "serviceInstanceDeployV2" => id } }
+
+    def serving(digest:, status: "SUCCESS", deploy_id: NEW_DEPLOY_ID)
+        {
+            data: {
+                "serviceInstance" => {
+                    "id" => "i",
+                    "source" => { "image" => "ghcr.io/copilotkit/x@#{digest}" },
+                    "updatedAt" => "2026-05-28T03:00:00Z",
+                    "latestDeployment" => {
+                        "id" => deploy_id, "status" => status,
+                        "meta" => { "imageDigest" => digest },
+                    },
+                },
+            },
+        }
+    end
+
+    def happy_plan
+        [
+            pre("2026-05-27T00:00:00Z"),
+            { data: { "serviceInstanceUpdate" => true } },
+            deploy_ok,
+            post(image: "ghcr.io/copilotkit/x@sha256:NEW", ts: "2026-05-28T01:00:00Z"),
+            serving(digest: "sha256:NEW"),
+        ]
+    end
+
+    # The serviceInstanceUpdate is always the 2nd GQL call (after the pre-update
+    # snapshot). Returns [query_string, vars] for that call.
+    def update_call(gql) = gql.calls[1]
+
+    def test_includes_healthcheck_path_when_ssot_declares_one
+        gql = FakeGQL.new(happy_plan)
+        Railway::PromoteCommand.pin_and_verify(gql,
+            service_id: "s", env_id: "e", image: "ghcr.io/copilotkit/x@sha256:NEW",
+            healthcheck_path: "/health", sleeper: ->(_n) {})
+
+        query, vars = update_call(gql)
+        # The whole input rides as a single $input: ServiceInstanceUpdateInput!
+        # variable; healthcheckPath is a NESTED key of that input (Railway infers
+        # its type from the input schema). See deploy-to-railway.ts /
+        # provision-starter-fleet.ts, which set healthcheckPath the same way.
+        assert_match(/\$input:\s*ServiceInstanceUpdateInput!/, query,
+            "update mutation should pass a single $input: ServiceInstanceUpdateInput! variable")
+        assert_match(/input:\s*\$input/, query,
+            "serviceInstanceUpdate should receive the input via $input")
+        assert_equal "/health", vars.dig(:input, :healthcheckPath),
+            "input.healthcheckPath should carry the SSOT healthcheckPath"
+    end
+
+    def test_omits_healthcheck_path_when_ssot_has_none
+        gql = FakeGQL.new(happy_plan)
+        # nil healthcheck_path == a live-null service (e.g. docs).
+        Railway::PromoteCommand.pin_and_verify(gql,
+            service_id: "s", env_id: "e", image: "ghcr.io/copilotkit/x@sha256:NEW",
+            healthcheck_path: nil, sleeper: ->(_n) {})
+
+        query, vars = update_call(gql)
+        # The image-only mutation must be used: NO healthcheckPath anywhere — we
+        # must NEVER send `healthcheckPath: null`, which would clear it.
+        refute_match(/healthcheckPath/, query,
+            "image-only mutation must not mention healthcheckPath")
+        refute vars.dig(:input)&.key?(:healthcheckPath),
+            "input must omit healthcheckPath for a live-null service"
+    end
+
+    def test_empty_string_path_is_treated_as_omitted
+        gql = FakeGQL.new(happy_plan)
+        Railway::PromoteCommand.pin_and_verify(gql,
+            service_id: "s", env_id: "e", image: "ghcr.io/copilotkit/x@sha256:NEW",
+            healthcheck_path: "", sleeper: ->(_n) {})
+
+        query, vars = update_call(gql)
+        refute_match(/healthcheckPath/, query)
+        refute vars.dig(:input)&.key?(:healthcheckPath)
+    end
+end

showcase/bin/spec/test_promote_p1.rb

@@ -0,0 +1,86 @@
+# frozen_string_literal: true
+
+require_relative "spec_helper"
+
+class PromoteP1Test < Minitest::Test
+    # Stub gql + ghcr clients used by PromoteCommand.
+    # Preflight checks accumulate findings before any short-circuit, so even
+    # on a P1 REFUSE the P2 deployments query is still issued. Provide a
+    # benign empty-deployments shape so the test focuses solely on P1.
+    class FakeGQLEmpty
+        def query(*); { "deployments" => { "edges" => [] } }; end
+    end
+
+    class FakeGHCR
+        def initialize(result); @result = result; end
+        def manifest_exists(_ref); @result; end
+        # Tag refs resolve to a synthetic digest; digest refs pass through.
+        # Returning a stable digest makes the resolved ref deterministic so
+        # the existing P1 tests (which staged digest-pinned images) continue
+        # to verify the same code path.
+        def resolve_digest(ref)
+            return ref.split("@", 2).last if ref.include?("@sha256:")
+            "sha256:fake_resolved_digest"
+        end
+        def parse_image_ref(ref); Railway::GHCR.allocate.parse_image_ref(ref); end
+    end
+
+    def test_refuses_when_digest_missing_in_ghcr
+        cmd = Railway::PromoteCommand.new(["--non-interactive", "--yes"])
+        # Inject deterministic precondition input — bypass live snapshot capture.
+        cmd.instance_variable_set(:@staging_snapshot, {
+            "services" => [{
+                "name"       => "showcase-shell",
+                "service_id" => "svc-1",
+                "image"      => "ghcr.io/copilotkit/showcase-shell@sha256:deadbeef",
+                "image_tag"  => "ghcr.io/copilotkit/showcase-shell:latest",
+                "digest"     => "sha256:deadbeef",
+                "env_keys"   => [],
+            }],
+        })
+        cmd.instance_variable_set(:@prod_snapshot, { "services" => [] })
+        cmd.instance_variable_set(:@gql,  FakeGQLEmpty.new)
+        cmd.instance_variable_set(:@ghcr, FakeGHCR.new(:missing))
+
+        cmd.define_singleton_method(:run_staging_probe) { |services:| { ok: true, summary: "" } }
+        out, _err = capture_io { @rc = cmd.run_with_preflight_only }
+        assert_equal 1, @rc, "promote must exit 1 on REFUSE"
+        assert_match(/REFUSE: P1.*ghcr\.io.*not found in GHCR/i, out)
+    end
+
+    def test_refuses_with_clear_message_on_auth_failed
+        cmd = Railway::PromoteCommand.new(["--non-interactive", "--yes"])
+        cmd.instance_variable_set(:@staging_snapshot, {
+            "services" => [{ "name" => "x", "service_id" => "s", "image" => "ghcr.io/copilotkit/x@sha256:abc", "env_keys" => [] }],
+        })
+        cmd.instance_variable_set(:@prod_snapshot, { "services" => [] })
+        # All preflight checks accumulate findings before any short-circuit;
+        # P2 will still query gql, so use the benign empty fake.
+        cmd.instance_variable_set(:@gql, FakeGQLEmpty.new)
+        cmd.instance_variable_set(:@ghcr, FakeGHCR.new(:auth_failed))
+
+        cmd.define_singleton_method(:run_staging_probe) { |services:| { ok: true, summary: "" } }
+        out, _err = capture_io { @rc = cmd.run_with_preflight_only }
+        assert_equal 1, @rc
+        assert_match(/REFUSE: P1.*GHCR.*auth/i, out)
+        assert_match(/GHCR_TOKEN.*or.*GITHUB_TOKEN/i, out)
+    end
+
+    def test_passes_p1_when_digest_exists
+        cmd = Railway::PromoteCommand.new(["--non-interactive", "--yes"])
+        cmd.instance_variable_set(:@staging_snapshot, {
+            "services" => [{ "name" => "x", "service_id" => "s", "image" => "ghcr.io/copilotkit/x@sha256:abc", "env_keys" => [] }],
+        })
+        cmd.instance_variable_set(:@prod_snapshot, { "services" => [] })
+        # P1 passes here so P2 will run — use the benign empty fake so
+        # the deployments query returns [].
+        cmd.instance_variable_set(:@gql, FakeGQLEmpty.new)
+        cmd.instance_variable_set(:@ghcr, FakeGHCR.new(:exists))
+
+        # P1 alone should not refuse; later checks (service-set parity) will,
+        # but P1's own gate is clean for this fixture.
+        cmd.define_singleton_method(:run_staging_probe) { |services:| { ok: true, summary: "" } }
+        out, _err = capture_io { cmd.run_with_preflight_only }
+        refute_match(/REFUSE: P1/, out)
+    end
+end

showcase/bin/spec/test_promote_p2.rb

@@ -0,0 +1,161 @@
+# frozen_string_literal: true
+
+require_relative "spec_helper"
+
+class PromoteP2Test < Minitest::Test
+    class FakeGQL
+        def initialize(deployments_by_svc)
+            @deployments_by_svc = deployments_by_svc
+            @calls = []
+        end
+
+        attr_reader :calls
+
+        def query(q, vars = {})
+            @calls << [q, vars]
+            if q.include?("query Deployments")
+                edges = (@deployments_by_svc[vars[:serviceId]] || []).map { |n| { "node" => n } }
+                return { "deployments" => { "edges" => edges } }
+            end
+            raise "unexpected query: #{q[0,40]}"
+        end
+    end
+
+    def cmd_with(staging:, deployments:)
+        c = Railway::PromoteCommand.new(["--non-interactive", "--yes"])
+        c.instance_variable_set(:@staging_snapshot, staging)
+        c.instance_variable_set(:@prod_snapshot, { "services" => [] })
+        c.instance_variable_set(:@gql, FakeGQL.new(deployments))
+        c.instance_variable_set(:@ghcr, Object.new.tap do |o|
+            def o.manifest_exists(_); :exists; end
+            def o.resolve_digest(ref); ref.include?("@sha256:") ? ref.split("@", 2).last : "sha256:fake"; end
+            def o.parse_image_ref(ref); Railway::GHCR.allocate.parse_image_ref(ref); end
+        end)
+        # Stub P3 probe so the test does not shell out to verify-deploy.ts.
+        c.define_singleton_method(:run_staging_probe) { |services:| { ok: true, summary: "" } }
+        c
+    end
+
+    def test_refuses_when_latest_deployment_not_success
+        staging = { "services" => [{
+            "name" => "x", "service_id" => "svc-1",
+            "image" => "ghcr.io/copilotkit/x@sha256:abc", "digest" => "sha256:abc",
+            "env_keys" => [],
+        }] }
+        deps = { "svc-1" => [
+            { "id" => "d2", "status" => "FAILED",  "meta" => { "image" => "ghcr.io/copilotkit/x@sha256:abc" }, "createdAt" => "2026-05-28T01:00:00Z" },
+            { "id" => "d1", "status" => "SUCCESS", "meta" => { "image" => "ghcr.io/copilotkit/x@sha256:abc" }, "createdAt" => "2026-05-28T00:00:00Z" },
+        ] }
+        out, err = capture_io { @rc = cmd_with(staging: staging, deployments: deps).run_with_preflight_only }
+        combined = out + err
+        assert_equal 1, @rc
+        assert_match(/REFUSE: P2.*x.*latest staging deployment.*FAILED/i, combined)
+    end
+
+    def test_refuses_when_latest_success_image_does_not_match
+        # In-flight race: newer build with a different digest is the latest.
+        staging = { "services" => [{
+            "name" => "x", "service_id" => "svc-1",
+            "image" => "ghcr.io/copilotkit/x@sha256:OLD", "digest" => "sha256:OLD",
+            "env_keys" => [],
+        }] }
+        deps = { "svc-1" => [
+            { "id" => "d2", "status" => "SUCCESS", "meta" => { "image" => "ghcr.io/copilotkit/x@sha256:NEW" }, "createdAt" => "2026-05-28T01:00:00Z" },
+        ] }
+        out, err = capture_io { @rc = cmd_with(staging: staging, deployments: deps).run_with_preflight_only }
+        combined = out + err
+        assert_equal 1, @rc
+        assert_match(/REFUSE: P2.*in-flight.*sha256:NEW.*sha256:OLD/i, combined)
+    end
+
+    def test_does_not_crash_when_deployment_meta_is_a_string
+        # Railway's Deployment.meta is a JSON scalar that can come back as a
+        # String (not a Hash). After FIX-2 we PARSE it first; a non-JSON
+        # string falls back to a WARN, but MUST NOT crash and MUST NOT
+        # produce a P2 REFUSE.
+        staging = { "services" => [{
+            "name" => "x", "service_id" => "svc-1",
+            "image" => "ghcr.io/copilotkit/x@sha256:abc", "digest" => "sha256:abc",
+            "env_keys" => [],
+        }] }
+        deps = { "svc-1" => [
+            { "id" => "d1", "status" => "SUCCESS",
+              "meta" => "raw-string-not-a-hash",
+              "createdAt" => "2026-05-28T01:00:00Z" },
+        ] }
+        out, err = capture_io { cmd_with(staging: staging, deployments: deps).run_with_preflight_only }
+        combined = out + err
+        # MUST NOT crash. P2 race-check is best-effort; SUCCESS is the real gate.
+        refute_match(/REFUSE: P2/, combined)
+        refute_match(/NoMethodError/, combined)
+    end
+
+    def test_passes_p2_when_latest_is_success_and_image_matches
+        staging = { "services" => [{
+            "name" => "x", "service_id" => "svc-1",
+            "image" => "ghcr.io/copilotkit/x@sha256:abc", "digest" => "sha256:abc",
+            "env_keys" => [],
+        }] }
+        deps = { "svc-1" => [
+            { "id" => "d1", "status" => "SUCCESS", "meta" => { "image" => "ghcr.io/copilotkit/x@sha256:abc" }, "createdAt" => "2026-05-28T01:00:00Z" },
+        ] }
+        out, err = capture_io { cmd_with(staging: staging, deployments: deps).run_with_preflight_only }
+        combined = out + err
+        refute_match(/REFUSE: P2/, combined)
+    end
+
+    def test_refuses_when_running_digest_from_meta_imageDigest_does_not_match
+        # REAL staging shape: the staging ref is TAG-FORM (`…:latest`), so the
+        # running digest is NOT in meta.image (which is also tag-form). It lives
+        # in meta.imageDigest. P1's resolved_prod_image pins staging's running
+        # digest (svc-1 below resolves to sha256:OLD via staging_running_digest),
+        # while the LATEST deployment is now running sha256:NEW — an in-flight
+        # race. Before the fix, deployed_digest read meta.image (no `@`) → nil →
+        # the guard was DEAD and never REFUSEd. Now it reads meta.imageDigest.
+        staging = { "services" => [{
+            "name" => "x", "service_id" => "svc-1",
+            "image" => "ghcr.io/copilotkit/x:latest", "digest" => nil,
+            "env_keys" => [],
+        }] }
+        deps = { "svc-1" => [
+            { "id" => "d2", "status" => "SUCCESS",
+              "meta" => { "image" => "ghcr.io/copilotkit/x:latest", "imageDigest" => "sha256:NEW" },
+              "createdAt" => "2026-05-28T02:00:00Z" },
+            { "id" => "d1", "status" => "SUCCESS",
+              "meta" => { "image" => "ghcr.io/copilotkit/x:latest", "imageDigest" => "sha256:OLD" },
+              "createdAt" => "2026-05-28T01:00:00Z" },
+        ] }
+        # Pin @promote_refs as if P1 resolved the OLDER running digest, so the
+        # latest deployment (sha256:NEW) is a genuine in-flight race.
+        c = cmd_with(staging: staging, deployments: deps)
+        c.instance_variable_set(:@promote_refs, { "x" => "ghcr.io/copilotkit/x@sha256:OLD" })
+        findings = c.send(:check_p2_staging_deployments, staging)
+        combined = findings.join("\n")
+        assert_match(/REFUSE: P2.*in-flight.*sha256:NEW.*sha256:OLD/i, combined)
+    end
+
+    def test_digest_override_skips_p2_race_check
+        # --digest override: operator deliberately pinned an explicit digest for
+        # this single service. @promote_refs then holds the operator's chosen
+        # digest (sha256:CHOSEN), which legitimately differs from staging's
+        # running digest (sha256:NEW). The P2 race comparison MUST be skipped —
+        # otherwise it would emit a spurious REFUSE for the exact case --digest
+        # exists to support.
+        staging = { "services" => [{
+            "name" => "x", "service_id" => "svc-1",
+            "image" => "ghcr.io/copilotkit/x:latest", "digest" => nil,
+            "env_keys" => [],
+        }] }
+        deps = { "svc-1" => [
+            { "id" => "d1", "status" => "SUCCESS",
+              "meta" => { "image" => "ghcr.io/copilotkit/x:latest", "imageDigest" => "sha256:NEW" },
+              "createdAt" => "2026-05-28T01:00:00Z" },
+        ] }
+        c = cmd_with(staging: staging, deployments: deps)
+        c.instance_variable_set(:@options, c.options.merge(digest: "ghcr.io/copilotkit/x@sha256:CHOSEN", service: "x"))
+        c.instance_variable_set(:@promote_refs, { "x" => "ghcr.io/copilotkit/x@sha256:CHOSEN" })
+        findings = c.send(:check_p2_staging_deployments, staging)
+        combined = findings.join("\n")
+        refute_match(/REFUSE: P2/, combined)
+    end
+end

showcase/bin/spec/test_promote_p3.rb

@@ -0,0 +1,104 @@
+# frozen_string_literal: true
+
+require_relative "spec_helper"
+
+class PromoteP3Test < Minitest::Test
+    def make_cmd(probe_result:, flag:)
+        argv = ["--non-interactive", "--yes"]
+        argv << flag if flag
+        c = Railway::PromoteCommand.new(argv)
+        # run_with_preflight_only skips parser.parse!; parse eagerly so the
+        # --no-require-staging-green / --confirm-divergence flags land.
+        c.parser.parse!(c.argv)
+        c.instance_variable_set(:@staging_snapshot, {
+            "services" => [{
+                "name" => "x", "service_id" => "svc-1",
+                "image" => "ghcr.io/copilotkit/x:latest", "digest" => "sha256:abc",
+                "env_keys" => [],
+                "start_command" => "node server.js", "healthcheck_path" => "/health",
+                "region" => "us-west", "replicas" => 1, "restart_policy" => "ON_FAILURE",
+            }],
+        })
+        c.instance_variable_set(:@prod_snapshot, {
+            "services" => [{
+                "name" => "x", "service_id" => "svc-1",
+                "image" => "ghcr.io/copilotkit/x@sha256:abc", "digest" => "sha256:abc",
+                "env_keys" => [],
+                "start_command" => "node server.js", "healthcheck_path" => "/health",
+                "region" => "us-west", "replicas" => 1, "restart_policy" => "ON_FAILURE",
+            }],
+        })
+        c.instance_variable_set(:@gql, Object.new.tap { |o| def o.query(*); { "deployments" => { "edges" => [{ "node" => { "id" => "d", "status" => "SUCCESS", "meta" => { "image" => "ghcr.io/copilotkit/x@sha256:abc" } } }] } }; end })
+        c.instance_variable_set(:@ghcr, Object.new.tap do |o|
+            def o.manifest_exists(_); :exists; end
+            def o.resolve_digest(ref); ref.include?("@sha256:") ? ref.split("@", 2).last : "sha256:abc"; end
+            def o.parse_image_ref(ref); Railway::GHCR.allocate.parse_image_ref(ref); end
+        end)
+        # Inject the probe result as a stub.
+        c.define_singleton_method(:run_staging_probe) { |services:| probe_result }
+        c
+    end
+
+    def test_refuses_on_red_probe_when_flag_default_on
+        cmd = make_cmd(probe_result: { ok: false, summary: "x: HTTP 502 from docs.staging.copilotkit.ai" }, flag: nil)
+        out, _ = capture_io { @rc = cmd.run_with_preflight_only }
+        assert_equal 1, @rc
+        assert_match(/REFUSE: P3.*staging.*not green.*HTTP 502/i, out)
+    end
+
+    def test_skips_probe_when_no_require_staging_green
+        cmd = make_cmd(probe_result: { ok: false, summary: "would have failed" }, flag: "--no-require-staging-green")
+        # Fail LOUD if the skip path ever calls the probe — the probe stub
+        # must be unreachable under --no-require-staging-green.
+        cmd.define_singleton_method(:run_staging_probe) do |services:|
+            raise "probe must not run under --no-require-staging-green"
+        end
+        out, _ = capture_io { cmd.run_with_preflight_only }
+        refute_match(/REFUSE: P3/, out)
+        assert_match(/P3 SKIPPED.*--no-require-staging-green/, out)
+    end
+
+    def test_passes_p3_on_green_probe
+        cmd = make_cmd(probe_result: { ok: true, summary: "all green" }, flag: nil)
+        out, _ = capture_io { cmd.run_with_preflight_only }
+        refute_match(/REFUSE: P3/, out)
+    end
+
+    # A service the SSOT marks probe.staging=false (harness-workers) must be
+    # treated as N/A by P3 — NOT handed to the probe (which crashes on a
+    # not-probe-eligible name) and NOT a REFUSE. Before the fix this REFUSEd
+    # via "staging is not green ... not probe-eligible", gating later tiers.
+    def test_ineligible_service_is_skipped_not_refused
+        skip "no probe.staging=false service in SSOT" if Railway::STAGING_PROBE_INELIGIBLE.empty?
+        ineligible = Railway::STAGING_PROBE_INELIGIBLE.first
+        cmd = make_cmd(probe_result: { ok: true, summary: "unused" }, flag: nil)
+        # Fail LOUD if P3 ever hands an ineligible-only set to the probe.
+        cmd.define_singleton_method(:run_staging_probe) do |services:|
+            raise "probe must not run for an ineligible-only set (#{services.inspect})"
+        end
+        staging = { "services" => [{ "name" => ineligible }] }
+        out, _ = capture_io { @findings = cmd.check_p3_staging_live_green(staging) }
+        assert_empty @findings, "P3 must produce no REFUSE for an ineligible-only service"
+        assert_match(/P3 N\/A \(#{Regexp.escape(ineligible)}\).*not staging-probe-eligible/, out)
+    end
+
+    # A mixed set (ineligible + eligible) must skip the ineligible one but
+    # STILL probe — and still gate on — the eligible one.
+    def test_mixed_set_still_probes_eligible
+        skip "no probe.staging=false service in SSOT" if Railway::STAGING_PROBE_INELIGIBLE.empty?
+        ineligible = Railway::STAGING_PROBE_INELIGIBLE.first
+        eligible = Railway::STAGING_SERVICES.find { |n| !Railway::STAGING_PROBE_INELIGIBLE.include?(n) }
+        cmd = make_cmd(probe_result: { ok: false, summary: "#{eligible}: HTTP 502" }, flag: nil)
+        probed = nil
+        cmd.define_singleton_method(:run_staging_probe) do |services:|
+            probed = services
+            { ok: false, summary: "#{eligible}: HTTP 502" }
+        end
+        staging = { "services" => [{ "name" => ineligible }, { "name" => eligible }] }
+        out, _ = capture_io { @findings = cmd.check_p3_staging_live_green(staging) }
+        assert_equal [eligible], probed, "P3 must probe only the eligible service"
+        assert_match(/P3 N\/A \(#{Regexp.escape(ineligible)}\)/, out)
+        assert_equal 1, @findings.size
+        assert_match(/REFUSE: P3.*#{Regexp.escape(eligible)}.*HTTP 502/, @findings.first)
+    end
+end

showcase/bin/spec/test_promote_p5.rb

@@ -0,0 +1,195 @@
+# frozen_string_literal: true
+
+require_relative "spec_helper"
+
+class PromoteP5Test < Minitest::Test
+    class FakeGQL
+        def initialize(plan); @plan = plan; @calls = []; end
+        attr_reader :calls
+
+        def query(q, vars = {})
+            @calls << [q, vars]
+            step = @plan.shift
+            raise "fake exhausted at call ##{@calls.size}: #{q[0,40]}" unless step
+            raise step[:raise] if step[:raise]
+            step[:data]   # inner data hash, matching GraphQL#query's return shape
+        end
+    end
+
+    NEW_DEPLOY_ID = "dep-new"
+
+    # Helper: a "pre-update snapshot" GQL response with a given updatedAt.
+    def pre(ts) = { data: { "serviceInstance" => { "id" => "i", "source" => { "image" => "ghcr.io/copilotkit/x@sha256:OLD" }, "updatedAt" => ts } } }
+
+    # Helper: a "post-update re-query" (config recheck) response.
+    def post(image:, ts:) = { data: { "serviceInstance" => { "id" => "i", "source" => { "image" => image }, "updatedAt" => ts } } }
+
+    # Helper: a successful DeployV2 mutation returning a new deployment id.
+    def deploy_ok(id = NEW_DEPLOY_ID) = { data: { "serviceInstanceDeployV2" => id } }
+
+    # Helper: a serving-digest recheck response. The NEW deployment has reached
+    # the given status and serves the given digest. Defaults to the SUCCESS +
+    # pinned-digest happy path.
+    def serving(digest:, status: "SUCCESS", deploy_id: NEW_DEPLOY_ID)
+        {
+            data: {
+                "serviceInstance" => {
+                    "id" => "i",
+                    "source" => { "image" => "ghcr.io/copilotkit/x@#{digest}" },
+                    "updatedAt" => "2026-05-28T03:00:00Z",
+                    "latestDeployment" => {
+                        "id" => deploy_id, "status" => status,
+                        "meta" => { "imageDigest" => digest },
+                    },
+                },
+            },
+        }
+    end
+
+    def test_refuses_when_update_returns_false
+        # Order: pre-update snapshot, then update (false) — refuse before deploy.
+        gql = FakeGQL.new([
+            pre("2026-05-27T00:00:00Z"),
+            { data: { "serviceInstanceUpdate" => false } },
+        ])
+        assert_raises(Railway::PromoteCommand::MutationError) do
+            Railway::PromoteCommand.pin_and_verify(gql,
+                service_id: "s", env_id: "e", image: "ghcr.io/copilotkit/x@sha256:NEW",
+                sleeper: ->(_n) {})
+        end
+    end
+
+    def test_refuses_when_deploy_v2_returns_no_id
+        # serviceInstanceDeployV2 must return a non-empty deployment id String.
+        gql = FakeGQL.new([
+            pre("2026-05-27T00:00:00Z"),
+            { data: { "serviceInstanceUpdate"   => true } },
+            { data: { "serviceInstanceDeployV2" => nil } },
+        ])
+        assert_raises(Railway::PromoteCommand::MutationError) do
+            Railway::PromoteCommand.pin_and_verify(gql,
+                service_id: "s", env_id: "e", image: "ghcr.io/copilotkit/x@sha256:NEW",
+                sleeper: ->(_n) {})
+        end
+    end
+
+    def test_verifies_image_AND_updatedAt_advanced_then_serving_digest
+        gql = FakeGQL.new([
+            pre("2026-05-27T00:00:00Z"),
+            { data: { "serviceInstanceUpdate" => true } },
+            deploy_ok,
+            post(image: "ghcr.io/copilotkit/x@sha256:NEW", ts: "2026-05-28T01:00:00Z"),
+            serving(digest: "sha256:NEW"),
+        ])
+        Railway::PromoteCommand.pin_and_verify(gql,
+            service_id: "s", env_id: "e", image: "ghcr.io/copilotkit/x@sha256:NEW",
+            sleeper: ->(_n) {})
+        assert_equal 5, gql.calls.size
+    end
+
+    def test_refuses_when_new_deployment_serves_wrong_digest
+        # Bug #2: config advanced + DeployV2 spawned, but the NEW deployment
+        # succeeded SERVING a stale digest. Must fail loud.
+        gql = FakeGQL.new([
+            pre("2026-05-27T00:00:00Z"),
+            { data: { "serviceInstanceUpdate" => true } },
+            deploy_ok,
+            post(image: "ghcr.io/copilotkit/x@sha256:NEW", ts: "2026-05-28T01:00:00Z"),
+            serving(digest: "sha256:STALE"),  # new deploy SUCCESS but wrong digest
+        ])
+        err = assert_raises(Railway::PromoteCommand::MutationError) do
+            Railway::PromoteCommand.pin_and_verify(gql,
+                service_id: "s", env_id: "e", image: "ghcr.io/copilotkit/x@sha256:NEW",
+                sleeper: ->(_n) {})
+        end
+        assert_match(/serving a stale image|SERVES/, err.message)
+    end
+
+    def test_refuses_when_new_deployment_crashes
+        gql = FakeGQL.new([
+            pre("2026-05-27T00:00:00Z"),
+            { data: { "serviceInstanceUpdate" => true } },
+            deploy_ok,
+            post(image: "ghcr.io/copilotkit/x@sha256:NEW", ts: "2026-05-28T01:00:00Z"),
+            serving(digest: "sha256:NEW", status: "CRASHED"),
+        ])
+        err = assert_raises(Railway::PromoteCommand::MutationError) do
+            Railway::PromoteCommand.pin_and_verify(gql,
+                service_id: "s", env_id: "e", image: "ghcr.io/copilotkit/x@sha256:NEW",
+                sleeper: ->(_n) {})
+        end
+        assert_match(/terminal status/, err.message)
+    end
+
+    def test_keeps_polling_when_stale_old_deployment_is_terminal
+        # Bug (false-abort): right after DeployV2 spawns new_deployment_id,
+        # latestDeployment may still briefly point to the OLD deployment, whose
+        # status flips to REMOVED as it's superseded. The terminal-status check
+        # must NOT raise on this stale old deployment (deploy.id != new id) — it
+        # must keep polling. On a later poll the NEW deployment becomes latest
+        # with SUCCESS + the pinned digest, so verify_serving_digest! converges.
+        gql = FakeGQL.new([
+            pre("2026-05-27T00:00:00Z"),
+            { data: { "serviceInstanceUpdate" => true } },
+            deploy_ok,
+            post(image: "ghcr.io/copilotkit/x@sha256:NEW", ts: "2026-05-28T01:00:00Z"),
+            # Early poll: OLD deployment is still latest, now being torn down.
+            serving(digest: "sha256:OLD", status: "REMOVED", deploy_id: "dep-old"),
+            # Later poll: NEW deployment is latest, SUCCESS, serving the pin.
+            serving(digest: "sha256:NEW"),
+        ])
+        Railway::PromoteCommand.pin_and_verify(gql,
+            service_id: "s", env_id: "e", image: "ghcr.io/copilotkit/x@sha256:NEW",
+            sleeper: ->(_n) {})
+    end
+
+    def test_refuses_when_image_advanced_but_updatedAt_did_NOT_advance
+        # P5 guard for the no-op re-pin / cache-shaped race: image-equality
+        # alone is insufficient. updatedAt MUST strictly advance past
+        # pre_update_ts; if not, three retries then refuse.
+        pre_ts = "2026-05-27T00:00:00Z"
+        stale_ts_post = post(image: "ghcr.io/copilotkit/x@sha256:NEW", ts: pre_ts)
+        gql = FakeGQL.new([
+            pre(pre_ts),
+            { data: { "serviceInstanceUpdate"   => true } },
+            deploy_ok,
+            stale_ts_post, stale_ts_post, stale_ts_post,
+        ])
+        assert_raises(Railway::PromoteCommand::MutationError) do
+            Railway::PromoteCommand.pin_and_verify(gql,
+                service_id: "s", env_id: "e", image: "ghcr.io/copilotkit/x@sha256:NEW",
+                sleeper: ->(_n) {})
+        end
+    end
+
+    def test_retries_then_refuses_on_stale_image
+        # Three re-queries all report stale image.
+        stale = post(image: "ghcr.io/copilotkit/x@sha256:OLD", ts: "2026-05-27T00:00:00Z")
+        gql = FakeGQL.new([
+            pre("2026-05-27T00:00:00Z"),
+            { data: { "serviceInstanceUpdate"   => true } },
+            deploy_ok,
+            stale, stale, stale,
+        ])
+        assert_raises(Railway::PromoteCommand::MutationError) do
+            Railway::PromoteCommand.pin_and_verify(gql,
+                service_id: "s", env_id: "e", image: "ghcr.io/copilotkit/x@sha256:NEW",
+                sleeper: ->(_n) {})
+        end
+    end
+
+    def test_accepts_when_third_requery_shows_advance
+        new_ok = post(image: "ghcr.io/copilotkit/x@sha256:NEW", ts: "2026-05-28T02:00:00Z")
+        stale  = post(image: "ghcr.io/copilotkit/x@sha256:OLD", ts: "2026-05-27T00:00:00Z")
+        gql = FakeGQL.new([
+            pre("2026-05-27T00:00:00Z"),
+            { data: { "serviceInstanceUpdate"   => true } },
+            deploy_ok,
+            stale, stale, new_ok,
+            serving(digest: "sha256:NEW"),
+        ])
+        Railway::PromoteCommand.pin_and_verify(gql,
+            service_id: "s", env_id: "e", image: "ghcr.io/copilotkit/x@sha256:NEW",
+            sleeper: ->(_n) {})
+    end
+end

showcase/bin/spec/test_promote_p6.rb

@@ -0,0 +1,162 @@
+# frozen_string_literal: true
+
+require_relative "spec_helper"
+
+class PromoteP6Test < Minitest::Test
+    def cmd_with(staging, prod, flag: nil)
+        argv = ["--non-interactive", "--yes"]
+        argv << flag if flag
+        c = Railway::PromoteCommand.new(argv)
+        # run_with_preflight_only skips the parser.parse!() that #run normally
+        # invokes; parse flags eagerly so --confirm-divergence etc. land in
+        # options[].
+        c.parser.parse!(c.argv)
+        c.instance_variable_set(:@staging_snapshot, staging)
+        c.instance_variable_set(:@prod_snapshot, prod)
+        c.instance_variable_set(:@gql, Object.new.tap { |o| def o.query(*); { "deployments" => { "edges" => [{ "node" => { "id" => "d", "status" => "SUCCESS", "meta" => { "image" => "ghcr.io/copilotkit/x@sha256:abc" } } }] } }; end })
+        c.instance_variable_set(:@ghcr, Object.new.tap do |o|
+            def o.manifest_exists(_); :exists; end
+            def o.resolve_digest(ref); ref.include?("@sha256:") ? ref.split("@", 2).last : "sha256:abc"; end
+            def o.parse_image_ref(ref); Railway::GHCR.allocate.parse_image_ref(ref); end
+        end)
+        c.define_singleton_method(:run_staging_probe) { |services:| { ok: true, summary: "" } }
+        c
+    end
+
+    # Defaults represent the "good" parity baseline: staging tag-floating,
+    # prod digest-pinned, all other fields identical. Tests override fields
+    # with the specific divergence under test.
+    def staging_svc(over = {})
+        {
+            "name" => "x", "service_id" => "s", "image" => "ghcr.io/copilotkit/x:latest",
+            "digest" => "sha256:abc", "env_keys" => [],
+            "start_command" => "node server.js", "healthcheck_path" => "/health",
+            "region" => "us-west", "replicas" => 1, "restart_policy" => "ON_FAILURE",
+        }.merge(over)
+    end
+
+    def prod_svc(over = {})
+        {
+            "name" => "x", "service_id" => "s", "image" => "ghcr.io/copilotkit/x@sha256:abc",
+            "digest" => "sha256:abc", "env_keys" => [],
+            "start_command" => "node server.js", "healthcheck_path" => "/health",
+            "region" => "us-west", "replicas" => 1, "restart_policy" => "ON_FAILURE",
+        }.merge(over)
+    end
+
+    def test_refuses_on_start_command_divergence
+        st = { "services" => [staging_svc("start_command" => "node staging.js")] }
+        pr = { "services" => [prod_svc("start_command" => "node prod.js")] }
+        out, _ = capture_io { @rc = cmd_with(st, pr).run_with_preflight_only }
+        assert_equal 1, @rc
+        assert_match(/REFUSE: P6.*x.*startCommand/i, out)
+    end
+
+    def test_refuses_on_healthcheck_path_divergence
+        st = { "services" => [staging_svc("healthcheck_path" => "/health")] }
+        pr = { "services" => [prod_svc("healthcheck_path" => "/healthz")] }
+        out, _ = capture_io { @rc = cmd_with(st, pr).run_with_preflight_only }
+        assert_equal 1, @rc
+        assert_match(/REFUSE: P6.*x.*healthcheckPath/i, out)
+    end
+
+    def test_refuses_on_image_shape_divergence
+        # staging digest-pinned (wrong; expected tag), prod tag-floating
+        # (wrong; expected digest). Both shapes invert the parity.
+        st = { "services" => [staging_svc("image" => "ghcr.io/copilotkit/x@sha256:abc")] }
+        pr = { "services" => [prod_svc("image" => "ghcr.io/copilotkit/x:latest")] }
+        out, _ = capture_io { @rc = cmd_with(st, pr).run_with_preflight_only }
+        assert_equal 1, @rc
+        assert_match(/REFUSE: P6.*x.*image shape/i, out)
+    end
+
+    def test_region_replicas_restartpolicy_are_advisory_non_blocking
+        # region/replicas/restartPolicy are ADVISORY: reported but never block,
+        # with NO --confirm-divergence. env-var key-set diff is DROPPED entirely.
+        # All CRITICAL_ENV_KEYS present so the only findings are advisory.
+        crit = Railway::CRITICAL_ENV_KEYS
+        st = { "services" => [staging_svc("region" => "us-west", "replicas" => 1, "restart_policy" => "ON_FAILURE", "env_keys" => crit + ["B"])] }
+        pr = { "services" => [prod_svc(   "region" => "us-east", "replicas" => 3, "restart_policy" => "ALWAYS",     "env_keys" => crit + ["C"])] }
+        c = cmd_with(st, pr)
+        c.define_singleton_method(:execute_promotion) { |_st, _pr| 0 }
+        out, _ = capture_io { @rc = c.run_with_preflight_only }
+        assert_equal 0, @rc, "ADVISORY findings must NOT block without --confirm-divergence"
+        assert_match(/ADVISORY.*x.*region/i, out)
+        assert_match(/ADVISORY.*x.*replicas/i, out)
+        assert_match(/ADVISORY.*x.*restartPolicy/i, out)
+        # env-var key-set diff is dropped: B/C divergence produces no finding.
+        refute_match(/env key set/i, out)
+    end
+
+    def test_env_var_values_never_compared_message_printed_every_run
+        # Carry CRITICAL_ENV_KEYS so the critical-key parity check passes and the
+        # run reaches the clean-promote path; stub execute_promotion to return 0.
+        # This proves the NOTE prints on a real (rc=0) promote, not just up-front
+        # before an early REFUSE.
+        crit = Railway::CRITICAL_ENV_KEYS
+        st = { "services" => [staging_svc("env_keys" => crit)] }
+        pr = { "services" => [prod_svc("env_keys" => crit)] }
+        c = cmd_with(st, pr)
+        c.define_singleton_method(:execute_promotion) { |_st, _pr| 0 }
+        out, _ = capture_io { @rc = c.run_with_preflight_only }
+        assert_equal 0, @rc, "clean-promote path must be reached (rc=0)"
+        assert_match(/env var VALUES are not compared/i, out)
+    end
+
+    # ── Whitelist parity policy (2026-06-22 prod↔staging comparison policy) ──
+
+    # (a) A prod-only extra env key (the NODE_ENV case) must NOT block after the
+    # env-key-set-diff WARN is dropped. Staging lacks it, prod carries it; the
+    # set diff used to flag this as a blocking WARN. No --confirm-divergence,
+    # so the run must exit 0 (no blocking finding) and emit no env-key-set WARN.
+    def test_prod_only_env_key_does_not_block
+        crit = Railway::CRITICAL_ENV_KEYS
+        st = { "services" => [staging_svc("env_keys" => crit + %w[A])] }
+        pr = { "services" => [prod_svc(   "env_keys" => crit + %w[A NODE_ENV])] }
+        c = cmd_with(st, pr)
+        c.define_singleton_method(:execute_promotion) { |_st, _pr| 0 }
+        out, _ = capture_io { @rc = c.run_with_preflight_only }
+        assert_equal 0, @rc, "prod-only env key (NODE_ENV) must not block without --confirm-divergence"
+        refute_match(/env key set divergence/i, out)
+    end
+
+    # (b1) Staging-gated contract: a CRITICAL_ENV_KEYS member present in STAGING
+    # but MISSING from PROD is a real, fixable divergence and must REFUSE.
+    def test_critical_key_in_staging_missing_in_prod_refuses
+        # OPENAI_API_KEY is a CRITICAL_ENV_KEYS member: present in staging, absent from prod.
+        st = { "services" => [staging_svc("env_keys" => %w[A OPENAI_API_KEY])] }
+        pr = { "services" => [prod_svc(   "env_keys" => %w[A])] }
+        out, _ = capture_io { @rc = cmd_with(st, pr).run_with_preflight_only }
+        assert_equal 1, @rc, "critical key in staging but missing from prod must REFUSE"
+        assert_match(/REFUSE.*critical env keys missing in prod.*OPENAI_API_KEY/i, out)
+    end
+
+    # (b2) Infra-token tolerance: a CRITICAL_ENV_KEYS member absent from BOTH
+    # staging AND prod (operator/CI/infra tokens like RAILWAY_TOKEN that no
+    # application container carries) must NOT drive the run to REFUSE.
+    def test_critical_key_absent_from_both_envs_does_not_refuse
+        # OPENAI_API_KEY is a CRITICAL_ENV_KEYS member; absent from staging AND prod.
+        st = { "services" => [staging_svc("env_keys" => %w[A])] }
+        pr = { "services" => [prod_svc(   "env_keys" => %w[A])] }
+        c = cmd_with(st, pr)
+        c.define_singleton_method(:execute_promotion) { |_st, _pr| 0 }
+        out, _ = capture_io { @rc = c.run_with_preflight_only }
+        assert_equal 0, @rc, "critical key absent from BOTH envs (infra token) must NOT refuse"
+        refute_match(/REFUSE.*critical env keys missing in prod/i, out)
+    end
+
+    # (c) Region/replicas divergence is ADVISORY (report-only, never blocks)
+    # after the change. Today it blocks as a WARN unless --confirm-divergence.
+    # All CRITICAL_ENV_KEYS present in prod so the only findings are advisory.
+    def test_region_replicas_divergence_is_advisory_non_blocking
+        crit = Railway::CRITICAL_ENV_KEYS
+        st = { "services" => [staging_svc("region" => "us-west", "replicas" => 1, "env_keys" => crit)] }
+        pr = { "services" => [prod_svc(   "region" => "us-east", "replicas" => 3, "env_keys" => crit)] }
+        c = cmd_with(st, pr)
+        c.define_singleton_method(:execute_promotion) { |_st, _pr| 0 }
+        out, _ = capture_io { @rc = c.run_with_preflight_only }
+        assert_equal 0, @rc, "region/replicas divergence must be ADVISORY (non-blocking) without --confirm-divergence"
+        assert_match(/ADVISORY.*x.*region/i, out)
+        assert_match(/ADVISORY.*x.*replicas/i, out)
+    end
+end

showcase/bin/spec/test_promote_replicas_reassert.rb

@@ -0,0 +1,168 @@
+# frozen_string_literal: true
+
+require_relative "spec_helper"
+
+# Re-assertion of the SSOT multiRegionConfig replica count on the promote pin
+# (the durable fix for the harness-workers -> 1 de-scale incident). A promote
+# that issues serviceInstanceUpdate WITHOUT a multiRegionConfig key lets Railway
+# fall back to its default region (us-west1) at 1 replica on the subsequent
+# serviceInstanceDeployV2 — collapsing the staged
+# `multiRegionConfig.us-west2.numReplicas = 6`. pin_and_verify must:
+#   - INCLUDE multiRegionConfig in the serviceInstanceUpdate input when the SSOT
+#     declares a replica override for the service+env (harness-workers -> 6 in
+#     us-west2), and
+#   - OMIT it entirely (never send a multiRegionConfig key) when the SSOT tracks
+#     no replica override, so a normal single-replica service is never touched.
+class PromoteReplicasReassertTest < Minitest::Test
+    class FakeGQL
+        def initialize(plan); @plan = plan; @calls = []; end
+        attr_reader :calls
+
+        def query(q, vars = {})
+            @calls << [q, vars]
+            step = @plan.shift
+            raise "fake exhausted at call ##{@calls.size}: #{q[0, 40]}" unless step
+            raise step[:raise] if step[:raise]
+            step[:data]
+        end
+    end
+
+    NEW_DEPLOY_ID = "dep-new"
+
+    def pre(ts) = { data: { "serviceInstance" => { "id" => "i", "source" => { "image" => "ghcr.io/copilotkit/x@sha256:OLD" }, "updatedAt" => ts } } }
+    def post(image:, ts:) = { data: { "serviceInstance" => { "id" => "i", "source" => { "image" => image }, "updatedAt" => ts } } }
+    def deploy_ok(id = NEW_DEPLOY_ID) = { data: { "serviceInstanceDeployV2" => id } }
+
+    def serving(digest:, status: "SUCCESS", deploy_id: NEW_DEPLOY_ID)
+        {
+            data: {
+                "serviceInstance" => {
+                    "id" => "i",
+                    "source" => { "image" => "ghcr.io/copilotkit/x@#{digest}" },
+                    "updatedAt" => "2026-05-28T03:00:00Z",
+                    "latestDeployment" => {
+                        "id" => deploy_id, "status" => status,
+                        "meta" => { "imageDigest" => digest },
+                    },
+                },
+            },
+        }
+    end
+
+    def happy_plan
+        [
+            pre("2026-05-27T00:00:00Z"),
+            { data: { "serviceInstanceUpdate" => true } },
+            deploy_ok,
+            post(image: "ghcr.io/copilotkit/x@sha256:NEW", ts: "2026-05-28T01:00:00Z"),
+            serving(digest: "sha256:NEW"),
+        ]
+    end
+
+    # The serviceInstanceUpdate is always the 2nd GQL call (after the pre-update
+    # snapshot). Returns [query_string, vars] for that call.
+    def update_call(gql) = gql.calls[1]
+
+    def test_includes_multiregion_replicas_when_ssot_declares_override
+        gql = FakeGQL.new(happy_plan)
+        Railway::PromoteCommand.pin_and_verify(gql,
+            service_id: "s", env_id: "e", image: "ghcr.io/copilotkit/x@sha256:NEW",
+            replica_config: { "us-west2" => 6 }, sleeper: ->(_n) {})
+
+        query, vars = update_call(gql)
+        # The mutation passes the WHOLE input object as a single $input variable
+        # typed ServiceInstanceUpdateInput! (the type Railway actually defines).
+        # multiRegionConfig is a NESTED key of that input — Railway infers its
+        # type from the ServiceInstanceUpdateInput schema, so we must NOT name a
+        # standalone `ServiceMultiRegionConfigInput` type (it DOES NOT EXIST and
+        # made the live promote 400). See deploy-to-railway.ts / provision-
+        # starter-fleet.ts, which set healthcheckPath/region/etc the same way.
+        refute_match(/ServiceMultiRegionConfigInput/, query,
+            "must not reference the nonexistent ServiceMultiRegionConfigInput type")
+        assert_match(/\$input:\s*ServiceInstanceUpdateInput!/, query,
+            "update mutation should pass a single $input: ServiceInstanceUpdateInput! variable")
+        assert_match(/input:\s*\$input/, query,
+            "serviceInstanceUpdate should receive the input via $input")
+        # The replica map rides INSIDE the input hash as the multiRegionConfig key.
+        assert_equal({ "us-west2" => { numReplicas: 6 } },
+            vars.dig(:input, :multiRegionConfig),
+            "input.multiRegionConfig should carry the SSOT region -> { numReplicas } map")
+        assert_equal({ image: "ghcr.io/copilotkit/x@sha256:NEW" },
+            vars.dig(:input, :source),
+            "input.source.image should always pin the target image")
+    end
+
+    def test_omits_multiregion_when_ssot_has_no_override
+        gql = FakeGQL.new(happy_plan)
+        # nil replica_config == a normal service with no replica override.
+        Railway::PromoteCommand.pin_and_verify(gql,
+            service_id: "s", env_id: "e", image: "ghcr.io/copilotkit/x@sha256:NEW",
+            replica_config: nil, sleeper: ->(_n) {})
+
+        query, vars = update_call(gql)
+        # NO multiRegionConfig anywhere — we must NEVER send a multiRegionConfig
+        # key for a service without an override (would risk clobbering config).
+        refute_match(/multiRegionConfig/, query,
+            "image-only mutation must not mention multiRegionConfig")
+        refute vars.dig(:input)&.key?(:multiRegionConfig),
+            "input must omit multiRegionConfig for a non-override service"
+    end
+
+    def test_empty_replica_config_is_treated_as_omitted
+        gql = FakeGQL.new(happy_plan)
+        Railway::PromoteCommand.pin_and_verify(gql,
+            service_id: "s", env_id: "e", image: "ghcr.io/copilotkit/x@sha256:NEW",
+            replica_config: {}, sleeper: ->(_n) {})
+
+        query, vars = update_call(gql)
+        refute_match(/multiRegionConfig/, query)
+        refute vars.dig(:input)&.key?(:multiRegionConfig)
+    end
+
+    # multiRegionConfig and healthcheckPath are INDEPENDENT re-assertion
+    # dimensions: a service can declare both (harness-workers tracks /health AND
+    # 6 replicas). Both must ride in the same serviceInstanceUpdate input.
+    def test_includes_both_healthcheck_and_replicas
+        gql = FakeGQL.new(happy_plan)
+        Railway::PromoteCommand.pin_and_verify(gql,
+            service_id: "s", env_id: "e", image: "ghcr.io/copilotkit/x@sha256:NEW",
+            healthcheck_path: "/health", replica_config: { "us-west2" => 6 },
+            sleeper: ->(_n) {})
+
+        query, vars = update_call(gql)
+        # Both ride as nested keys inside the single $input object.
+        assert_match(/\$input:\s*ServiceInstanceUpdateInput!/, query)
+        refute_match(/ServiceMultiRegionConfigInput/, query)
+        assert_equal "/health", vars.dig(:input, :healthcheckPath)
+        assert_equal({ "us-west2" => { numReplicas: 6 } },
+            vars.dig(:input, :multiRegionConfig))
+    end
+
+    # SSOT resolution: the promote loop pulls replica intent from the REAL SSOT
+    # (railway-envs.generated.json) via ssot_replica_config. harness-workers
+    # carries workerProvisioning.prod.effectiveReplicas=6 -> { us-west2 => 6 };
+    # every other service tracks no override -> nil (so multiRegionConfig is
+    # omitted and their live config is never touched).
+    def cmd
+        c = Railway::PromoteCommand.new(["--non-interactive", "--yes"])
+        c.parser.parse!(c.argv)
+        c
+    end
+
+    def test_ssot_resolves_harness_workers_to_six_replicas_in_us_west2
+        assert_equal({ "us-west2" => 6 },
+            cmd.ssot_replica_config("harness-workers", "prod"),
+            "harness-workers prod must resolve to 6 replicas in us-west2 from SSOT")
+    end
+
+    def test_ssot_replica_config_is_nil_for_a_non_override_service
+        # Pick any non-worker service the SSOT tracks; it must have NO override.
+        sample = (Railway::SSOT_DATA["services"] || [])
+            .map { |s| s["name"] }
+            .reject { |n| n == "harness-workers" }
+            .first
+        refute_nil sample, "expected at least one non-worker service in the SSOT"
+        assert_nil cmd.ssot_replica_config(sample, "prod"),
+            "#{sample} has no replica override -> ssot_replica_config must be nil"
+    end
+end

showcase/bin/spec/test_promote_resolve_once.rb

@@ -0,0 +1,317 @@
+# frozen_string_literal: true
+
+require_relative "spec_helper"
+
+# Covers five CR fixes against PromoteCommand:
+#   FIX-1: digest is resolved EXACTLY ONCE per staging service across the
+#          whole preflight+execute, and the SAME ref P1 verified is what
+#          gets pinned by execute_promotion (no TOCTOU).
+#   FIX-2: a Railway::GHCR::Error raised mid-loop in P1 produces a per-
+#          service REFUSE finding and the loop continues for remaining
+#          services — earlier findings are NOT discarded.
+#   FIX-3: pin_and_verify asserts the serviceInstanceDeployV2 result is a
+#          non-empty deployment id (symmetric with the serviceInstanceUpdate
+#          result check).
+#   FIX-4: P2 emits a WARN finding when deployment meta is not a Hash, so
+#          the silent skip of the in-flight race-check is visible.
+#   FIX-5: when pin_and_verify raises mid-loop in execute_promotion, a
+#          loud PARTIAL-PROMOTION report names the already-pinned services.
+class PromoteResolveOnceTest < Minitest::Test
+    # FakeGHCR that counts resolve_digest calls per ref.
+    class CountingGHCR
+        attr_reader :resolve_calls
+
+        def initialize(resolve_map:, exists_set: nil)
+            @resolve_map   = resolve_map
+            @exists_set    = exists_set
+            @resolve_calls = Hash.new(0)
+        end
+
+        def resolve_digest(ref)
+            @resolve_calls[ref] += 1
+            return ref.split("@", 2).last if ref.include?("@sha256:")
+            @resolve_map[ref]
+        end
+
+        def manifest_exists(ref)
+            return :missing if @exists_set && !@exists_set.include?(ref)
+            :exists
+        end
+
+        def parse_image_ref(ref)
+            Railway::GHCR.allocate.parse_image_ref(ref)
+        end
+    end
+
+    # GQL fake reusing the pattern from PromoteExecuteTest::RecordingGQL,
+    # but with optional knobs for FIX-3 (redeploy returns false) and FIX-5
+    # (per-service update failure).
+    class RecordingGQL
+        def initialize(redeploy_result: true, update_fail_for: nil)
+            @calls            = []
+            # Per-service post-update state so multi-service test cases don't
+            # leak each other's pinned image into pre-update rechecks. Keyed by
+            # serviceId. Value is { image:, post_ts: }. Pre-update returns the
+            # "OLD" snapshot (with an early ts) for any service not yet pinned.
+            @post            = {}
+            @ts_counter      = 0
+            @redeploy_result = redeploy_result
+            @update_fail_for = update_fail_for  # service_id whose update returns false
+        end
+        attr_reader :calls
+
+        def query(q, vars = {})
+            @calls << [q, vars]
+            sid = vars[:serviceId]
+            if q.include?("serviceInstanceUpdate")
+                if @update_fail_for && sid == @update_fail_for
+                    return { "serviceInstanceUpdate" => false }
+                end
+                @ts_counter += 1
+                @post[sid] = { image: vars.dig(:input, :source, :image), ts: "2026-05-29T00:00:%02dZ" % @ts_counter }
+                { "serviceInstanceUpdate" => true }
+            elsif q.include?("serviceInstanceDeployV2")
+                # @redeploy_result false → return nil id so the DeployV2 guard
+                # fails loud (mirrors the prior redeploy-false fail path).
+                { "serviceInstanceDeployV2" => (@redeploy_result ? "dep-#{sid}" : nil) }
+            elsif q.include?("ServiceInstanceRecheck")
+                if (entry = @post[sid])
+                    {
+                        "serviceInstance" => {
+                            "id" => "i",
+                            "source" => { "image" => entry[:image] },
+                            "updatedAt" => entry[:ts],
+                            "latestDeployment" => {
+                                "id" => "dep-#{sid}", "status" => "SUCCESS",
+                                "meta" => {
+                                    "imageDigest" => (entry[:image].include?("@") ? entry[:image].split("@", 2).last : nil),
+                                },
+                            },
+                        },
+                    }
+                else
+                    {
+                        "serviceInstance" => {
+                            "id" => "i",
+                            "source" => { "image" => "ghcr.io/copilotkit/x@sha256:OLD" },
+                            "updatedAt" => "2026-05-28T00:00:00Z",
+                        },
+                    }
+                end
+            else
+                {}
+            end
+        end
+
+        def pinned_images
+            @calls.select { |q, _| q.include?("serviceInstanceUpdate") }.map { |_, v| v.dig(:input, :source, :image) }
+        end
+    end
+
+    # Silence pin_and_verify's 10s-per-retry waits.
+    def with_fast_sleeper
+        original = Railway::PromoteCommand.const_get(:RETRY_DELAY_SEC)
+        Railway::PromoteCommand.send(:remove_const, :RETRY_DELAY_SEC)
+        Railway::PromoteCommand.const_set(:RETRY_DELAY_SEC, 0)
+        yield
+    ensure
+        Railway::PromoteCommand.send(:remove_const, :RETRY_DELAY_SEC)
+        Railway::PromoteCommand.const_set(:RETRY_DELAY_SEC, original)
+    end
+
+    def make_svc(name, image:)
+        {
+            "name" => name, "service_id" => "svc-stg-#{name}",
+            "image" => image,
+            # All CRITICAL_ENV_KEYS present so the (now unconditional) critical
+            # env-key presence assertion does not fire — these tests isolate
+            # the resolve-once / pin behavior, not env-key parity.
+            "env_keys" => Railway::CRITICAL_ENV_KEYS.dup,
+            "start_command" => "node server.js", "healthcheck_path" => "/health",
+            "region" => "us-west", "replicas" => 1, "restart_policy" => "ON_FAILURE",
+        }
+    end
+
+    def make_prod_svc(name)
+        {
+            "name" => name, "service_id" => "svc-prod-#{name}",
+            "image" => "ghcr.io/copilotkit/#{name}@sha256:OLD",
+            "env_keys" => Railway::CRITICAL_ENV_KEYS.dup,
+            "start_command" => "node server.js", "healthcheck_path" => "/health",
+            "region" => "us-west", "replicas" => 1, "restart_policy" => "ON_FAILURE",
+        }
+    end
+
+    # =================== FIX-1 ===================
+
+    def test_fix1_resolve_digest_called_exactly_once_per_service
+        # Single :latest staging service. After full preflight + execute, the
+        # ghcr.resolve_digest call counter MUST be exactly 1 for that ref —
+        # not 2 (which was the pre-fix TOCTOU duplicate-resolve behavior).
+        ghcr = CountingGHCR.new(resolve_map: { "ghcr.io/copilotkit/x:latest" => "sha256:abc123" })
+        gql  = RecordingGQL.new
+
+        cmd = Railway::PromoteCommand.new(["--non-interactive", "--yes", "--confirm-divergence"])
+        cmd.parser.parse!(cmd.argv)
+        cmd.instance_variable_set(:@staging_snapshot, {
+            "services" => [make_svc("x", image: "ghcr.io/copilotkit/x:latest")],
+        })
+        cmd.instance_variable_set(:@prod_snapshot, { "services" => [make_prod_svc("x")] })
+        cmd.instance_variable_set(:@gql, gql)
+        cmd.instance_variable_set(:@ghcr, ghcr)
+        cmd.define_singleton_method(:fetch_latest_staging_deployments) do |_svc_id|
+            [{ "id" => "d", "status" => "SUCCESS",
+               "meta" => { "image" => "ghcr.io/copilotkit/x@sha256:abc123" } }]
+        end
+        cmd.define_singleton_method(:run_staging_probe) { |services:| { ok: true, summary: "" } }
+
+        out, _ = with_fast_sleeper { capture_io { @rc = cmd.run_with_preflight_only } }
+        assert_equal 0, @rc, "promote should succeed; got out=#{out}"
+
+        # The exact ref pinned MUST be what P1 verified (resolve-once).
+        assert_equal ["ghcr.io/copilotkit/x@sha256:abc123"], gql.pinned_images,
+            "must pin the SAME digest-form ref P1 verified"
+
+        # The :latest tag must have been resolved EXACTLY ONCE — not twice.
+        assert_equal 1, ghcr.resolve_calls["ghcr.io/copilotkit/x:latest"],
+            "resolve_digest must be called exactly once for the staging tag; calls=#{ghcr.resolve_calls.inspect}"
+    end
+
+    # =================== FIX-2 ===================
+
+    # GHCR fake whose manifest_exists raises Railway::GHCR::Error on the
+    # SECOND service. The first service's findings must survive.
+    class RaisingOnSecondGHCR
+        def initialize
+            @manifest_calls = 0
+        end
+
+        def resolve_digest(ref)
+            return ref.split("@", 2).last if ref.include?("@sha256:")
+            "sha256:resolved_for_#{ref}"
+        end
+
+        def manifest_exists(_ref)
+            @manifest_calls += 1
+            raise Railway::GHCR::Error, "boom on call ##{@manifest_calls}" if @manifest_calls == 2
+            :exists
+        end
+
+        def parse_image_ref(ref); Railway::GHCR.allocate.parse_image_ref(ref); end
+    end
+
+    def test_fix2_p1_rescue_is_per_service_not_method_level
+        # Two services. The SECOND triggers a GHCR::Error in manifest_exists.
+        # Before the fix: method-level rescue replaced ALL findings with a
+        # single "REFUSE: P1: GHCR check raised ..." entry, dropping the
+        # first service's clean record AND scoping the error to "P1" with no
+        # service name. After the fix: the first service has no finding
+        # (it passed), the second service has a per-service REFUSE finding
+        # that names the service.
+        cmd = Railway::PromoteCommand.new([])
+        cmd.instance_variable_set(:@ghcr, RaisingOnSecondGHCR.new)
+        # resolved_prod_image now pins staging's RUNNING digest (meta.imageDigest);
+        # stub the deployment lookup so the flow reaches manifest_exists (which
+        # raises on the 2nd service under test).
+        cmd.define_singleton_method(:fetch_latest_staging_deployments) do |svc_id|
+            name = svc_id.sub("svc-stg-", "")
+            [{ "id" => "d", "status" => "SUCCESS",
+               "meta" => { "image" => "ghcr.io/copilotkit/#{name}:latest",
+                           "imageDigest" => "sha256:running_#{name}" } }]
+        end
+        staging = {
+            "services" => [
+                make_svc("a", image: "ghcr.io/copilotkit/a:latest"),
+                make_svc("b", image: "ghcr.io/copilotkit/b:latest"),
+            ],
+        }
+        findings = cmd.send(:check_p1_ghcr_digests, staging)
+
+        refute(findings.any? { |f| f =~ /REFUSE: P1 \(a\)/ },
+            "service 'a' passed P1 and must have no REFUSE; findings=#{findings.inspect}")
+        assert(findings.any? { |f| f =~ /REFUSE: P1 \(b\).*boom on call #2/ },
+            "service 'b' must have a per-service P1 REFUSE naming it; findings=#{findings.inspect}")
+    end
+
+    # =================== FIX-3 ===================
+
+    def test_fix3_pin_and_verify_raises_when_deploy_returns_no_id
+        gql = RecordingGQL.new(redeploy_result: false)
+        err = assert_raises(Railway::PromoteCommand::MutationError) do
+            Railway::PromoteCommand.pin_and_verify(gql,
+                service_id: "svc-x", env_id: "env-prod",
+                image: "ghcr.io/copilotkit/x@sha256:abc",
+                sleeper: ->(_) {})
+        end
+        assert_match(/serviceInstanceDeployV2/i, err.message,
+            "MutationError must reference the deploy mutation; got: #{err.message}")
+    end
+
+    # =================== FIX-4 ===================
+
+    def test_fix4_p2_warns_when_meta_is_a_string
+        # P2 already guards meta.is_a?(Hash) (no crash), but the silent skip
+        # of the race-check should produce a WARN finding so it's visible.
+        cmd = Railway::PromoteCommand.new([])
+        cmd.define_singleton_method(:fetch_latest_staging_deployments) do |_svc_id|
+            [{ "id" => "d", "status" => "SUCCESS",
+               "meta" => "raw-string-not-a-hash" }]
+        end
+        staging = {
+            "services" => [{
+                "name" => "x", "service_id" => "svc-1",
+                "image" => "ghcr.io/copilotkit/x@sha256:abc", "digest" => "sha256:abc",
+                "env_keys" => [],
+            }],
+        }
+        findings = cmd.send(:check_p2_staging_deployments, staging)
+        assert(findings.any? { |f| f =~ /WARN: P2 \(x\).*meta.*String.*Hash/ },
+            "expected WARN finding for non-Hash meta; got: #{findings.inspect}")
+    end
+
+    # =================== FIX-5 ===================
+
+    def test_fix5_partial_promotion_report_on_midloop_failure
+        # Three staging services [a, b, c]; pin succeeds for a and b, fails
+        # for c (serviceInstanceUpdate returns false → MutationError).
+        # The output must name a and b as already-pinned and c as failed.
+        ghcr = CountingGHCR.new(resolve_map: {
+            "ghcr.io/copilotkit/a:latest" => "sha256:aaa",
+            "ghcr.io/copilotkit/b:latest" => "sha256:bbb",
+            "ghcr.io/copilotkit/c:latest" => "sha256:ccc",
+        })
+        gql = RecordingGQL.new(update_fail_for: "svc-prod-c")
+
+        cmd = Railway::PromoteCommand.new(["--non-interactive", "--yes", "--confirm-divergence"])
+        cmd.parser.parse!(cmd.argv)
+        cmd.instance_variable_set(:@staging_snapshot, {
+            "services" => [
+                make_svc("a", image: "ghcr.io/copilotkit/a:latest"),
+                make_svc("b", image: "ghcr.io/copilotkit/b:latest"),
+                make_svc("c", image: "ghcr.io/copilotkit/c:latest"),
+            ],
+        })
+        cmd.instance_variable_set(:@prod_snapshot, {
+            "services" => [make_prod_svc("a"), make_prod_svc("b"), make_prod_svc("c")],
+        })
+        cmd.instance_variable_set(:@gql, gql)
+        cmd.instance_variable_set(:@ghcr, ghcr)
+        cmd.define_singleton_method(:fetch_latest_staging_deployments) do |svc_id|
+            svc = svc_id.sub("svc-stg-", "")
+            digest = { "a" => "sha256:aaa", "b" => "sha256:bbb", "c" => "sha256:ccc" }.fetch(svc)
+            [{ "id" => "d", "status" => "SUCCESS",
+               "meta" => { "image" => "ghcr.io/copilotkit/#{svc}@#{digest}" } }]
+        end
+        cmd.define_singleton_method(:run_staging_probe) { |services:| { ok: true, summary: "" } }
+
+        out, err = with_fast_sleeper { capture_io { @rc = cmd.run_with_preflight_only } }
+        assert_equal 1, @rc, "execute_promotion must return 1 when a mid-loop pin fails"
+        combined = out + err
+        assert_match(/PARTIAL PROMOTION/i, combined,
+            "must emit a loud PARTIAL PROMOTION report; output=#{combined}")
+        assert_match(/already pinned.*\ba\b.*\bb\b/m, combined,
+            "report must name the already-pinned services a and b; output=#{combined}")
+        assert_match(/FAILED on c/, combined,
+            "report must name the failing service c; output=#{combined}")
+    end
+end

showcase/bin/spec/test_promote_single_service.rb

@@ -0,0 +1,277 @@
+# frozen_string_literal: true
+
+# bin/railway promote — single-service positional + --digest support.
+#
+# Context: showcase_promote.yml loops services and invokes
+#   bin/railway promote "$svc" [--digest "$DIGEST"]
+# but the original PromoteCommand parser accepts NEITHER. The positional
+# was silently discarded (promoting the ENTIRE fleet per iteration) and
+# --digest raised OptionParser::InvalidOption (the workflow aborts under
+# set -euo pipefail). These tests pin the fix contract.
+
+require_relative "spec_helper"
+require "stringio"
+
+class PromoteSingleServiceTest < Minitest::Test
+    # Minimal benign GQL fake (P2 deployments query, etc.). The promotion
+    # tests below stub fetch_latest_staging_deployments and run_staging_probe
+    # so this is only used as a fallback.
+    class FakeGQLBenign
+        attr_reader :calls
+        def initialize
+            @calls = []
+            @pinned_by_service = {}  # serviceId => image; tracks per-service pin
+            @ts_counter = 0
+        end
+        def query(q, vars = {})
+            @calls << [q, vars]
+            sid = vars[:serviceId]
+            if q.include?("serviceInstanceUpdate")
+                @pinned_by_service[sid] = vars.dig(:input, :source, :image)
+                { "serviceInstanceUpdate" => true }
+            elsif q.include?("serviceInstanceDeployV2")
+                { "serviceInstanceDeployV2" => "dep-#{sid}" }
+            elsif q.include?("ServiceInstanceRecheck")
+                pinned = @pinned_by_service[sid]
+                if pinned.nil?
+                    {
+                        "serviceInstance" => {
+                            "id" => "i",
+                            "source" => { "image" => "ghcr.io/copilotkit/old@sha256:OLD" },
+                            "updatedAt" => "2026-05-28T00:00:00Z",
+                        },
+                    }
+                else
+                    @ts_counter += 1
+                    pinned_digest = pinned.include?("@") ? pinned.split("@", 2).last : nil
+                    {
+                        "serviceInstance" => {
+                            "id" => "i",
+                            "source" => { "image" => pinned },
+                            "updatedAt" => "2026-05-29T00:00:#{format('%02d', @ts_counter)}Z",
+                            "latestDeployment" => {
+                                "id" => "dep-#{sid}", "status" => "SUCCESS",
+                                "meta" => { "imageDigest" => pinned_digest },
+                            },
+                        },
+                    }
+                end
+            else
+                { "deployments" => { "edges" => [] } }
+            end
+        end
+
+        def pinned_services
+            @calls.select { |q, _| q.include?("serviceInstanceUpdate") }
+                  .map { |_, vars| [vars[:serviceId], vars.dig(:input, :source, :image)] }
+        end
+
+        def pinned_image_for(service_id)
+            row = @calls.find { |q, vars| q.include?("serviceInstanceUpdate") && vars[:serviceId] == service_id }
+            row && row[1].dig(:input, :source, :image)
+        end
+    end
+
+    class FakeGHCR
+        def initialize(resolve_map: {})
+            @resolve_map = resolve_map
+        end
+        def resolve_digest(ref)
+            return ref.split("@", 2).last if ref.include?("@sha256:")
+            @resolve_map[ref] || "sha256:default_digest_for_#{ref.sub(/[^a-z0-9]/i, '_')[0, 16]}"
+        end
+        def manifest_exists(_ref); :exists; end
+        def parse_image_ref(ref); Railway::GHCR.allocate.parse_image_ref(ref); end
+    end
+
+    def make_service(name, prod_image: "ghcr.io/copilotkit/#{name}@sha256:OLD#{name.gsub('-', '')}")
+        {
+            "name" => name, "service_id" => "svc-#{name}",
+            "image" => "ghcr.io/copilotkit/#{name}:latest",
+            # All CRITICAL_ENV_KEYS present so the (now unconditional) critical
+            # env-key presence assertion does not fire — this spec isolates the
+            # single-service narrowing behavior, not env-key parity.
+            "env_keys" => Railway::CRITICAL_ENV_KEYS.dup,
+            "start_command" => "node server.js", "healthcheck_path" => "/health",
+            "region" => "us-west", "replicas" => 1, "restart_policy" => "ON_FAILURE",
+        }
+    end
+
+    def make_prod(name)
+        {
+            "name" => name, "service_id" => "prod-#{name}",
+            "image" => "ghcr.io/copilotkit/#{name}@sha256:OLD#{name.gsub('-', '')}",
+            # All CRITICAL_ENV_KEYS present so the (now unconditional) critical
+            # env-key presence assertion does not fire — this spec isolates the
+            # single-service narrowing behavior, not env-key parity.
+            "env_keys" => Railway::CRITICAL_ENV_KEYS.dup,
+            "start_command" => "node server.js", "healthcheck_path" => "/health",
+            "region" => "us-west", "replicas" => 1, "restart_policy" => "ON_FAILURE",
+        }
+    end
+
+    def build_cmd(argv)
+        cmd = Railway::PromoteCommand.new(argv + ["--non-interactive", "--yes", "--confirm-divergence"])
+        # IMPORTANT: do NOT call parser.parse! here — tests must exercise the
+        # full `run` path so argv parsing (positional + --digest) is covered.
+        cmd
+    end
+
+    def install_two_service_fixture(cmd, gql, ghcr)
+        # Two staging services — "aimock" and "harness" — both also in prod.
+        staging_svcs = [make_service("aimock"), make_service("harness")]
+        prod_svcs    = [make_prod("aimock"),    make_prod("harness")]
+        cmd.instance_variable_set(:@staging_snapshot, { "services" => staging_svcs })
+        cmd.instance_variable_set(:@prod_snapshot,    { "services" => prod_svcs })
+        cmd.instance_variable_set(:@gql, gql)
+        cmd.instance_variable_set(:@ghcr, ghcr)
+        # Skip P2 race-check: stub deployments to SUCCESS with the digest the
+        # promote will actually pin (which, when --digest is set, is the
+        # override — NOT the GHCR-resolved one). Read options[:digest] off the
+        # command at call-time so this works for both default and override
+        # paths without test-side branching.
+        cmd.define_singleton_method(:fetch_latest_staging_deployments) do |service_id|
+            name = service_id.sub(/^svc-/, "")
+            override = options[:digest]
+            if override && options[:service] == name && override.include?("@")
+                ref = override
+            else
+                ghcr_obj = instance_variable_get(:@ghcr)
+                digest = ghcr_obj.resolve_digest("ghcr.io/copilotkit/#{name}:latest")
+                ref = "ghcr.io/copilotkit/#{name}@#{digest}"
+            end
+            [{ "id" => "d", "status" => "SUCCESS", "meta" => { "image" => ref } }]
+        end
+        cmd.define_singleton_method(:run_staging_probe) { |services:| { ok: true, summary: "" } }
+    end
+
+    def with_fast_sleeper
+        original = Railway::PromoteCommand.const_get(:RETRY_DELAY_SEC)
+        Railway::PromoteCommand.send(:remove_const, :RETRY_DELAY_SEC)
+        Railway::PromoteCommand.const_set(:RETRY_DELAY_SEC, 0)
+        yield
+    ensure
+        Railway::PromoteCommand.send(:remove_const, :RETRY_DELAY_SEC)
+        Railway::PromoteCommand.const_set(:RETRY_DELAY_SEC, original)
+    end
+
+    # ── (a) Positional service: only that service is promoted. ────────────
+
+    def test_positional_service_promotes_only_that_service
+        gql  = FakeGQLBenign.new
+        ghcr = FakeGHCR.new(resolve_map: {
+            "ghcr.io/copilotkit/aimock:latest"  => "sha256:NEW_AIMOCK",
+            "ghcr.io/copilotkit/harness:latest" => "sha256:NEW_HARNESS",
+        })
+        cmd = build_cmd(["aimock"])
+        install_two_service_fixture(cmd, gql, ghcr)
+        rc = nil
+        out, _ = with_fast_sleeper { capture_io { rc = cmd.run } }
+        assert_equal 0, rc, "single-service promote should succeed; got out=#{out}"
+
+        pinned = gql.pinned_services
+        assert_equal 1, pinned.size, "must pin exactly ONE service when positional is given (got #{pinned.inspect})"
+        sid, image = pinned.first
+        assert_equal "prod-aimock", sid, "must target prod-aimock, not the whole fleet"
+        assert_equal "ghcr.io/copilotkit/aimock@sha256:NEW_AIMOCK", image
+        # Loud regression guard against silent fleet-wide pin.
+        refute gql.pinned_image_for("prod-harness"), "harness must NOT be touched when only aimock was requested"
+    end
+
+    # ── (b) Positional + --digest: digest overrides resolved ref. ─────────
+
+    def test_positional_service_with_digest_pins_that_digest
+        gql  = FakeGQLBenign.new
+        ghcr = FakeGHCR.new(resolve_map: {
+            "ghcr.io/copilotkit/aimock:latest"  => "sha256:NEW_AIMOCK",
+            "ghcr.io/copilotkit/harness:latest" => "sha256:NEW_HARNESS",
+        })
+        override = "ghcr.io/copilotkit/aimock@sha256:OVERRIDE_DIGEST"
+        cmd = build_cmd(["aimock", "--digest", override])
+        install_two_service_fixture(cmd, gql, ghcr)
+        rc = nil
+        out, _ = with_fast_sleeper { capture_io { rc = cmd.run } }
+        assert_equal 0, rc, "promote with --digest should succeed; got out=#{out}"
+
+        pinned = gql.pinned_services
+        assert_equal 1, pinned.size, "--digest must still pin only the named service"
+        sid, image = pinned.first
+        assert_equal "prod-aimock", sid
+        assert_equal override, image,
+            "must pin the EXPLICIT --digest ref, not the GHCR-resolved one"
+    end
+
+    # ── (c) --digest without positional → fail fast. ──────────────────────
+
+    def test_digest_without_service_fails_fast
+        cmd = build_cmd(["--digest", "ghcr.io/copilotkit/x@sha256:abc"])
+        ghcr = FakeGHCR.new
+        gql  = FakeGQLBenign.new
+        install_two_service_fixture(cmd, gql, ghcr)
+        ex = nil
+        # die! calls Kernel#exit which raises SystemExit; capture it so the
+        # test can assert the exit code AND the error message together.
+        out, err = capture_io { ex = assert_raises(SystemExit) { cmd.run } }
+        refute_equal 0, ex.status, "--digest without a service must NOT promote the fleet"
+        combined = out + err
+        assert_match(/--digest.*requires.*service|--digest.*without.*service|service.*required.*--digest/i,
+            combined, "must surface a clear error explaining --digest needs a positional service")
+        # And it must NOT have issued any pin mutations.
+        assert_empty gql.pinned_services, "no pin mutations should run on the fail-fast path"
+    end
+
+    # ── (d) Unknown positional service → fail fast with valid names. ──────
+
+    def test_unknown_positional_service_fails_with_valid_names_listed
+        cmd = build_cmd(["this-service-does-not-exist"])
+        ghcr = FakeGHCR.new
+        gql  = FakeGQLBenign.new
+        install_two_service_fixture(cmd, gql, ghcr)
+        ex = nil
+        out, err = capture_io { ex = assert_raises(SystemExit) { cmd.run } }
+        refute_equal 0, ex.status, "unknown service must fail, not silently no-op or fall through to fleet"
+        combined = out + err
+        assert_match(/unknown.*service|not.*known|invalid.*service/i, combined)
+        # The error must list at least one canonical staging name (e.g. aimock)
+        # so the operator can self-correct.
+        assert_match(/aimock/, combined,
+            "valid-names enumeration must include canonical staging services")
+        assert_empty gql.pinned_services, "no pin mutations on unknown-service error path"
+    end
+
+    # ── (e) No positional → preserve full-fleet behavior (regression). ────
+
+    def test_no_positional_preserves_full_fleet_promote
+        gql  = FakeGQLBenign.new
+        ghcr = FakeGHCR.new(resolve_map: {
+            "ghcr.io/copilotkit/aimock:latest"  => "sha256:NEW_AIMOCK",
+            "ghcr.io/copilotkit/harness:latest" => "sha256:NEW_HARNESS",
+        })
+        cmd = build_cmd([])
+        install_two_service_fixture(cmd, gql, ghcr)
+        rc = nil
+        out, _ = with_fast_sleeper { capture_io { rc = cmd.run } }
+        assert_equal 0, rc, "no-arg promote must keep working for operators; got out=#{out}"
+
+        pinned_ids = gql.pinned_services.map(&:first).sort
+        assert_equal ["prod-aimock", "prod-harness"], pinned_ids,
+            "no-arg promote must continue to pin the whole fleet"
+    end
+
+    # ── Parser-level coverage for --digest flag and positional acceptance. ─
+
+    def test_parser_accepts_digest_flag
+        c = Railway::PromoteCommand.new(["--digest", "ghcr.io/copilotkit/x@sha256:abc"])
+        # Must NOT raise InvalidOption.
+        c.parser.parse!(c.argv)
+        assert_equal "ghcr.io/copilotkit/x@sha256:abc", c.options[:digest]
+    end
+
+    def test_parser_leaves_positional_in_argv
+        c = Railway::PromoteCommand.new(["aimock", "--yes"])
+        c.parser.parse!(c.argv)
+        # OptionParser#parse! consumes known flags; the positional must remain.
+        assert_includes c.argv, "aimock"
+        assert c.options[:yes]
+    end
+end

showcase/bin/spec/test_promote_single_service_fleet_invariants.rb

@@ -0,0 +1,510 @@
+# frozen_string_literal: true
+
+# bin/railway promote — fleet-scoped invariants must read the FULL
+# un-narrowed snapshots, while set-parity is target-scoped for a
+# single-service promote.
+#
+# Background: the per-service `promote <svc>` feature narrows the
+# target_*/per-service views to one service for the P1..P3/P6 and
+# critical-env-key checks, but the FLEET-SHAPE checks
+# (check_expected_prod_domains, check_service_set_parity) read the FULL
+# un-narrowed fleet_* snapshots — they describe the env, not the promote
+# target. (#5322 removed the earlier co-narrowing of these checks; do NOT
+# reintroduce snapshot narrowing for fleet-scoped checks.)
+#
+# Mechanism under test:
+#
+#   (1) check_expected_prod_domains compares the FLEET-WIDE public-host set
+#       (EXPECTED_DOMAINS[PRODUCTION_ENV_ID]) against the union of
+#       custom_domains across the FULL prod snapshot. The fixture's full
+#       prod fleet carries the full public-host set (every host in
+#       EXPECTED_DOMAINS[PRODUCTION_ENV_ID], currently 5); a narrowed
+#       single-service view of the target owns 0 of them, so reading the
+#       narrowed view would make the entire set look "missing" → spurious
+#       WARN → and because the
+#       real workflow (.github/workflows/showcase_promote.yml) does NOT
+#       pass --confirm-divergence, promote would refuse. Reading the FULL
+#       fleet avoids that.
+#
+#   (2) check_service_set_parity reads the FULL un-narrowed fleet_staging /
+#       fleet_prod snapshots and applies `& [target]` scoping to BOTH the
+#       staging-only and prod-only arms when a single service is targeted.
+#       So a single-service promote REFUSEs only on parity violations
+#       involving the TARGET itself, and tolerates unrelated single-env
+#       services (staging-only harness-workers / starter-* demos, or a
+#       deprecated prod-only service). Full-fleet promotes (target nil)
+#       keep both arms at full fleet strictness.
+#
+# Red-green coverage for THIS PR is provided by the two tolerance tests:
+#   - test_single_service_promote_tolerates_unrelated_staging_only_services
+#   - test_single_service_promote_tolerates_unrelated_prod_only_service
+# The test_healthy_* and test_target_absent_* tests are #5322 regression
+# guards (they pin that fleet-scoped checks read the FULL snapshots and
+# that an absent target still fails loud), not red-green for this change.
+
+require_relative "spec_helper"
+require "stringio"
+
+class PromoteSingleServiceFleetInvariantsTest < Minitest::Test
+    # Reuse the same minimal benign GQL/GHCR fakes as
+    # test_promote_single_service.rb (parallel fixture style — kept inline
+    # here so this spec is self-contained).
+    class FakeGQLBenign
+        attr_reader :calls
+        def initialize
+            @calls = []
+            @pinned_by_service = {}
+            @ts_counter = 0
+        end
+        def query(q, vars = {})
+            @calls << [q, vars]
+            sid = vars[:serviceId]
+            if q.include?("serviceInstanceUpdate")
+                @pinned_by_service[sid] = vars.dig(:input, :source, :image)
+                { "serviceInstanceUpdate" => true }
+            elsif q.include?("serviceInstanceDeployV2")
+                { "serviceInstanceDeployV2" => "dep-#{sid}" }
+            elsif q.include?("ServiceInstanceRecheck")
+                pinned = @pinned_by_service[sid]
+                if pinned.nil?
+                    {
+                        "serviceInstance" => {
+                            "id" => "i",
+                            "source" => { "image" => "ghcr.io/copilotkit/old@sha256:OLD" },
+                            "updatedAt" => "2026-05-28T00:00:00Z",
+                        },
+                    }
+                else
+                    @ts_counter += 1
+                    pinned_digest = pinned.include?("@") ? pinned.split("@", 2).last : nil
+                    {
+                        "serviceInstance" => {
+                            "id" => "i",
+                            "source" => { "image" => pinned },
+                            "updatedAt" => "2026-05-29T00:00:#{format('%02d', @ts_counter)}Z",
+                            "latestDeployment" => {
+                                "id" => "dep-#{sid}", "status" => "SUCCESS",
+                                "meta" => { "imageDigest" => pinned_digest },
+                            },
+                        },
+                    }
+                end
+            else
+                { "deployments" => { "edges" => [] } }
+            end
+        end
+
+        def pinned_services
+            @calls.select { |q, _| q.include?("serviceInstanceUpdate") }
+                  .map { |_, vars| [vars[:serviceId], vars.dig(:input, :source, :image)] }
+        end
+
+        def pinned_image_for(service_id)
+            row = @calls.find { |q, vars| q.include?("serviceInstanceUpdate") && vars[:serviceId] == service_id }
+            row && row[1].dig(:input, :source, :image)
+        end
+    end
+
+    class FakeGHCR
+        def initialize(resolve_map: {})
+            @resolve_map = resolve_map
+        end
+        def resolve_digest(ref)
+            return ref.split("@", 2).last if ref.include?("@sha256:")
+            @resolve_map[ref] || "sha256:default_digest_for_#{ref.sub(/[^a-z0-9]/i, '_')[0, 16]}"
+        end
+        def manifest_exists(_ref); :exists; end
+        def parse_image_ref(ref); Railway::GHCR.allocate.parse_image_ref(ref); end
+    end
+
+    def make_staging_service(name)
+        {
+            "name" => name, "service_id" => "svc-#{name}",
+            "image" => "ghcr.io/copilotkit/#{name}:latest",
+            # All CRITICAL_ENV_KEYS present so the (now unconditional) critical
+            # env-key presence assertion does not fire — this spec isolates the
+            # fleet-shape invariants, not env-key parity.
+            "env_keys" => Railway::CRITICAL_ENV_KEYS.dup,
+            "start_command" => "node server.js", "healthcheck_path" => "/health",
+            "region" => "us-west", "replicas" => 1, "restart_policy" => "ON_FAILURE",
+        }
+    end
+
+    def make_prod_service(name, custom_domains: [])
+        {
+            "name" => name, "service_id" => "prod-#{name}",
+            "image" => "ghcr.io/copilotkit/#{name}@sha256:OLD#{name.gsub(/[^a-z0-9]/i, '')}",
+            # All CRITICAL_ENV_KEYS present so the (now unconditional) critical
+            # env-key presence assertion does not fire — this spec isolates the
+            # fleet-shape invariants, not env-key parity.
+            "env_keys" => Railway::CRITICAL_ENV_KEYS.dup,
+            "start_command" => "node server.js", "healthcheck_path" => "/health",
+            "region" => "us-west", "replicas" => 1, "restart_policy" => "ON_FAILURE",
+            "custom_domains" => custom_domains,
+        }
+    end
+
+    # Build a FLEET-SHAPED snapshot pair. The full prod snapshot's union of
+    # custom_domains across services EXACTLY satisfies the fleet-wide
+    # EXPECTED_DOMAINS[PRODUCTION_ENV_ID] set, so a healthy fleet must NOT
+    # produce a domain WARN. Two of those services ("aimock", "harness")
+    # also exist in staging — promote will target one of them.
+    #
+    # `staging_only_extras:` injects services into the STAGING fleet that
+    # have NO prod counterpart — modeling the live staging-only services
+    # (harness-workers + the 12 starter-* demos). They land in the full
+    # (pre-narrow) staging snapshot, exactly where the fleet-scoped
+    # set-parity check reads. They are legitimately staging-only and a
+    # single-service promote must tolerate them.
+    #
+    # `prod_only_extras:` is the mirror: it injects services into the FULL
+    # prod snapshot that have NO staging counterpart — modeling a
+    # deprecated/prod-only service. They land in the full (pre-narrow) prod
+    # snapshot where the prod-only arm of set-parity reads. They are
+    # legitimately prod-only and a single-service promote must tolerate them.
+    def install_fleet_fixture(cmd, gql, ghcr, prod_includes_target: true, target: "aimock",
+                              staging_only_extras: [], prod_only_extras: [])
+        # DERIVE-FROM-SSOT INVARIANT: the "domain-bearing" prod services and
+        # their public hosts are derived from the SAME constant the prod code
+        # reads — Railway::EXPECTED_DOMAINS[PRODUCTION_ENV_ID] — rather than
+        # re-hardcoding host literals here. This keeps the fixture coupled to
+        # the SSOT (showcase/scripts/railway-envs.generated.json): when the
+        # SSOT public-host set changes, this fixture moves with it instead of
+        # synthesizing phantom-domain WARNs or breaking the parity die!.
+        #
+        # Each public host is mapped back to its owning SSOT service name so
+        # the fixture's prod fleet legitimately carries all public prod hosts,
+        # even when promote is narrowed to a service that owns no public host.
+        public_hosts = Railway::EXPECTED_DOMAINS[Railway::PRODUCTION_ENV_ID]
+        ssot_services = Railway::SSOT_DATA.fetch("services")
+        domain_services_prod = public_hosts.map do |host|
+            owner = ssot_services.find { |s| s.dig("domains", "prod") == host }
+            make_prod_service(owner.fetch("name"), custom_domains: [host])
+        end
+        # The promote target must be a real SSOT staging service (this is what
+        # the prod code validates against), so assert it rather than trusting a
+        # bare literal that could silently drift from the SSOT.
+        assert_includes Railway::STAGING_SERVICES, target,
+            "fixture target #{target.inspect} must be a real SSOT staging service"
+        # Plus the targeted service (no public domain of its own — e.g. an
+        # internal aimock). Optionally OMIT it from prod to exercise BUG #2.
+        #
+        # GUARD: if `target` already owns a public prod host it is ALREADY in
+        # `domain_services_prod` above. Appending make_prod_service(target)
+        # again would produce a malformed fleet with the SAME prod service
+        # (service_id "prod-#{target}") listed twice — once domain-bearing,
+        # once with custom_domains:[] — contradicting this helper's own
+        # "no public domain of its own" contract. So only append the bare
+        # target when it is NOT already a derived domain owner.
+        derived_owner_names = domain_services_prod.map { |s| s["name"] }
+        if prod_includes_target && !derived_owner_names.include?(target)
+            domain_services_prod << make_prod_service(target)
+        end
+        # And a non-targeted sibling that exists in both staging and prod —
+        # picked from STAGING_SERVICES as a real service that owns no public
+        # prod host, so it stays distinct from the domain-bearing set above.
+        domain_owner_names = domain_services_prod.map { |s| s["name"] }
+        sibling = Railway::STAGING_SERVICES.find do |n|
+            !domain_owner_names.include?(n) && n != target &&
+                ssot_services.find { |s| s["name"] == n }&.dig("domains", "prod")&.end_with?(".up.railway.app")
+        end
+        domain_services_prod << make_prod_service(sibling)
+
+        # Staging mirrors prod's service NAMES (set parity) — every prod
+        # service has a corresponding staging entry. (For BUG #2, target is
+        # in staging but absent from prod by design.) `prod_only_extras` are
+        # deliberately NOT mirrored into staging, so they land in the full
+        # prod snapshot with no staging counterpart.
+        staging_names = (domain_services_prod.map { |s| s["name"] } + [target] + staging_only_extras).uniq
+        staging_services = staging_names.map { |n| make_staging_service(n) }
+
+        # Inject prod-only extras into the FULL prod snapshot AFTER deriving
+        # staging_names, so they have no staging counterpart.
+        prod_only_extras.each { |n| domain_services_prod << make_prod_service(n) }
+
+        cmd.instance_variable_set(:@staging_snapshot, { "services" => staging_services })
+        cmd.instance_variable_set(:@prod_snapshot,    { "services" => domain_services_prod })
+        cmd.instance_variable_set(:@gql, gql)
+        cmd.instance_variable_set(:@ghcr, ghcr)
+
+        # P2 race-check stub — see test_promote_single_service.rb for the
+        # same pattern. Mirror the pin the promote will actually issue.
+        cmd.define_singleton_method(:fetch_latest_staging_deployments) do |service_id|
+            name = service_id.sub(/^svc-/, "")
+            override = options[:digest]
+            if override && options[:service] == name && override.include?("@")
+                ref = override
+            else
+                ghcr_obj = instance_variable_get(:@ghcr)
+                digest = ghcr_obj.resolve_digest("ghcr.io/copilotkit/#{name}:latest")
+                ref = "ghcr.io/copilotkit/#{name}@#{digest}"
+            end
+            [{ "id" => "d", "status" => "SUCCESS", "meta" => { "image" => ref } }]
+        end
+        cmd.define_singleton_method(:run_staging_probe) { |services:| { ok: true, summary: "" } }
+    end
+
+    def build_cmd(argv)
+        # CRITICAL: this test MUST mirror the real workflow invocation —
+        # showcase_promote.yml calls `promote <svc>` WITHOUT
+        # --confirm-divergence. Do NOT add --confirm-divergence here.
+        Railway::PromoteCommand.new(argv + ["--non-interactive", "--yes"])
+    end
+
+    # Zero the promote retry back-off for the duration of the block so the 3x
+    # eventual-consistency retry loop in pin_and_verify doesn't add real wall
+    # time. RETRY_DELAY_SEC is a process-global constant, so this is a global
+    # mutation; the clean seam would be pin_and_verify(sleeper:), but the
+    # cmd.run → pin loop constructs that call internally with no injection
+    # point we can reach without changing bin/railway production logic. So we
+    # keep the swap but make it BULLETPROOF against run-order state leakage:
+    #
+    #   - capture `original` BEFORE any mutation;
+    #   - track whether the swap actually happened (`swapped`) so a failure
+    #     between capture and set never leaves the const removed/zeroed;
+    #   - restore in `ensure` (runs even if the block raises);
+    #   - swap via const_set with warnings silenced (no remove_const churn /
+    #     "already initialized constant" warning), and restore the EXACT prior
+    #     value so no perturbed state can leak into a later test in the
+    #     randomized run order.
+    def with_fast_sleeper
+        original = Railway::PromoteCommand.const_get(:RETRY_DELAY_SEC)
+        swapped  = false
+        silence_const_redefinition { Railway::PromoteCommand.const_set(:RETRY_DELAY_SEC, 0) }
+        swapped = true
+        yield
+    ensure
+        silence_const_redefinition { Railway::PromoteCommand.const_set(:RETRY_DELAY_SEC, original) } if swapped
+    end
+
+    # const_set over an existing constant emits a Ruby "already initialized
+    # constant" warning; suppress it locally so the suite output stays clean
+    # without touching the process-global $VERBOSE beyond this block.
+    def silence_const_redefinition
+        prev = $VERBOSE
+        $VERBOSE = nil
+        yield
+    ensure
+        $VERBOSE = prev
+    end
+
+    # ── #5322 regression guard: healthy fleet, single-service promote, NO
+    # --confirm-divergence must succeed. check_expected_prod_domains reads the
+    # FULL prod snapshot (the full EXPECTED_DOMAINS[PRODUCTION_ENV_ID]
+    # public-host set present), so no phantom domain
+    # WARN is synthesized and the promote proceeds: rc=0, target pinned,
+    # siblings untouched. (This pins that fleet-scoped checks read the full
+    # snapshot; it is a guard, not red-green for the current parity change.)
+    def test_healthy_fleet_single_service_promote_without_confirm_divergence_succeeds
+        gql  = FakeGQLBenign.new
+        ghcr = FakeGHCR.new(resolve_map: {
+            "ghcr.io/copilotkit/aimock:latest" => "sha256:NEW_AIMOCK",
+        })
+        cmd = build_cmd(["aimock"])
+        install_fleet_fixture(cmd, gql, ghcr, prod_includes_target: true, target: "aimock")
+
+        rc = nil
+        out, _ = with_fast_sleeper { capture_io { rc = cmd.run } }
+
+        assert_equal 0, rc,
+            "single-service promote against a HEALTHY fleet must NOT be " \
+            "refused for missing fleet-wide domains (workflow doesn't pass " \
+            "--confirm-divergence). Got rc=#{rc.inspect}; out=\n#{out}"
+
+        # Spurious-WARN smoke check: we must NOT have emitted a fleet-domains
+        # WARN since the un-narrowed fleet satisfies EXPECTED_DOMAINS.
+        refute_match(/WARN: production missing expected custom domains/, out,
+            "fleet-domain check should evaluate the FULL prod snapshot; the " \
+            "narrowed view must not synthesize a phantom 'missing' WARN")
+
+        pinned = gql.pinned_services
+        assert_equal 1, pinned.size,
+            "exactly one service should be promoted (target only); got #{pinned.inspect}"
+        sid, image = pinned.first
+        assert_equal "prod-aimock", sid
+        assert_equal "ghcr.io/copilotkit/aimock@sha256:NEW_AIMOCK", image
+
+        # Siblings must remain untouched.
+        refute gql.pinned_image_for("prod-harness"),
+            "harness must not be pinned when only aimock was promoted"
+        refute gql.pinned_image_for("prod-dashboard"),
+            "dashboard must not be pinned when only aimock was promoted"
+    end
+
+    # ── #5322 regression guard: target in staging but ABSENT from prod must
+    # FAIL LOUD. check_service_set_parity reads the FULL un-narrowed
+    # fleet_staging / fleet_prod and applies `& [target]` scoping: the target
+    # is in the staging-only set, so the staging-not-in-prod REFUSE survives
+    # scoping and surfaces (never a silent rc=0 with no pin). To prove the
+    # check truly reads the FULL fleet AND applies target-scoping (not merely
+    # a trivially-narrowed pair), the fixture also carries an UNRELATED
+    # staging-only sibling: it must be IGNORED while the target REFUSE fires —
+    # the REFUSE names ONLY the target.
+    def test_target_absent_from_prod_fails_loud
+        gql  = FakeGQLBenign.new
+        ghcr = FakeGHCR.new(resolve_map: {
+            "ghcr.io/copilotkit/aimock:latest" => "sha256:NEW_AIMOCK",
+        })
+        cmd = build_cmd(["aimock"])
+        # prod_includes_target=false → "aimock" exists in staging but NOT
+        # in the full prod snapshot. An unrelated staging-only sibling
+        # ("harness-workers") is injected too: the full fleet thus has TWO
+        # staging-only names, but target-scoping must REFUSE on ONLY the
+        # target. This distinguishes the fix from the buggy full-vs-narrowed
+        # behavior — a check that ignored target-scoping would also name the
+        # sibling.
+        install_fleet_fixture(cmd, gql, ghcr, prod_includes_target: false, target: "aimock",
+                              staging_only_extras: %w[harness-workers])
+
+        rc = nil
+        out, _ = with_fast_sleeper { capture_io { rc = cmd.run } }
+
+        refute_equal 0, rc,
+            "target service present in staging but ABSENT from prod must " \
+            "FAIL LOUD (nonzero rc), not silently succeed with no pin. " \
+            "Got rc=#{rc.inspect}; out=\n#{out}"
+
+        # The error message must be informative — surface the parity
+        # violation rather than a generic / opaque failure. AND it must name
+        # ONLY the target: target-scoping means the unrelated staging-only
+        # sibling is excluded from the REFUSE even though it is in the FULL
+        # fleet's staging-only set. (A check that read the full fleet without
+        # target-scoping would also list "harness-workers" here.)
+        assert_match(/REFUSE: services in staging not in prod: aimock\b/, out,
+            "must surface a clear REFUSE naming the target missing from prod")
+        refute_match(/harness-workers/, out,
+            "target-scoping must exclude unrelated staging-only siblings from " \
+            "the REFUSE — only the target should be named")
+
+        assert_empty gql.pinned_services,
+            "no pin mutations may be issued when target is absent from prod"
+    end
+
+    # ── Single-service promote must TOLERATE unrelated staging-only services.
+    # The live staging fleet legitimately carries services with no prod
+    # counterpart — harness-workers (SSOT-modeled staging-only) and the 12
+    # starter-* demos (not in SSOT). For a single-service promote of a
+    # healthy target (present in both staging and prod) these are irrelevant.
+    #
+    # Red-green for the STAGING-only arm of the target-scoping fix (#5324):
+    # before scoping, check_service_set_parity diffs the FULL staging fleet vs
+    # the FULL prod fleet and REFUSEs because the extras are "in staging not
+    # in prod" — blocking an otherwise-clean single-service promote. After
+    # the fix, the staging-only REFUSE is target-scoped, so the unrelated
+    # extras are ignored; rc=0, target pinned, no REFUSE.
+    def test_single_service_promote_tolerates_unrelated_staging_only_services
+        gql  = FakeGQLBenign.new
+        ghcr = FakeGHCR.new(resolve_map: {
+            "ghcr.io/copilotkit/docs:latest" => "sha256:NEW_DOCS",
+        })
+        cmd = build_cmd(["docs"])
+        # Target "docs" is present in BOTH staging and prod (the domain-
+        # bearing prod fixture already includes a "docs" service, mirrored
+        # into staging). Inject unrelated staging-only siblings.
+        install_fleet_fixture(cmd, gql, ghcr, prod_includes_target: true, target: "docs",
+                              staging_only_extras: %w[harness-workers starter-adk starter-langgraph-python])
+
+        rc = nil
+        out, _ = with_fast_sleeper { capture_io { rc = cmd.run } }
+
+        assert_equal 0, rc,
+            "a single-service promote of a healthy target must NOT be refused " \
+            "because UNRELATED staging-only services (harness-workers, " \
+            "starter-* demos) exist in staging but not prod. " \
+            "Got rc=#{rc.inspect}; out=\n#{out}"
+
+        refute_match(/REFUSE: services in staging not in prod/, out,
+            "staging-only services unrelated to the promote target must not " \
+            "trigger the set-parity REFUSE on a single-service promote")
+
+        pinned = gql.pinned_services
+        assert_equal 1, pinned.size,
+            "exactly one service should be promoted (target only); got #{pinned.inspect}"
+        sid, image = pinned.first
+        assert_equal "prod-docs", sid
+        assert_equal "ghcr.io/copilotkit/docs@sha256:NEW_DOCS", image
+    end
+
+    # ── Single-service promote must TOLERATE unrelated prod-only services.
+    # This is the MIRROR of the staging-only tolerance test and the red-green
+    # for the PROD-only arm of the target-scoping fix (#5324). A prod-only
+    # service (e.g. a deprecated "deprecated-prod-only-svc" still present in
+    # prod but removed from staging) has no bearing on a single-service promote of an
+    # unrelated healthy target.
+    #
+    # Red (before this PR's source change): the prod-only arm of
+    # check_service_set_parity was computed over the FULL fleet
+    # (`(p_names - s_names)`) with NO target scoping, so ANY prod-only service
+    # REFUSEs EVERY single-service promote — "services in prod not in staging"
+    # fires and rc=1. Green (after): the prod-only arm is target-scoped too,
+    # so the unrelated prod-only service is ignored; rc=0, target pinned, no
+    # "in prod not in staging" REFUSE.
+    def test_single_service_promote_tolerates_unrelated_prod_only_service
+        gql  = FakeGQLBenign.new
+        ghcr = FakeGHCR.new(resolve_map: {
+            "ghcr.io/copilotkit/docs:latest" => "sha256:NEW_DOCS",
+        })
+        cmd = build_cmd(["docs"])
+        # Target "docs" is present in BOTH staging and prod. Inject an
+        # unrelated prod-only sibling (no staging counterpart).
+        install_fleet_fixture(cmd, gql, ghcr, prod_includes_target: true, target: "docs",
+                              prod_only_extras: %w[deprecated-prod-only-svc])
+
+        rc = nil
+        out, _ = with_fast_sleeper { capture_io { rc = cmd.run } }
+
+        assert_equal 0, rc,
+            "a single-service promote of a healthy target must NOT be refused " \
+            "because an UNRELATED prod-only service (deprecated-prod-only-svc) exists in " \
+            "prod but not staging. Got rc=#{rc.inspect}; out=\n#{out}"
+
+        refute_match(/REFUSE: services in prod not in staging/, out,
+            "prod-only services unrelated to the promote target must not " \
+            "trigger the set-parity REFUSE on a single-service promote")
+
+        pinned = gql.pinned_services
+        assert_equal 1, pinned.size,
+            "exactly one service should be promoted (target only); got #{pinned.inspect}"
+        sid, image = pinned.first
+        assert_equal "prod-docs", sid
+        assert_equal "ghcr.io/copilotkit/docs@sha256:NEW_DOCS", image
+    end
+
+    # ── Fixture well-formedness: the derived prod snapshot must NEVER list the
+    # same prod service twice. When the promote target already owns a public
+    # prod host (e.g. "docs" owns docs.copilotkit.ai), the SSOT-derivation
+    # already includes it among the domain-bearing services; the helper must
+    # not ALSO append a bare make_prod_service(target), which would yield two
+    # services with the same name + service_id ("prod-docs") — one
+    # domain-bearing, one with custom_domains:[] — a malformed fleet shape that
+    # contradicts the helper's "no public domain of its own" contract and is
+    # only masked downstream by find_service first-match + .uniq. We assert the
+    # invariant directly against the snapshot the fixture installs.
+    def test_install_fleet_fixture_produces_no_duplicate_prod_services
+        gql  = FakeGQLBenign.new
+        ghcr = FakeGHCR.new
+        cmd  = build_cmd(["docs"])
+        # "docs" owns a public prod host, so it is the case that previously
+        # produced a duplicate prod-docs entry.
+        install_fleet_fixture(cmd, gql, ghcr, prod_includes_target: true, target: "docs")
+
+        prod = cmd.instance_variable_get(:@prod_snapshot)
+        names = (prod["services"] || []).map { |s| s["name"] }
+        dups  = names.tally.select { |_, count| count > 1 }.keys
+        assert_empty dups,
+            "derived prod snapshot must not list any service name twice; " \
+            "duplicates=#{dups.inspect} in #{names.inspect}"
+
+        ids = (prod["services"] || []).map { |s| s["service_id"] }
+        id_dups = ids.tally.select { |_, count| count > 1 }.keys
+        assert_empty id_dups,
+            "derived prod snapshot must not list any service_id twice; " \
+            "duplicate ids=#{id_dups.inspect} in #{ids.inspect}"
+
+        # The target must still be present in prod exactly once (a healthy
+        # single-service promote target), so the tolerance tests above keep
+        # genuinely exercising a present-in-both target.
+        assert_equal 1, names.count("docs"),
+            "the promote target must appear in the prod fleet exactly once"
+    end
+end

showcase/bin/spec/test_promote_staging_running_digest.rb

@@ -0,0 +1,241 @@
+# frozen_string_literal: true
+
+require_relative "spec_helper"
+
+# Covers the staging-running-digest pin + drift warning:
+#   CASE-A: staging RUNNING digest == current :latest  -> pin it, NO warning.
+#   CASE-B: staging RUNNING digest != current :latest  -> pin the RUNNING
+#           digest (NOT :latest) + emit the loud STAGING DRIFT warning.
+#   CASE-C: resolved_prod_image pins meta.imageDigest, not resolve_digest(tag).
+#   CASE-D: no SUCCESS staging deployment -> resolved_prod_image returns nil
+#           (caller REFUSEs rather than pin a mutable tag).
+class PromoteStagingRunningDigestTest < Minitest::Test
+    # GHCR fake: resolve_digest(:latest-tag) returns a configurable "current
+    # latest" digest; manifest_exists is always :exists. Counts resolve_digest
+    # calls so we can assert the tag is NOT used to pick the prod pin.
+    class FakeGHCR
+        attr_reader :resolve_calls
+
+        def initialize(current_latest:)
+            @current_latest = current_latest
+            @resolve_calls  = Hash.new(0)
+        end
+
+        def resolve_digest(ref)
+            @resolve_calls[ref] += 1
+            return ref.split("@", 2).last if ref.include?("@sha256:")
+            @current_latest
+        end
+
+        def manifest_exists(_ref); :exists; end
+        def parse_image_ref(ref); Railway::GHCR.allocate.parse_image_ref(ref); end
+    end
+
+    # Benign GQL: P2 issues a deployments query; return empty so P2 is a no-op.
+    class FakeGQLEmpty
+        def query(*); { "deployments" => { "edges" => [] } }; end
+    end
+
+    def make_cmd(staging_services:, prod_services: [], current_latest:, running_meta:)
+        cmd = Railway::PromoteCommand.new(["--non-interactive", "--yes", "--confirm-divergence"])
+        cmd.parser.parse!(cmd.argv)
+        cmd.instance_variable_set(:@staging_snapshot, { "services" => staging_services })
+        cmd.instance_variable_set(:@prod_snapshot,    { "services" => prod_services })
+        cmd.instance_variable_set(:@gql,  FakeGQLEmpty.new)
+        cmd.instance_variable_set(:@ghcr, FakeGHCR.new(current_latest: current_latest))
+        # The RUNNING digest source: latest SUCCESS staging deployment's
+        # meta.imageDigest. Keyed implicitly by the single service under test.
+        cmd.define_singleton_method(:fetch_latest_staging_deployments) do |_svc_id|
+            [{ "id" => "d", "status" => "SUCCESS", "meta" => running_meta }]
+        end
+        cmd.define_singleton_method(:run_staging_probe) { |services:| { ok: true, summary: "" } }
+        cmd
+    end
+
+    STG = lambda do |name|
+        {
+            "name" => name, "service_id" => "svc-stg-#{name}",
+            "image" => "ghcr.io/copilotkit/#{name}:latest",
+            "env_keys" => [],
+        }
+    end
+
+    # =================== CASE-C: pins meta.imageDigest, not :latest ===========
+
+    def test_resolved_prod_image_pins_running_digest_not_latest
+        cmd = make_cmd(
+            staging_services: [STG.call("x")],
+            current_latest:   "sha256:261ccdef",  # the WRONG (drifted) :latest
+            running_meta:     { "image" => "ghcr.io/copilotkit/x:latest",
+                                "imageDigest" => "sha256:f9454e79" }, # the RUNNING digest
+        )
+        resolved = cmd.send(:resolved_prod_image, STG.call("x"))
+        assert_equal "ghcr.io/copilotkit/x@sha256:f9454e79", resolved,
+            "must pin staging RUNNING digest (meta.imageDigest), NOT current :latest"
+        # resolve_digest(:latest-tag) must NOT be what picks the prod pin.
+        refute_includes resolved, "261ccdef",
+            "prod pin must not be the current :latest digest"
+    end
+
+    # =================== CASE-D: no SUCCESS deploy -> nil =====================
+
+    def test_resolved_prod_image_nil_when_no_running_digest
+        cmd = make_cmd(
+            staging_services: [STG.call("x")],
+            current_latest:   "sha256:261ccdef",
+            running_meta:     { "image" => "ghcr.io/copilotkit/x:latest" }, # no imageDigest
+        )
+        # Drop the SUCCESS deployment entirely.
+        cmd.define_singleton_method(:fetch_latest_staging_deployments) { |_| [] }
+        assert_nil cmd.send(:resolved_prod_image, STG.call("x")),
+            "must return nil (caller REFUSEs) when no running digest is resolvable"
+    end
+
+    # =================== CASE-A: running == :latest -> no warning ============
+
+    def test_no_drift_warning_when_running_equals_latest
+        same = "sha256:cafebabe"
+        cmd = make_cmd(
+            staging_services: [STG.call("x")],
+            prod_services:    [{ "name" => "x", "service_id" => "svc-prod-x",
+                                 "image" => "ghcr.io/copilotkit/x@sha256:OLD", "env_keys" => [] }],
+            current_latest:   same,
+            running_meta:     { "image" => "ghcr.io/copilotkit/x:latest", "imageDigest" => same },
+        )
+        out, _err = capture_io { cmd.send(:check_p1_ghcr_digests, cmd.instance_variable_get(:@staging_snapshot)) }
+        cmd.send(:emit_staging_drift_warnings)
+        out2, _ = capture_io { cmd.send(:emit_staging_drift_warnings) }
+        refute_match(/STAGING DRIFT/, out + out2,
+            "no drift warning when staging RUNNING == current :latest")
+        # And the pinned ref is still the running digest.
+        assert_equal "ghcr.io/copilotkit/x@sha256:cafebabe",
+            cmd.instance_variable_get(:@promote_refs)["x"]
+    end
+
+    # =================== CASE-B: running != :latest -> warning ===============
+
+    def test_drift_warning_when_running_differs_from_latest
+        cmd = make_cmd(
+            staging_services: [STG.call("ms-agent-dotnet")],
+            prod_services:    [{ "name" => "ms-agent-dotnet", "service_id" => "svc-prod-d",
+                                 "image" => "ghcr.io/copilotkit/ms-agent-dotnet@sha256:OLD", "env_keys" => [] }],
+            current_latest:   "sha256:261ccdef3f9a",  # drifted :latest
+            running_meta:     { "image" => "ghcr.io/copilotkit/ms-agent-dotnet:latest",
+                                "imageDigest" => "sha256:f9454e79fbf5" }, # RUNNING
+        )
+        # Run P1 (populates @promote_refs + @staging_drift) then emit the warning.
+        capture_io { cmd.send(:check_p1_ghcr_digests, cmd.instance_variable_get(:@staging_snapshot)) }
+        out, _err = capture_io { cmd.send(:emit_staging_drift_warnings) }
+
+        assert_match(/STAGING DRIFT/, out, "must emit the loud drift block")
+        assert_match(/ms-agent-dotnet/, out, "must name the drifted service")
+        assert_match(/f9454e79/, out, "must name the staging RUNNING digest")
+        assert_match(/261ccdef/, out, "must name the current :latest digest")
+        # Promote still pins the RUNNING digest (drift is a warning, not a refuse).
+        assert_equal "ghcr.io/copilotkit/ms-agent-dotnet@sha256:f9454e79fbf5",
+            cmd.instance_variable_get(:@promote_refs)["ms-agent-dotnet"],
+            "promote must STILL pin the staging RUNNING digest despite drift"
+    end
+
+    # =================== CASE-F: --digest override -> NO drift signal ========
+
+    # When the operator supplies an explicit --digest for a single service, the
+    # pinned ref is the operator's CHOSEN override (not staging's running
+    # digest), so "drift from :latest" is meaningless. detect_staging_drift must
+    # skip — even when the override digest differs from current :latest — and
+    # emit_staging_drift_warnings must say nothing for it.
+    def test_no_drift_signal_when_digest_override_active
+        override_ref = "ghcr.io/copilotkit/x@sha256:0verr1de00"
+        cmd = make_cmd(
+            staging_services: [STG.call("x")],
+            current_latest:   "sha256:261ccdef",  # differs from the override digest
+            running_meta:     { "image" => "ghcr.io/copilotkit/x:latest",
+                                "imageDigest" => "sha256:f9454e79" },
+        )
+        # Operator pinned an explicit digest for this single service (mirrors the
+        # `run`-time options after `promote x --digest <ref>`).
+        cmd.options[:service] = "x"
+        cmd.options[:digest]  = override_ref
+
+        capture_io { cmd.send(:check_p1_ghcr_digests, cmd.instance_variable_get(:@staging_snapshot)) }
+
+        # No drift entry recorded — the override is a deliberate choice, not drift.
+        assert_empty Array(cmd.instance_variable_get(:@staging_drift)),
+            "a --digest override must NOT be recorded as staging drift"
+
+        # And the loud warning emits nothing for it.
+        out, _ = capture_io { cmd.send(:emit_staging_drift_warnings) }
+        refute_match(/STAGING DRIFT/, out,
+            "a --digest override must not produce a (spurious) drift warning")
+        refute_match(/STAGING_DRIFT_MARKER/, out,
+            "a --digest override must not emit a drift marker")
+
+        # The override digest is what got pinned (verbatim).
+        assert_equal override_ref, cmd.instance_variable_get(:@promote_refs)["x"],
+            "promote must pin the operator's chosen override digest"
+    end
+
+    # =================== CASE-E: GHCR resolve fails -> loud WARN, no abort ====
+
+    # GHCR fake whose tag resolution RAISES (auth/transport failure), but digest
+    # refs (@sha256:...) still resolve — so resolved_prod_image's running-digest
+    # pin succeeds while detect_staging_drift's :latest comparison blows up.
+    class FakeGHCRRaisingTag
+        def resolve_digest(ref)
+            return ref.split("@", 2).last if ref.include?("@sha256:")
+            raise StandardError, "GHCR auth failed (boom)"
+        end
+
+        def manifest_exists(_ref); :exists; end
+        def parse_image_ref(ref); Railway::GHCR.allocate.parse_image_ref(ref); end
+    end
+
+    def test_drift_check_failure_warns_loudly_and_does_not_abort
+        cmd = make_cmd(
+            staging_services: [STG.call("x")],
+            current_latest:   "sha256:unused",
+            running_meta:     { "image" => "ghcr.io/copilotkit/x:latest",
+                                "imageDigest" => "sha256:f9454e79" },
+        )
+        cmd.instance_variable_set(:@ghcr, FakeGHCRRaisingTag.new)
+
+        out, err = capture_io do
+            cmd.send(:check_p1_ghcr_digests, cmd.instance_variable_get(:@staging_snapshot))
+        end
+
+        # Fail loud: the drift-check failure must surface, with the error message.
+        assert_match(/WARN: staging drift check failed for x/, out + err,
+            "must emit a loud WARN when the GHCR :latest resolve fails")
+        assert_match(/GHCR auth failed \(boom\)/, out + err,
+            "WARN must carry the underlying error message")
+
+        # Promote PROCEEDS: the running-digest pin still happened (no abort).
+        assert_equal "ghcr.io/copilotkit/x@sha256:f9454e79",
+            cmd.instance_variable_get(:@promote_refs)["x"],
+            "promote must STILL pin the running digest despite the drift-check failure"
+
+        # No drift entry recorded (the check could not be performed).
+        assert_empty Array(cmd.instance_variable_get(:@staging_drift)),
+            "a check FAILURE must not be recorded as a drift"
+        out2, _ = capture_io { cmd.send(:emit_staging_drift_warnings) }
+        refute_match(/STAGING DRIFT/, out2,
+            "a check failure must not produce a (false) drift warning")
+    end
+
+    # =================== CASE-B (machine marker for Slack payload) ===========
+
+    def test_drift_emits_machine_readable_marker
+        # The bordered block is for humans; promote-fleet.sh greps the
+        # STAGING_DRIFT_MARKER: line and aggregates it into the Slack payload.
+        cmd = make_cmd(
+            staging_services: [STG.call("x")],
+            current_latest:   "sha256:aaaa1111",
+            running_meta:     { "image" => "ghcr.io/copilotkit/x:latest", "imageDigest" => "sha256:bbbb2222" },
+        )
+        capture_io { cmd.send(:check_p1_ghcr_digests, cmd.instance_variable_get(:@staging_snapshot)) }
+        out, _ = capture_io { cmd.send(:emit_staging_drift_warnings) }
+        assert_match(/^STAGING_DRIFT_MARKER: /, out, "must emit a machine-readable drift marker line")
+        assert_match(/running=bbbb2222/, out, "marker must carry the RUNNING digest")
+        assert_match(/latest=aaaa1111/, out, "marker must carry the :latest digest")
+    end
+end

showcase/bin/spec/test_promote_u5_env_serviceref.rb

@@ -0,0 +1,198 @@
+# frozen_string_literal: true
+
+require_relative "spec_helper"
+
+# U5 (spec §5) — Ruby env + service-ref + resource preflight (Stage 2).
+#
+#   §5.2 service-ref assertion (REFUSE): each promote target carrying
+#        `serviceRefs` must have its prod env var (e.g. OPENAI_BASE_URL) point at
+#        the env-LOCAL target host (prod->prod aimock). A prod ref pointing at
+#        the STAGING target host => REFUSE (the ms-agent-dotnet class). ASSERT,
+#        never copy.
+#   §5.3.1 replicate-class env-write (NEW capability): for declared replicate
+#        keys, copy staging->prod value via the variable*Upsert family, verified
+#        by re-read. Default replicate set is EMPTY (opt-in per service); the
+#        MECHANISM + assertion path lands even with no opt-in keys.
+#   §5.3.2/5.3.3 assert prod-specific keys present + prod-shaped (NEVER copy);
+#        assert secret KEY presence (never value).
+#   §5.4 concurrency DETECT-and-WARN: concurrency env vars (e.g.
+#        BROWSER_POOL_SIZE) join the WARN-class divergence set
+#        (region/replicas/restartPolicy already WARN). DETECT ONLY — never set.
+#        (A limitOverride CPU/memory-cap WARN was dropped: Railway exposes no
+#        readable limit field on ServiceInstance, so it was unimplementable
+#        dead code — see test_snapshot_graphql.rb regression guard.)
+#
+# RISK R-A (HIGH — env clobber): a prod-specific key (e.g. a domain like
+# NEXT_PUBLIC_POCKETBASE_URL) must be ASSERTED, NEVER written/copied. The
+# fourth test below proves no upsert is issued for a prod-specific mismatch.
+class PromoteU5EnvServiceRefTest < Minitest::Test
+    # A fake gql that records every mutation issued and answers variable reads
+    # from an in-memory per-(serviceId,envId) value map. Tests seed values and
+    # assert on @mutations to prove "asserted, never copied".
+    class FakeGql
+        attr_reader :mutations
+
+        # values: { [service_id, env_id] => { "KEY" => "value", ... } }
+        def initialize(values: {})
+            @values = values
+            @mutations = []
+        end
+
+        def query(query, variables = {})
+            if query.include?("variableUpsert") || query.include?("variableCollectionUpsert")
+                @mutations << { kind: :upsert, variables: variables }
+                sid = variables[:serviceId] || variables["serviceId"]
+                eid = variables[:envId] || variables["environmentId"] || variables["envId"]
+                name = variables[:name] || variables["name"]
+                val  = variables[:value] || variables["value"]
+                (@values[[sid, eid]] ||= {})[name] = val if name
+                return { "variableUpsert" => true }
+            end
+
+            if query.include?("query EnvServiceVariables")
+                sid = variables[:serviceId] || variables["serviceId"]
+                eid = variables[:envId] || variables["environmentId"] || variables["envId"]
+                # Railway returns a flat JSON scalar map { "NAME" => "value" }.
+                return { "variables" => (@values[[sid, eid]] || {}) }
+            end
+
+            @mutations << { kind: :other, query: query, variables: variables }
+            {}
+        end
+    end
+
+    def cmd
+        c = Railway::PromoteCommand.new(["--non-interactive", "--yes"])
+        c.parser.parse!(c.argv)
+        c
+    end
+
+    PROD_ENV = Railway::PRODUCTION_ENV_ID
+    STG_ENV  = Railway::STAGING_ENV_ID
+
+    # ── §5.2 service-ref assertion (REFUSE) ─────────────────────────────────
+
+    def test_serviceref_prod_pointing_at_staging_aimock_refuses
+        # showcase-ag2 carries serviceRefs:[{key:OPENAI_BASE_URL,target:aimock}].
+        # aimock prod host = showcase-aimock-production.up.railway.app.
+        c = cmd
+        c.instance_variable_set(:@gql, FakeGql.new(values: {
+            ["ag2-prod", PROD_ENV] => {
+                # WRONG: prod points at the STAGING aimock host.
+                "OPENAI_BASE_URL" => "https://aimock-staging.up.railway.app",
+            },
+        }))
+        staging = { "services" => [{ "name" => "showcase-ag2", "service_id" => "ag2-stg" }] }
+        prod    = { "services" => [{ "name" => "showcase-ag2", "service_id" => "ag2-prod" }] }
+        findings = c.check_service_refs(staging, prod)
+        assert(findings.any? { |f| f.start_with?("REFUSE") && f =~ /OPENAI_BASE_URL/ },
+               "expected REFUSE for prod serviceRef pointing at staging host, got #{findings.inspect}")
+        assert(c.instance_variable_get(:@gql).mutations.none? { |m| m[:kind] == :upsert },
+               "service-ref check must ASSERT, never upsert")
+    end
+
+    def test_serviceref_prod_pointing_at_prod_aimock_passes
+        c = cmd
+        c.instance_variable_set(:@gql, FakeGql.new(values: {
+            ["ag2-prod", PROD_ENV] => {
+                "OPENAI_BASE_URL" => "https://showcase-aimock-production.up.railway.app",
+            },
+        }))
+        staging = { "services" => [{ "name" => "showcase-ag2", "service_id" => "ag2-stg" }] }
+        prod    = { "services" => [{ "name" => "showcase-ag2", "service_id" => "ag2-prod" }] }
+        findings = c.check_service_refs(staging, prod)
+        assert_empty findings.select { |f| f.start_with?("REFUSE") },
+                     "prod->prod aimock ref must not REFUSE, got #{findings.inspect}"
+    end
+
+    # ── §5.3.1 replicate-class env-write (NEW capability) ───────────────────
+
+    def test_replicate_key_value_diff_upserts_and_reread_green
+        # Inject a replicate set (default is EMPTY) so the mechanism is exercised.
+        c = cmd
+        fake = FakeGql.new(values: {
+            ["x-stg", STG_ENV]  => { "FEATURE_FLAG" => "on" },
+            ["x-prod", PROD_ENV] => { "FEATURE_FLAG" => "off" },
+        })
+        c.instance_variable_set(:@gql, fake)
+        staging = { "services" => [{ "name" => "x", "service_id" => "x-stg" }] }
+        prod    = { "services" => [{ "name" => "x", "service_id" => "x-prod" }] }
+        findings = c.replicate_env_keys(staging, prod, replicate_keys: %w[FEATURE_FLAG])
+        # mechanism issued the upsert to prod with staging's value
+        upserts = fake.mutations.select { |m| m[:kind] == :upsert }
+        assert_equal 1, upserts.size, "expected exactly one upsert, got #{fake.mutations.inspect}"
+        up = upserts.first[:variables]
+        assert_equal "FEATURE_FLAG", (up[:name] || up["name"])
+        assert_equal "on", (up[:value] || up["value"])
+        assert_equal PROD_ENV, (up[:envId] || up["environmentId"] || up["envId"])
+        # re-read confirms prod now carries staging's value (verified-by-re-read)
+        assert(findings.any? { |f| f =~ /replicat/i && f =~ /FEATURE_FLAG/ },
+               "expected a replicate report line, got #{findings.inspect}")
+        refute(findings.any? { |f| f.start_with?("REFUSE") }, "replicate must not REFUSE on success")
+    end
+
+    def test_default_replicate_set_is_empty_no_writes
+        c = cmd
+        fake = FakeGql.new(values: {
+            ["x-stg", STG_ENV]  => { "FEATURE_FLAG" => "on" },
+            ["x-prod", PROD_ENV] => { "FEATURE_FLAG" => "off" },
+        })
+        c.instance_variable_set(:@gql, fake)
+        staging = { "services" => [{ "name" => "x", "service_id" => "x-stg" }] }
+        prod    = { "services" => [{ "name" => "x", "service_id" => "x-prod" }] }
+        # default (no replicate_keys argument) => empty set => no writes
+        c.replicate_env_keys(staging, prod)
+        assert(fake.mutations.none? { |m| m[:kind] == :upsert },
+               "default replicate set must be EMPTY (no upserts), got #{fake.mutations.inspect}")
+    end
+
+    # ── R-A (HIGH): prod-specific key ASSERTED, NEVER written ───────────────
+
+    def test_prod_specific_key_mismatch_refuses_and_never_writes
+        c = cmd
+        fake = FakeGql.new(values: {
+            ["dash-stg", STG_ENV]  => { "NEXT_PUBLIC_POCKETBASE_URL" => "https://pb.staging.copilotkit.ai" },
+            # prod MISSING the prod-specific key entirely => mismatch.
+            ["dash-prod", PROD_ENV] => {},
+        })
+        c.instance_variable_set(:@gql, fake)
+        staging = { "services" => [{ "name" => "dashboard", "service_id" => "dash-stg" }] }
+        prod    = { "services" => [{ "name" => "dashboard", "service_id" => "dash-prod" }] }
+        findings = c.assert_prod_specific_keys(staging, prod, prod_specific_keys: %w[NEXT_PUBLIC_POCKETBASE_URL])
+        assert(findings.any? { |f| f.start_with?("REFUSE") && f =~ /NEXT_PUBLIC_POCKETBASE_URL/ },
+               "expected REFUSE for missing prod-specific key, got #{findings.inspect}")
+        # CRITICAL R-A: no upsert/write issued for the prod-specific key.
+        assert(fake.mutations.none? { |m| m[:kind] == :upsert },
+               "prod-specific key must be ASSERTED, NEVER copied/written, got #{fake.mutations.inspect}")
+    end
+
+    # ── §5.4 resource/concurrency DETECT-and-WARN (never REFUSE) ────────────
+
+    # NOTE: a limitOverride (CPU/memory cap) divergence WARN was originally
+    # specified for check_resource_divergence, but Railway's GraphQL schema
+    # exposes no readable limit field on ServiceInstance (verified by
+    # introspection — the only related surface is the write-only
+    # serviceInstanceLimitsUpdate mutation). build_snapshot therefore cannot
+    # capture the cap, so the WARN was dead code and has been dropped. See
+    # test_snapshot_graphql.rb#test_limit_override_warn_is_unimplementable_via_real_snapshot
+    # for the regression guard proving it through the real snapshot path.
+
+    def test_concurrency_env_divergence_is_advisory_not_refuses
+        # Concurrency knobs (BROWSER_POOL_SIZE) are a RESOURCE/scaling signal,
+        # not a functional contract — divergence is ADVISORY (report-only,
+        # never blocks) per the 2026-06-22 prod↔staging comparison policy.
+        c = cmd
+        staging = { "services" => [svc("x", "x-stg", "env_keys" => %w[BROWSER_POOL_SIZE])] }
+        prod    = { "services" => [svc("x", "x-prod", "env_keys" => [])] }
+        findings = c.check_resource_divergence(staging, prod)
+        assert(findings.any? { |f| f.start_with?("ADVISORY") && f =~ /BROWSER_POOL_SIZE/ },
+               "expected ADVISORY for concurrency env divergence, got #{findings.inspect}")
+        assert(findings.none? { |f| f.start_with?("REFUSE") || f.start_with?("WARN") })
+    end
+
+    private
+
+    def svc(name, sid, over = {})
+        { "name" => name, "service_id" => sid }.merge(over)
+    end
+end

showcase/bin/spec/test_reconcile_prod.rb

@@ -0,0 +1,206 @@
+# frozen_string_literal: true
+
+require_relative "spec_helper"
+
+# Covers `bin/railway reconcile-prod` — the prod-vs-staging drift comparator
+# (Lever 1 of the promote-reliability hardening plan).
+#
+# Contract: for every prod-eligible (`probe.prod == true`) service, compare the
+# prod SERVING digest (the immutable `@sha256:` prod is pinned to) against the
+# staging RUNNING digest (staging's latest SUCCESS deployment's
+# meta.imageDigest — the same source PromoteCommand#staging_running_digest
+# reads). Classify each service:
+#
+#   green  — prod digest == staging running digest (in sync).
+#   stale  — prod digest != staging running digest AND staging IS resolvable
+#            (prod has drifted behind a green staging — the thing we alert on).
+#   gray   — staging running digest not resolvable (no SUCCESS deploy / no
+#            imageDigest) — informational, NOT stale (we can't prove drift).
+#
+# Exit code contract (the whole point of the gate):
+#   exit 0  — no `stale` services (all green, or only green+gray).
+#   exit 1  — at least one `stale` service (prod drifted behind green staging).
+#
+# Read-only: NO promotes / mutations. --json emits machine output.
+#
+# The test injects fakes the same way the promote suite does
+# (test_promote_staging_running_digest.rb): instance_variable_set the prod +
+# staging snapshots and a fake that resolves the staging running digest.
+class ReconcileProdTest < Minitest::Test
+    # Build a ReconcileProdCommand with injected prod + staging snapshots and a
+    # per-service staging-running-digest map (sid => "sha256:..." or nil).
+    #
+    # `eligible` is the list of SSOT-eligible service descriptors the command
+    # iterates: each is { "name" =>, "service_id" => }. We inject it directly so
+    # the test does not depend on the real generated SSOT JSON.
+    def make_cmd(eligible:, prod_services:, running_by_sid:, argv: [])
+        cmd = Railway::ReconcileProdCommand.new(argv)
+        cmd.parser.parse!(cmd.argv)
+        # Inject the prod snapshot the comparator reads (LintProd path).
+        cmd.define_singleton_method(:build_prod_snapshot) do
+            { "services" => prod_services }
+        end
+        # Inject the prod-eligible service set (normally derived from the SSOT
+        # generated.json by probe.prod == true).
+        cmd.define_singleton_method(:eligible_services) { eligible }
+        # Inject the staging running digest lookup (normally
+        # PromoteCommand#staging_running_digest reading Railway deployments).
+        cmd.define_singleton_method(:staging_running_digest_for) do |svc|
+            running_by_sid[svc["service_id"]]
+        end
+        cmd
+    end
+
+    PROD = lambda do |name, sid, digest|
+        # A prod snapshot service is pinned to an immutable digest:
+        # image = ghcr.io/org/name@sha256:..., digest = sha256:...
+        {
+            "name"       => name,
+            "service_id" => sid,
+            "image"      => "ghcr.io/copilotkit/#{name}@#{digest}",
+            "digest"     => digest,
+        }
+    end
+
+    ELIG = lambda do |name, sid|
+        { "name" => name, "service_id" => sid }
+    end
+
+    # ===================== RED-anchor: STALE => exit 1 ========================
+    # Prod is pinned to digest A; staging is RUNNING green digest B. prod !=
+    # staging-green => STALE. The comparator MUST classify it stale and exit 1.
+    def test_stale_when_prod_differs_from_green_staging
+        cmd = make_cmd(
+            eligible:       [ELIG.call("shell", "sid-shell")],
+            prod_services:  [PROD.call("shell", "sid-shell", "sha256:aaaa1111")],
+            running_by_sid: { "sid-shell" => "sha256:bbbb2222" }, # green staging, DIFFERENT
+        )
+        rows = cmd.classify_all
+        row = rows.find { |r| r["name"] == "shell" }
+        assert_equal "stale", row["status"],
+            "prod digest != staging green digest must classify STALE"
+        assert_equal 1, cmd.run_classification(rows),
+            "any stale service must exit non-zero (1)"
+    end
+
+    # ===================== GREEN => exit 0 ====================================
+    def test_green_when_prod_matches_staging
+        cmd = make_cmd(
+            eligible:       [ELIG.call("shell", "sid-shell"),
+                             ELIG.call("docs", "sid-docs")],
+            prod_services:  [PROD.call("shell", "sid-shell", "sha256:aaaa1111"),
+                             PROD.call("docs", "sid-docs", "sha256:cccc3333")],
+            running_by_sid: { "sid-shell" => "sha256:aaaa1111",
+                              "sid-docs"  => "sha256:cccc3333" },
+        )
+        rows = cmd.classify_all
+        assert(rows.all? { |r| r["status"] == "green" },
+            "all matching => all green, got #{rows.map { |r| r['status'] }.inspect}")
+        assert_equal 0, cmd.run_classification(rows),
+            "no stale service => exit 0"
+    end
+
+    # ===================== GRAY (staging not green) => exit 0 =================
+    # Staging has no resolvable running digest (nil). That is NOT drift we can
+    # prove — classify gray (informational), NOT stale. Must NOT red the run.
+    def test_gray_when_staging_not_resolvable
+        cmd = make_cmd(
+            eligible:       [ELIG.call("shell", "sid-shell")],
+            prod_services:  [PROD.call("shell", "sid-shell", "sha256:aaaa1111")],
+            running_by_sid: { "sid-shell" => nil }, # staging not green/resolvable
+        )
+        rows = cmd.classify_all
+        row = rows.find { |r| r["name"] == "shell" }
+        assert_equal "gray", row["status"],
+            "unresolvable staging digest must be gray, not stale"
+        assert_equal 0, cmd.run_classification(rows),
+            "gray (not stale) must NOT red the run"
+    end
+
+    # ===================== mixed: one stale among green/gray => exit 1 ========
+    def test_mixed_with_one_stale_exits_nonzero
+        cmd = make_cmd(
+            eligible:       [ELIG.call("shell", "sid-shell"),
+                             ELIG.call("docs", "sid-docs"),
+                             ELIG.call("dojo", "sid-dojo")],
+            prod_services:  [PROD.call("shell", "sid-shell", "sha256:aaaa1111"),
+                             PROD.call("docs", "sid-docs", "sha256:cccc3333"),
+                             PROD.call("dojo", "sid-dojo", "sha256:dddd4444")],
+            running_by_sid: { "sid-shell" => "sha256:aaaa1111",   # green
+                              "sid-docs"  => "sha256:9999ffff",   # STALE
+                              "sid-dojo"  => nil },                # gray
+        )
+        rows = cmd.classify_all
+        by_name = rows.each_with_object({}) { |r, h| h[r["name"]] = r["status"] }
+        assert_equal "green", by_name["shell"]
+        assert_equal "stale", by_name["docs"]
+        assert_equal "gray",  by_name["dojo"]
+        assert_equal 1, cmd.run_classification(rows),
+            "one stale among green/gray => exit 1"
+    end
+
+    # ===================== --json machine output =============================
+    def test_json_output_shape
+        cmd = make_cmd(
+            eligible:       [ELIG.call("shell", "sid-shell")],
+            prod_services:  [PROD.call("shell", "sid-shell", "sha256:aaaa1111")],
+            running_by_sid: { "sid-shell" => "sha256:bbbb2222" },
+            argv:           ["--json"],
+        )
+        out, = capture_io { cmd.run }
+        payload = JSON.parse(out)
+        assert_equal 1, payload["stale"], "stale count surfaced in JSON"
+        svc = payload["services"].find { |s| s["name"] == "shell" }
+        assert_equal "stale", svc["status"]
+        assert_equal "sha256:aaaa1111", svc["prod"]
+        assert_equal "sha256:bbbb2222", svc["staging"]
+    end
+
+    # ===================== run() end-to-end exit code =========================
+    def test_run_exits_nonzero_on_stale
+        cmd = make_cmd(
+            eligible:       [ELIG.call("shell", "sid-shell")],
+            prod_services:  [PROD.call("shell", "sid-shell", "sha256:aaaa1111")],
+            running_by_sid: { "sid-shell" => "sha256:bbbb2222" },
+        )
+        rc = nil
+        capture_io { rc = cmd.run }
+        assert_equal 1, rc, "run() must exit 1 when a service is stale"
+    end
+
+    def test_run_exits_zero_when_all_green
+        cmd = make_cmd(
+            eligible:       [ELIG.call("shell", "sid-shell")],
+            prod_services:  [PROD.call("shell", "sid-shell", "sha256:aaaa1111")],
+            running_by_sid: { "sid-shell" => "sha256:aaaa1111" },
+        )
+        rc = nil
+        capture_io { rc = cmd.run }
+        assert_equal 0, rc, "run() must exit 0 when all services green"
+    end
+
+    # ===================== prod service missing from snapshot =================
+    # A prod-eligible service that has NO prod snapshot entry (never deployed to
+    # prod) has no prod digest to compare. It must NOT be classified stale
+    # (we can't prove drift) — classify gray (informational).
+    def test_missing_prod_service_is_gray_not_stale
+        cmd = make_cmd(
+            eligible:       [ELIG.call("newsvc", "sid-new")],
+            prod_services:  [], # newsvc not in prod yet
+            running_by_sid: { "sid-new" => "sha256:bbbb2222" },
+        )
+        rows = cmd.classify_all
+        row = rows.find { |r| r["name"] == "newsvc" }
+        assert_equal "gray", row["status"],
+            "prod-eligible service absent from prod snapshot must be gray, not stale"
+        assert_equal 0, cmd.run_classification(rows)
+    end
+
+    # The dispatcher must register the subcommand.
+    def test_subcommand_registered
+        assert Railway::SUBCOMMANDS.key?("reconcile-prod"),
+            "reconcile-prod must be registered in the dispatcher"
+        assert_equal Railway::ReconcileProdCommand,
+            Railway::SUBCOMMANDS["reconcile-prod"]
+    end
+end

showcase/bin/spec/test_rollback_commit_injection.rb

@@ -0,0 +1,350 @@
+# frozen_string_literal: true
+
+require_relative "spec_helper"
+
+# RollbackCommitCommand takes attacker-influenced operator input (--sha, --env)
+# and used to interpolate both into shell strings (`git ls-tree ... #{sha}` and
+# `git show #{sha}:#{path}`). A malformed --sha like "abc; touch /tmp/pwn"
+# would be parsed by the shell. These tests pin the invariant:
+#
+#   1. Happy path: a valid hex --sha + known --env locates the snapshot via
+#      IO.popen (argv form, no shell) and hands off to RestoreCommand.
+#   2. Malformed --sha is rejected BEFORE any subprocess is spawned.
+#   3. Unknown --env is rejected BEFORE any subprocess is spawned.
+class RollbackCommitInjectionTest < Minitest::Test
+    # Capture every IO.popen invocation issued during the test so we can both
+    # stub out git AND assert that injection attempts never reach a subprocess.
+    #
+    # Hardening notes:
+    #   * `expected_subcmds` is the set of git subcommands the test
+    #     explicitly configured (via PopenSpy.responses or PopenSpy.exits).
+    #     Any `git <subcmd>` call NOT in that set raises UnexpectedPopen
+    #     instead of silently returning nil — so a future code path that
+    #     starts shelling out to e.g. `git rev-parse` is caught loudly
+    #     at the boundary rather than producing a confusing downstream
+    #     YAML.safe_load failure on an empty string.
+    #   * `exits` yields an honest `$?.exitstatus` for the configured code
+    #     by shelling to a tiny `ruby -e "exit N"` — `true`/`false` only
+    #     produce 0/1 and so failed to surface bugs sensitive to the
+    #     specific code (e.g. git's 128 for "bad object").
+    class UnexpectedPopen < StandardError; end
+
+    module PopenSpy
+        @calls = []
+        @responses = {}
+        @exits = {}
+
+        class << self
+            attr_reader :calls
+            attr_accessor :responses, :exits
+
+            def reset!
+                @calls = []
+                @responses = {}
+                @exits = {}
+            end
+
+            def record(args)
+                @calls << args
+            end
+
+            # Subcommands the current test has explicitly accounted for.
+            # A response of "" or an exit of 0 counts as an explicit
+            # opt-in: the test author has thought about that subcmd.
+            def expected_subcmds
+                (@responses.keys + @exits.keys).uniq
+            end
+
+            # Set $?.exitstatus to `code` by running a real, short-lived
+            # subprocess that exits with that code. Using `system("true")`
+            # / `system("false")` only ever yields 0 or 1 — too lossy for
+            # bug-fidelity assertions (e.g. git's exit 128 on bad object).
+            def stamp_exit_status!(code)
+                # `ruby -e "exit N"` is portable across CI runners and
+                # avoids relying on shell builtins. Suppress stderr just
+                # in case (shouldn't print anything, but defensive).
+                # Fail loud if the spawn itself fails: `system` returns
+                # `nil` on exec failure (command-not-found / interpreter
+                # unresolvable), in which case `$?` reflects a
+                # ~127 exec failure rather than the configured code and
+                # silently corrupts the spy contract. `false` (the
+                # child ran and exited non-zero with the configured
+                # code) is the happy path here and must NOT raise.
+                result = system(RbConfig.ruby, "-e", "exit #{Integer(code)}", out: File::NULL, err: File::NULL)
+                raise "stamp_exit_status! failed to spawn #{RbConfig.ruby}" if result.nil?
+            end
+        end
+    end
+
+    # A stand-in for RestoreCommand.run so we can detect successful hand-off
+    # without touching Railway's GraphQL API.
+    class FakeRestore
+        @last_argv = nil
+        @ran = false
+        class << self
+            attr_accessor :last_argv, :ran
+            def reset!
+                @last_argv = nil
+                @ran = false
+            end
+        end
+
+        def initialize(argv)
+            @argv = argv
+        end
+
+        def run
+            FakeRestore.last_argv = @argv
+            FakeRestore.ran = true
+        end
+    end
+
+    def setup
+        PopenSpy.reset!
+        FakeRestore.reset!
+
+        # Monkey-patch IO.popen ONLY for the duration of each test.
+        # The real RollbackCommitCommand uses the argv-array form:
+        #   IO.popen(["git", "ls-tree", ...], err: [:child, :out]) { |io| io.read }
+        # We intercept that and return canned output keyed by the first non-git
+        # subcommand ("ls-tree" or "show").
+        #
+        # Hardening: any `git <subcmd>` NOT in PopenSpy.expected_subcmds
+        # raises UnexpectedPopen. That makes "a new subprocess shows up
+        # in the code path" a loud failure instead of a silent nil read.
+        # Non-git popens fall through to the real implementation (we
+        # want to keep e.g. minitest's own bookkeeping intact, though
+        # nothing currently relies on it).
+        @original_popen = IO.method(:popen)
+        spy = PopenSpy
+        IO.singleton_class.send(:define_method, :popen) do |*args, **kwargs, &block|
+            spy.record(args)
+            argv = args.first
+            if argv.is_a?(Array) && argv.first == "git"
+                subcmd = argv[1]
+                unless spy.expected_subcmds.include?(subcmd)
+                    raise UnexpectedPopen,
+                        "PopenSpy received an UNEXPECTED `git #{subcmd}` invocation. " \
+                        "The test only configured: #{spy.expected_subcmds.inspect}. " \
+                        "If this is a legitimate new subprocess, opt in by setting " \
+                        "PopenSpy.responses[#{subcmd.inspect}] (and/or exits) in " \
+                        "the test setup. Full argv: #{argv.inspect}"
+                end
+                response = spy.responses[subcmd] || ""
+                # Stamp $?.exitstatus with the configured code (default 0).
+                # Critical for tests asserting on the *specific* exit code
+                # (e.g. git's 128 for "fatal: bad object") rather than a
+                # generic 0/1 success/fail.
+                exit_code = spy.exits[subcmd] || 0
+                spy.stamp_exit_status!(exit_code)
+                # Mimic the block form used by the production code.
+                if block
+                    require "stringio"
+                    block.call(StringIO.new(response))
+                else
+                    response
+                end
+            else
+                # Defer to the real implementation for anything we don't expect.
+                spy.instance_variable_get(:@original_popen)&.call(*args, **kwargs, &block)
+            end
+        end
+
+        # Stub RestoreCommand so the integration boundary never tries to hit
+        # Railway. We swap the constant and restore in teardown.
+        @original_restore = Railway::RestoreCommand
+        Railway.send(:remove_const, :RestoreCommand)
+        Railway.const_set(:RestoreCommand, FakeRestore)
+    end
+
+    def teardown
+        # Restore IO.popen.
+        original = @original_popen
+        IO.singleton_class.send(:define_method, :popen) do |*args, **kwargs, &block|
+            original.call(*args, **kwargs, &block)
+        end
+        # Restore RestoreCommand.
+        Railway.send(:remove_const, :RestoreCommand)
+        Railway.const_set(:RestoreCommand, @original_restore)
+    end
+
+    # ── Happy path ─────────────────────────────────────────────────────────
+
+    def test_valid_sha_and_env_invokes_git_via_argv_and_hands_off_to_restore
+        PopenSpy.responses["ls-tree"] = "showcase/.railway-snapshots/20260101T000000Z-staging.yaml\n"
+        PopenSpy.responses["show"] = "schema_version: 1\nservices: []\n"
+
+        cmd = Railway::RollbackCommitCommand.new(
+            ["--env", "staging", "--sha", "abc1234", "--yes", "--non-interactive", "--dry-run"]
+        )
+        cmd.run
+
+        # 1. RestoreCommand got the expected argv (proves hand-off happened).
+        assert FakeRestore.ran, "expected RestoreCommand to be invoked on happy path"
+        assert_includes FakeRestore.last_argv, "--env"
+        assert_includes FakeRestore.last_argv, "staging"
+        assert_includes FakeRestore.last_argv, "--snapshot"
+        assert_includes FakeRestore.last_argv, "--dry-run"
+
+        # 2. Every git invocation used the argv-array form (no shell string).
+        git_calls = PopenSpy.calls.map(&:first).select { |a| a.is_a?(Array) && a.first == "git" }
+        refute_empty git_calls, "expected at least one git subprocess via IO.popen argv-array"
+        git_calls.each do |argv|
+            assert argv.is_a?(Array), "git call must be an argv array, got #{argv.inspect}"
+            assert argv.all? { |a| a.is_a?(String) }, "all argv elements must be strings"
+        end
+
+        # 3. The sha appears as a literal argv element somewhere (not glued).
+        ls_call = git_calls.find { |a| a[1] == "ls-tree" }
+        assert ls_call, "expected a `git ls-tree` invocation"
+        assert_includes ls_call, "abc1234"
+
+        show_call = git_calls.find { |a| a[1] == "show" }
+        assert show_call, "expected a `git show` invocation"
+        # `git show` takes sha:path as a single token by design; ensure it's
+        # the FULL token (not concatenated with anything else like `; rm -rf`).
+        assert_includes show_call, "abc1234:showcase/.railway-snapshots/20260101T000000Z-staging.yaml"
+    end
+
+    # ── Rejection: malformed --sha ─────────────────────────────────────────
+
+    def test_malformed_sha_is_rejected_before_any_subprocess
+        malicious = "abc; touch /tmp/pwn"
+        cmd = Railway::RollbackCommitCommand.new(["--env", "staging", "--sha", malicious])
+
+        exited = assert_raises(SystemExit) { cmd.run }
+        refute_equal 0, exited.status, "rejection must exit nonzero"
+
+        # No subprocess of any kind should have been launched.
+        assert_empty PopenSpy.calls.select { |c| c.first.is_a?(Array) && c.first.first == "git" },
+            "no git subprocess should be spawned for malformed --sha; got #{PopenSpy.calls.inspect}"
+        refute FakeRestore.ran, "RestoreCommand must not run when --sha is rejected"
+    end
+
+    def test_sha_with_uppercase_is_rejected
+        cmd = Railway::RollbackCommitCommand.new(["--env", "staging", "--sha", "ABC1234"])
+        exited = assert_raises(SystemExit) { cmd.run }
+        refute_equal 0, exited.status, "rejection must exit nonzero"
+        assert_empty PopenSpy.calls.select { |c| c.first.is_a?(Array) && c.first.first == "git" },
+            "no git subprocess should be spawned for uppercase --sha"
+        refute FakeRestore.ran, "RestoreCommand must not run when --sha is rejected"
+    end
+
+    def test_sha_too_short_is_rejected
+        cmd = Railway::RollbackCommitCommand.new(["--env", "staging", "--sha", "abc12"])
+        exited = assert_raises(SystemExit) { cmd.run }
+        refute_equal 0, exited.status, "rejection must exit nonzero"
+        assert_empty PopenSpy.calls.select { |c| c.first.is_a?(Array) && c.first.first == "git" },
+            "no git subprocess should be spawned for too-short --sha"
+        refute FakeRestore.ran, "RestoreCommand must not run when --sha is rejected"
+    end
+
+    # ── Subprocess-failure gating ──────────────────────────────────────────
+
+    # Regression: a failed `git show` previously slipped past the nil/empty
+    # guard because `err: [:child, :out]` merges stderr into stdout, so a
+    # non-zero exit produces a non-empty `yaml` containing git's error text
+    # which then flowed into YAML.safe_load. The fix gates on $?.exitstatus.
+    def test_git_show_nonzero_exit_dies_before_yaml_parse
+        PopenSpy.responses["ls-tree"] = "showcase/.railway-snapshots/20260101T000000Z-staging.yaml\n"
+        # Simulate a corrupt/missing blob: git prints an error to stderr
+        # (merged into stdout via err: [:child, :out]) and exits non-zero.
+        PopenSpy.responses["show"] = "fatal: bad object abc1234:showcase/.railway-snapshots/20260101T000000Z-staging.yaml\n"
+        PopenSpy.exits["show"] = 128
+
+        cmd = Railway::RollbackCommitCommand.new(
+            ["--env", "staging", "--sha", "abc1234", "--yes", "--non-interactive", "--dry-run"]
+        )
+
+        exited = assert_raises(SystemExit) { cmd.run }
+        refute_equal 0, exited.status, "git-show failure must exit nonzero"
+        refute FakeRestore.ran, "RestoreCommand must not run when git show fails"
+    end
+
+    def test_empty_snapshot_listing_dies_before_git_show
+        # ls-tree succeeds but returns no entries → die before any git show.
+        PopenSpy.responses["ls-tree"] = ""
+
+        cmd = Railway::RollbackCommitCommand.new(
+            ["--env", "staging", "--sha", "abc1234", "--yes", "--non-interactive", "--dry-run"]
+        )
+
+        exited = assert_raises(SystemExit) { cmd.run }
+        refute_equal 0, exited.status, "no-snapshot must exit nonzero"
+        refute FakeRestore.ran, "RestoreCommand must not run when no snapshot found"
+
+        # No `git show` should have been spawned.
+        show_calls = PopenSpy.calls.select do |c|
+            c.first.is_a?(Array) && c.first.first == "git" && c.first[1] == "show"
+        end
+        assert_empty show_calls, "no git show should run when ls-tree returns empty"
+    end
+
+    # ── Rejection: unknown --env ───────────────────────────────────────────
+
+    def test_unknown_env_is_rejected_before_any_subprocess
+        cmd = Railway::RollbackCommitCommand.new(["--env", "evil; rm -rf ~", "--sha", "abc1234"])
+
+        exited = assert_raises(SystemExit) { cmd.run }
+        refute_equal 0, exited.status, "rejection must exit nonzero"
+
+        assert_empty PopenSpy.calls.select { |c| c.first.is_a?(Array) && c.first.first == "git" },
+            "no git subprocess should be spawned for unknown --env"
+        refute FakeRestore.ran
+    end
+
+    # ── PopenSpy self-test: hardening guarantees ──────────────────────────
+    #
+    # These tests pin the spy's own contract so it can't silently rot.
+    # The spy is the only thing standing between a future subprocess
+    # addition and a test that "passes" with a wrong answer.
+
+    # If the production code adds a NEW git subprocess (e.g. rev-parse)
+    # without the test opting in, the spy must raise — not silently
+    # return nil/"" which would corrupt downstream assertions.
+    def test_popen_spy_raises_on_unexpected_git_subcommand
+        # Only "ls-tree" and "show" are configured here.
+        PopenSpy.responses["ls-tree"] = ""
+        PopenSpy.responses["show"] = ""
+
+        # Direct invocation simulates the "new subprocess slipped in"
+        # scenario without needing to add a real call site to railway.
+        err = assert_raises(UnexpectedPopen) do
+            IO.popen(["git", "rev-parse", "HEAD"], err: [:child, :out]) { |io| io.read }
+        end
+        assert_match(/UNEXPECTED `git rev-parse`/, err.message,
+            "spy must name the offending subcommand in its error")
+        assert_match(/ls-tree/, err.message,
+            "spy must list the configured subcommands so the operator " \
+            "can decide whether to opt the new one in")
+    end
+
+    # Exit-code fidelity: configuring `exits["show"] = 128` must yield
+    # an honest `$?.exitstatus == 128`, not a generic 1. The git-show
+    # failure-gate test depends on this fidelity to be a meaningful
+    # regression test of the gate (a gate keyed on `!= 0` would pass
+    # against a fake 1, but a gate keyed on `== 128` would not).
+    def test_popen_spy_stamps_real_exit_status_for_configured_code
+        PopenSpy.responses["show"] = "fatal: whatever\n"
+        PopenSpy.exits["show"] = 128
+
+        IO.popen(["git", "show", "abc1234:foo"], err: [:child, :out]) { |io| io.read }
+        assert_equal 128, $?.exitstatus,
+            "PopenSpy.stamp_exit_status! must reflect the *configured* " \
+            "exit code in $?.exitstatus (got #{$?.exitstatus.inspect}). " \
+            "Without this, tests asserting on specific git exit codes " \
+            "(e.g. 128 for bad object) are vacuous."
+    end
+
+    # Default behaviour: when `exits[subcmd]` is unset, the call must
+    # behave like a successful git invocation (`$?.exitstatus == 0`).
+    def test_popen_spy_defaults_to_exit_zero_when_exits_unset
+        PopenSpy.responses["ls-tree"] = "snap.yaml\n"
+
+        IO.popen(["git", "ls-tree", "abc1234"], err: [:child, :out]) { |io| io.read }
+        assert_equal 0, $?.exitstatus,
+            "default exit status for an un-configured subcmd response " \
+            "must be 0 (success), matching how real git behaves on a " \
+            "successful call"
+    end
+end

showcase/bin/spec/test_rollback_sort.rb

@@ -0,0 +1,106 @@
+# frozen_string_literal: true
+
+require_relative "spec_helper"
+
+# RollbackCommand#find_previous_deployment must pick the newest SUCCESS
+# deployment STRICTLY OLDER than the current HEAD deploy (by createdAt) — i.e.
+# the last-known-good deploy to roll back to. Railway's GraphQL `deployments`
+# connection returns nodes in an arbitrary order, so the method MUST sort by
+# createdAt descending before selecting, exactly as the sibling
+# fetch_latest_staging_deployments documents and does.
+#
+# Selection logic: sort newest-first by createdAt, drop the HEAD (index 0),
+# then take the first SUCCESS in the remainder. This is correct in BOTH
+# directions:
+#   - head=SUCCESS  -> the previous SUCCESS (one deploy back)
+#   - head=FAILED   -> the newest SUCCESS below the failed head (the
+#                      last-known-good — NOT one good deploy too far)
+#
+# Truncation hazard: the query only fetches `first: N` deployments in arbitrary
+# order, so a >N-deploy service may not contain the true previous within the
+# window. When no target is found AND the window is saturated (returned count
+# >= N), the method must die! loud rather than return nil — otherwise rollback
+# becomes a confusing no-op or rolls to the wrong place.
+class RollbackSortTest < Minitest::Test
+    # Mirror the production query's `first:` limit so the saturated-window test
+    # stays in lockstep with bin/railway.
+    WINDOW = Railway::RollbackCommand::DEPLOYMENTS_WINDOW
+
+    # Minimal fake GraphQL client: returns canned DEPLOYMENTS_QUERY edges.
+    class FakeGQL
+        def initialize(nodes)
+            @nodes = nodes
+        end
+
+        def query(_query_str, _variables = {})
+            { "deployments" => { "edges" => @nodes.map { |n| { "node" => n } } } }
+        end
+    end
+
+    def cmd_for(nodes)
+        cmd = Railway::RollbackCommand.new([])
+        cmd.instance_variable_set(:@gql, FakeGQL.new(nodes))
+        cmd
+    end
+
+    def test_head_success_picks_previous_success
+        # Edge order is deliberately scrambled (NOT chronological). By createdAt
+        # the SUCCESS deployments are, newest-first:
+        #   dep-newest (T5) > dep-prev (T3) > dep-old (T1)
+        # HEAD is dep-newest (SUCCESS), so the rollback target is the previous
+        # SUCCESS, dep-prev. A FAILED deploy at T4 must be ignored.
+        nodes = [
+            { "id" => "dep-old",    "status" => "SUCCESS", "createdAt" => "2026-01-01T00:00:00Z" },
+            { "id" => "dep-newest", "status" => "SUCCESS", "createdAt" => "2026-01-05T00:00:00Z" },
+            { "id" => "dep-failed", "status" => "FAILED",  "createdAt" => "2026-01-04T00:00:00Z" },
+            { "id" => "dep-prev",   "status" => "SUCCESS", "createdAt" => "2026-01-03T00:00:00Z" },
+        ]
+
+        result = cmd_for(nodes).find_previous_deployment("svc-1", "env-1")
+        assert_equal "dep-prev", result,
+            "expected the previous SUCCESS below the SUCCESS head (dep-prev), got #{result.inspect}"
+    end
+
+    def test_head_failed_picks_newest_success_below_head
+        # HEAD is FAILED (this is exactly when rollback is invoked). The target
+        # MUST be the newest SUCCESS strictly below the failed head — SUCCESS_A
+        # at T4 — NOT SUCCESS_B at T3 (the old `successes[1]` behavior would
+        # skip a good deploy and roll back one too far).
+        nodes = [
+            { "id" => "head-failed", "status" => "FAILED",  "createdAt" => "2026-01-05T00:00:00Z" },
+            { "id" => "success-a",   "status" => "SUCCESS", "createdAt" => "2026-01-04T00:00:00Z" },
+            { "id" => "success-b",   "status" => "SUCCESS", "createdAt" => "2026-01-03T00:00:00Z" },
+        ]
+
+        result = cmd_for(nodes).find_previous_deployment("svc-1", "env-1")
+        assert_equal "success-a", result,
+            "head=FAILED must roll back to the newest SUCCESS below it (success-a), " \
+            "not skip it to success-b; got #{result.inspect}"
+    end
+
+    def test_returns_nil_when_no_success_below_head_and_window_not_saturated
+        # Fewer than WINDOW deployments returned => genuine "no previous",
+        # returning nil is correct (the caller die!s with a clear message).
+        nodes = [
+            { "id" => "dep-only",   "status" => "SUCCESS", "createdAt" => "2026-01-05T00:00:00Z" },
+            { "id" => "dep-failed", "status" => "FAILED",  "createdAt" => "2026-01-04T00:00:00Z" },
+        ]
+        assert_nil cmd_for(nodes).find_previous_deployment("svc-1", "env-1")
+    end
+
+    def test_raises_when_window_saturated_and_no_target_found
+        # Exactly WINDOW deployments returned with no SUCCESS below the head =>
+        # the true previous may have been truncated out of the window. The
+        # method must die! (SystemExit) rather than silently return nil.
+        head = { "id" => "head", "status" => "FAILED", "createdAt" => "2026-02-#{WINDOW + 1}T00:00:00Z" }
+        rest = (1...WINDOW).map do |i|
+            { "id" => "crashed-#{i}", "status" => "CRASHED", "createdAt" => format("2026-02-%02dT00:00:00Z", i) }
+        end
+        nodes = [head] + rest
+        assert_equal WINDOW, nodes.size, "test must return exactly WINDOW deployments"
+
+        assert_raises(SystemExit) do
+            cmd_for(nodes).find_previous_deployment("svc-1", "env-1")
+        end
+    end
+end

showcase/bin/spec/test_snapshot_graphql.rb

@@ -0,0 +1,369 @@
+# frozen_string_literal: true
+
+require_relative "spec_helper"
+
+# Verifies the GraphQL queries used by SnapshotCommand match Railway's
+# public schema shape (as of 2026-05). The mocks here mirror real Railway
+# responses; any drift between this file and the live schema means the
+# tool will fail at runtime — which is exactly the bug this PR fixes.
+class SnapshotGraphqlTest < Minitest::Test
+    # Fake GraphQL client that returns canned responses keyed by query name.
+    class FakeGQL
+        def initialize(responses)
+            @responses = responses
+            @calls = []
+        end
+
+        attr_reader :calls
+
+        def query(query_str, variables = {})
+            @calls << [query_str, variables]
+            # Match on the operation name (the line after `query `) so we can
+            # serve different mocks for SERVICES_LIST_QUERY,
+            # SERVICE_INSTANCE_QUERY, ENVIRONMENT_VARIABLES_QUERY.
+            op = query_str[/query\s+(\w+)/, 1]
+            response = @responses[op] || @responses[:default]
+            raise "no fake response for op=#{op.inspect}" unless response
+            response.respond_to?(:call) ? response.call(variables) : response
+        end
+    end
+
+    def services_list_response
+        {
+            "project" => {
+                "id"   => Railway::PROJECT_ID,
+                "name" => "showcase",
+                "services" => {
+                    "edges" => [
+                        { "node" => { "id" => "svc-aimock", "name" => "aimock" } },
+                        { "node" => { "id" => "svc-shell",  "name" => "shell" } },
+                    ],
+                },
+            },
+        }
+    end
+
+    def env_vars_response_with_per_service_keys
+        {
+            "environment" => {
+                "id"   => Railway::PRODUCTION_ENV_ID,
+                "name" => "production",
+                "variables" => {
+                    "edges" => [
+                        { "node" => { "name" => "PORT",        "serviceId" => "svc-aimock", "isSealed" => false } },
+                        { "node" => { "name" => "NODE_OPTIONS","serviceId" => "svc-aimock", "isSealed" => false } },
+                        { "node" => { "name" => "API_KEY",     "serviceId" => "svc-shell",  "isSealed" => true } },
+                    ],
+                },
+            },
+        }
+    end
+
+    def service_instance_response(image:, start_cmd: nil, domains: [],
+                                  healthcheck_path: nil, region: nil,
+                                  num_replicas: nil, restart_policy: nil)
+        {
+            "serviceInstance" => {
+                "id"                 => "inst-#{image[/sha256:[a-f0-9]+/] || 'tag'}",
+                "serviceId"          => "svc-aimock",
+                "environmentId"      => Railway::PRODUCTION_ENV_ID,
+                "startCommand"       => start_cmd,
+                "healthcheckPath"    => healthcheck_path,
+                "region"             => region,
+                "numReplicas"        => num_replicas,
+                "restartPolicyType"  => restart_policy,
+                "source"             => { "image" => image, "repo" => nil },
+                "latestDeployment"   => { "id" => "dep-1", "status" => "SUCCESS" },
+                "domains" => {
+                    "customDomains"  => domains.map { |d| { "id" => "cd-#{d}", "domain" => d } },
+                    "serviceDomains" => [],
+                },
+            },
+        }
+    end
+
+    def test_limit_override_warn_is_unimplementable_via_real_snapshot
+        # Regression guard for the DROPPED limitOverride WARN. A
+        # limitOverride (CPU/memory cap) divergence WARN was originally
+        # specified for check_resource_divergence, but it was dead code: it
+        # compared snapshot["limit_override"] values that build_snapshot never
+        # captures, because Railway's GraphQL ServiceInstance type exposes NO
+        # readable limit field (verified by introspection 2026-06 — the only
+        # related surface is the write-only serviceInstanceLimitsUpdate mutation
+        # taking ServiceInstanceLimitsUpdateInput{memoryGB,vCPUs}; the
+        # ServiceInstanceLimit type is an opaque scalar with no readable fields).
+        #
+        # This guard drives a divergence through the REAL capture path
+        # (build_snapshot) — even injecting a hypothetical limit into the fake
+        # serviceInstance response — and asserts: (a) build_snapshot drops it
+        # (no limit_override key survives), and (b) check_resource_divergence
+        # emits NO limitOverride WARN. If Railway ever adds a readable limit
+        # field and someone wires it through SERVICE_INSTANCE_QUERY + the
+        # snapshot hash, this guard flips and signals the WARN can be re-added.
+        build = lambda do |limit|
+            fake = FakeGQL.new(
+                "ProjectServices" => {
+                    "project" => { "id" => Railway::PROJECT_ID, "name" => "showcase",
+                        "services" => { "edges" => [{ "node" => { "id" => "svc-x", "name" => "x" } }] } },
+                },
+                "EnvVariables" => { "environment" => { "id" => "e", "name" => "n",
+                    "variables" => { "edges" => [] } } },
+                "ServiceInstance" => service_instance_response(
+                    image: "ghcr.io/copilotkit/x@sha256:cafef00d",
+                ).tap { |r| r["serviceInstance"]["serviceLimitOverride"] = limit },
+            )
+            cmd = Railway::SnapshotCommand.new(["--env", "production", "--dry-run"])
+            cmd.instance_variable_set(:@gql, fake)
+            cmd.build_snapshot(Railway::PRODUCTION_ENV_ID)
+        end
+        staging = build.call("memoryGB" => 4.0, "vCPUs" => 2.0)
+        prod    = build.call("memoryGB" => 1.0, "vCPUs" => 1.0)
+
+        # (a) the real snapshot path captures no limit field at all.
+        staging["services"].each { |s| assert_nil s["limit_override"] }
+        prod["services"].each    { |s| assert_nil s["limit_override"] }
+
+        # (b) consequently the dropped WARN cannot (and does not) fire.
+        c = Railway::PromoteCommand.new(["--non-interactive", "--yes"])
+        c.parser.parse!(c.argv)
+        findings = c.check_resource_divergence(staging, prod)
+        assert(findings.none? { |f| f =~ /limitOverride/i },
+            "limitOverride WARN was dropped as unimplementable dead code; it must " \
+            "not reappear unless build_snapshot genuinely captures a limit field — " \
+            "got #{findings.inspect}")
+    end
+
+    def test_build_snapshot_uses_corrected_field_names_and_produces_pinned_entries
+        fake = FakeGQL.new(
+            "ProjectServices" => services_list_response,
+            "EnvVariables"    => env_vars_response_with_per_service_keys,
+            "ServiceInstance" => lambda do |vars|
+                if vars[:serviceId] == "svc-aimock"
+                    service_instance_response(
+                        image: "ghcr.io/copilotkit/showcase-aimock@sha256:cafef00d",
+                        start_cmd: "node /app/dist/cli.js",
+                        domains: ["aimock.showcase.copilotkit.ai"],
+                    )
+                else
+                    service_instance_response(
+                        image: "ghcr.io/copilotkit/showcase-shell@sha256:beef1234",
+                        domains: [],
+                    )
+                end
+            end,
+        )
+
+        cmd = Railway::SnapshotCommand.new(["--env", "production", "--dry-run"])
+        cmd.instance_variable_set(:@gql, fake)
+
+        snap = cmd.build_snapshot(Railway::PRODUCTION_ENV_ID)
+
+        assert_equal 2, snap["version"]
+        assert_equal 2, snap["services"].length
+
+        aimock = snap["services"].find { |s| s["name"] == "aimock" }
+        assert_equal "ghcr.io/copilotkit/showcase-aimock@sha256:cafef00d", aimock["image"]
+        assert_equal "sha256:cafef00d", aimock["digest"]
+        assert_equal "ghcr.io/copilotkit/showcase-aimock", aimock["image_tag"]
+        assert_equal "node /app/dist/cli.js", aimock["start_command"]
+        assert_equal ["aimock.showcase.copilotkit.ai"], aimock["custom_domains"]
+        assert_equal %w[NODE_OPTIONS PORT], aimock["env_keys"]
+        assert_equal "dep-1", aimock["latest_deployment_id"]
+
+        shell = snap["services"].find { |s| s["name"] == "shell" }
+        assert_equal ["API_KEY"], shell["env_keys"]
+        assert_equal [], shell["custom_domains"]
+    end
+
+    def test_lint_prod_marks_mutable_tag_when_image_is_unpinned
+        fake = FakeGQL.new(
+            "ProjectServices" => services_list_response,
+            "EnvVariables"    => env_vars_response_with_per_service_keys,
+            "ServiceInstance" => lambda do |vars|
+                # Both return an unpinned :latest tag.
+                service_instance_response(
+                    image: "ghcr.io/copilotkit/#{vars[:serviceId]}:latest",
+                )
+            end,
+        )
+        snap_cmd = Railway::SnapshotCommand.new(["--env", "production", "--dry-run"])
+        snap_cmd.instance_variable_set(:@gql, fake)
+        snap = snap_cmd.build_snapshot(Railway::PRODUCTION_ENV_ID)
+
+        # All services are mutable-tag because none have @sha256:.
+        snap["services"].each do |svc|
+            refute svc["image"].include?("@sha256:"), "expected mutable tag for #{svc['name']}"
+            assert_nil svc["digest"], "expected nil digest for #{svc['name']}"
+        end
+    end
+
+    def test_build_snapshot_queries_use_only_supported_fields
+        # Regression guard: the previous version of this tool referenced
+        # `Project.domains` and `Service.serviceInstances`, both of which do
+        # NOT exist in Railway's public GraphQL schema. Ensure those tokens
+        # never reappear in the query constants.
+        sources = [
+            Railway::SERVICES_LIST_QUERY,
+            Railway::SERVICE_INSTANCE_QUERY,
+            Railway::ENVIRONMENT_VARIABLES_QUERY,
+        ]
+        sources.each do |q|
+            refute_match(/project\s*\([^)]*\)\s*\{[^}]*\bdomains\b/m, q,
+                "Project has no `domains` field — use serviceInstance.domains or the top-level domains query.")
+            refute_match(/\bserviceInstances\b/m, q,
+                "Service has no `serviceInstances` field — use serviceInstance(serviceId, environmentId) directly.")
+        end
+    end
+
+    def test_deploy_mutation_uses_serviceInstanceDeployV2_not_redeploy
+        # Bug #2: serviceInstanceRedeploy replays the EXISTING deployment's
+        # snapshot (its OLD image), so a freshly pinned source.image never
+        # reaches the running container. Pinning must go through
+        # serviceInstanceUpdate (source.image) + serviceInstanceDeployV2, which
+        # spawns a NEW deployment that PULLS the updated source.image.
+        #
+        # serviceInstanceDeployV2 has signature (serviceId, environmentId,
+        # commitSha?) and does NOT accept an `image` arg — the image is carried
+        # by the preceding serviceInstanceUpdate.
+        refute_match(/serviceInstanceDeployV2\s*\([^)]*\bimage\b/m,
+            Railway::RestoreCommand::DEPLOY_V2_MUTATION,
+            "DeployV2 must not be called with an image arg.")
+        assert_match(/serviceInstanceUpdate\s*\(/, Railway::RestoreCommand::UPDATE_IMAGE_MUTATION)
+        assert_match(/source:\s*\{\s*image:/, Railway::RestoreCommand::UPDATE_IMAGE_MUTATION)
+        assert_match(/serviceInstanceDeployV2\s*\(/, Railway::RestoreCommand::DEPLOY_V2_MUTATION)
+        # The bug-#2 trap: serviceInstanceRedeploy must NOT be the deploy path.
+        refute Railway::RestoreCommand.const_defined?(:REDEPLOY_MUTATION),
+            "serviceInstanceRedeploy must be removed — it replays the old image (bug #2)."
+    end
+
+    def test_snapshot_v2_captures_healthcheck_region_replicas_restart_policy
+        # P6 parity-matrix needs these four fields on every snapshot service.
+        # Snapshot schema is v2; SnapshotCommand#build_snapshot must map them
+        # from the new SERVICE_INSTANCE_QUERY selection set.
+        fake = FakeGQL.new(
+            "ProjectServices" => services_list_response,
+            "EnvVariables"    => env_vars_response_with_per_service_keys,
+            "ServiceInstance" => lambda do |vars|
+                if vars[:serviceId] == "svc-aimock"
+                    service_instance_response(
+                        image: "ghcr.io/copilotkit/showcase-aimock@sha256:cafef00d",
+                        start_cmd: "node /app/dist/cli.js",
+                        domains: ["aimock.showcase.copilotkit.ai"],
+                        healthcheck_path: "/healthz",
+                        region: "us-west2",
+                        num_replicas: 2,
+                        restart_policy: "ON_FAILURE",
+                    )
+                else
+                    service_instance_response(
+                        image: "ghcr.io/copilotkit/showcase-shell@sha256:beef1234",
+                        healthcheck_path: "/health",
+                        region: "us-east1",
+                        num_replicas: 1,
+                        restart_policy: "ALWAYS",
+                    )
+                end
+            end,
+        )
+
+        cmd = Railway::SnapshotCommand.new(["--env", "production", "--dry-run"])
+        cmd.instance_variable_set(:@gql, fake)
+        snap = cmd.build_snapshot(Railway::PRODUCTION_ENV_ID)
+
+        aimock = snap["services"].find { |s| s["name"] == "aimock" }
+        assert_equal "/healthz",    aimock["healthcheck_path"]
+        assert_equal "us-west2",    aimock["region"]
+        assert_equal 2,             aimock["replicas"]
+        assert_equal "ON_FAILURE",  aimock["restart_policy"]
+
+        shell = snap["services"].find { |s| s["name"] == "shell" }
+        assert_equal "/health",   shell["healthcheck_path"]
+        assert_equal "us-east1",  shell["region"]
+        assert_equal 1,           shell["replicas"]
+        assert_equal "ALWAYS",    shell["restart_policy"]
+    end
+
+    def test_snapshot_io_read_accepts_v1_and_v2_snapshots
+        # rollback-commit replays historical snapshots from arbitrary git SHAs,
+        # so SnapshotIO.read MUST stay backwards-compat with v1 even though
+        # all NEW snapshots are written as v2.
+        require "tempfile"
+        [1, 2].each do |ver|
+            Tempfile.create(["snap-v#{ver}", ".yaml"]) do |f|
+                f.write(YAML.dump("version" => ver, "services" => []))
+                f.flush
+                snap = Railway::SnapshotIO.read(f.path)
+                assert_equal ver, snap["version"]
+            end
+        end
+    end
+
+    def test_deploymentRollback_mutation_has_no_selection_set_because_it_returns_boolean
+        # deploymentRollback's return type is Boolean (scalar). GraphQL
+        # forbids a selection set on scalar fields.
+        refute_match(/deploymentRollback\s*\([^)]*\)\s*\{/m,
+            Railway::RollbackCommand::ROLLBACK_MUTATION,
+            "deploymentRollback returns Boolean; no selection set allowed.")
+    end
+
+    def test_build_snapshot_skips_service_whose_instance_throws_not_found_instead_of_aborting
+        # Regression: run 27144525566. A single odd/half-deleted service in the
+        # project list threw `GraphQL: ServiceInstance not found` from the
+        # per-service serviceInstance query. The only guard was `next if
+        # inst.nil?` — it handled a NULL result but not a THROWN error, so the
+        # error bubbled to Railway.run's top-level `rescue GraphQL::Error` and
+        # aborted the ENTIRE promote (opaque exit 2) before any preflight ran.
+        #
+        # build_snapshot must tolerate a thrown per-service `ServiceInstance not
+        # found` the same way it tolerates nil: skip that one service and keep
+        # going, so the healthy services still produce a usable snapshot.
+        fake = FakeGQL.new(
+            "ProjectServices" => services_list_response,
+            "EnvVariables"    => env_vars_response_with_per_service_keys,
+            "ServiceInstance" => lambda do |vars|
+                if vars[:serviceId] == "svc-aimock"
+                    # Half-deleted / instance-less service: Railway throws this
+                    # exact GraphQL error rather than returning null.
+                    raise Railway::GraphQL::Error, "GraphQL: ServiceInstance not found"
+                else
+                    service_instance_response(
+                        image: "ghcr.io/copilotkit/showcase-shell@sha256:beef1234",
+                        domains: [],
+                    )
+                end
+            end,
+        )
+
+        cmd = Railway::SnapshotCommand.new(["--env", "production", "--dry-run"])
+        cmd.instance_variable_set(:@gql, fake)
+
+        snap = cmd.build_snapshot(Railway::PRODUCTION_ENV_ID)
+
+        # The thrown service is skipped; the healthy one survives.
+        names = snap["services"].map { |s| s["name"] }
+        assert_equal ["shell"], names, "the instance-less service must be skipped, not abort the snapshot"
+        shell = snap["services"].find { |s| s["name"] == "shell" }
+        assert_equal "ghcr.io/copilotkit/showcase-shell@sha256:beef1234", shell["image"]
+    end
+
+    def test_build_snapshot_does_not_swallow_unrelated_graphql_errors
+        # The per-service tolerance MUST be narrow: only a `ServiceInstance not
+        # found` error for an individual service is skippable. Any OTHER GraphQL
+        # failure (auth, rate-limit, schema drift, transient 5xx surfaced as a
+        # GraphQL error) must still propagate fail-loud — never silently hidden.
+        fake = FakeGQL.new(
+            "ProjectServices" => services_list_response,
+            "EnvVariables"    => env_vars_response_with_per_service_keys,
+            "ServiceInstance" => lambda do |_vars|
+                raise Railway::GraphQL::Error, "GraphQL: Not Authorized"
+            end,
+        )
+
+        cmd = Railway::SnapshotCommand.new(["--env", "production", "--dry-run"])
+        cmd.instance_variable_set(:@gql, fake)
+
+        assert_raises(Railway::GraphQL::Error) do
+            cmd.build_snapshot(Railway::PRODUCTION_ENV_ID)
+        end
+    end
+end

showcase/bin/spec/test_snapshot_ivar_lint.rb

@@ -0,0 +1,132 @@
+# frozen_string_literal: true
+
+# Snapshot-ivar enforcement lint.
+#
+# Background: PromoteCommand has two snapshot "views" — the FULL un-narrowed
+# fleet snapshot and the (optionally) target-narrowed snapshot. Two prior
+# regressions came from a `check_*` method reading the wrong raw ivar
+# (`@staging_snapshot` / `@prod_snapshot`) inside a fleet-scoped invariant
+# and accidentally evaluating it against the narrowed view, producing
+# spurious WARN/REFUSE findings on single-service promotes.
+#
+# The fix introduced four accessors — `fleet_staging`, `fleet_prod`,
+# `target_staging`, `target_prod` — and the convention is that ALL reads of
+# the four backing ivars go through one of those accessors. This lint test
+# pins the convention as an executable invariant: any direct read of
+# `@staging_snapshot`, `@prod_snapshot`, `@full_staging_snapshot`, or
+# `@full_prod_snapshot` outside the explicit allowlist below FAILS the
+# suite.
+#
+# The allowlist is keyed by `<line-number>:<exact-line-content>` and the
+# check requires BOTH to match. NOTE: the allowlist is keyed by both line
+# number and stripped content. Any line shift in bin/railway above the
+# allowlisted region requires renumbering every entry by hand; the
+# `test_allowlist_entries_match_current_file_content` self-check fails
+# loud when this drifts. Nothing refreshes the allowlist mechanically —
+# this dual self-check + offender-sweep is intentional, so a new
+# offender (even one with identical surrounding text) fails the suite.
+#
+# To intentionally add a NEW legitimate write/accessor site, also add its
+# `<line>:<content>` entry to ALLOWED_LINES.
+
+require_relative "spec_helper"
+
+class SnapshotIvarLintTest < Minitest::Test
+    RAILWAY_PATH = File.expand_path("../railway", __dir__)
+
+    # The four protected ivars. Anything matching one of these names is a
+    # candidate offender unless it appears on an allowlisted line.
+    IVAR_PATTERN = /@(?:full_)?(?:staging|prod)_snapshot\b/.freeze
+
+    # Allowlist: every legitimate site that mentions one of the four
+    # ivars. Format = "<1-indexed line>:<stripped line content>". When the
+    # railway file legitimately changes, update this list to match.
+    #
+    # Categories (must remain in sync with bin/railway):
+    #   - run             : initial @full_*_snapshot capture at promote start
+    #   - capture_snapshots
+    #                      : the single test-seam assignment site
+    #   - narrow_snapshots_to_single_service!
+    #                      : the narrowing reads + writes of @{staging,prod}_snapshot
+    #   - fleet_staging / fleet_prod / target_staging / target_prod
+    #                      : the four accessors themselves (the ONLY sanctioned reads)
+    #   - comments        : block/inline comments that name the ivar in
+    #                       prose (do not perform a read)
+    ALLOWED_LINES = [
+        # `run` — capture full-fleet view before optional narrowing.
+        '1504:@full_staging_snapshot = @staging_snapshot',
+        '1505:@full_prod_snapshot    = @prod_snapshot',
+
+        # Doc comment above narrow_snapshots_to_single_service!.
+        '1513:# Narrow @staging_snapshot and @prod_snapshot to only the named',
+
+        # narrow_snapshots_to_single_service! — the WRITE site.
+        '1521:staging_match = (@staging_snapshot["services"] || []).select { |s| s["name"] == name }',
+        '1526:@staging_snapshot = @staging_snapshot.merge("services" => staging_match)',
+        '1527:prod_match = (@prod_snapshot["services"] || []).select { |s| s["name"] == name }',
+        '1528:@prod_snapshot = @prod_snapshot.merge("services" => prod_match)',
+
+        # Doc comment above capture_snapshots.
+        '1532:# @staging_snapshot / @prod_snapshot directly.',
+
+        # capture_snapshots — single test-seam assignment site.
+        '1534:@staging_snapshot ||= SnapshotCommand.new(["--env", "staging", "--dry-run"]).build_snapshot(STAGING_ENV_ID)',
+        '1535:@prod_snapshot    ||= SnapshotCommand.new(["--env", "production", "--dry-run"]).build_snapshot(PRODUCTION_ENV_ID)',
+
+        # Doc comment above the accessor block (explains test seam).
+        '1559:# promote tests stub @staging_snapshot/@prod_snapshot directly',
+
+        # The four accessor bodies — the ONLY sanctioned reads.
+        '1571:@full_staging_snapshot || @staging_snapshot',
+        '1575:@full_prod_snapshot || @prod_snapshot',
+        '1579:@staging_snapshot',
+        '1583:@prod_snapshot',
+    ].freeze
+
+    def setup
+        @lines = File.readlines(RAILWAY_PATH).each_with_index.map { |l, i| [i + 1, l.chomp] }
+        @allowed = ALLOWED_LINES.each_with_object({}) do |entry, h|
+            lineno, content = entry.split(":", 2)
+            h[Integer(lineno)] = content
+        end
+    end
+
+    def test_allowlist_entries_match_current_file_content
+        # Defensive: prove the allowlist itself is correct. If someone
+        # reformats bin/railway and the allowlist drifts, surface that as
+        # a clear assertion rather than a spurious lint failure later.
+        @allowed.each do |lineno, expected|
+            actual = @lines.find { |n, _| n == lineno }
+            assert actual, "allowlist references line #{lineno} but railway has no such line"
+            assert_equal expected, actual[1].strip,
+                "allowlist content for line #{lineno} does not match railway file " \
+                "(allowlist=#{expected.inspect}, file=#{actual[1].strip.inspect}). " \
+                "If the file legitimately changed, update ALLOWED_LINES."
+        end
+    end
+
+    def test_no_direct_ivar_reads_outside_allowlist
+        offenders = []
+        @lines.each do |lineno, content|
+            next unless content =~ IVAR_PATTERN
+            next if @allowed.key?(lineno) && @allowed[lineno] == content.strip
+            offenders << "  #{RAILWAY_PATH}:#{lineno}: #{content.strip}"
+        end
+
+        assert_empty offenders, <<~MSG
+            Direct read of @{,full_}{staging,prod}_snapshot found outside the
+            sanctioned write/accessor sites. ALL reads must go through one
+            of the four accessors:
+
+                fleet_staging  / fleet_prod   — FLEET-shape invariants
+                target_staging / target_prod  — per-service checks
+
+            Offenders:
+            #{offenders.join("\n")}
+
+            If this site is a legitimate new write or accessor, add its
+            "<line>:<stripped content>" entry to ALLOWED_LINES in
+            #{__FILE__} and document why.
+        MSG
+    end
+end

showcase/bin/spec/test_snapshot_roundtrip.rb

@@ -0,0 +1,61 @@
+# frozen_string_literal: true
+
+require_relative "spec_helper"
+require "tempfile"
+require "stringio"
+
+class SnapshotRoundtripTest < Minitest::Test
+    def sample_snapshot
+        {
+            "version"     => 1,
+            "captured_at" => "2026-01-01T00:00:00Z",
+            "project_id"  => Railway::PROJECT_ID,
+            "environment" => { "id" => Railway::STAGING_ENV_ID, "name" => "staging" },
+            "services" => [
+                {
+                    "name"                  => "showcase-shell",
+                    "service_id"            => "svc-1",
+                    "image"                 => "ghcr.io/copilotkit/showcase-shell@sha256:abc",
+                    "image_tag"             => "ghcr.io/copilotkit/showcase-shell",
+                    "digest"                => "sha256:abc",
+                    "start_command"         => "node server.js",
+                    "auto_updates_disabled" => nil,
+                    "latest_deployment_id"  => "dep-1",
+                    "env_keys"              => %w[KEY1 KEY2],
+                    "custom_domains"        => ["showcase.staging.copilotkit.ai"],
+                },
+            ],
+        }
+    end
+
+    def test_write_and_read_roundtrip
+        snap = sample_snapshot
+        Tempfile.create(["snap", ".yaml"]) do |f|
+            Railway::SnapshotIO.write(f.path, snap)
+            loaded = Railway::SnapshotIO.read(f.path)
+            assert_equal snap["version"], loaded["version"]
+            assert_equal "showcase-shell", loaded["services"][0]["name"]
+            assert_equal "sha256:abc", loaded["services"][0]["digest"]
+        end
+    end
+
+    def test_read_rejects_wrong_schema_version
+        bad = sample_snapshot
+        bad["version"] = 999
+        Tempfile.create(["snap", ".yaml"]) do |f|
+            File.write(f.path, YAML.dump(bad))
+            orig = $stderr
+            $stderr = StringIO.new
+            ex = assert_raises(SystemExit) { Railway::SnapshotIO.read(f.path) }
+            $stderr = orig
+            assert_equal 2, ex.status
+        end
+    end
+
+    def test_find_service_by_name
+        snap = sample_snapshot
+        svc = Railway.find_service(snap, "showcase-shell")
+        assert_equal "svc-1", svc["service_id"]
+        assert_nil Railway.find_service(snap, "does-not-exist")
+    end
+end