light-heart-labs--dreamserver/ods/docs/RELEASE_VALIDATION.md

# Release Validation

Last updated: 2026-05-25

ODS is validated as an installed appliance, not only as a collection of
unit tests. The release-grade path combines CI, clean distro bootstrap checks,
a distro lab, and a private real-hardware fleet so operational changes are
tested against the surfaces users actually touch: the public install command,
service startup, dashboard flows, model routing, Hermes, full-model
capabilities, reinstall, restart, and `ods doctor`.

This document is a public, sanitized summary of that release gate. It describes
what a green run proves without publishing private hostnames, LAN addresses,
usernames, local paths, or raw run logs. For the broader hardware and distro
surface, see [VALIDATION-MATRIX.md](VALIDATION-MATRIX.md). For day-to-day local
test commands, see [TESTING.md](TESTING.md).

## When We Run It

Run the release-grade fleet after operational code changes: installer phases,
bootstrap logic, Docker Compose stack generation, service manifests, dashboard
API behavior, Hermes, model routing, GPU/runtime detection, lifecycle commands,
or anything that can affect a user's install or running stack.

Docs-only, comment-only, and narrow test-only changes usually use focused
validation instead. Dependency or runtime wiring changes should use the
release-grade gate even when the code diff is small.

PRs should state their changed surface and validation level explicitly. Use
[HIGH_RISK_CHANGE_MAP.md](HIGH_RISK_CHANGE_MAP.md) to decide whether focused
checks are enough or whether the candidate needs release-grade fleet validation
before it is treated as releasable.

## User Green

`User Green` is the top-level release-readiness result. It is not a marketing
claim that every possible machine or network will work. It means the enabled
release surfaces passed or were explicitly accounted for in the current
candidate run.

| Gate | What it proves |
|---|---|
| Zero-prereq bootstrap | Bare Linux distro containers can fetch the public installer and provision missing prerequisites such as Git, Python, Docker, and Compose. |
| Install Green | Enabled real-hardware hosts can fresh-install from the public bootstrap path. |
| Product Green | Core services, cloud-mode contracts, dashboard flows, Hermes auth/chat, and UI checks pass after install. |
| Capability Green | Full-model capability probes pass after large model downloads and swaps complete. |
| Lifecycle Green | Idempotent reinstall, `ods restart`, and `ods doctor` recover cleanly after state changes. |
| User Green | The combined release gate is clean, with failures, skips, and deferrals resolved or documented. |

## Release-Grade Surfaces

The release harness normally combines these layers:

| Layer | Coverage | Why it matters |
|---|---|---|
| CI | Fast syntax, contract, dashboard, shell, Python, and PowerShell checks | Catches cheap regressions before hardware time is spent. |
| Zero-prereq bootstrap | Clean Ubuntu, Debian, Fedora, Rocky, Arch, and openSUSE containers | Proves the public `curl` path does not assume a developer workstation. |
| Distro lab | 10 Linux container lanes plus systemd-capable Incus VM lanes | Exercises package-manager, systemd, Docker daemon, and Compose behavior across distro families. |
| Real hardware fleet | Linux NVIDIA, Linux AMD/ROCm-Lemonade, ARM Linux NVIDIA, and Apple Silicon hardware classes | Proves accelerator/runtime behavior and the installed product on actual machines. |

The latest release-grade fleet run for the current candidate should be cited in
release notes with its commit, date, enabled hardware classes, and any skipped
or deferred surfaces. Public docs should summarize the sanitized evidence
rather than linking raw private run artifacts.

## What We Check

A release-grade run includes:

- public bootstrap and zero-prereq install checks;
- fresh install on enabled real-hardware targets;
- core service health and generated config contracts;
- cloud and hybrid mode contracts;
- dashboard API flows such as model download, model switch, and extension install;
- Hermes authentication and agent chat with seed verification;
- browser UI checks on the default UI target;
- full-model capabilities such as chat, search, file read/write, code execution,
  skills, model identity, and context;
- lifecycle checks: idempotent reinstall, `ods restart`, and `ods doctor`;
- regression replay for previously fixed fleet failures.

Capability probes can be deferred while a large model is still downloading or
hot-swapping. The capabilities watcher polls until the model is ready, reruns
the probes, and updates the report so a release is not marked User Green just
because the first pass arrived before the model did.

## Known Limits

- ODS Talk and owner-card probes only gate when the owner-card surface is
  enabled and `ods-proxy` is actually available, such as a LAN-enabled install.
  Default non-LAN installs skip those probes instead of false-failing a surface
  the user did not expose.
- Vision probes are opt-in and should be called out explicitly when enabled for
  a candidate run.
- Incus Arch and openSUSE VM lanes can hit nested Docker limitations in the lab.
  The container lanes and real-hardware fleet are used to separate those lab
  limitations from product regressions.
- The fleet is not an exhaustive promise for every driver, firewall, storage
  layout, Docker Desktop version, home router, or unsupported hardware
  combination.
- A green release run is a strong release signal, not a soak test. Long-running
  thermal, benchmark, and overnight stability evidence is tracked separately.

## Reading A Green Run

A current User Green pass should give contributors and auditors high confidence
that the main supported install paths are working at release time:

- clean machines can reach the installer through the public bootstrap path;
- supported hardware classes can install and start the product;
- users can exercise dashboard, Hermes, model, and extension workflows;
- full-model AI capabilities are validated after downloads complete;
- reinstall, restart, and diagnostic paths recover after state changes.

It should not be read as a claim that all optional modes ran in that candidate.
Release notes should name any skipped or opt-in surfaces, especially Windows
fleet targets, ODS Talk owner-card flows, vision probes, AP mode, and network
topologies that vary by lab.