wehub-resource-sync
9e8f1bbeed
3.7 KiB
Validation Reproducibility
ODS's full release-grade validation uses public CI, distro labs, and a private real-hardware fleet. Not every fork can reproduce every lane, but every operator should be able to understand what was tested and run an appropriate subset for their own hardware.
Use this guide with RELEASE_VALIDATION.md, VALIDATION-MATRIX.md, and HIGH_RISK_CHANGE_MAP.md.
Validation Layers
| Layer | Publicly reproducible | Purpose |
|---|---|---|
| Static checks | Yes | Syntax, contracts, docs, manifest health, config drift |
| Dashboard tests | Yes | UI/API unit and build confidence |
| Compose matrix | Yes | Validates stack file combinations without full hardware |
| Distro containers | Mostly | Package-manager and bootstrap behavior across distro families |
| Incus/systemd VMs | Lab-dependent | Docker daemon and systemd behavior |
| Real hardware fleet | Hardware-dependent | GPU/backend, model, lifecycle, and full-product behavior |
| Full-model capabilities | Hardware/model-dependent | Proves agent, search, files, code, model identity, and context after model swap |
Minimum Local Audit Set
From ods/:
git diff --check
python scripts/audit-extensions.py --project-dir .
python scripts/validate-generated-configs.py
python scripts/validate-golden-paths.py
python scripts/validate-dependency-pins.py
Add dashboard checks when dashboard code or dashboard-api behavior changes.
Reproducing Installer Confidence
For a local fork or appliance candidate:
- Start from a clean checkout.
- Record the commit.
- Run a fresh install on representative hardware.
- Confirm service health.
- Exercise dashboard model and extension flows.
- Confirm Hermes can answer a seed prompt.
- Wait for full model download and hot-swap.
- Run capability probes or an equivalent manual checklist.
- Run idempotent reinstall,
ods restart, andods doctor. - Record skipped surfaces honestly.
If you cannot run the full fleet, say which hardware and distro classes were not tested.
Capability Deferrals
Capability checks should not fail just because a large model is still downloading. A valid report distinguishes:
- bootstrap model active;
- full model downloading;
- full model downloaded but not served yet;
- full model served and capability probes passed;
- full model served and a probe failed.
Do not mark a release as fully validated until deferred full-model checks have resolved or are explicitly excluded from that release.
Owner-Card And Talk Probes
ODS Talk owner-card probes only gate releases when the owner-card surface is
enabled and ods-proxy is healthy. A default non-LAN install should skip
those probes rather than failing a surface the user did not expose.
If your fork depends on owner-card access, add a dedicated validation lane for the LAN/proxy path.
Recording A Validation Receipt
Use a concise receipt:
Project ref:
Downstream ref:
Date:
Hardware:
OS/distro:
Install command:
Services enabled:
Model selected:
Gates passed:
Gates skipped/deferred:
Known limitations:
Logs/report path:
Validation receipts are most valuable when they are boring, specific, and easy to compare with the next run.
How To Read Upstream Claims
Upstream validation proves the upstream candidate on the tested hardware and software surface. It is a strong signal for forks, but it does not automatically validate:
- private extensions;
- changed model catalogs;
- changed compose overlays;
- custom LAN exposure;
- different Docker Desktop versions;
- different GPU drivers;
- unsupported distros or hardware.
Forks should cite upstream receipts and add their own downstream receipt for local changes.