312 lines
23 KiB
Markdown
312 lines
23 KiB
Markdown
---
|
||
title: "Quality Gate Playbook"
|
||
---
|
||
|
||
# Quality-Gate System — Critical Assessment, Catalog and Replication Playbook
|
||
|
||
> **What this document is.** A critical assessment of OmniRoute's quality-gate system,
|
||
> compared to industry best practices, **plus** a comprehensive catalog of all quality
|
||
> checkpoints and a **tool-agnostic replication plan** to apply the same system to
|
||
> any project. Generated on 2026-06-16 from the real repository state (not from memory).
|
||
>
|
||
> Benchmarks: OWASP DSOMM · OpenSSF Scorecard · SLSA · SonarQube "Clean as You Code" ·
|
||
> Quality-Ratchet pattern · DORA 2024 · OWASP LLM Top 10 (2025) · mutation-testing best practices.
|
||
|
||
---
|
||
|
||
## Part 1 — Verdict and Maturity Classification
|
||
|
||
**Overall grade: A− / "Advanced". Top ~5–10% of projects.** The system independently
|
||
implements several patterns that the industry explicitly names — which is the strongest
|
||
alignment signal (we didn't copy a checklist; we converged on the right practices).
|
||
|
||
| Reference framework | Where we stand | Grade |
|
||
| ---------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------ |
|
||
| **OWASP DSOMM** (5 levels, 5 dimensions) | Solid Level 3, reaching 4 in _Test Intensity_ and _Static Depth_. Most orgs sit at 1–2. | **L3→L4** |
|
||
| **OpenSSF Scorecard** (18 checks) | We pass CI-Tests, Code-Review, Dependency-Update-Tool, Fuzzing, SAST, Signed-Releases (provenance), Token-Permissions, Vulnerabilities, Dangerous-Workflow. **Gaps:** Branch-Protection on `main` OFF; some actions not pinned. | **~7–8/10** |
|
||
| **SLSA** (4 levels) | `npm publish --provenance` + `id-token: write` + GitHub-hosted build = **L2**, approaching L3. Missing hardened/hermetic builder for L3+. | **L2→L3** |
|
||
| **SonarQube "Clean as You Code"** | Identical philosophy: the ratchet gates _non-regression_ (new code doesn't worsen the metric). **Divergence:** Sonar recommends **few** conditions; we have ~46 gates (fatigue risk). | **Aligned, with caveat** |
|
||
| **Quality-Ratchet pattern** | Reference implementation: ratchet + `dedicatedGate` + `tightenSlack` + `--require-tighten` + graceful-skip. More sophisticated than most public examples. | **Exemplary** |
|
||
| **DORA 2024** | Very strong on _stability_ axis. Risk: heavy gates can cost _lead time_ — mitigated by fast-gates split, but with coverage gap (see Part 2). | **Strong (stability)** |
|
||
| **OWASP LLM Top 10 (2025)** | We cover risk #1 (prompt-injection) with runtime guard + promptfoo (eval) + garak (red-team). Standard industry tools. | **Covered** |
|
||
| **Mutation testing** | Stryker nightly, thresholds 70/50, 8 critical modules. Industry consensus (60% existing / 80% new, nightly) — **we beat it**. **Gap:** score is not yet a ratchet. | **Almost there** |
|
||
|
||
---
|
||
|
||
## Part 2 — Critical Assessment (strengths + honest weaknesses)
|
||
|
||
### Strengths (what's above average)
|
||
|
||
1. **Multi-metric ratchet engine.** The heart of the system. 24 metrics in `quality-baseline.json`
|
||
- 4 dedicated baselines, each with direction (`up`/`down`), tolerance (`eps`), slack
|
||
(`tightenSlack`) and `dedicatedGate` flag. Things that get fixed **stay** fixed — it's the
|
||
antidote to codebase entropy.
|
||
2. **Defense-in-depth for supply-chain.** SAST (CodeQL/Sonar) + secrets (gitleaks with
|
||
`useDefault`) + SCA (osv/npm-audit/Trivy/Dependabot) + licenses + lockfile + SBOM + SLSA
|
||
provenance + Scorecard + workflow hardening (zizmor). Few codebases have this complete stack.
|
||
3. **Antidotes against Goodhart's Law.** Coverage as a target is a classic anti-pattern
|
||
("when the measure becomes the target, it ceases to be a good measure"). We have the
|
||
counterweights: **mutation testing** (measures whether the test catches the bug, not just
|
||
whether it executes the line), **`check-test-masking`** (blocks weakening asserts to pass),
|
||
**per-module coverage floors** (forces testing HIGH-risk code, not just the easy parts) and
|
||
**`check-pr-evidence`** (Hard Rule #18).
|
||
4. **Anti-hallucination / consistency gates.** A rare and valuable category: `check-known-symbols`,
|
||
`check-fetch-targets`, `check-openapi-routes`, `check-docs-symbols` ensure that docs, specs and
|
||
string dispatches point to living symbols. Catches "rot" that lint/test don't.
|
||
5. **Advisory→blocking lifecycle.** New gates enter as advisory (don't block merges while
|
||
maturing), then become blocking at cycle end. Reduces friction without losing the ceiling.
|
||
6. **Graceful skip when infra is missing.** Scanners (`--ratchet`) exit `exit 0` if the binary/network
|
||
fails — missing infra never blocks a legitimate PR. Mature engineering.
|
||
7. **Codified culture.** Hard Rules + `trust-but-verify` + stale-allowlist + evidence-gate
|
||
turn discipline into automated verification.
|
||
|
||
### Honest weaknesses (real gaps)
|
||
|
||
1. **🔴 The fast-gates split is a structural hole.** `quality.yml` (PR→`release/**`) runs **only
|
||
filesystem gates** — no typecheck, no tests, no build, no coverage. A typecheck/test regression
|
||
passes in a release PR and only blows up on the forward-merge to `main`. The motivation
|
||
(speed) is valid, but the gate should be where the merge happens (shift-left). **Largest
|
||
pending structural fix.**
|
||
2. **🟠 Gate sprawl/fatigue risk.** ~46 gates + 25 jobs is A LOT. Sonar itself warns:
|
||
too many conditions cause "gate fatigue" and priority debates, with risk of a gate being
|
||
ignored. DORA warns that heavy gates cost lead-time. We mitigate with advisory tiers and
|
||
non-absolute ratchets, but a **periodic ROI review per gate** is missing (some micro-gates for
|
||
doc-sync are consolidatable).
|
||
3. **🟠 Mutation score is not yet a ratchet.** The strongest antidote against coverage-gaming is
|
||
**advisory**. It's the highest-value pending item (and already 90% built).
|
||
4. **🟡 Advisories that should block (with the right scope).** `osv` (vulnCount) and `oasdiff` are
|
||
advisory despite frozen baselines. osv-advisory makes sense (a new CVE on an old dep would block
|
||
an unrelated PR) — but there's a middle ground (block only CRITICAL+fixable, as we did with
|
||
Trivy). oasdiff advisory means a contract-breaking change can pass.
|
||
5. **🟡 Runtime security is nightly-only.** schemathesis/garak/promptfoo/chaos/k6 run at night.
|
||
Correct decision (slow, need a live server), but a PR can introduce an injection-guard regression
|
||
that only gets caught the following night.
|
||
6. **🟡 Branch-protection on `main` is OFF.** `BRANCH_LOCK_TOKEN` locks _release_ branches, but
|
||
`main` itself is unprotected. Scorecard/DSOMM ding. Owner action required.
|
||
7. **🟡 CodeQL default-setup; semgrep not codified.** default-setup works (0 alerts), but a
|
||
committed `codeql.yml` gives more control; semgrep runs via an external cloud platform, not
|
||
versioned in the repo.
|
||
|
||
---
|
||
|
||
## Part 3 — Complete Catalog of Quality Checkpoints (portable)
|
||
|
||
The 12 categories below are the "quality system" in reusable form. Each lists the
|
||
**objective** (what to protect), the **tools we use** and the **tool-agnostic equivalent**
|
||
to replicate on any stack.
|
||
|
||
### 1. Style & formatting (deterministic, fast)
|
||
|
||
- **OmniRoute:** Prettier + ESLint via lint-staged (pre-commit), 2-spaces/double-quotes/100col.
|
||
- **Generic:** one auto-fixable formatter + one linter, running in pre-commit on staged files.
|
||
|
||
### 2. Types
|
||
|
||
- **OmniRoute:** `typecheck:core` (blocking) + `typecheck:noimplicit:core` (advisory) + `type-coverage` ratchet 92.17% + per-file any-budget.
|
||
- **Generic:** strict typecheck in CI + ratcheted type-coverage metric + per-file `any`/escape-hatch budget.
|
||
|
||
### 3. Tests (intensity)
|
||
|
||
- **OmniRoute:** 2 non-overlapping runners (Node native + vitest), 8 shards, global coverage 60/60/60/60 + ratchet ~76% + **8 per-module floors for critical modules** + nightly property tests + **mutation testing** nightly.
|
||
- **Generic:** test runner(s) + **absolute** coverage floor (anti-zero) + coverage **ratchet** (anti-regression) + **per-module floors for high-risk code** (anti-Goodhart) + property-based for pure logic + **mutation testing** nightly as the real measure of test quality.
|
||
|
||
### 4. Test policy (anti-gaming)
|
||
|
||
- **OmniRoute:** `pr-test-policy` (prod code requires a test), `check-test-masking` (blocks weakened asserts), `pr-evidence` (success claim requires evidence block), `test-discovery` (every test collected by a runner).
|
||
- **Generic:** "new code ⇒ new test" gate + assert-removed/tautology detector + evidence requirement (TDD or living test) + guarantee that no test is orphaned outside the globs.
|
||
|
||
### 5. Complexity & code health (ratchets)
|
||
|
||
- **OmniRoute:** ESLint-warnings (3769↓), jscpd duplication (5.72%↓), cyclomatic+max-lines complexity (1800↓), cognitive complexity sonarjs (753↓), dead-code/unused-exports knip (339↓), per-file file-size (frozen, shrink-only), circular-deps (custom Tarjan, blocking).
|
||
- **Generic:** ratchet every health metric (warnings, duplication, cyclomatic **and** cognitive complexity, dead code, file size, import cycles). Direction always "don't regress".
|
||
|
||
### 6. Static security (SAST + secrets)
|
||
|
||
- **OmniRoute:** CodeQL (ratchet alerts = 0), gitleaks (`[extend] useDefault=true` — critical!), SonarQube, custom security rules (public-creds, error-helper, route-guard-membership, route-validation).
|
||
- **Generic:** SAST (CodeQL/Sonar/semgrep) with alert ratchet + secrets scanner with **inherited default ruleset** (custom config that overrides the default = blind) + project-specific Hard Rule security gates.
|
||
|
||
### 7. Supply-chain (dependencies)
|
||
|
||
- **OmniRoute:** osv-scanner + npm-audit + Trivy + Dependabot (SCA), license-checker (SPDX allowlist), lockfile-lint (HTTPS+sha512+registry), `check-deps` anti-slopsquatting (allowlist + age ≥72h).
|
||
- **Generic:** multi-source SCA + license allowlist + lockfile integrity check + dependency allowlist with age/typosquatting check + grouped update bot.
|
||
|
||
### 8. Supply-chain (build & release)
|
||
|
||
- **OmniRoute:** SBOM (CycloneDX + syft), SLSA provenance (`--provenance`), OpenSSF Scorecard (weekly), workflow hardening (zizmor: artipacked→`persist-credentials:false`, cache-poisoning, token-permissions).
|
||
- **Generic:** generate SBOM on publish + signed provenance (SLSA L2+) + scheduled Scorecard + harden all workflows (minimum-privilege tokens, no persisted credentials on non-pusher checkout, actions pinned by SHA).
|
||
|
||
### 9. Contracts & API
|
||
|
||
- **OmniRoute:** oasdiff (breaking-change OpenAPI), schemathesis (contract fuzz nightly), openapi-coverage (% documented routes, ratchet 38.3%), openapi-security-tiers (spec vs route-guard).
|
||
- **Generic:** breaking-change contract diff (oasdiff/buf) + property-based fuzz against the spec (schemathesis) + ratcheted documentation coverage + spec↔code consistency.
|
||
|
||
### 10. Docs & i18n (anti-rot)
|
||
|
||
- **OmniRoute:** docs-sync (mirrored versions), docs-counts-sync (numbers in docs vs code), env-doc-sync, doc-links, fabricated-docs, cli-i18n, i18n-ui-coverage (`--threshold=65` + ratchet 80.1%).
|
||
- **Generic:** sync versions/counts/env-vars between docs and code (gate, not trust) + validate internal links + ratcheted i18n coverage.
|
||
|
||
### 11. Anti-hallucination / consistency (the rare category)
|
||
|
||
- **OmniRoute:** known-symbols (string dispatch ⇒ living symbol), provider-consistency, fetch-targets (client fetch ⇒ real route), docs-symbols, db-rules (Hard Rules #2/#5), migration-numbering.
|
||
- **Generic:** for every "duplicated source of truth" (registry, string dispatch, cross-layer references), a gate that proves both sides match. Catches the rot that typecheck/test don't.
|
||
|
||
### 12. Resilience & domain (product-specific)
|
||
|
||
- **OmniRoute:** chaos (fault-injection), heap-growth (leak), k6 (soak), promptfoo+garak (LLM red-team OWASP LLM Top 10), the 3 resilience laws (circuit-breaker/cooldown/lockout).
|
||
- **Generic:** identify the failure modes of **your** domain and have a gate (even if nightly) for each. For AI apps: injection red-team. For distributed systems: chaos + leak + soak.
|
||
|
||
---
|
||
|
||
## Part 4 — Replication Plan for Any Project
|
||
|
||
Build in **phases**, each delivering value on its own. Don't try all 12 categories at once —
|
||
that causes exactly the gate fatigue Part 2 warns about. Every new gate enters **advisory** and
|
||
becomes **blocking** when stable.
|
||
|
||
### The reusable centerpiece: the "anatomy of a ratchet gate"
|
||
|
||
The entire system revolves around this 3-file pattern. Copy it first:
|
||
|
||
1. **`baseline.json`** — the frozen metric value + `direction` (`up`/`down`) + `eps` (anti-flake) + `tightenSlack` + `dedicatedGate`.
|
||
2. **`collect-metrics.<ext>`** — runs the tool, extracts the number, writes `metrics.json`.
|
||
3. **`check-ratchet.<ext>`** — compares `metrics.json` vs `baseline.json`; `exit 1` **only** if regressed beyond `eps`; `exit 0` (graceful skip) if the tool/infra was missing; with `--require-tighten`, `exit 1` if it **improved** without updating the baseline (locks in the gain).
|
||
|
||
With this in place, **every** new metric (coverage, complexity, warnings, SAST alerts, bundle size, mutation score…) is just one line in the baseline.
|
||
|
||
### Phase 0 — Foundation (week 1)
|
||
|
||
CI exists; formatter + linter + typecheck + 1 test runner + **absolute** coverage floor
|
||
(e.g., 60%). Pre-commit runs fast auto-fixable checks. _Output: no PR breaks the basics._
|
||
|
||
### Phase 1 — The ratchet engine (week 2) — **the foundation of everything**
|
||
|
||
Implement the 3 files above. Freeze baselines for: warnings, coverage, complexity, duplication,
|
||
dead code, file size. _Output: the codebase can only improve from here._
|
||
|
||
### Phase 2 — Static depth (week 3)
|
||
|
||
SAST (CodeQL/Sonar/semgrep) with alert ratchet; secrets scanner (**inherit the default ruleset**);
|
||
SCA (osv/Dependabot) + license allowlist + lockfile-lint. _Output: known vulnerabilities and
|
||
leaked secrets don't pass._
|
||
|
||
### Phase 3 — Build supply-chain (week 4)
|
||
|
||
SBOM on publish + signed provenance (SLSA L2) + scheduled Scorecard + workflow hardening
|
||
(zizmor: minimum tokens, no persisted credentials, pinned actions). _Output: traceable and
|
||
tamper-proof releases._
|
||
|
||
### Phase 4 — Test intensity (week 5–6)
|
||
|
||
2nd runner if useful; **per-module coverage floors for critical modules** (anti-Goodhart);
|
||
property-based for pure logic; **mutation testing nightly** → when the 1st score arrives, make
|
||
`mutationScore` a ratchet. _Output: coverage stops being a vanity metric; tests provably catch bugs._
|
||
|
||
### Phase 5 — Contract & dynamic (week 7)
|
||
|
||
If there's a public API: oasdiff (breaking-change, **blocking**) + schemathesis (nightly fuzz).
|
||
DAST/red-team nightly as appropriate for the domain. _Output: contracts don't break silently._
|
||
|
||
### Phase 6 — Anti-hallucination & domain (week 8)
|
||
|
||
One consistency gate for each "duplicated truth" in the project. Domain-specific failure-mode
|
||
gates (for AI: injection red-team). _Output: structural rot and domain failures have a safety net._
|
||
|
||
### Phase 7 — Governance (ongoing)
|
||
|
||
- Advisory→blocking cycle for every new gate.
|
||
- `stale-allowlist`: every suppression has a justification + issue; obsolete suppression is caught.
|
||
- `evidence-gate`: success claim in a PR requires proof (test or living test).
|
||
- **Quarterly ROI review per gate** (kill/defund those that don't pay back — fights fatigue).
|
||
- Promote your project's Hard Rules into executable gates.
|
||
|
||
### Cross-cutting principles (non-negotiable)
|
||
|
||
- **Ratchet, not absolute.** Gate _non-regression_, not a fixed number (except anti-zero floors).
|
||
- **Absolute floor + ratchet together.** The floor prevents collapse; the ratchet prevents slow erosion.
|
||
- **Anti-Goodhart by design.** Every target metric needs a counterweight (coverage ⇒ mutation + anti-masking; per-module floors to force testing the hard code).
|
||
- **Graceful skip.** Missing infra never blocks; only real regression blocks.
|
||
- **`dedicatedGate` for expensive metrics.** Metrics that need an external binary get their own script (with skip), outside the synchronous central ratchet.
|
||
- **Gate where the merge happens.** Don't leave a gap between the fast gate and the actual merge (the lesson from the fast-gates split).
|
||
- **Few blocking gates, well-chosen.** Sonar/DORA: too many conditions = fatigue. Prefer advisory + ratchet over a wall of blocking gates.
|
||
|
||
---
|
||
|
||
## Part 5 — Recommended improvements (prioritized, compatible)
|
||
|
||
**P0 — highest ROI, almost ready**
|
||
|
||
1. **Mutation score ratchet** (after the 1st nightly Stryker produces values). Key antidote against coverage-Goodhart; ~90% done.
|
||
2. **Close the fast-gates hole** — add typecheck + impacted tests to `quality.yml` (PR→release).
|
||
3. **Branch-protection on `main`** (owner setting) — boosts Scorecard, closes the DSOMM gap.
|
||
|
||
**P1 — valuable** 4. **osv/oasdiff → blocking with the right scope** — osv only CRITICAL+fixable (two-step like Trivy); oasdiff blocks breaking-changes. 5. **`require-tighten` → blocking** (end of cycle) — locks in metric gains. 6. **ROI/timing review per-gate** in `ci-summary` — find and prune slow/low-value gates.
|
||
|
||
**P2 — diminishing returns** 7. **SLSA L3** — hermetic/reproducible builder (GitHub SLSA generator) if you want to move up from L2. 8. **Committed CodeQL config + versioned semgrep** — more control/reproducibility. 9. **Per-PR DAST smoke** — fast subset of schemathesis/promptfoo on highest-risk endpoints (not just nightly). 10. **Flakiness dashboard + DORA metrics** — ensure gates aren't eroding speed.
|
||
|
||
---
|
||
|
||
## Part 6 — Concrete release lessons (gates to add in Phase 9)
|
||
|
||
> This section records real incidents from release closures where a gate **was missing**,
|
||
> with concrete evidence and the proposed gate. Each item is a candidate for Part 5.
|
||
|
||
### Lesson v3.8.27 (2026-06-17) — the "fast-gates hole" lets deterministic regressions reach release day
|
||
|
||
**What happened.** During the v3.8.27 `/generate-release`, the release PR (`release/v3.8.27` → `main`)
|
||
was the **first** execution of the full `ci.yml` matrix in the integrated cycle. Result: 12 failures
|
||
at once — **3 deterministic tests** + ~9 flakes/env. None were live product regressions, but
|
||
all went unnoticed because cycle PRs enter `release/**` via the **Fast QG
|
||
(`quality.yml`)**, which does NOT run the full unit suite, nor `pr-test-policy` (test-masking), nor the
|
||
full integration suite, nor schema parity checking. The 3 deterministic ones:
|
||
|
||
1. **Test outdated by UI change** — `permissions modal switch buttons declare button type`:
|
||
#4034 added a 4th switch (a11y `type="button"` maintained); the test's `=== 3` count became
|
||
outdated. Static analysis should have caught this in the #4034 PR.
|
||
2. **Test outdated by packaging change** — `findMissingArtifactPaths ... root runtime files`:
|
||
`dist/http-method-guard.cjs` became a legitimate required-path; the test's expected list became
|
||
outdated.
|
||
3. **Lossy modularization divergence (most serious)** — `settings schemas accept ... unprefixed
|
||
toggle`: the **modularized** `updateSettingsSchema` (`schemas/settings.ts`, created by #3988) diverged
|
||
from the canonical one (`settingsSchemas.ts`): **45 fields vs 85 — 40 dropped + 6 divergent (qdrant\*)**. It was
|
||
**dead-code** (runtime uses the canonical one), so no live impact, but only a hand-written parity
|
||
test caught it. #4030 restored 16 analogous drops from #3988/#3993, but this one slipped through.
|
||
|
||
**Proposed gates (Phase 9):**
|
||
|
||
- **G1 — Actually close the fast-gates hole (extends P0 #2).** In `quality.yml` (PR→`release/**`),
|
||
beyond typecheck + impacted tests, run **`pr-test-policy` (test-masking) + the full deterministic
|
||
unit suite** (or at least the static/parity files, which are fast and non-flaky).
|
||
This way, outdated tests and assert removal are caught in the PR that introduces them — not on
|
||
release day. Keep integration/e2e out (slow/flaky), but the deterministic layer CANNOT stay only
|
||
in PR→main.
|
||
- **G2 — Modularization parity gate (NEW, not covered today).** A check that, for each symbol
|
||
re-exported by a modularized barrel (`src/shared/validation/schemas/*`, `providerRegistry`
|
||
modules, etc.), compares the **shape** (`z.object` keys, registry entries) against the canonical
|
||
source and **fails on divergence** (dropped/extra field). Would have caught the 40-field drop from
|
||
#3988 in that very PR. Generalizes the hand-written parity tests (which only exist where someone
|
||
remembered to write them). Cheap: imports both and diffs `Object.keys(shape)`.
|
||
- **G3 — Deterministic flake triage (support).** LiveWS-startup and the integration-combo/breaker
|
||
tests fail due to server timeout/cascade in CI (env), not logic. Mark these as
|
||
`known-flaky` (quarantined with issue) so the release-PR red is **only real signals**, not noise
|
||
masking deterministic regressions in the middle.
|
||
|
||
**Principle:** _the gate has to run where the merge happens_ (already in "Cross-cutting principles"). The
|
||
v3.8.27 incident shows this also applies to the **deterministic test layer**, not just lint/typecheck —
|
||
otherwise the debt of outdated tests + lossy modularization only appears in PR→main, in batch, at
|
||
the worst moment.
|
||
|
||
---
|
||
|
||
## Sources (industry best practices)
|
||
|
||
- OWASP DevSecOps Maturity Model (DSOMM) — https://dsomm.owasp.org/about
|
||
- OpenSSF Scorecard / SLSA — https://openssf.org · https://slsa.dev
|
||
- SonarQube "Clean as You Code" — https://docs.sonarsource.com/sonarqube-server/latest/user-guide/clean-as-you-code
|
||
- Quality Ratchets (LeadDev) — https://leaddev.com/software-quality/introducing-quality-ratchets-tool-managing-complex-systems
|
||
- Continuous Code Improvement Using Ratcheting (Greiner) — https://robertgreiner.com/continuous-code-improvement-using-ratcheting/
|
||
- DORA 2024 State of DevOps — https://cloud.google.com/blog/products/devops-sre/announcing-the-2024-dora-report
|
||
- Mutation testing best practices (Stryker) — https://stryker-mutator.io
|
||
- Coverage as anti-pattern (Goodhart) — https://www.industriallogic.com/blog/code-coverage-complications/
|
||
- OWASP Top 10 for LLM Applications (2025) — https://owasp.org/www-project-top-10-for-large-language-model-applications/
|
||
- Contract testing (oasdiff/schemathesis) — https://www.oasdiff.com · https://schemathesis.readthedocs.io
|