wehub/diegosouzapw--omniroute
1
0
Fork 0
Code Issues Pull Requests Actions 10 Deployments Models Agent Notes Packages Projects Releases Wiki Activity
Files
main
diegosouzapw--omniroute/docs/ops/QUALITY_GATE_PLAYBOOK.md
T
wehub-resource-sync d8dcd5f6d1 chore: import upstream snapshot with attribution
2026-07-13 13:39:12 +08:00

23 KiB
Raw Permalink Blame History

title
title
Quality Gate Playbook

Quality-Gate System — Critical Assessment, Catalog and Replication Playbook

What this document is. A critical assessment of OmniRoute's quality-gate system, compared to industry best practices, plus a comprehensive catalog of all quality checkpoints and a tool-agnostic replication plan to apply the same system to any project. Generated on 2026-06-16 from the real repository state (not from memory).

Benchmarks: OWASP DSOMM · OpenSSF Scorecard · SLSA · SonarQube "Clean as You Code" · Quality-Ratchet pattern · DORA 2024 · OWASP LLM Top 10 (2025) · mutation-testing best practices.

Part 1 — Verdict and Maturity Classification

Overall grade: A− / "Advanced". Top ~5–10% of projects. The system independently implements several patterns that the industry explicitly names — which is the strongest alignment signal (we didn't copy a checklist; we converged on the right practices).

Reference framework Where we stand Grade
OWASP DSOMM (5 levels, 5 dimensions) Solid Level 3, reaching 4 in Test Intensity and Static Depth. Most orgs sit at 1–2. L3→L4
OpenSSF Scorecard (18 checks) We pass CI-Tests, Code-Review, Dependency-Update-Tool, Fuzzing, SAST, Signed-Releases (provenance), Token-Permissions, Vulnerabilities, Dangerous-Workflow. Gaps: Branch-Protection on main OFF; some actions not pinned. ~7–8/10
SLSA (4 levels) npm publish --provenance + id-token: write + GitHub-hosted build = L2, approaching L3. Missing hardened/hermetic builder for L3+. L2→L3
SonarQube "Clean as You Code" Identical philosophy: the ratchet gates non-regression (new code doesn't worsen the metric). Divergence: Sonar recommends few conditions; we have ~46 gates (fatigue risk). Aligned, with caveat
Quality-Ratchet pattern Reference implementation: ratchet + dedicatedGate + tightenSlack + --require-tighten + graceful-skip. More sophisticated than most public examples. Exemplary
DORA 2024 Very strong on stability axis. Risk: heavy gates can cost lead time — mitigated by fast-gates split, but with coverage gap (see Part 2). Strong (stability)
OWASP LLM Top 10 (2025) We cover risk #1 (prompt-injection) with runtime guard + promptfoo (eval) + garak (red-team). Standard industry tools. Covered
Mutation testing Stryker nightly, thresholds 70/50, 8 critical modules. Industry consensus (60% existing / 80% new, nightly) — we beat it. Gap: score is not yet a ratchet. Almost there

Part 2 — Critical Assessment (strengths + honest weaknesses)

Strengths (what's above average)

  1. Multi-metric ratchet engine. The heart of the system. 24 metrics in quality-baseline.json
    • 4 dedicated baselines, each with direction (up/down), tolerance (eps), slack (tightenSlack) and dedicatedGate flag. Things that get fixed stay fixed — it's the antidote to codebase entropy.
  2. Defense-in-depth for supply-chain. SAST (CodeQL/Sonar) + secrets (gitleaks with useDefault) + SCA (osv/npm-audit/Trivy/Dependabot) + licenses + lockfile + SBOM + SLSA provenance + Scorecard + workflow hardening (zizmor). Few codebases have this complete stack.
  3. Antidotes against Goodhart's Law. Coverage as a target is a classic anti-pattern ("when the measure becomes the target, it ceases to be a good measure"). We have the counterweights: mutation testing (measures whether the test catches the bug, not just whether it executes the line), check-test-masking (blocks weakening asserts to pass), per-module coverage floors (forces testing HIGH-risk code, not just the easy parts) and check-pr-evidence (Hard Rule #18).
  4. Anti-hallucination / consistency gates. A rare and valuable category: check-known-symbols, check-fetch-targets, check-openapi-routes, check-docs-symbols ensure that docs, specs and string dispatches point to living symbols. Catches "rot" that lint/test don't.
  5. Advisory→blocking lifecycle. New gates enter as advisory (don't block merges while maturing), then become blocking at cycle end. Reduces friction without losing the ceiling.
  6. Graceful skip when infra is missing. Scanners (--ratchet) exit exit 0 if the binary/network fails — missing infra never blocks a legitimate PR. Mature engineering.
  7. Codified culture. Hard Rules + trust-but-verify + stale-allowlist + evidence-gate turn discipline into automated verification.

Honest weaknesses (real gaps)

  1. 🔴 The fast-gates split is a structural hole. quality.yml (PR→release/**) runs only filesystem gates — no typecheck, no tests, no build, no coverage. A typecheck/test regression passes in a release PR and only blows up on the forward-merge to main. The motivation (speed) is valid, but the gate should be where the merge happens (shift-left). Largest pending structural fix.
  2. 🟠 Gate sprawl/fatigue risk. ~46 gates + 25 jobs is A LOT. Sonar itself warns: too many conditions cause "gate fatigue" and priority debates, with risk of a gate being ignored. DORA warns that heavy gates cost lead-time. We mitigate with advisory tiers and non-absolute ratchets, but a periodic ROI review per gate is missing (some micro-gates for doc-sync are consolidatable).
  3. 🟠 Mutation score is not yet a ratchet. The strongest antidote against coverage-gaming is advisory. It's the highest-value pending item (and already 90% built).
  4. 🟡 Advisories that should block (with the right scope). osv (vulnCount) and oasdiff are advisory despite frozen baselines. osv-advisory makes sense (a new CVE on an old dep would block an unrelated PR) — but there's a middle ground (block only CRITICAL+fixable, as we did with Trivy). oasdiff advisory means a contract-breaking change can pass.
  5. 🟡 Runtime security is nightly-only. schemathesis/garak/promptfoo/chaos/k6 run at night. Correct decision (slow, need a live server), but a PR can introduce an injection-guard regression that only gets caught the following night.
  6. 🟡 Branch-protection on main is OFF. BRANCH_LOCK_TOKEN locks release branches, but main itself is unprotected. Scorecard/DSOMM ding. Owner action required.
  7. 🟡 CodeQL default-setup; semgrep not codified. default-setup works (0 alerts), but a committed codeql.yml gives more control; semgrep runs via an external cloud platform, not versioned in the repo.

Part 3 — Complete Catalog of Quality Checkpoints (portable)

The 12 categories below are the "quality system" in reusable form. Each lists the objective (what to protect), the tools we use and the tool-agnostic equivalent to replicate on any stack.

1. Style & formatting (deterministic, fast)

  • OmniRoute: Prettier + ESLint via lint-staged (pre-commit), 2-spaces/double-quotes/100col.
  • Generic: one auto-fixable formatter + one linter, running in pre-commit on staged files.

2. Types

  • OmniRoute: typecheck:core (blocking) + typecheck:noimplicit:core (advisory) + type-coverage ratchet 92.17% + per-file any-budget.
  • Generic: strict typecheck in CI + ratcheted type-coverage metric + per-file any/escape-hatch budget.

3. Tests (intensity)

  • OmniRoute: 2 non-overlapping runners (Node native + vitest), 8 shards, global coverage 60/60/60/60 + ratchet ~76% + 8 per-module floors for critical modules + nightly property tests + mutation testing nightly.
  • Generic: test runner(s) + absolute coverage floor (anti-zero) + coverage ratchet (anti-regression) + per-module floors for high-risk code (anti-Goodhart) + property-based for pure logic + mutation testing nightly as the real measure of test quality.

4. Test policy (anti-gaming)

  • OmniRoute: pr-test-policy (prod code requires a test), check-test-masking (blocks weakened asserts), pr-evidence (success claim requires evidence block), test-discovery (every test collected by a runner).
  • Generic: "new code ⇒ new test" gate + assert-removed/tautology detector + evidence requirement (TDD or living test) + guarantee that no test is orphaned outside the globs.

5. Complexity & code health (ratchets)

  • OmniRoute: ESLint-warnings (3769↓), jscpd duplication (5.72%↓), cyclomatic+max-lines complexity (1800↓), cognitive complexity sonarjs (753↓), dead-code/unused-exports knip (339↓), per-file file-size (frozen, shrink-only), circular-deps (custom Tarjan, blocking).
  • Generic: ratchet every health metric (warnings, duplication, cyclomatic and cognitive complexity, dead code, file size, import cycles). Direction always "don't regress".

6. Static security (SAST + secrets)

  • OmniRoute: CodeQL (ratchet alerts = 0), gitleaks ([extend] useDefault=true — critical!), SonarQube, custom security rules (public-creds, error-helper, route-guard-membership, route-validation).
  • Generic: SAST (CodeQL/Sonar/semgrep) with alert ratchet + secrets scanner with inherited default ruleset (custom config that overrides the default = blind) + project-specific Hard Rule security gates.

7. Supply-chain (dependencies)

  • OmniRoute: osv-scanner + npm-audit + Trivy + Dependabot (SCA), license-checker (SPDX allowlist), lockfile-lint (HTTPS+sha512+registry), check-deps anti-slopsquatting (allowlist + age ≥72h).
  • Generic: multi-source SCA + license allowlist + lockfile integrity check + dependency allowlist with age/typosquatting check + grouped update bot.

8. Supply-chain (build & release)

  • OmniRoute: SBOM (CycloneDX + syft), SLSA provenance (--provenance), OpenSSF Scorecard (weekly), workflow hardening (zizmor: artipacked→persist-credentials:false, cache-poisoning, token-permissions).
  • Generic: generate SBOM on publish + signed provenance (SLSA L2+) + scheduled Scorecard + harden all workflows (minimum-privilege tokens, no persisted credentials on non-pusher checkout, actions pinned by SHA).

9. Contracts & API

  • OmniRoute: oasdiff (breaking-change OpenAPI), schemathesis (contract fuzz nightly), openapi-coverage (% documented routes, ratchet 38.3%), openapi-security-tiers (spec vs route-guard).
  • Generic: breaking-change contract diff (oasdiff/buf) + property-based fuzz against the spec (schemathesis) + ratcheted documentation coverage + spec↔code consistency.

10. Docs & i18n (anti-rot)

  • OmniRoute: docs-sync (mirrored versions), docs-counts-sync (numbers in docs vs code), env-doc-sync, doc-links, fabricated-docs, cli-i18n, i18n-ui-coverage (--threshold=65 + ratchet 80.1%).
  • Generic: sync versions/counts/env-vars between docs and code (gate, not trust) + validate internal links + ratcheted i18n coverage.

11. Anti-hallucination / consistency (the rare category)

  • OmniRoute: known-symbols (string dispatch ⇒ living symbol), provider-consistency, fetch-targets (client fetch ⇒ real route), docs-symbols, db-rules (Hard Rules #2/#5), migration-numbering.
  • Generic: for every "duplicated source of truth" (registry, string dispatch, cross-layer references), a gate that proves both sides match. Catches the rot that typecheck/test don't.

12. Resilience & domain (product-specific)

  • OmniRoute: chaos (fault-injection), heap-growth (leak), k6 (soak), promptfoo+garak (LLM red-team OWASP LLM Top 10), the 3 resilience laws (circuit-breaker/cooldown/lockout).
  • Generic: identify the failure modes of your domain and have a gate (even if nightly) for each. For AI apps: injection red-team. For distributed systems: chaos + leak + soak.

Part 4 — Replication Plan for Any Project

Build in phases, each delivering value on its own. Don't try all 12 categories at once — that causes exactly the gate fatigue Part 2 warns about. Every new gate enters advisory and becomes blocking when stable.

The reusable centerpiece: the "anatomy of a ratchet gate"

The entire system revolves around this 3-file pattern. Copy it first:

  1. baseline.json — the frozen metric value + direction (up/down) + eps (anti-flake) + tightenSlack + dedicatedGate.
  2. collect-metrics.<ext> — runs the tool, extracts the number, writes metrics.json.
  3. check-ratchet.<ext> — compares metrics.json vs baseline.json; exit 1 only if regressed beyond eps; exit 0 (graceful skip) if the tool/infra was missing; with --require-tighten, exit 1 if it improved without updating the baseline (locks in the gain).

With this in place, every new metric (coverage, complexity, warnings, SAST alerts, bundle size, mutation score…) is just one line in the baseline.

Phase 0 — Foundation (week 1)

CI exists; formatter + linter + typecheck + 1 test runner + absolute coverage floor (e.g., 60%). Pre-commit runs fast auto-fixable checks. Output: no PR breaks the basics.

Phase 1 — The ratchet engine (week 2) — the foundation of everything

Implement the 3 files above. Freeze baselines for: warnings, coverage, complexity, duplication, dead code, file size. Output: the codebase can only improve from here.

Phase 2 — Static depth (week 3)

SAST (CodeQL/Sonar/semgrep) with alert ratchet; secrets scanner (inherit the default ruleset); SCA (osv/Dependabot) + license allowlist + lockfile-lint. Output: known vulnerabilities and leaked secrets don't pass.

Phase 3 — Build supply-chain (week 4)

SBOM on publish + signed provenance (SLSA L2) + scheduled Scorecard + workflow hardening (zizmor: minimum tokens, no persisted credentials, pinned actions). Output: traceable and tamper-proof releases.

Phase 4 — Test intensity (week 5–6)

2nd runner if useful; per-module coverage floors for critical modules (anti-Goodhart); property-based for pure logic; mutation testing nightly → when the 1st score arrives, make mutationScore a ratchet. Output: coverage stops being a vanity metric; tests provably catch bugs.

Phase 5 — Contract & dynamic (week 7)

If there's a public API: oasdiff (breaking-change, blocking) + schemathesis (nightly fuzz). DAST/red-team nightly as appropriate for the domain. Output: contracts don't break silently.

Phase 6 — Anti-hallucination & domain (week 8)

One consistency gate for each "duplicated truth" in the project. Domain-specific failure-mode gates (for AI: injection red-team). Output: structural rot and domain failures have a safety net.

Phase 7 — Governance (ongoing)

  • Advisory→blocking cycle for every new gate.
  • stale-allowlist: every suppression has a justification + issue; obsolete suppression is caught.
  • evidence-gate: success claim in a PR requires proof (test or living test).
  • Quarterly ROI review per gate (kill/defund those that don't pay back — fights fatigue).
  • Promote your project's Hard Rules into executable gates.

Cross-cutting principles (non-negotiable)

  • Ratchet, not absolute. Gate non-regression, not a fixed number (except anti-zero floors).
  • Absolute floor + ratchet together. The floor prevents collapse; the ratchet prevents slow erosion.
  • Anti-Goodhart by design. Every target metric needs a counterweight (coverage ⇒ mutation + anti-masking; per-module floors to force testing the hard code).
  • Graceful skip. Missing infra never blocks; only real regression blocks.
  • dedicatedGate for expensive metrics. Metrics that need an external binary get their own script (with skip), outside the synchronous central ratchet.
  • Gate where the merge happens. Don't leave a gap between the fast gate and the actual merge (the lesson from the fast-gates split).
  • Few blocking gates, well-chosen. Sonar/DORA: too many conditions = fatigue. Prefer advisory + ratchet over a wall of blocking gates.

Part 5 — Recommended improvements (prioritized, compatible)

P0 — highest ROI, almost ready

  1. Mutation score ratchet (after the 1st nightly Stryker produces values). Key antidote against coverage-Goodhart; ~90% done.
  2. Close the fast-gates hole — add typecheck + impacted tests to quality.yml (PR→release).
  3. Branch-protection on main (owner setting) — boosts Scorecard, closes the DSOMM gap.

P1 — valuable 4. osv/oasdiff → blocking with the right scope — osv only CRITICAL+fixable (two-step like Trivy); oasdiff blocks breaking-changes. 5. require-tighten → blocking (end of cycle) — locks in metric gains. 6. ROI/timing review per-gate in ci-summary — find and prune slow/low-value gates.

P2 — diminishing returns 7. SLSA L3 — hermetic/reproducible builder (GitHub SLSA generator) if you want to move up from L2. 8. Committed CodeQL config + versioned semgrep — more control/reproducibility. 9. Per-PR DAST smoke — fast subset of schemathesis/promptfoo on highest-risk endpoints (not just nightly). 10. Flakiness dashboard + DORA metrics — ensure gates aren't eroding speed.

Part 6 — Concrete release lessons (gates to add in Phase 9)

This section records real incidents from release closures where a gate was missing, with concrete evidence and the proposed gate. Each item is a candidate for Part 5.

Lesson v3.8.27 (2026-06-17) — the "fast-gates hole" lets deterministic regressions reach release day

What happened. During the v3.8.27 /generate-release, the release PR (release/v3.8.27 → main) was the first execution of the full ci.yml matrix in the integrated cycle. Result: 12 failures at once — 3 deterministic tests + ~9 flakes/env. None were live product regressions, but all went unnoticed because cycle PRs enter release/** via the Fast QG (quality.yml), which does NOT run the full unit suite, nor pr-test-policy (test-masking), nor the full integration suite, nor schema parity checking. The 3 deterministic ones:

  1. Test outdated by UI change — permissions modal switch buttons declare button type: #4034 added a 4th switch (a11y type="button" maintained); the test's === 3 count became outdated. Static analysis should have caught this in the #4034 PR.
  2. Test outdated by packaging change — findMissingArtifactPaths ... root runtime files: dist/http-method-guard.cjs became a legitimate required-path; the test's expected list became outdated.
  3. Lossy modularization divergence (most serious) — settings schemas accept ... unprefixed toggle: the modularized updateSettingsSchema (schemas/settings.ts, created by #3988) diverged from the canonical one (settingsSchemas.ts): 45 fields vs 85 — 40 dropped + 6 divergent (qdrant*). It was dead-code (runtime uses the canonical one), so no live impact, but only a hand-written parity test caught it. #4030 restored 16 analogous drops from #3988/#3993, but this one slipped through.

Proposed gates (Phase 9):

  • G1 — Actually close the fast-gates hole (extends P0 #2). In quality.yml (PR→release/**), beyond typecheck + impacted tests, run pr-test-policy (test-masking) + the full deterministic unit suite (or at least the static/parity files, which are fast and non-flaky). This way, outdated tests and assert removal are caught in the PR that introduces them — not on release day. Keep integration/e2e out (slow/flaky), but the deterministic layer CANNOT stay only in PR→main.
  • G2 — Modularization parity gate (NEW, not covered today). A check that, for each symbol re-exported by a modularized barrel (src/shared/validation/schemas/*, providerRegistry modules, etc.), compares the shape (z.object keys, registry entries) against the canonical source and fails on divergence (dropped/extra field). Would have caught the 40-field drop from #3988 in that very PR. Generalizes the hand-written parity tests (which only exist where someone remembered to write them). Cheap: imports both and diffs Object.keys(shape).
  • G3 — Deterministic flake triage (support). LiveWS-startup and the integration-combo/breaker tests fail due to server timeout/cascade in CI (env), not logic. Mark these as known-flaky (quarantined with issue) so the release-PR red is only real signals, not noise masking deterministic regressions in the middle.

Principle: the gate has to run where the merge happens (already in "Cross-cutting principles"). The v3.8.27 incident shows this also applies to the deterministic test layer, not just lint/typecheck — otherwise the debt of outdated tests + lossy modularization only appears in PR→main, in batch, at the worst moment.

Sources (industry best practices)

Reference in New Issue View Git Blame Copy Permalink