ruvnet--ruflo/verification/README.md

Ruflo Verification System

A cross-platform, cryptographically-signed regression-protection toolkit. Every documented fix in ruflo gets attested by a SHA-256 + marker substring + Ed25519 signature. Anyone with the same git commit can re-derive the public key and verify it independently.

Three-layer regression protection:

1. Behavioral smoke tests (CI) — exercise user-visible failure modes
2. Cryptographic witness manifest (this folder) — attest fix presence
3. Temporal history (<os>/history.jsonl) — bisect when a regression was introduced

See docs/validation/README.md for the full architecture, ADR-102 for the smoke-harness pattern, and ADR-103 for the temporal-history layer.

---

Why this exists

Three regressions filed on 2026-05-08 (#1859, #1862, #1867) all passed unit tests on the broken commits but broke for users on first install. Unit tests verify code paths; users hit subprocess flag parsers, fresh npm install resolution, plugin/CLI version drift. The witness layer attests that every documented fix's load-bearing code is still present — even when the fix has no dedicated smoke test.

A regression that deletes the load-bearing line of a fix flips its markerVerified to false. The CI's witness-verify job blocks publish.

---

Folder layout

verification/
  README.md                  ← you are here
  witness-fixes.json         ← input config (OS-independent)
  mcp-tool-baseline.json     ← ADR-112 — monotone-decreasing tool-discoverability baseline
  results.md                 ← human-readable last-run report
  inventory.json             ← reserved for capability inventory
  linux/
    manifest.md.json         ← signed by Linux CI runner
    history.jsonl            ← Linux temporal log
  macos/
    manifest.md.json         ← signed by macOS CI runner
    history.jsonl            ← macOS temporal log
  windows/
    manifest.md.json         ← signed by Windows CI runner
    history.jsonl            ← Windows temporal log

Why per-OS? File hashes drift across platforms (LF vs CRLF normalization, path separators, prebuilt binary differences). Each OS produces its own truthful snapshot. The witness-verify CI job runs once per OS and writes/verifies that OS's bundle, catching platform-specific marker drift before it reaches users on that platform.

Why JSONL inside each OS folder? The witness use case is structured key-value retrieval (snapshot N's commit + fix statuses), not vector similarity search. JSONL is git-diff-friendly, append-cheap, and human-readable — the better fit. The folder layout adopts the RVF-style cognitive container concept (per-OS bundle of manifest + history + metadata); the file format inside is JSONL.

---

Tutorial — adopt the witness in your own project

1. Bootstrap

The witness toolkit lives in plugins/ruflo-core/scripts/witness/ — copy it into your project (or install the ruflo-core plugin).

# Initialize per-OS folders + sample fix template
node plugins/ruflo-core/scripts/witness/init.mjs --root .

# Single runtime dependency
npm i @noble/ed25519

Outputs:

verification/
  witness-fixes.json     # template — edit with your fixes
  <os>/
    manifest.md.json     # empty seed
    history.jsonl        # empty seed

2. Register your first fix

Open verification/witness-fixes.json:

{
  "fixes": [
    {
      "id": "MY-001",
      "desc": "Race condition in token refresh — early-return on in-flight refresh",
      "file": "dist/auth.js",
      "marker": "if (this._refreshing) return this._refreshing;"
    }
  ]
}

Choosing a marker is the load-bearing skill:

Bad marker	Why it fails
'function'	False-positives in any file with a function
'TODO'	Flaps as TODOs come and go
'fix'	Too generic

Good marker	Why it works
(await import('better-sqlite3')).default	Distinctive AND specific to the fix mechanism
(ctx.flags.file as string) || ctx.args[0]	The exact swap that fixes the bug
import * as bcrypt from 'bcryptjs'	Proves the bcrypt → bcryptjs migration is in dist

Pick a unique substring from the diff that's specifically created by the fix.

3. Generate the signed manifest

In ruflo's monorepo there's a wrapper that hard-codes paths:

node scripts/regen-witness.mjs

For external projects, call the plugin script with explicit paths:

node plugins/ruflo-core/scripts/witness/regen.mjs \
  --manifest verification/macos/manifest.md.json \
  --history  verification/macos/history.jsonl \
  --fixes    verification/witness-fixes.json

Output:

witness regen summary
─────────────────────
  gitCommit:    abc123def456…
  branch:       main
  issuedAt:     2026-05-09T00:00:00.000Z
  total fixes:  1  (was 0)
  verified:     1
  missing:      0
  new entries:  MY-001
written:  verification/macos/manifest.md.json
appended: verification/macos/history.jsonl

verified: N/N confirms every marker is present in its cited file.

4. Verify

node plugins/ruflo-core/scripts/witness/verify.mjs \
  --manifest verification/macos/manifest.md.json

Output:

Manifest signature:
  hash matches:                    yes
  public key reproducible:         yes
  Ed25519 signature valid:         yes

Summary: pass=1 drift=0 regressed=0 missing=0

If regressed > 0 or signatureValid: no, the manifest has been tampered with or a documented fix's marker has been deleted.

5. Investigate a regression

When CI reports F12 regressed:

node plugins/ruflo-core/scripts/witness/history.mjs \
  --history verification/macos/history.jsonl regressions

Output:

F12
  last pass:    a1b2c3d4  2026-05-07T14:23:11.000Z
  regressed at: 9f8e7d6c  2026-05-08T09:14:55.000Z

Then git log lastPass..regressedAt -- <file> collapses bisect from "many commits" to "the few in this 18-hour window."

6. Wire into CI

Add to .github/workflows/<your-pipeline>.yml:

witness-verify:
  strategy:
    matrix:
      os: [ubuntu-latest, macos-latest, windows-latest]
  runs-on: ${{ matrix.os }}
  steps:
    - uses: actions/checkout@v4
    - uses: actions/setup-node@v4
      with: { node-version: '22' }
    - run: npm i @noble/ed25519
    - shell: bash
      run: |
        case "$RUNNER_OS" in
          Linux)   OS_DIR=linux ;;
          macOS)   OS_DIR=macos ;;
          Windows) OS_DIR=windows ;;
        esac
        node plugins/ruflo-core/scripts/witness/verify.mjs \
          --manifest "verification/$OS_DIR/manifest.md.json" \
          --json > /tmp/result.json
        node -e "if (!require('/tmp/result.json').ok) process.exit(1)"

Block publish on failure. Every commit on every OS gets attested.

---

Features

Feature	Mechanism	What it catches
SHA-256 fingerprint per fix	Per-file hash recorded at issuance	Any change to the file
Marker substring	Distinctive code substring required to be present	Refactors that delete the fix
Ed25519 signature	Deterministic seed sha256(gitCommit + ':ruflo-witness/v1')	Manifest tampering, key-derivation drift
Reproducible public key	Anyone with the git commit re-derives the same pubkey	No committed private key needed
Per-OS bundles	verification/{linux,macos,windows}/	Platform-specific marker drift (CRLF, path separators)
Temporal history (JSONL)	One snapshot appended per regen	When a regression was introduced
Drift vs regression	sha256 changed but marker present → drift; marker missing → regression	False alarms suppressed; real regressions flagged
Status timeline per fix	history.mjs timeline --id F12	Flapping markers (too brittle) and persistent drift (too unstable)

---

Daily workflow cheat sheet

# Add a new fix
$EDITOR verification/witness-fixes.json
node scripts/regen-witness.mjs

# Verify the current tree
node plugins/ruflo-core/scripts/witness/verify.mjs \
  --manifest verification/macos/manifest.md.json

# Diff against the previous snapshot
node plugins/ruflo-core/scripts/witness/history.mjs \
  --history verification/macos/history.jsonl summary

# Find when a regression was introduced
node plugins/ruflo-core/scripts/witness/history.mjs \
  --history verification/macos/history.jsonl regressions

# Status timeline for one fix
node plugins/ruflo-core/scripts/witness/history.mjs \
  --history verification/macos/history.jsonl timeline --id F12

---

Schema

<os>/manifest.md.json

{
  "manifest": {
    "schema": "ruflo-witness/v1",
    "issuedAt": "2026-05-09T00:00:00.000Z",
    "gitCommit": "<full sha>",
    "branch": "main",
    "os": "macos",                              // matches the folder
    "releases": { "ruflo": "3.7.0-alpha.21", "@claude-flow/plugin-agent-federation": "1.0.0-alpha.15" },
    "summary": { "totalFixes": 102, "verified": 102, "missing": 0 },
    "fixes": [
      {
        "id": "F1",
        "desc": "...",
        "file": "v3/@claude-flow/cli/dist/...",
        "sha256": "<64 hex>",
        "marker": "<distinctive substring>",
        "markerVerified": true
      }
    ]
  },
  "integrity": {
    "manifestHashAlgo": "sha256",
    "manifestHash": "<64 hex of canonical manifest>",
    "signature": "<64-byte hex of Ed25519 signature over canonical manifest>",
    "signatureAlgo": "ed25519",
    "publicKey": "<32-byte hex>",
    "signature": "<64-byte hex>",
    "seedDerivation": "sha256(gitCommit + ':ruflo-witness/v1')"
  }
}

<os>/history.jsonl (one entry per line)

{
  "v": 1,
  "commit": "<full sha>",
  "issuedAt": "2026-05-09T00:00:00.000Z",
  "branch": "main",
  "os": "macos",
  "manifestHash": "<hex>",
  "summary": { "totalFixes": 82, "verified": 82, "missing": 0 },
  "fixes": {
    "F1":     { "sha256": "...", "markerVerified": true },
    "#1867":  { "sha256": "...", "markerVerified": true }
  }
}

witness-fixes.json (input, OS-independent)

{
  "fixes": [
    {
      "id": "MY-001",
      "desc": "human-readable description",
      "file": "path/relative/to/repo/root.js",
      "marker": "distinctive substring that must remain in the file"
    }
  ]
}

---

Anti-patterns

• Hand-editing manifest.md.json — breaks the signature. Always regenerate via regen.mjs.
• Markers that match generic patterns ('function', 'import', 'TODO') — false positives or flapping.
• Skipping the history append — without --history, you lose Layer 3 (regression introduction time).
• Committing the manifest without the history — they belong in the same commit.
• Cross-OS manifest editing — the manifest is signed for the OS that generated it. Don't copy a Linux manifest into the macOS folder.

---

How it integrates with the rest of ruflo's regression stack

Layer	Where	What it catches
Layer 1 — install smoke	v3/@claude-flow/memory/scripts/smoke-no-bsqlite.mjs + CI smoke-install-no-bsqlite	npm install failures on platforms without prebuilds
Layer 1 — hook smoke	plugins/ruflo-core/scripts/test-hooks.mjs + CI plugin-hooks-smoke	Plugin/CLI flag drift, parser ambiguity
Layer 1 — MCP protocol smoke	plugins/ruflo-core/scripts/test-mcp-protocol.mjs + CI mcp-protocol-smoke	HTTP MCP wire-format compliance
Layer 1 — memory-import smoke	plugins/ruflo-core/scripts/test-memory-import.mjs + CI memory-import-smoke	Memory_import_claude WSL path + key sanitization regressions
Layer 1 — tool-description audit	scripts/audit-tool-descriptions.mjs + CI tool-descriptions-audit (ADR-112)	Every MCP tool description must have "Use when … is wrong because …" guidance + length ≥ 80 + unique. Baseline at verification/mcp-tool-baseline.json monotone-decreasing
Layer 2 — witness manifest	verification/<os>/manifest.md.json + CI witness-verify	Documented fix marker disappearing
Layer 3 — temporal history	verification/<os>/history.jsonl + history.mjs	When a regression was introduced

All CI jobs gate publish. A regression in any layer blocks the release on the platform where it was caught.

---

References

• docs/validation/README.md — full architecture overview (in-repo)
• Public gist — external-friendly version
• ADR-102 — smoke harness pattern + flag-priority CLI convention
• ADR-103 — JSONL history layer + plugin distribution
• plugins/ruflo-core/skills/witness/SKILL.md — Claude Code skill workflow
• plugins/ruflo-core/agents/witness-curator.md — curator agent definition
• .github/workflows/v3-ci.yml — smoke-install-no-bsqlite, plugin-hooks-smoke, witness-verify jobs