chore: import upstream snapshot with attribution

scripts/README.md

@@ -0,0 +1,75 @@
+# scripts/
+
+Utility scripts bundled with the Headroom repo. Most are one-off operator
+tools; a few are runnable as part of development workflows.
+
+## Reproducing the reconnect storm
+
+`repro_codex_replay.py` reproduces the multi-agent Codex reconnect/retry storm
+against a local Headroom proxy (default `http://127.0.0.1:8787`). Use it to:
+
+- Regression-check that `/livez` stays responsive under a cold-start storm.
+- Empirically tune the Unit 4 pre-upstream semaphore default
+  (`HEADROOM_ANTHROPIC_PRE_UPSTREAM_CONCURRENCY`).
+- Exercise the Codex WS lifecycle + Anthropic HTTP path simultaneously
+  without needing to replay captured production traffic.
+
+### Run
+
+```bash
+# Default: 8 WS + 4 HTTP clients, 30s storm, p99 /livez must stay <= 500ms.
+python scripts/repro_codex_replay.py
+
+# Tighter budget, shorter run:
+python scripts/repro_codex_replay.py \
+    --url http://127.0.0.1:8787 \
+    --ws-clients 16 \
+    --anthropic-clients 8 \
+    --duration 60 \
+    --livez-threshold-ms 100
+
+# Dump the full summary as JSON for downstream tooling:
+python scripts/repro_codex_replay.py --json
+```
+
+Exit code:
+
+- `0` — warmup succeeded (or was skipped), storm ran for the requested
+  duration, and `/livez` p99 stayed under `--livez-threshold-ms`.
+- `1` — soft assertion failed, proxy unreachable, or unhandled exception.
+  Proxy-unreachable is detected and reported within ~5 seconds.
+
+### Fixtures
+
+The script loads two hand-crafted, fully synthetic JSON fixtures:
+
+- `scripts/fixtures/anthropic_replay_body.json` — shape of a large agent
+  reconnect replay `/v1/messages?beta=true` POST body.
+- `scripts/fixtures/codex_response_create_frame.json` — first Codex WS frame
+  with the `{"type": "response.create", "response": {...}}` envelope.
+
+Override via `--ws-frame-fixture` / `--anthropic-body-fixture` if you have
+captured traffic to replay instead.
+
+### Interpretation
+
+- `/livez p99` under threshold means the event loop is not starved during the
+  storm. If it rises with the semaphore unbounded
+  (`HEADROOM_ANTHROPIC_PRE_UPSTREAM_CONCURRENCY=10000`) and drops back under
+  the default, Unit 4's backpressure is working.
+- `Codex WS: opened` should equal `--ws-clients`. `response.completed`
+  typically stays low when upstream auth isn't configured locally — the goal
+  is handshake + relay wiring, not real upstream traffic.
+- `Anthropic HTTP: ok_2xx + non_2xx + timed_out + errors` should roughly equal
+  `attempted`. Sustained non-zero `timed_out` during the storm is the failure
+  signal the plan targets.
+
+A smoke test at `tests/test_scripts/test_repro_codex_replay_smoke.py`
+exercises the script against a mock FastAPI server on every PR.
+
+## Install scripts
+
+- `install.sh` — POSIX installer.
+- `install.ps1` — Windows PowerShell installer.
+
+These are generated by the release pipeline; edit with care.

scripts/audit_wheel_glibc_symbols.py

@@ -0,0 +1,217 @@
+#!/usr/bin/env python3
+"""Audit a manylinux Python wheel for glibc symbol references that exceed
+the wheel's declared manylinux ABI floor.
+
+This catches the bug class from issue #355: a wheel tagged
+`manylinux_2_28_x86_64` whose `_core.so` references `__isoc23_strtoll`
+(introduced in glibc 2.38) — the link succeeds because the manylinux
+build host has 2.38+, but every end user with glibc < 2.38 sees an
+`ImportError: undefined symbol` at `import headroom._core`.
+
+How it works
+------------
+
+1. Parse the wheel filename to extract the manylinux tag
+   (`manylinux_2_28` → glibc floor 2.28).
+2. Run `objdump -T` (or `nm -D`) on every `.so` inside the wheel.
+3. For every `UND` symbol, check whether it's allowed at the glibc
+   floor. Allowed:
+   - Symbols with a `GLIBC_x.y` version tag where `x.y <= floor`.
+   - Symbols defined by us locally (i.e. NOT marked `UND`).
+4. Reject any `UND` symbol that's:
+   - Tagged with a GLIBC version > floor.
+   - Or in the `__isoc23_*` family (no version tag, but introduced
+     in glibc 2.38 — special-cased).
+   - Or in any other known "introduced after floor" symbol list
+     (extensible).
+
+Exit 0 = wheel is portable. Exit 1 = wheel will break on some users.
+
+Usage
+-----
+
+    scripts/audit_wheel_glibc_symbols.py path/to/headroom_ai-*.whl
+
+Run on Linux only — `objdump` from binutils is the audit tool. macOS's
+default `objdump` is llvm-objdump, also works on ELF.
+"""
+
+from __future__ import annotations
+
+import argparse
+import re
+import shutil
+import subprocess
+import sys
+import tempfile
+import zipfile
+from pathlib import Path
+
+# Symbols introduced after specific glibc versions, beyond what
+# `auditwheel` already checks (auditwheel relies on the GLIBC_x.y
+# version tag baked into versioned symbols; the entries below are
+# either tagless or family-tagged, neither of which auditwheel
+# catches). Add here as new bug classes surface.
+#
+# Each entry is `(symbol_name_prefix, min_glibc_version_introduced, justification_url)`.
+# `startswith(prefix)` is used to match — for a single symbol use the
+# full name as the prefix (no other symbol starts with it).
+POST_FLOOR_SYMBOLS = [
+    # C23 strtol family. Issue #355.
+    (
+        "__isoc23_",
+        (2, 38),
+        "https://sourceware.org/glibc/wiki/Release/2.38",
+    ),
+    # Single-threaded fast-path flag read by libstdc++ (gcc 11+).
+    # Caught by the X1 smoke gate on PR #396 (X2 dry-run) on the
+    # manylinux_2_28 floor entry — the audit had let the wheel
+    # through because it didn't know about this symbol.
+    (
+        "__libc_single_threaded",
+        (2, 32),
+        "https://sourceware.org/glibc/wiki/Release/2.32",
+    ),
+]
+
+
+def parse_manylinux_floor(wheel_filename: str) -> tuple[int, int] | None:
+    """Extract glibc floor from a manylinux wheel filename.
+
+    `headroom_ai-0.20.26-cp312-cp312-manylinux_2_28_x86_64.whl` → (2, 28).
+    Returns None for non-manylinux wheels (macOS, Windows, sdist).
+    """
+    m = re.search(r"manylinux_(\d+)_(\d+)_", wheel_filename)
+    if m:
+        return (int(m.group(1)), int(m.group(2)))
+    # `manylinux2014_x86_64` is the legacy alias for manylinux_2_17.
+    if "manylinux2014_" in wheel_filename:
+        return (2, 17)
+    if "manylinux1_" in wheel_filename:
+        return (2, 5)
+    return None
+
+
+def list_undef_symbols(so_path: Path) -> list[tuple[str, str]]:
+    """Return [(symbol_name, glibc_version_or_empty), ...] for every
+    UND (undefined) dynamic symbol in `so_path`.
+    """
+    objdump = shutil.which("objdump") or shutil.which("llvm-objdump")
+    if not objdump:
+        raise RuntimeError(
+            "neither `objdump` nor `llvm-objdump` is on PATH; "
+            "install binutils (Linux) or LLVM (macOS) to run this audit"
+        )
+    out = subprocess.run(
+        [objdump, "-T", str(so_path)],
+        check=True,
+        capture_output=True,
+        text=True,
+    ).stdout
+    found = []
+    for line in out.splitlines():
+        # `objdump -T` lines: `address SECTION ... NAME` where SECTION
+        # contains `*UND*` for undefined references and a versioned
+        # symbol name like `__isoc23_strtoll@GLIBC_2.38` or unversioned.
+        if "*UND*" not in line:
+            continue
+        # The last whitespace-separated token is the (versioned) symbol.
+        token = line.split()[-1]
+        if "@" in token:
+            name, _, ver = token.partition("@")
+            # `@@` indicates the default version; strip the second `@`.
+            ver = ver.lstrip("@")
+        else:
+            name, ver = token, ""
+        found.append((name, ver))
+    return found
+
+
+def glibc_version_from_token(ver: str) -> tuple[int, int] | None:
+    """`GLIBC_2.28` → (2, 28). Returns None for non-GLIBC tokens."""
+    m = re.match(r"^GLIBC_(\d+)\.(\d+)", ver)
+    if m:
+        return (int(m.group(1)), int(m.group(2)))
+    return None
+
+
+def audit_so(so_path: Path, floor: tuple[int, int]) -> list[str]:
+    """Return a list of human-readable violation strings."""
+    violations: list[str] = []
+    for name, ver in list_undef_symbols(so_path):
+        v = glibc_version_from_token(ver)
+        if v is not None and v > floor:
+            violations.append(
+                f"  {name}@{ver} requires glibc {v[0]}.{v[1]} > floor {floor[0]}.{floor[1]}"
+            )
+            continue
+        # Versionless symbols: cross-check the post-floor list.
+        for prefix, introduced, url in POST_FLOOR_SYMBOLS:
+            if name.startswith(prefix) and introduced > floor:
+                violations.append(
+                    f"  {name} (no version tag, introduced in glibc "
+                    f"{introduced[0]}.{introduced[1]} > floor "
+                    f"{floor[0]}.{floor[1]} — see {url})"
+                )
+                break
+    return violations
+
+
+def main() -> int:
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("wheel", type=Path, help="path to the .whl to audit")
+    args = parser.parse_args()
+
+    wheel: Path = args.wheel
+    if not wheel.exists():
+        print(f"ERROR: wheel not found: {wheel}", file=sys.stderr)
+        return 1
+
+    floor = parse_manylinux_floor(wheel.name)
+    if floor is None:
+        print(
+            f"OK: {wheel.name} is not a manylinux wheel; nothing to audit "
+            "(macOS / Windows / sdist run on a different runtime ABI)."
+        )
+        return 0
+
+    print(f"Auditing {wheel.name} (glibc floor: {floor[0]}.{floor[1]})")
+    with tempfile.TemporaryDirectory() as td:
+        td_path = Path(td)
+        with zipfile.ZipFile(wheel) as zf:
+            so_members = [m for m in zf.namelist() if m.endswith(".so")]
+            if not so_members:
+                print(
+                    f"WARN: {wheel.name} contains no .so files; "
+                    "nothing to audit (pure-Python wheel?)"
+                )
+                return 0
+            zf.extractall(td_path, members=so_members)
+
+        all_violations: list[tuple[str, list[str]]] = []
+        for so_rel in so_members:
+            so_path = td_path / so_rel
+            violations = audit_so(so_path, floor)
+            if violations:
+                all_violations.append((so_rel, violations))
+
+    if not all_violations:
+        print(f"OK: all .so files in {wheel.name} are within glibc {floor[0]}.{floor[1]}.")
+        return 0
+
+    print(f"\nFAIL: {wheel.name} references symbols above its glibc floor:")
+    for so_name, viols in all_violations:
+        print(f"\n  {so_name}:")
+        for v in viols:
+            print(f"  {v}")
+    print(
+        "\nThis wheel will fail to import on end-user systems with the "
+        "older glibc. Fix the build (or add a compat shim) before "
+        "publishing to PyPI. See issue #355 for the canonical example "
+        "and `crates/headroom-py/glibc_compat.c` for the shim pattern."
+    )
+    return 1
+
+
+if __name__ == "__main__":
+    sys.exit(main())

scripts/build_rust_extension.sh

@@ -0,0 +1,58 @@
+#!/usr/bin/env bash
+# Build + install the Rust extension (headroom._core) into the active venv.
+#
+# With single-wheel architecture (post-#355), `pip install -e .` invokes
+# maturin (declared in pyproject.toml's `[build-system]`) which builds the
+# Rust extension and installs it into site-packages alongside the Python
+# source. Earlier versions of this script symlinked the .so into the
+# in-tree `headroom/` directory because the dual-package layout left the
+# .so in `crates/headroom-py/python/headroom/`. That dance is no longer
+# needed — maturin places the .so directly in the editable install's
+# overlay and Python's import system finds it.
+#
+# Idempotent. Safe to run repeatedly.
+
+set -euo pipefail
+
+cd "$(dirname "$0")/.."
+
+log() {
+    printf '[build_rust_extension] %s\n' "$*" >&2
+}
+
+fail() {
+    printf '[build_rust_extension] error: %s\n' "$*" >&2
+    exit 1
+}
+
+# Pre-flight: a venv should be active. The install would otherwise write
+# into the system Python.
+if [[ -z "${VIRTUAL_ENV:-}" ]]; then
+    log "warning: VIRTUAL_ENV is unset; pip will install into the system Python."
+    log "         If that is not what you want, abort and 'source .venv/bin/activate' first."
+fi
+
+if ! command -v cargo >/dev/null 2>&1; then
+    fail "cargo not found on PATH. Install Rust toolchain (rustup) first."
+fi
+
+# Build + install in one shot. The `[build-system] build-backend = "maturin"`
+# in pyproject.toml means pip drives maturin under the hood. The resulting
+# wheel contains both the Python source and the compiled `headroom/_core.so`,
+# and pip installs them into the editable overlay together.
+log "pip install -e . (drives maturin via build-backend)"
+python -m pip install -e . || fail "pip install -e . failed (see output above)"
+
+# End-to-end verification — same shape as Phase A0's startup smoke check.
+log "verifying \`from headroom._core import DiffCompressor, SmartCrusher\`"
+python -c '
+import sys
+try:
+    from headroom._core import DiffCompressor, SmartCrusher
+except Exception as exc:
+    print(f"verify FAILED: {type(exc).__name__}: {exc}", file=sys.stderr)
+    sys.exit(1)
+print(f"verify OK: DiffCompressor={DiffCompressor!r}, SmartCrusher={SmartCrusher!r}")
+' || fail "import verification failed (see above)"
+
+log "headroom._core build + install + verify: OK"

scripts/changelog-gen.py

@@ -0,0 +1,203 @@
+#!/usr/bin/env python3
+"""Generate changelog from conventional commits."""
+
+from __future__ import annotations
+
+import argparse
+import re
+import subprocess
+from datetime import date
+from pathlib import Path
+from typing import NamedTuple
+
+ROOT = Path(__file__).parent.parent
+
+COMMIT_PATTERN = re.compile(
+    r"^(feat|fix|ci|chore|perf|refactor|docs|style|test)(\(.+\))?(!)?:\s*(.+)$"
+)
+BREAKING_CHANGE_PATTERN = re.compile(r"^BREAKING CHANGE:\s*(.+)$", re.MULTILINE)
+COMMIT_ENTRY_PATTERN = re.compile(r"^(.+?)(?:\n(.+))?\|(\w+)$", re.MULTILINE)
+FIELD_SEP = "\x1f"
+RECORD_SEP = "\x1e"
+GIT_LOG_FORMAT = "%s%x1f%b%x1f%h%x1e"
+
+TYPE_LABELS: dict[str, str] = {
+    "feat": "Features",
+    "fix": "Bug Fixes",
+    "ci": "CI/CD",
+    "chore": "Chores",
+    "perf": "Performance",
+    "refactor": "Refactors",
+    "docs": "Documentation",
+    "style": "Styles",
+    "test": "Tests",
+    "other": "Other Changes",
+}
+
+
+class ParsedCommit(NamedTuple):
+    type: str
+    scope: str | None
+    breaking: bool
+    message: str
+    hash: str
+
+
+def iter_commit_entries(log_output: str) -> list[tuple[str, str, str]]:
+    """Split raw git log output into (subject, body, hash) tuples."""
+
+    if not log_output.strip():
+        return []
+
+    if RECORD_SEP in log_output and FIELD_SEP in log_output:
+        entries: list[tuple[str, str, str]] = []
+        for raw_entry in log_output.split(RECORD_SEP):
+            if not raw_entry:
+                continue
+            if FIELD_SEP not in raw_entry:
+                continue
+            subject, body_and_hash = raw_entry.split(FIELD_SEP, 1)
+            if FIELD_SEP not in body_and_hash:
+                continue
+            body, commit_hash = body_and_hash.rsplit(FIELD_SEP, 1)
+            entries.append((subject.strip(), body.strip(), commit_hash.strip()))
+        return entries
+
+    return [
+        (
+            match.group(1).strip(),
+            (match.group(2) or "").strip(),
+            match.group(3).strip(),
+        )
+        for match in COMMIT_ENTRY_PATTERN.finditer(log_output)
+    ]
+
+
+def get_merge_summary(subject: str, body: str) -> str:
+    """Return the first meaningful summary line for a merge commit."""
+
+    if not subject.startswith("Merge "):
+        return ""
+
+    for line in body.splitlines():
+        stripped = line.strip()
+        if stripped:
+            return stripped
+    return ""
+
+
+def parse_commits(log_output: str) -> list[ParsedCommit]:
+    """Parse git log output into structured commits."""
+
+    commits: list[ParsedCommit] = []
+
+    for subject, body, commit_hash in iter_commit_entries(log_output):
+        is_breaking = bool(BREAKING_CHANGE_PATTERN.search(body))
+        merge_summary = get_merge_summary(subject, body)
+        candidates = [subject]
+        if merge_summary:
+            candidates.insert(0, merge_summary)
+
+        for candidate in candidates:
+            commit_match = COMMIT_PATTERN.match(candidate)
+            if not commit_match:
+                continue
+
+            scope = commit_match.group(2)
+            if scope:
+                scope = scope[1:-1]
+            commits.append(
+                ParsedCommit(
+                    type=commit_match.group(1),
+                    scope=scope,
+                    breaking=is_breaking or bool(commit_match.group(3)),
+                    message=commit_match.group(4),
+                    hash=commit_hash,
+                )
+            )
+            break
+        else:
+            fallback_message = merge_summary or subject
+            if not fallback_message or fallback_message.startswith("Merge "):
+                continue
+            commits.append(
+                ParsedCommit(
+                    type="other",
+                    scope=None,
+                    breaking=is_breaking,
+                    message=fallback_message,
+                    hash=commit_hash,
+                )
+            )
+
+    return commits
+
+
+def generate_changelog(version: str, commits: list[ParsedCommit]) -> str:
+    """Generate markdown changelog from parsed commits."""
+    today = date.today().isoformat()
+    lines = [f"## [{version}] - {today}", ""]
+
+    # Collect breaking changes
+    breaking_commits = [c for c in commits if c.breaking]
+    if breaking_commits:
+        lines.append("### Breaking Changes")
+        for commit in breaking_commits:
+            if commit.scope:
+                lines.append(f"- **{commit.scope}**: {commit.message} ({commit.hash})")
+            else:
+                lines.append(f"- {commit.message} ({commit.hash})")
+        lines.append("")
+
+    # Group by type
+    by_type: dict[str, list[ParsedCommit]] = {}
+    for commit in commits:
+        by_type.setdefault(commit.type, []).append(commit)
+
+    for commit_type, label in TYPE_LABELS.items():
+        type_commits = by_type.get(commit_type, [])
+        if not type_commits:
+            continue
+        lines.append(f"### {label}")
+        for commit in type_commits:
+            if commit.scope:
+                lines.append(f"- **{commit.scope}**: {commit.message} ({commit.hash})")
+            else:
+                lines.append(f"- {commit.message} ({commit.hash})")
+        lines.append("")
+
+    return "\n".join(lines) + "\n"
+
+
+def run_git_log(since: str | None, cwd: Path) -> str:
+    """Run git log command and return output."""
+    cmd = ["git", "log", "--first-parent", f"--pretty=format:{GIT_LOG_FORMAT}"]
+    if since:
+        cmd.append(f"{since}..HEAD")
+    else:
+        cmd.append("HEAD")
+    result = subprocess.run(cmd, cwd=cwd, capture_output=True, text=True)
+    return result.stdout
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser(description="Generate changelog from conventional commits")
+    parser.add_argument("--version", required=True, help="Version number (e.g., 0.6.0)")
+    parser.add_argument("--since", help="Starting tag (exclusive)")
+    parser.add_argument("--dry-run", action="store_true", help="Print to stdout instead of writing")
+    args = parser.parse_args()
+
+    log_output = run_git_log(args.since, ROOT)
+    commits = parse_commits(log_output)
+    changelog = generate_changelog(args.version, commits)
+
+    if args.dry_run:
+        print(changelog)
+    else:
+        output_path = ROOT / ".changelog.md"
+        output_path.write_text(changelog, encoding="utf-8")
+        print(f"Changelog written to {output_path}")
+
+
+if __name__ == "__main__":
+    main()

scripts/eval_output_shaper.py

@@ -0,0 +1,235 @@
+"""Live before/after eval for the output shaper.
+
+Sends the SAME request to the Anthropic API twice — once as a client would
+send it (baseline) and once after `shape_request` rewrites it (exactly what
+the proxy forwards upstream) — and compares `usage.output_tokens`, which
+includes thinking tokens.
+
+Scenario A (verbosity steering): a complex code-review ask. Baseline vs
+verbosity levels 2 and 3.
+
+Scenario B (effort routing): an agentic transcript whose last message is a
+clean tool_result (mechanical continuation) with `output_config.effort` set
+to "xhigh" the way Claude Code pins it. The shaper lowers effort to "low"
+for this turn only.
+
+Usage:
+    source .venv/bin/activate && python scripts/eval_output_shaper.py
+Requires ANTHROPIC_API_KEY in the environment or in ./.env.
+"""
+
+from __future__ import annotations
+
+import copy
+import os
+import statistics
+import sys
+from pathlib import Path
+from typing import Any
+
+sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
+
+import anthropic  # noqa: E402
+
+from headroom.proxy.output_shaper import OutputShaperSettings, shape_request  # noqa: E402
+
+MODEL = "claude-opus-4-8"
+TRIALS = 2
+
+BUGGY_CODE = '''\
+import threading
+from collections import OrderedDict
+
+class TTLCache:
+    """LRU cache with per-entry TTL."""
+
+    def __init__(self, max_size=128, ttl=300):
+        self.max_size = max_size
+        self.ttl = ttl
+        self._store = OrderedDict()
+        self._lock = threading.Lock()
+
+    def get(self, key, now):
+        entry = self._store.get(key)
+        if entry is None:
+            return None
+        value, expires_at = entry
+        if now > expires_at:
+            del self._store[key]
+            return None
+        self._store.move_to_end(key)
+        return value
+
+    def put(self, key, value, now):
+        with self._lock:
+            if key in self._store:
+                self._store.move_to_end(key)
+            self._store[key] = (value, now + self.ttl)
+            if len(self._store) > self.max_size:
+                self._store.popitem(last=True)
+
+    def cleanup(self, now):
+        for key, (_, expires_at) in self._store.items():
+            if now > expires_at:
+                del self._store[key]
+'''
+
+
+def load_env() -> None:
+    env_path = Path(__file__).resolve().parent.parent / ".env"
+    if not env_path.exists() or os.environ.get("ANTHROPIC_API_KEY"):
+        return
+    for line in env_path.read_text().splitlines():
+        line = line.strip()
+        if line and not line.startswith("#") and "=" in line:
+            key, _, value = line.partition("=")
+            value = value.strip().strip("'\"")
+            os.environ.setdefault(key.strip(), value)
+
+
+def scenario_a_body() -> dict[str, Any]:
+    """Complex single-turn ask — exercises verbosity steering."""
+    return {
+        "model": MODEL,
+        "max_tokens": 8000,
+        "system": "You are a senior Python engineer doing code review.",
+        "messages": [
+            {
+                "role": "user",
+                "content": (
+                    "Review this cache implementation. Identify every bug and "
+                    "thread-safety issue, then show how to fix each one:\n\n"
+                    f"```python\n{BUGGY_CODE}```"
+                ),
+            }
+        ],
+    }
+
+
+def scenario_b_body() -> dict[str, Any]:
+    """Agentic mechanical continuation — exercises effort routing."""
+    return {
+        "model": MODEL,
+        "max_tokens": 8000,
+        "thinking": {"type": "adaptive"},
+        "output_config": {"effort": "xhigh"},
+        "system": (
+            "You are a coding agent. Use the Read tool to inspect files, then "
+            "report findings concisely."
+        ),
+        "tools": [
+            {
+                "name": "Read",
+                "description": "Read a file from the repository.",
+                "input_schema": {
+                    "type": "object",
+                    "properties": {"path": {"type": "string"}},
+                    "required": ["path"],
+                },
+            }
+        ],
+        "messages": [
+            {
+                "role": "user",
+                "content": "Check whether cache.py has thread-safety issues.",
+            },
+            {
+                "role": "assistant",
+                "content": [
+                    {"type": "text", "text": "Reading cache.py first."},
+                    {
+                        "type": "tool_use",
+                        "id": "toolu_eval_01",
+                        "name": "Read",
+                        "input": {"path": "cache.py"},
+                    },
+                ],
+            },
+            {
+                "role": "user",
+                "content": [
+                    {
+                        "type": "tool_result",
+                        "tool_use_id": "toolu_eval_01",
+                        "content": BUGGY_CODE,
+                    }
+                ],
+            },
+        ],
+    }
+
+
+def run(client: anthropic.Anthropic, body: dict[str, Any]) -> dict[str, int]:
+    # The installed SDK may predate output_config as a typed kwarg; the API
+    # accepts it either way, so pass it through extra_body.
+    body = dict(body)
+    extra_body = None
+    if "output_config" in body:
+        extra_body = {"output_config": body.pop("output_config")}
+    response = client.messages.create(**body, extra_body=extra_body)
+    if response.stop_reason == "refusal":
+        raise RuntimeError("request was refused by safety classifiers")
+    return {
+        "input_tokens": response.usage.input_tokens,
+        "output_tokens": response.usage.output_tokens,
+    }
+
+
+def main() -> int:
+    load_env()
+    if not os.environ.get("ANTHROPIC_API_KEY"):
+        print("ANTHROPIC_API_KEY not found (env or .env)", file=sys.stderr)
+        return 1
+    client = anthropic.Anthropic()
+    which = sys.argv[1].upper() if len(sys.argv) > 1 else "ALL"
+
+    conditions: list[tuple[str, str, dict[str, Any]]] = []
+
+    if which in ("A", "ALL"):
+        # Scenario A: baseline vs steered.
+        conditions.append(("A:verbosity", "baseline", scenario_a_body()))
+        for level in (2, 3):
+            body = scenario_a_body()
+            shape_request(body, OutputShaperSettings(enabled=True, verbosity_level=level))
+            conditions.append(("A:verbosity", f"shaped L{level}", body))
+
+    if which in ("B", "ALL"):
+        # Scenario B: baseline (effort=xhigh) vs shaped (effort routed to low).
+        conditions.append(("B:effort-routing", "baseline xhigh", scenario_b_body()))
+        body = scenario_b_body()
+        result = shape_request(body, OutputShaperSettings(enabled=True, verbosity_level=0))
+        assert body["output_config"]["effort"] == "low", result.labels
+        conditions.append(("B:effort-routing", "shaped low", body))
+
+    print(f"model={MODEL}  trials={TRIALS}\n")
+    print(f"{'scenario':<18} {'condition':<16} {'trial':<6} {'in_tok':>7} {'out_tok':>8}")
+    print("-" * 60)
+
+    results: dict[tuple[str, str], list[int]] = {}
+    for scenario, condition, body in conditions:
+        for trial in range(1, TRIALS + 1):
+            usage = run(client, copy.deepcopy(body))
+            results.setdefault((scenario, condition), []).append(usage["output_tokens"])
+            print(
+                f"{scenario:<18} {condition:<16} {trial:<6} "
+                f"{usage['input_tokens']:>7} {usage['output_tokens']:>8}"
+            )
+
+    print("\n=== Summary (mean output tokens, reduction vs baseline) ===")
+    baselines: dict[str, float] = {}
+    for (scenario, condition), outs in results.items():
+        if condition.startswith("baseline"):
+            baselines[scenario] = statistics.mean(outs)
+    for (scenario, condition), outs in results.items():
+        mean = statistics.mean(outs)
+        base = baselines.get(scenario, 0)
+        if condition.startswith("baseline") or not base:
+            print(f"{scenario:<18} {condition:<16} {mean:>8.0f}  (baseline)")
+        else:
+            pct = (base - mean) / base * 100
+            print(f"{scenario:<18} {condition:<16} {mean:>8.0f}  ({pct:+.1f}% vs baseline)")
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

scripts/export_kompress_v2_onnx.py

@@ -0,0 +1,267 @@
+#!/usr/bin/env python
+"""Export a Kompress PyTorch checkpoint to ONNX INT8 for Headroom's light path.
+
+Why this exists
+---------------
+Headroom's ``[proxy]`` extra ships ``onnxruntime`` but **not** torch — the
+proxy runs Kompress text compression on ONNX Runtime alone. The loader
+(``headroom/transforms/kompress_compressor.py``) downloads
+``onnx/kompress-int8.onnx`` from the model repo and runs it through
+``_OnnxModel``, which expects a single graph output named ``final_scores``
+(per-token importance in ``[0, 1]``, kept when ``> 0.5``).
+
+``chopratejas/kompress-v2-base`` ships only PyTorch weights
+(``model.safetensors`` / ``merged.pt``) — no ONNX. So pointing Headroom at v2
+without an ONNX export would silently force the heavier ``[ml]`` (torch) path
+on every proxy install. This script reproduces v1's exact ONNX contract from
+the v2 PyTorch checkpoint, so a default swap stays zero-cost for light installs.
+
+The model is a *custom* dual-head ModernBERT (token classifier + span CNN), not
+a standard HF architecture, so ``optimum-cli export onnx`` does not apply — we
+trace the real module from ``kompress_compressor._get_model_class()``.
+
+Requires
+--------
+    pip install headroom-ai[ml] onnxruntime    # torch + transformers + onnxruntime
+
+Usage
+-----
+    # Convert + verify locally (writes onnx/kompress-int8.onnx):
+    python scripts/export_kompress_v2_onnx.py --model-id chopratejas/kompress-v2-base
+
+    # Convert, verify, and upload back to the HF repo (needs `huggingface-cli login`):
+    python scripts/export_kompress_v2_onnx.py --model-id chopratejas/kompress-v2-base --upload
+"""
+
+from __future__ import annotations
+
+import argparse
+import logging
+import sys
+from pathlib import Path
+
+logging.basicConfig(level=logging.INFO, format="%(levelname)s %(message)s")
+logger = logging.getLogger("export_kompress_v2_onnx")
+
+# ModernBERT encoder + tokenizer base (must match training and the loader).
+BASE_MODEL = "answerdotai/ModernBERT-base"
+DEFAULT_MODEL_ID = "chopratejas/kompress-v2-base"
+
+
+def _build_core(model_id: str):
+    """Instantiate HeadroomCompressorModel and load the merged v2 weights.
+
+    The v2 repo's ``model.safetensors`` is the *unmerged* PEFT structure
+    (``encoder.base_model.model...`` with separate ``base_layer`` + LoRA
+    adapters), which does not map onto ``HeadroomCompressorModel``. The
+    canonical artifact is ``merged.pt`` — a structured checkpoint with already
+    LoRA-merged sub-state-dicts:
+
+        {"encoder_state_dict", "token_head_state_dict",
+         "span_conv_state_dict", "config", "checkpoint_kind"}
+
+    Each loads cleanly (0 missing / 0 unexpected) into the encoder + heads.
+    """
+    import torch
+    from huggingface_hub import hf_hub_download
+
+    from headroom.transforms.kompress_compressor import _get_model_class
+
+    ckpt_path = hf_hub_download(model_id, "merged.pt")
+    ckpt = torch.load(ckpt_path, map_location="cpu")
+    for key in ("encoder_state_dict", "token_head_state_dict", "span_conv_state_dict"):
+        if key not in ckpt:
+            raise RuntimeError(
+                f"merged.pt missing '{key}'. Found: {sorted(ckpt)}. "
+                "This script targets the v2 'merged' checkpoint format."
+            )
+
+    core = _get_model_class()(model_name=BASE_MODEL)
+
+    def _strict_load(module, sd, label: str) -> None:
+        missing, unexpected = module.load_state_dict(sd, strict=False)
+        if missing or unexpected:
+            raise RuntimeError(
+                f"{label}: state_dict mismatch (missing={list(missing)[:5]}, "
+                f"unexpected={list(unexpected)[:5]}). Architecture drifted from the checkpoint."
+            )
+        logger.info("  %s loaded (%d tensors, exact match)", label, len(sd))
+
+    logger.info("Loading merged.pt (checkpoint_kind=%s)", ckpt.get("checkpoint_kind"))
+    _strict_load(core.encoder, ckpt["encoder_state_dict"], "encoder")
+    _strict_load(core.token_head, ckpt["token_head_state_dict"], "token_head")
+    _strict_load(core.span_conv, ckpt["span_conv_state_dict"], "span_conv")
+
+    core.eval()
+    return core
+
+
+def _export_wrapper(core):
+    """Wrap the dual head so forward() returns `final_scores` (== get_scores)."""
+    import torch
+    import torch.nn as nn
+
+    class ExportWrapper(nn.Module):
+        def __init__(self, inner):
+            super().__init__()
+            self.inner = inner
+
+        def forward(self, input_ids, attention_mask):  # noqa: ANN001
+            hidden = self.inner.encoder(input_ids, attention_mask=attention_mask).last_hidden_state
+            token_probs = torch.softmax(self.inner.token_head(hidden), dim=-1)[:, :, 1]
+            span_scores = self.inner.span_conv(hidden.transpose(1, 2)).squeeze(1)
+            return token_probs * (0.5 + 0.5 * span_scores)
+
+    return ExportWrapper(core).eval()
+
+
+def export(model_id: str, out_path: Path, opset: int, precision: str) -> None:
+    import numpy as np
+    import torch
+
+    core = _build_core(model_id)
+    wrapper = _export_wrapper(core)
+
+    out_path.parent.mkdir(parents=True, exist_ok=True)
+
+    # fp32 path: trace straight to the final artifact (lossless — verified 100%
+    # keep-decision agreement with PyTorch). int8 path: trace to a temp fp32
+    # graph, then dynamically quantize into the final artifact.
+    trace_target = out_path if precision == "fp32" else out_path.with_name("kompress-fp32-tmp.onnx")
+
+    dummy_ids = torch.randint(0, 1000, (1, 64), dtype=torch.long)
+    dummy_mask = torch.ones((1, 64), dtype=torch.long)
+
+    logger.info("Tracing → ONNX (opset %d, precision=%s) ...", opset, precision)
+    with torch.no_grad():
+        torch.onnx.export(
+            wrapper,
+            (dummy_ids, dummy_mask),
+            str(trace_target),
+            input_names=["input_ids", "attention_mask"],
+            output_names=["final_scores"],
+            dynamic_axes={
+                "input_ids": {0: "batch", 1: "seq"},
+                "attention_mask": {0: "batch", 1: "seq"},
+                "final_scores": {0: "batch", 1: "seq"},
+            },
+            opset_version=opset,
+            do_constant_folding=True,
+            dynamo=False,
+        )
+
+    if precision == "int8":
+        from onnxruntime.quantization import QuantType, quantize_dynamic
+
+        logger.info("INT8 dynamic quantization (MatMul only) → %s", out_path)
+        # Restrict to MatMul: the encoder's linear layers carry ~all the weight
+        # mass and ORT's CPU provider implements MatMulInteger. Quantizing the
+        # tiny span_conv Conv1d layers would emit ConvInteger, which ORT CPU
+        # cannot run. per_channel recovers transformer accuracy at the 0.5 boundary.
+        quantize_dynamic(
+            str(trace_target),
+            str(out_path),
+            weight_type=QuantType.QInt8,
+            op_types_to_quantize=["MatMul"],
+            per_channel=True,
+        )
+        trace_target.unlink(missing_ok=True)
+
+    _verify(model_id, core, out_path, np, torch)
+
+
+def _verify(model_id: str, core, out_path: Path, np, torch) -> None:
+    """Compare ONNX scores against PyTorch get_scores on a real tokenized sample."""
+    import onnxruntime as ort
+    from transformers import AutoTokenizer
+
+    tok = AutoTokenizer.from_pretrained(BASE_MODEL)
+    sample = (
+        "The proxy compresses tool outputs before they reach the model. "
+        "Errors and stack traces should survive; boilerplate should not. "
+    ) * 6
+    words = sample.split()
+    enc = tok(
+        words,
+        is_split_into_words=True,
+        truncation=True,
+        max_length=512,
+        padding=True,
+        return_tensors="pt",
+    )
+
+    with torch.no_grad():
+        torch_scores = core.get_scores(enc["input_ids"], enc["attention_mask"])[0].cpu().numpy()
+
+    sess = ort.InferenceSession(str(out_path), providers=["CPUExecutionProvider"])
+    onnx_scores = sess.run(
+        ["final_scores"],
+        {
+            "input_ids": enc["input_ids"].numpy().astype(np.int64),
+            "attention_mask": enc["attention_mask"].numpy().astype(np.int64),
+        },
+    )[0][0]
+
+    max_abs = float(np.max(np.abs(torch_scores - onnx_scores)))
+    keep_torch = torch_scores > 0.5
+    keep_onnx = onnx_scores > 0.5
+    agree = float((keep_torch == keep_onnx).mean())
+    logger.info(
+        "Verify: max|Δscore|=%.4f  keep-decision agreement=%.1f%% (fp32 ~100%%, int8 ~98-100%%)",
+        max_abs,
+        agree * 100,
+    )
+    if agree < 0.98:
+        logger.warning(
+            "Keep-decision agreement below 98%% — for fp32 this means a tracing "
+            "problem; for int8 consider per_channel/fp32. Inspect before publishing."
+        )
+
+
+def upload(model_id: str, out_path: Path) -> None:
+    from huggingface_hub import upload_file
+
+    # Publish under onnx/<artifact filename> so int8 and fp32 can coexist.
+    repo_path = f"onnx/{out_path.name}"
+    logger.info("Uploading %s → %s:%s", out_path, model_id, repo_path)
+    upload_file(
+        path_or_fileobj=str(out_path),
+        path_in_repo=repo_path,
+        repo_id=model_id,
+        commit_message="Add ONNX export for Headroom lightweight (no-torch) path",
+    )
+    logger.info("Uploaded. Headroom's ONNX loader will now find it on next cold start.")
+
+
+def main() -> int:
+    ap = argparse.ArgumentParser(description=__doc__)
+    ap.add_argument("--model-id", default=DEFAULT_MODEL_ID)
+    ap.add_argument(
+        "--precision",
+        choices=["fp32", "int8"],
+        default="fp32",
+        help="fp32 = lossless, larger artifact. int8 = ~2x smaller, tiny accuracy cost.",
+    )
+    ap.add_argument(
+        "--out",
+        type=Path,
+        default=None,
+        help="Local output path. Defaults to onnx/kompress-<precision>.onnx.",
+    )
+    ap.add_argument("--opset", type=int, default=17)
+    ap.add_argument(
+        "--upload",
+        action="store_true",
+        help="Upload to the HF repo under onnx/<filename> (needs HF write auth).",
+    )
+    args = ap.parse_args()
+
+    out_path = args.out or Path(f"onnx/kompress-{args.precision}.onnx")
+    export(args.model_id, out_path, args.opset, args.precision)
+    if args.upload:
+        upload(args.model_id, out_path)
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

scripts/fixtures/anthropic_replay_body.json

@@ -0,0 +1,133 @@
+{
+  "model": "claude-opus-4-7",
+  "max_tokens": 4096,
+  "stream": true,
+  "system": "You are Claude Code, Anthropic's official CLI for disciplined software engineering. You are helping the operator triage a resilience bug in a Python/FastAPI LLM proxy that handles both Anthropic /v1/messages traffic and OpenAI Codex /v1/responses WebSocket traffic on the same process.\n\nWorking style: Read code with symbolic tools when available; prefer small, reviewable diffs; never bypass compression or add fast paths that skip transforms; keep /livez trivial and IO-free; treat the WebSocket relay lifecycle as load-bearing. When you are uncertain about upstream behavior, instrument first and mitigate second. Multi-agent reconnect storms arrive as a burst of /v1/messages?beta=true POSTs immediately after process restart; these are the failure mode under investigation.\n\nInvariants: compression stays on; /livez must respond under 100ms even during cold-start replay storms; memory-context lookups are wrapped in wait_for with a bounded timeout; upstream WS handshake has retry/open-timeout hardening; debug endpoints are loopback-only.",
+  "tools": [
+    {
+      "name": "read_file",
+      "description": "Read a file from the local filesystem. Returns file contents as a string. Use when you need to inspect exact text of a source file before editing.",
+      "input_schema": {
+        "type": "object",
+        "properties": {
+          "path": {"type": "string", "description": "Absolute file path."},
+          "start_line": {"type": "integer", "description": "Optional 1-indexed start line."},
+          "end_line": {"type": "integer", "description": "Optional inclusive end line."}
+        },
+        "required": ["path"]
+      }
+    },
+    {
+      "name": "search_code",
+      "description": "Search the codebase for a regex pattern. Returns matching file paths and surrounding lines. Prefer over read_file when you don't know the file location yet.",
+      "input_schema": {
+        "type": "object",
+        "properties": {
+          "pattern": {"type": "string"},
+          "glob": {"type": "string", "description": "Optional glob filter, e.g. '**/*.py'."},
+          "max_results": {"type": "integer", "default": 50}
+        },
+        "required": ["pattern"]
+      }
+    },
+    {
+      "name": "run_tests",
+      "description": "Run the pytest suite or a subset. Returns summary output including passed/failed/errored counts and tracebacks for failures.",
+      "input_schema": {
+        "type": "object",
+        "properties": {
+          "expression": {"type": "string", "description": "pytest -k expression."},
+          "paths": {"type": "array", "items": {"type": "string"}, "description": "Specific test files or directories."}
+        }
+      }
+    },
+    {
+      "name": "edit_file",
+      "description": "Apply an exact-string replacement to a file. The old_string must be unique within the file; otherwise use replace_all. Prefer over write_file for modifications.",
+      "input_schema": {
+        "type": "object",
+        "properties": {
+          "path": {"type": "string"},
+          "old_string": {"type": "string"},
+          "new_string": {"type": "string"},
+          "replace_all": {"type": "boolean", "default": false}
+        },
+        "required": ["path", "old_string", "new_string"]
+      }
+    },
+    {
+      "name": "run_command",
+      "description": "Execute a shell command and return stdout, stderr, and exit code. Use for git, build, and verification commands.",
+      "input_schema": {
+        "type": "object",
+        "properties": {
+          "command": {"type": "string"},
+          "timeout_seconds": {"type": "integer", "default": 120}
+        },
+        "required": ["command"]
+      }
+    }
+  ],
+  "messages": [
+    {
+      "role": "user",
+      "content": [
+        {
+          "type": "text",
+          "text": "The shared proxy on 127.0.0.1:8787 started acting like a zombie this morning after a restart. New /livez probes time out, new /v1/responses WebSocket handshakes hang for ~20 seconds before the client gives up, but the existing agent sessions that were already streaming when the restart happened keep working fine. Two of us were connected at the time so there was a reconnect burst of Anthropic /v1/messages?beta=true POSTs carrying multi-turn history (~30 KB each). I suspect the replay storm is interacting badly with cold-start ONNX/embedder loads. Start by reading headroom/proxy/handlers/anthropic.py:handle_anthropic_messages and headroom/proxy/server.py:startup — I want to know whether the Anthropic pre-upstream stages (deep_copy, first compression pass, memory-context lookup) run before or after the Kompress/Magika preload settles, and whether there is any concurrency limit in front of that path. Don't propose a fix yet; just map the stages and tell me where the event loop would plausibly starve."
+        }
+      ]
+    },
+    {
+      "role": "assistant",
+      "content": [
+        {
+          "type": "text",
+          "text": "I've read both files. Mapping the pre-upstream stages on the Anthropic path: (1) handle_anthropic_messages reads the raw request bytes, (2) _read_request_json decodes and json.loads — this is synchronous and runs on the event loop thread, (3) a deep copy of the body is taken for logging, (4) the first compression stage fires through anthropic_pipeline.transforms, which on cold start is the moment Kompress ONNX and Magika get lazy-loaded, (5) memory-context lookup runs via MemoryHandler.search_and_format_context inside asyncio.wait_for, (6) only then does the handler reach http_client.stream() for upstream.\n\nStages 3 and 4 are the choke: deep_copy is synchronous Python on 30KB bodies (fine individually, but N concurrent replays mean N deep copies on one event loop), and the first compression stage blocks on Kompress load when the eager preload hasn't finished yet. There is currently no asyncio.Semaphore in front of stages 1-5. /livez does not go through this path, but the event loop is shared — if stages 3-5 hog it long enough, /livez's response coroutine doesn't get scheduled until they yield.\n\nI have not instrumented timings yet. Do you want me to add a StageTimer pass first, or jump straight to proposing a pre-upstream semaphore?"
+        }
+      ]
+    },
+    {
+      "role": "user",
+      "content": [
+        {
+          "type": "text",
+          "text": "Instrument first — that's the whole point of not guessing. Add per-stage timings for read_request_json, deep_copy, compression_first_stage, memory_context, upstream_connect, upstream_first_byte on the Anthropic HTTP path, and accept -> first_client_frame -> upstream_connect -> upstream_first_event -> total_session on the Codex WS path. Both should share a single StageTimer utility, log one structured line per request with request_id + session_id + all stage fields, and emit Prometheus histograms. Don't change any behavior. Once that's in, we can let the harness run and read the real numbers before touching concurrency."
+        }
+      ]
+    },
+    {
+      "role": "assistant",
+      "content": [
+        {
+          "type": "tool_use",
+          "id": "toolu_anthropic_replay_01",
+          "name": "search_code",
+          "input": {
+            "pattern": "class PrometheusMetrics",
+            "glob": "headroom/proxy/**/*.py"
+          }
+        }
+      ]
+    },
+    {
+      "role": "user",
+      "content": [
+        {
+          "type": "tool_result",
+          "tool_use_id": "toolu_anthropic_replay_01",
+          "content": "headroom/proxy/prometheus_metrics.py:41:class PrometheusMetrics:\n  Holds in-process counters/gauges/histograms plus async record_* helpers. Backend-agnostic; OTLP export is deferred. Pattern: register each histogram once in __init__, update via record_request / record_stage_timings."
+        }
+      ]
+    },
+    {
+      "role": "assistant",
+      "content": [
+        {
+          "type": "text",
+          "text": "Plan: create headroom/proxy/stage_timer.py with a StageTimer context manager exposing .measure(stage_name) as both a sync and async context manager, accumulating into a dict[str, float] keyed by stage with millisecond values. Extend PrometheusMetrics with one histogram per stage, keyed by path ('anthropic' | 'codex_ws'). Extend request_logger with a stage_timings field. Then plumb stage_timer through handle_anthropic_messages and handle_openai_responses_ws without changing any existing control flow — the timer is a pure observer. I'll land the util + unit tests first, then thread it through the two handlers in separate commits so the diff stays reviewable."
+        }
+      ]
+    }
+  ]
+}

scripts/fixtures/codex_response_create_frame.json

@@ -0,0 +1,52 @@
+{
+  "type": "response.create",
+  "response": {
+    "model": "gpt-5.3-codex",
+    "stream": true,
+    "store": false,
+    "parallel_tool_calls": true,
+    "tools": [],
+    "input": [
+      {
+        "type": "message",
+        "role": "system",
+        "content": [
+          {
+            "type": "input_text",
+            "text": "You are Codex, a proxy-aware coding agent. The upstream connection is a WebSocket proxied through a local Headroom process. Keep responses incremental and stream-friendly; the proxy compresses large tool outputs on the way back and expects well-formed response.created / response.output_text.delta / response.completed envelopes."
+          }
+        ]
+      },
+      {
+        "type": "message",
+        "role": "user",
+        "content": [
+          {
+            "type": "input_text",
+            "text": "We're reproducing a reconnect storm against the local proxy at 127.0.0.1:8787. Open a WebSocket session, send one response.create frame, and stay connected. The goal is not to generate useful output — it's to make sure the proxy's WS lifecycle (accept, upstream connect, relay-task wiring, registry register/deregister) survives N concurrent sessions plus a parallel burst of large /v1/messages POSTs without /livez going dark. Do not expect a real model response; upstream auth will fail and the proxy should gracefully close. Just hold the slot for the storm duration."
+          }
+        ]
+      },
+      {
+        "type": "message",
+        "role": "assistant",
+        "content": [
+          {
+            "type": "output_text",
+            "text": "Acknowledged. I'll keep this session open, avoid generating extra frames, and let the harness measure /livez latency and registry drain time. If the upstream closes with an auth error I'll expect a clean response.completed or error frame from the proxy, not a dangling task."
+          }
+        ]
+      },
+      {
+        "type": "message",
+        "role": "user",
+        "content": [
+          {
+            "type": "input_text",
+            "text": "Confirm the session. No further work required — the harness is driving."
+          }
+        ]
+      }
+    ]
+  }
+}

scripts/install-git-hooks.sh

@@ -0,0 +1,104 @@
+#!/usr/bin/env bash
+# Install git hooks for the Headroom repo:
+#   1. pre-commit  — repo pre-commit checks (ruff, mypy, sync-plugin-versions)
+#   2. commit-msg  — conventional-commit enforcement via commitlint
+#   3. pre-push    — full ci-precheck (cargo fmt/clippy/test + python suite)
+#
+# Why pre-push was added: the 2026-04-27 push hit five CI failures that could
+# all have been caught locally — cargo fmt drift, an x86_64-apple-darwin wheel
+# that the project doesn't actually need, missing Rust extension in two CI
+# lanes, and a commitlint warning treated as an error.
+#
+# Why pre-commit was added: PR #772 merged with inline-comment spacing and
+# import-order violations because the ruff pre-commit hook in
+# .pre-commit-config.yaml was never installed for contributors.
+#
+# Idempotent. Re-running is safe. Skips installation if `.git/hooks/` is
+# missing (e.g. running outside a git checkout).
+
+set -euo pipefail
+
+cd "$(dirname "$0")/.."
+
+if [[ ! -d .git/hooks ]]; then
+    echo "error: .git/hooks/ not found — run from a git checkout root" >&2
+    exit 1
+fi
+
+if ! command -v npx &>/dev/null; then
+    echo "error: npx not found — install Node 18+ before installing Headroom's git hooks." >&2
+    exit 1
+fi
+
+HOOK_PATH=".git/hooks/pre-push"
+
+cat > "$HOOK_PATH" <<'HOOK_EOF'
+#!/usr/bin/env bash
+# Headroom pre-push hook — runs `make ci-precheck` so CI never finds a
+# bug a local check could have caught.
+#
+# Skip with: `git push --no-verify`. Use sparingly — every skip is a roll
+# of the dice on a CI break.
+
+set -euo pipefail
+
+# Skip the hook entirely when push goes to a ref that is not on the main
+# tracking branches we gate. Adjust the pattern below if more branches
+# need gating.
+remote="$1"
+url="$2"
+
+while IFS=' ' read -r local_ref local_sha remote_ref remote_sha; do
+    # Empty local_sha means a delete; nothing to verify.
+    if [[ "$local_sha" == "0000000000000000000000000000000000000000" ]]; then
+        continue
+    fi
+    echo "── pre-push: running 'make ci-precheck' before pushing $local_ref → $remote_ref"
+done
+
+if [[ -z "${VIRTUAL_ENV:-}" ]]; then
+    if [[ -f .venv/bin/activate ]]; then
+        # shellcheck disable=SC1091
+        source .venv/bin/activate
+    else
+        echo "warn: no VIRTUAL_ENV set and no .venv/ found — python checks may use the wrong interpreter" >&2
+    fi
+fi
+
+if make ci-precheck; then
+    exit 0
+else
+    echo ""
+    echo "❌ pre-push: 'make ci-precheck' failed. Fix the issues above before pushing."
+    echo "   To bypass (NOT recommended): git push --no-verify"
+    exit 1
+fi
+HOOK_EOF
+
+chmod +x "$HOOK_PATH"
+
+echo "✅ installed: $HOOK_PATH"
+echo "   Runs 'make ci-precheck' before every git push."
+echo "   Bypass (use sparingly): git push --no-verify"
+
+# Install pre-commit hooks (repo checks on every commit).
+# Prefer the project venv over a global install so contributors always run the
+# pinned version. Resolution order: active $VIRTUAL_ENV → .venv → global PATH.
+PRE_COMMIT_BIN=""
+if [[ -n "${VIRTUAL_ENV:-}" && -x "${VIRTUAL_ENV}/bin/pre-commit" ]]; then
+    PRE_COMMIT_BIN="${VIRTUAL_ENV}/bin/pre-commit"
+elif [[ -x .venv/bin/pre-commit ]]; then
+    PRE_COMMIT_BIN=".venv/bin/pre-commit"
+elif command -v pre-commit &>/dev/null; then
+    PRE_COMMIT_BIN="pre-commit"
+fi
+
+if [[ -n "$PRE_COMMIT_BIN" ]]; then
+    "$PRE_COMMIT_BIN" install
+    "$PRE_COMMIT_BIN" install --hook-type commit-msg
+    echo "✅ installed: .git/hooks/pre-commit (repo pre-commit checks via pre-commit)"
+    echo "✅ installed: .git/hooks/commit-msg (conventional commit enforcement via commitlint)"
+else
+    echo "error: pre-commit not found — run 'pip install -e .[dev]' first, then re-run this script." >&2
+    exit 1
+fi

scripts/pr-governance.py

@@ -0,0 +1,329 @@
+#!/usr/bin/env python3
+"""Validate Headroom PR template compliance for GitHub Actions."""
+
+from __future__ import annotations
+
+import argparse
+import json
+import os
+import re
+import sys
+from dataclasses import asdict, dataclass, field
+from pathlib import Path
+from typing import Any
+
+COMMENT_MARKER = "<!-- headroom-pr-governance -->"
+READY_LABEL = "status: ready for review"
+AUTHOR_ACTION_LABEL = "status: needs author action"
+
+REQUIRED_SECTIONS = (
+    "Description",
+    "Type of Change",
+    "Changes Made",
+    "Testing",
+    "Real Behavior Proof",
+    "Review Readiness",
+)
+PROOF_FIELDS = (
+    "Environment",
+    "Exact command / steps",
+    "Observed result",
+    "Not tested",
+)
+
+SECTION_RE = re.compile(r"^##\s+(.+?)\s*$", re.MULTILINE)
+CHECKBOX_RE = re.compile(r"^- \[(?P<checked>[ xX])\] (?P<label>.+)$", re.MULTILINE)
+HTML_COMMENT_RE = re.compile(r"<!--.*?-->", re.DOTALL)
+CODE_BLOCK_RE = re.compile(r"```(?:[\w.+-]+)?\n(?P<content>.*?)```", re.DOTALL)
+
+
+@dataclass(slots=True)
+class GovernanceReport:
+    """Serializable PR governance result."""
+
+    comment_marker: str
+    valid: bool
+    is_draft: bool
+    is_bot_pr: bool
+    ready_for_review: bool
+    needs_author_action: bool
+    problems: list[str] = field(default_factory=list)
+    labels_to_add: list[str] = field(default_factory=list)
+    labels_to_remove: list[str] = field(default_factory=list)
+    comment_markdown: str = ""
+    summary_markdown: str = ""
+
+    def to_dict(self) -> dict[str, Any]:
+        return asdict(self)
+
+
+def load_event(path: Path) -> dict[str, Any]:
+    return json.loads(path.read_text(encoding="utf-8"))
+
+
+def extract_sections(body: str) -> dict[str, str]:
+    matches = list(SECTION_RE.finditer(body))
+    sections: dict[str, str] = {}
+    for index, match in enumerate(matches):
+        start = match.end()
+        end = matches[index + 1].start() if index + 1 < len(matches) else len(body)
+        sections[match.group(1).strip()] = body[start:end].strip()
+    return sections
+
+
+def strip_html_comments(text: str) -> str:
+    return HTML_COMMENT_RE.sub("", text).strip()
+
+
+def non_empty_lines(text: str) -> list[str]:
+    return [line.strip() for line in strip_html_comments(text).splitlines() if line.strip()]
+
+
+def checked_items(section: str) -> list[str]:
+    return [
+        match.group("label").strip()
+        for match in CHECKBOX_RE.finditer(section)
+        if match.group("checked").lower() == "x"
+    ]
+
+
+def has_descriptive_text(section: str) -> bool:
+    ignored_prefixes = ("closes #", "fixes #", "resolves #", "related to #")
+    for line in non_empty_lines(section):
+        lowered = line.lower()
+        if line.startswith("#"):
+            continue
+        if lowered.startswith(ignored_prefixes):
+            continue
+        if len(line) >= 10:
+            return True
+    return False
+
+
+def has_non_placeholder_bullets(section: str) -> bool:
+    placeholders = {"change 1", "change 2", "change 3"}
+    for line in non_empty_lines(section):
+        if not line.startswith("- "):
+            continue
+        bullet = line[2:].strip().lower()
+        if bullet and bullet not in placeholders:
+            return True
+    return False
+
+
+def has_test_output(section: str) -> bool:
+    for match in CODE_BLOCK_RE.finditer(section):
+        content = strip_html_comments(match.group("content")).strip()
+        if not content:
+            continue
+        if "paste relevant command output or artifact links here" in content.lower():
+            continue
+        return True
+    return False
+
+
+def proof_field_values(section: str) -> dict[str, str]:
+    values: dict[str, str] = {}
+    for line in non_empty_lines(section):
+        if not line.startswith("- ") or ":" not in line:
+            continue
+        label, value = line[2:].split(":", 1)
+        values[label.strip()] = value.strip()
+    return values
+
+
+def normalize_checkbox_map(items: list[str]) -> set[str]:
+    return {item.lower() for item in items}
+
+
+def validate_pull_request(event: dict[str, Any]) -> GovernanceReport:
+    pull_request = event["pull_request"]
+    author = pull_request["user"]["login"]
+    is_draft = bool(pull_request.get("draft", False))
+    is_bot_pr = author.endswith("[bot]")
+    body = pull_request.get("body") or ""
+    # Normalize Windows line endings so regex patterns expecting \n
+    # (particularly the code-block fence regex) match correctly.
+    body = body.replace("\r\n", "\n")
+
+    if is_bot_pr:
+        summary = "### PR governance\n\nBot-authored PR detected; template enforcement is skipped."
+        return GovernanceReport(
+            comment_marker=COMMENT_MARKER,
+            valid=True,
+            is_draft=is_draft,
+            is_bot_pr=True,
+            ready_for_review=False,
+            needs_author_action=False,
+            comment_markdown=summary,
+            summary_markdown=summary,
+        )
+
+    sections = extract_sections(body)
+    problems: list[str] = []
+
+    for section_name in REQUIRED_SECTIONS:
+        if section_name not in sections:
+            problems.append(f"Missing required section `{section_name}`.")
+
+    description = sections.get("Description", "")
+    if description and not has_descriptive_text(description):
+        problems.append("Fill in `Description` with a real summary of the change.")
+
+    changes_made = sections.get("Changes Made", "")
+    if changes_made and not has_non_placeholder_bullets(changes_made):
+        problems.append(
+            "Replace the placeholder bullets in `Changes Made` with the actual changes."
+        )
+
+    type_of_change_checked = checked_items(sections.get("Type of Change", ""))
+    if sections.get("Type of Change") and not type_of_change_checked:
+        problems.append("Check at least one box in `Type of Change`.")
+
+    testing_section = sections.get("Testing", "")
+    testing_checked = checked_items(testing_section)
+    if testing_section and not testing_checked:
+        problems.append("Check at least one verification item in `Testing`.")
+    if testing_section and not has_test_output(testing_section):
+        problems.append("Paste real command output or artifact links in `Testing` → `Test Output`.")
+
+    proof_section = sections.get("Real Behavior Proof", "")
+    proof_values = proof_field_values(proof_section)
+    for field_name in PROOF_FIELDS:
+        if proof_section and not proof_values.get(field_name):
+            problems.append(f"Fill in `Real Behavior Proof` → `{field_name}`.")
+
+    readiness_checked = normalize_checkbox_map(checked_items(sections.get("Review Readiness", "")))
+    has_self_review = "i have performed a self-review" in readiness_checked
+    has_ready_checkbox = "this pr is ready for human review" in readiness_checked
+    if not is_draft:
+        if not has_self_review:
+            problems.append(
+                "Check `I have performed a self-review` before requesting human review."
+            )
+        if not has_ready_checkbox:
+            problems.append(
+                "Check `This PR is ready for human review` or convert the PR back to draft."
+            )
+
+    valid = not problems
+    ready_for_review = valid and not is_draft and has_ready_checkbox and has_self_review
+    needs_author_action = not valid
+
+    if valid and ready_for_review:
+        status_lines = [
+            "### PR governance",
+            "",
+            "This PR follows the template and is marked ready for human review.",
+        ]
+    elif valid:
+        status_lines = [
+            "### PR governance",
+            "",
+            "This draft PR follows the template so far. Keep it in draft until it is ready for human review.",
+        ]
+    else:
+        status_lines = [
+            "### PR governance",
+            "",
+            "This PR does not yet satisfy the required template fields:",
+            "",
+            *[f"- {problem}" for problem in problems],
+            "",
+            "Please update the PR body, or move the PR back to draft while it is still in progress.",
+        ]
+
+    labels_to_add: list[str] = []
+    labels_to_remove: list[str] = []
+    if needs_author_action:
+        labels_to_add.append(AUTHOR_ACTION_LABEL)
+        labels_to_remove.append(READY_LABEL)
+    else:
+        labels_to_remove.append(AUTHOR_ACTION_LABEL)
+        if ready_for_review:
+            labels_to_add.append(READY_LABEL)
+        else:
+            labels_to_remove.append(READY_LABEL)
+
+    comment_markdown = "\n".join(status_lines)
+    return GovernanceReport(
+        comment_marker=COMMENT_MARKER,
+        valid=valid,
+        is_draft=is_draft,
+        is_bot_pr=False,
+        ready_for_review=ready_for_review,
+        needs_author_action=needs_author_action,
+        problems=problems,
+        labels_to_add=labels_to_add,
+        labels_to_remove=labels_to_remove,
+        comment_markdown=comment_markdown,
+        summary_markdown=comment_markdown,
+    )
+
+
+def validate_pull_request_body(event: dict[str, Any], body: str | None = None) -> GovernanceReport:
+    """Validate a PR event, optionally replacing the event payload body.
+
+    GitHub reruns use the original event payload. That makes a governance rerun
+    keep validating an old PR body even after maintainers fix the live body.
+    The workflow fetches the current body via the API and passes it here so the
+    check reflects what reviewers see on the PR page.
+    """
+    if body is None:
+        return validate_pull_request(event)
+
+    event_copy = dict(event)
+    pull_request = dict(event["pull_request"])
+    pull_request["body"] = body
+    event_copy["pull_request"] = pull_request
+    return validate_pull_request(event_copy)
+
+
+def emit_outputs(report: GovernanceReport) -> None:
+    output_path = os.environ.get("GITHUB_OUTPUT")
+    lines = [
+        f"valid={str(report.valid).lower()}",
+        f"ready_for_review={str(report.ready_for_review).lower()}",
+        f"needs_author_action={str(report.needs_author_action).lower()}",
+        f"is_bot_pr={str(report.is_bot_pr).lower()}",
+    ]
+    if not output_path:
+        for line in lines:
+            print(line)
+        return
+
+    with Path(output_path).open("a", encoding="utf-8") as output_file:
+        for line in lines:
+            output_file.write(f"{line}\n")
+
+
+def parse_args(argv: list[str]) -> argparse.Namespace:
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument(
+        "--event", type=Path, required=True, help="Path to the GitHub event payload JSON."
+    )
+    parser.add_argument(
+        "--body-file",
+        type=Path,
+        help=(
+            "Optional file containing the current PR body. Use this in GitHub Actions "
+            "so reruns validate the live PR body instead of the stale event payload."
+        ),
+    )
+    parser.add_argument("--report", type=Path, required=True, help="Path to write the JSON report.")
+    return parser.parse_args(argv)
+
+
+def main(argv: list[str] | None = None) -> int:
+    args = parse_args(argv or sys.argv[1:])
+    body_override = (
+        args.body_file.read_text(encoding="utf-8") if args.body_file is not None else None
+    )
+    report = validate_pull_request_body(load_event(args.event), body_override)
+    args.report.write_text(json.dumps(report.to_dict(), indent=2), encoding="utf-8")
+    emit_outputs(report)
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

scripts/record_fixtures.py

@@ -0,0 +1,59 @@
+#!/usr/bin/env python3
+"""Entry point for recording Rust-vs-Python parity fixtures.
+
+Usage:
+    python scripts/record_fixtures.py
+
+This driver:
+  1. Monkey-patches the Phase-1 transform classes via
+     `tests.parity.recorder.record_all()`.
+  2. Runs a small deterministic synthetic workload (no network, no real LLM
+     calls) that hits each transform at least 20 times with varied inputs.
+  3. Prints a summary and exits non-zero if any transform was blocked.
+"""
+
+from __future__ import annotations
+
+import logging
+import sys
+from pathlib import Path
+
+# Make `tests/parity/recorder.py` importable without installing it.
+_REPO_ROOT = Path(__file__).resolve().parent.parent
+sys.path.insert(0, str(_REPO_ROOT))
+
+from tests.parity.recorder import record_all, run_default_workload  # noqa: E402
+
+
+def main() -> int:
+    logging.basicConfig(level=logging.INFO, format="%(levelname)s %(name)s: %(message)s")
+    log = logging.getLogger("record_fixtures")
+
+    statuses = record_all()
+    log.info("patch status:")
+    blocked = []
+    for name, status in statuses.items():
+        log.info("  %-16s %s", name, status)
+        if status.startswith("blocked:"):
+            blocked.append((name, status))
+
+    counts = run_default_workload()
+    log.info("fixture counts:")
+    shortfall = []
+    for name, n in counts.items():
+        log.info("  %-16s %d", name, n)
+        if n < 20:
+            shortfall.append((name, n))
+
+    if blocked:
+        log.error("blocked transforms: %s", blocked)
+    if shortfall:
+        log.error("transforms with <20 fixtures: %s", shortfall)
+
+    # Exit non-zero if anything was wholly blocked; short recordings are
+    # a soft warning.
+    return 1 if blocked else 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

scripts/refresh_model_limits.sh

@@ -0,0 +1,48 @@
+#!/usr/bin/env bash
+#
+# Refresh the vendored LiteLLM model_prices_and_context_window.json
+# used by `crates/headroom-proxy/src/compression/model_limits.rs`.
+#
+# We vendor the snapshot rather than fetching at build/runtime so the
+# proxy binary ships with no network dependency at startup. Operators
+# tracking new model releases run this script and commit the diff.
+#
+# Validation:
+#   1. JSON parses
+#   2. Contains a known-stable Claude model entry
+#   3. Contains a known-stable GPT model entry
+# These guard against accidentally vendoring an empty / malformed file.
+
+set -euo pipefail
+
+REPO_ROOT="$(cd "$(dirname "$0")/.." && pwd)"
+DEST="$REPO_ROOT/crates/headroom-proxy/data/model_prices_and_context_window.json"
+URL="https://raw.githubusercontent.com/BerriAI/litellm/main/model_prices_and_context_window.json"
+
+echo "Fetching $URL"
+TMP="$(mktemp)"
+trap 'rm -f "$TMP"' EXIT
+curl -fsSL "$URL" -o "$TMP"
+
+# Validate the snapshot before swapping it in.
+python3 -c "
+import json, sys
+with open('$TMP') as f:
+    data = json.load(f)
+if not isinstance(data, dict):
+    sys.exit('top-level not an object')
+if 'sample_spec' not in data:
+    sys.exit('missing sample_spec entry — schema may have changed')
+# Spot-check stable entries.
+required = ['claude-sonnet-4-5-20250929', 'gpt-4o-mini', 'gpt-4-turbo']
+missing = [k for k in required if k not in data]
+if missing:
+    sys.exit(f'missing required entries: {missing!r}')
+print(f'OK: {len(data)} entries, including {required}')
+"
+
+mv "$TMP" "$DEST"
+trap - EXIT
+
+echo "Updated $DEST"
+echo "Run 'cargo test -p headroom-proxy --lib compression::model_limits' to verify."

scripts/replay_codex_ws_load.py

@@ -0,0 +1,445 @@
+#!/usr/bin/env python3
+"""Tier-3 replay: reproduce Codex /v1/responses compression load.
+
+Parses a production proxy log to extract per-session frame-size scenarios,
+generates synthetic payloads matching those sizes/shapes, and concurrently
+drives the proxy's _compress_openai_responses_payload entry point. Reports
+per-frame latency percentiles, timeout count, and total wall time so a
+before/after comparison proves the P2 scheduler fix.
+
+Why this lives in scripts/ (not tests/):
+    - It is a measurement tool, not a correctness test.
+    - It needs to run against multiple branches (main baseline vs fix
+      branch) and report comparable numbers.
+    - It exercises the *real* compression dispatch by booting a proxy
+      instance via create_app() and calling the handler method directly —
+      no HTTP/WS layer, because the bug is in the dispatch, not the wire.
+
+Usage:
+    .venv/bin/python scripts/replay_codex_ws_load.py \\
+        --log "/Users/tchopra/Downloads/proxy (1).log" \\
+        --concurrency 10 \\
+        --frames-per-session 20
+"""
+
+from __future__ import annotations
+
+import argparse
+import concurrent.futures
+import json
+import os
+import statistics
+import sys
+import time
+from dataclasses import dataclass, field
+from pathlib import Path
+
+REPO_ROOT = Path(__file__).resolve().parent.parent
+sys.path.insert(0, str(REPO_ROOT))
+
+
+# Telemetry off so we don't pollute the user's metrics during replay.
+os.environ.setdefault("HEADROOM_DISABLE_TELEMETRY", "true")
+os.environ.setdefault("HEADROOM_REQUIRE_RUST_CORE", "false")
+
+
+@dataclass
+class Frame:
+    bytes_estimate: int
+    text_shape: str  # plain_text_like | code_fence | traceback | jsonl_like
+
+
+@dataclass
+class Scenario:
+    request_id: str
+    frames: list[Frame] = field(default_factory=list)
+
+
+# ── Log parser ─────────────────────────────────────────────────────────
+
+
+# Marker columns. We are not using regex here per the design constraints —
+# the log shape is a single deterministic format set by code we own. If
+# the format changes the parser fails loud, not silently.
+_FRAME_TOKEN = " WS /v1/responses "
+_REQID_OPEN = "["
+_REQID_CLOSE = "]"
+
+
+def _parse_kv(text: str) -> dict[str, str]:
+    """Parse ``key=value`` pairs out of a slow-unit log tail. Stops at the
+    first unquoted space after a value. Quoted values not supported because
+    the log never emits them; if it ever does, this raises.
+    """
+    out: dict[str, str] = {}
+    for token in text.split():
+        if "=" not in token:
+            continue
+        k, _, v = token.partition("=")
+        out[k] = v
+    return out
+
+
+def parse_log(log_path: Path) -> dict[str, Scenario]:
+    """Group ``WS /v1/responses slow compression unit`` entries by request_id.
+
+    Each ``slow compression unit`` line carries the per-unit byte count and
+    text_shape — exactly what we need to reconstruct a payload of similar
+    compression cost. We deliberately ignore the ``compressed`` / ``frame
+    compressed`` lines because they report POST-compression bytes, not the
+    pre-compression input the dispatcher sees.
+
+    Format:
+        ... [hr_..._...] WS /v1/responses slow compression unit elapsed_ms=N
+            strategy=X category=Y modified=Z content_type=T text_shape=S
+            bytes=B min_bytes=N tokens_before=T tokens_after=T tokens_saved=S
+            strategy_chain=[...]
+    """
+    scenarios: dict[str, Scenario] = {}
+    with log_path.open("r", encoding="utf-8", errors="replace") as fh:
+        for line in fh:
+            if "slow compression unit" not in line:
+                continue
+            if _FRAME_TOKEN not in line:
+                continue
+            req_open = line.find(_REQID_OPEN)
+            req_close = line.find(_REQID_CLOSE, req_open + 1)
+            if req_open < 0 or req_close < 0:
+                continue
+            request_id = line[req_open + 1 : req_close]
+            tail = line[req_close + 1 :]
+            kv = _parse_kv(tail)
+            try:
+                size = int(kv["bytes"])
+            except (KeyError, ValueError):
+                continue
+            shape = kv.get("text_shape", "plain_text_like")
+            scen = scenarios.setdefault(request_id, Scenario(request_id=request_id))
+            scen.frames.append(Frame(bytes_estimate=size, text_shape=shape))
+    return scenarios
+
+
+# ── Payload synthesizer ────────────────────────────────────────────────
+
+
+_LOREM = (
+    "Lorem ipsum dolor sit amet, consectetur adipiscing elit. "
+    "Sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. "
+)
+_CODE_LINE = "def compute_metric_{i}(value: int) -> int:\n    return value * {i}\n\n"
+_TRACEBACK_LINE = (
+    '  File "/app/handler.py", line {i}, in process_request\n    raise RuntimeError(f"oops {i}")\n'
+)
+
+
+def _text_for_shape(shape: str, target_bytes: int) -> str:
+    """Generate a string roughly ``target_bytes`` long, shaped like the
+    production observation. No randomness — same input produces same output
+    so the replay is reproducible.
+    """
+    if target_bytes < 64:
+        # Below size_floor — generator just returns a short token.
+        return "ok"
+    if shape == "code_fence":
+        body_target = max(target_bytes - 12, 0)  # "```python\n" + closing
+        repeats = max(body_target // 50, 1)
+        body = "".join(_CODE_LINE.format(i=i) for i in range(repeats))
+        return "```python\n" + body[:body_target] + "\n```"
+    if shape == "traceback":
+        header = "Traceback (most recent call last):\n"
+        body_target = max(target_bytes - len(header), 0)
+        repeats = max(body_target // 65, 1)
+        body = "".join(_TRACEBACK_LINE.format(i=i) for i in range(repeats))
+        return header + body[:body_target]
+    # plain_text_like / unknown / jsonl_like → lorem ipsum is fine as a
+    # neutral payload; we are measuring scheduler contention, not compressor
+    # quality, so the content shape just needs to traverse the same router.
+    repeats = max(target_bytes // len(_LOREM), 1)
+    raw = _LOREM * repeats
+    return raw[:target_bytes]
+
+
+def synthesize_payload(frame: Frame, turn_no: int) -> dict:
+    """Build the *inner* Responses payload (no `response.create` envelope)
+    with one function_call_output of the target byte size.
+
+    ``_compress_openai_responses_payload`` is envelope-agnostic but routes
+    by inspecting ``input``/``messages`` at the top level. The WS handler
+    extracts ``payload["response"]`` and passes that downstream — we pass
+    the same shape directly so the router actually sees compressible
+    candidates instead of a single opaque ``response`` key.
+    """
+    output_text = _text_for_shape(frame.text_shape, frame.bytes_estimate)
+    return {
+        "model": "gpt-4o-mini",
+        "input": [
+            {
+                "type": "message",
+                "role": "user",
+                "content": [
+                    {
+                        "type": "input_text",
+                        "text": f"Turn {turn_no} — please summarize.",
+                    }
+                ],
+            },
+            {
+                "type": "function_call",
+                "call_id": f"call_replay_{turn_no}",
+                "name": "shell",
+                "arguments": '{"command": "build"}',
+            },
+            {
+                "type": "function_call_output",
+                "call_id": f"call_replay_{turn_no}",
+                "output": output_text,
+            },
+        ],
+        "instructions": "Be brief.",
+        "max_output_tokens": 30,
+    }
+
+
+# ── Proxy bring-up ─────────────────────────────────────────────────────
+
+
+def boot_proxy():
+    """Build a HeadroomProxy instance with optimize=True so the compression
+    dispatch is actually exercised.
+
+    This deliberately does NOT start the FastAPI server. We only need the
+    in-process handler methods. Lifecycle hooks (background tasks, model
+    pre-loading) that fire on startup are not required for the dispatch
+    method we exercise — Kompress will lazy-load on first use, which we
+    explicitly warm up below.
+    """
+    from headroom.proxy.server import ProxyConfig, create_app
+
+    config = ProxyConfig(
+        optimize=True,
+        cache_enabled=False,
+        rate_limit_enabled=False,
+    )
+    app = create_app(config)
+    return app.state.proxy
+
+
+def warmup(proxy, model: str = "gpt-4o-mini") -> float:
+    """Issue one small compression call so model weights are loaded.
+
+    Returns the warmup wall time so the caller can sanity-check the
+    measurements (warmup time is NOT counted toward replay metrics).
+    """
+    payload = synthesize_payload(
+        Frame(bytes_estimate=4096, text_shape="plain_text_like"), turn_no=0
+    )
+    started = time.perf_counter()
+    proxy._compress_openai_responses_payload(payload, model=model, request_id="replay-warmup")
+    return (time.perf_counter() - started) * 1000.0
+
+
+# ── Replay driver ──────────────────────────────────────────────────────
+
+
+@dataclass
+class FrameResult:
+    request_id: str
+    frame_index: int
+    bytes_in: int
+    elapsed_ms: float
+    error: str | None = None
+
+
+def replay_session(proxy, scenario: Scenario, model: str) -> list[FrameResult]:
+    out: list[FrameResult] = []
+    for idx, frame in enumerate(scenario.frames):
+        payload = synthesize_payload(frame, turn_no=idx + 1)
+        started = time.perf_counter()
+        err: str | None = None
+        try:
+            proxy._compress_openai_responses_payload(
+                payload, model=model, request_id=scenario.request_id
+            )
+        except Exception as e:  # noqa: BLE001 — surface ALL failure modes
+            err = f"{type(e).__name__}: {e}"
+        elapsed_ms = (time.perf_counter() - started) * 1000.0
+        out.append(
+            FrameResult(
+                request_id=scenario.request_id,
+                frame_index=idx,
+                bytes_in=frame.bytes_estimate,
+                elapsed_ms=elapsed_ms,
+                error=err,
+            )
+        )
+    return out
+
+
+def _percentile(values: list[float], pct: float) -> float:
+    if not values:
+        return 0.0
+    s = sorted(values)
+    k = max(0, min(len(s) - 1, int(round(pct / 100.0 * (len(s) - 1)))))
+    return s[k]
+
+
+# ── Reporting ──────────────────────────────────────────────────────────
+
+
+def print_report(
+    results: list[FrameResult],
+    wall_time_s: float,
+    concurrency: int,
+    warmup_ms: float,
+    out_json: Path | None,
+) -> None:
+    elapsed = [r.elapsed_ms for r in results]
+    errors = [r for r in results if r.error]
+    total_bytes = sum(r.bytes_in for r in results)
+    by_session: dict[str, list[float]] = {}
+    for r in results:
+        by_session.setdefault(r.request_id, []).append(r.elapsed_ms)
+    session_totals = [sum(v) for v in by_session.values()]
+
+    summary = {
+        "concurrency": concurrency,
+        "warmup_ms": round(warmup_ms, 1),
+        "frames_total": len(results),
+        "sessions": len(by_session),
+        "wall_time_s": round(wall_time_s, 2),
+        "errors": len(errors),
+        "error_classes": sorted({type(e.error).__name__: 1 for e in errors if e.error}.keys()),
+        "input_bytes_total": total_bytes,
+        "per_frame_elapsed_ms": {
+            "p50": round(_percentile(elapsed, 50), 1),
+            "p90": round(_percentile(elapsed, 90), 1),
+            "p99": round(_percentile(elapsed, 99), 1),
+            "max": round(max(elapsed) if elapsed else 0.0, 1),
+            "mean": round(statistics.mean(elapsed) if elapsed else 0.0, 1),
+        },
+        "per_session_total_ms": {
+            "p50": round(_percentile(session_totals, 50), 1),
+            "p90": round(_percentile(session_totals, 90), 1),
+            "max": round(max(session_totals) if session_totals else 0.0, 1),
+        },
+    }
+
+    print("─── Codex compression replay summary ───")
+    print(f"Concurrency:           {summary['concurrency']}")
+    print(f"Sessions replayed:     {summary['sessions']}")
+    print(f"Frames replayed:       {summary['frames_total']}")
+    print(f"Wall time:             {summary['wall_time_s']}s")
+    print(f"Warmup wall time:      {summary['warmup_ms']}ms (NOT counted in metrics)")
+    print(f"Failures:              {summary['errors']}")
+    print(f"Input bytes total:     {summary['input_bytes_total']:,}")
+    print("Per-frame elapsed_ms:")
+    for k, v in summary["per_frame_elapsed_ms"].items():
+        print(f"  {k:5}                {v}")
+    print("Per-session total_ms:")
+    for k, v in summary["per_session_total_ms"].items():
+        print(f"  {k:5}                {v}")
+    if errors:
+        print("\nFirst 5 errors:")
+        for e in errors[:5]:
+            print(f"  [{e.request_id}] frame {e.frame_index}: {e.error}")
+
+    if out_json:
+        out_json.write_text(json.dumps(summary, indent=2))
+        print(f"\nWrote machine-readable summary to {out_json}")
+
+
+# ── Main ───────────────────────────────────────────────────────────────
+
+
+def main() -> int:
+    parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
+    parser.add_argument(
+        "--log",
+        type=Path,
+        required=True,
+        help="Path to production proxy log; per-session frame sizes are extracted from "
+        "`slow compression unit` lines.",
+    )
+    parser.add_argument(
+        "--concurrency",
+        type=int,
+        default=10,
+        help="Number of concurrent sessions to replay (default: 10).",
+    )
+    parser.add_argument(
+        "--frames-per-session",
+        type=int,
+        default=20,
+        help="Cap frames per session for bounded run-time (default: 20). "
+        "Sessions with more frames are truncated; with fewer are padded.",
+    )
+    parser.add_argument(
+        "--model",
+        default="gpt-4o-mini",
+        help="Model name passed through the dispatcher (default: gpt-4o-mini).",
+    )
+    parser.add_argument(
+        "--out-json",
+        type=Path,
+        help="Write machine-readable summary JSON here for before/after comparison.",
+    )
+    args = parser.parse_args()
+
+    if not args.log.exists():
+        print(f"error: log file not found: {args.log}", file=sys.stderr)
+        return 2
+
+    print(f"[replay] parsing {args.log} ...", flush=True)
+    scenarios = parse_log(args.log)
+    if not scenarios:
+        print(
+            "error: no scenarios extracted from log (no `slow compression unit` lines)",
+            file=sys.stderr,
+        )
+        return 2
+
+    # Pick the top-N sessions by frame count — those exercised the bug
+    # hardest in production and give the most representative replay.
+    ranked = sorted(scenarios.values(), key=lambda s: -len(s.frames))
+    picked = ranked[: args.concurrency]
+    # Cap each scenario's frame count for bounded runtime.
+    for s in picked:
+        s.frames = s.frames[: args.frames_per_session]
+    print(
+        f"[replay] picked {len(picked)} scenarios "
+        f"(total frames: {sum(len(s.frames) for s in picked)})",
+        flush=True,
+    )
+
+    print("[replay] booting proxy in-process ...", flush=True)
+    proxy = boot_proxy()
+
+    print("[replay] warming up Kompress + router ...", flush=True)
+    warmup_ms = warmup(proxy, model=args.model)
+    print(f"[replay] warmup done in {warmup_ms:.1f}ms", flush=True)
+
+    print(
+        f"[replay] starting replay: {len(picked)} concurrent sessions x "
+        f"{args.frames_per_session} frames",
+        flush=True,
+    )
+    results: list[FrameResult] = []
+    wall_started = time.perf_counter()
+    with concurrent.futures.ThreadPoolExecutor(max_workers=args.concurrency) as pool:
+        futures = [pool.submit(replay_session, proxy, s, args.model) for s in picked]
+        for fut in concurrent.futures.as_completed(futures):
+            results.extend(fut.result())
+    wall_time_s = time.perf_counter() - wall_started
+
+    print_report(
+        results,
+        wall_time_s=wall_time_s,
+        concurrency=args.concurrency,
+        warmup_ms=warmup_ms,
+        out_json=args.out_json,
+    )
+    return 0 if all(r.error is None for r in results) else 1
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

scripts/repro_codex_replay.py

@@ -0,0 +1,699 @@
+#!/usr/bin/env python3
+"""Repro harness: multi-agent Codex reconnect storm against a local proxy.
+
+This script reproduces the failure class described in
+``wiki/plans/2026-04-17-codex-proxy-runtime-analysis.md`` ("Latest Correction"):
+a burst of concurrent Codex WebSocket sessions plus a parallel burst of large
+Anthropic ``/v1/messages`` replays immediately after a fresh proxy restart, with
+``/livez`` probed continuously to detect event-loop starvation.
+
+Usage::
+
+    python scripts/repro_codex_replay.py \\
+        --url http://127.0.0.1:8787 \\
+        --ws-clients 8 \\
+        --anthropic-clients 4 \\
+        --duration 30
+
+Exit code is ``0`` iff the warmup phase succeeded (or was skipped), the storm
+phase ran for the requested duration, and ``/livez`` p99 stayed at or below the
+configured threshold (``--livez-threshold-ms``, default 500ms).
+
+The harness adds no new pip dependencies: it uses ``asyncio`` + ``websockets``
++ ``httpx`` only, all already available in the Headroom dev environment.
+"""
+
+from __future__ import annotations
+
+import argparse
+import asyncio
+import json
+import random
+import statistics
+import sys
+import time
+from contextlib import asynccontextmanager, suppress
+from dataclasses import dataclass, field
+from pathlib import Path
+from typing import Any
+from urllib.parse import urlparse, urlunparse
+
+import httpx
+
+try:
+    from websockets.asyncio.client import connect as ws_connect
+except ImportError:  # pragma: no cover - older websockets fallback
+    from websockets.client import connect as ws_connect  # type: ignore[no-redef]
+
+# asyncio.timeout was added in Python 3.11; provide a thin shim for 3.10.
+if sys.version_info >= (3, 11):
+    _asyncio_timeout = asyncio.timeout
+else:
+
+    @asynccontextmanager
+    async def _asyncio_timeout(delay: float | None):  # type: ignore[no-redef]
+        """Minimal shim that mimics ``asyncio.timeout`` using ``wait_for``."""
+        # For the single call-site we have, wrapping the entire body in a task
+        # and cancelling it on timeout is sufficient.  We yield control, and if
+        # the caller's block exceeds *delay* seconds the enclosing task is
+        # cancelled.
+        loop = asyncio.get_running_loop()
+        deadline = loop.time() + delay if delay is not None else None
+        task = asyncio.current_task()
+        handle = None
+        if deadline is not None and task is not None:
+            handle = loop.call_at(deadline, task.cancel)
+        try:
+            yield
+        except asyncio.CancelledError:
+            raise asyncio.TimeoutError() from None
+        finally:
+            if handle is not None:
+                handle.cancel()
+
+
+from websockets.exceptions import ConnectionClosed, InvalidStatus, WebSocketException
+
+# ---------------------------------------------------------------------------
+# Fixtures
+# ---------------------------------------------------------------------------
+
+SCRIPT_DIR = Path(__file__).resolve().parent
+DEFAULT_WS_FRAME_FIXTURE = SCRIPT_DIR / "fixtures" / "codex_response_create_frame.json"
+DEFAULT_ANTHROPIC_BODY_FIXTURE = SCRIPT_DIR / "fixtures" / "anthropic_replay_body.json"
+
+
+def _load_json(path: Path) -> dict[str, Any]:
+    with path.open("r", encoding="utf-8") as f:
+        return json.load(f)
+
+
+# ---------------------------------------------------------------------------
+# Stats
+# ---------------------------------------------------------------------------
+
+
+@dataclass
+class LatencyHistogram:
+    """Minimal fixed-list histogram with p50/p95/p99/max computation."""
+
+    samples_ms: list[float] = field(default_factory=list)
+
+    def record(self, value_ms: float) -> None:
+        self.samples_ms.append(value_ms)
+
+    @property
+    def count(self) -> int:
+        return len(self.samples_ms)
+
+    def percentile(self, p: float) -> float:
+        if not self.samples_ms:
+            return 0.0
+        ordered = sorted(self.samples_ms)
+        if p <= 0:
+            return ordered[0]
+        if p >= 100:
+            return ordered[-1]
+        idx = max(0, min(len(ordered) - 1, int(round((p / 100.0) * (len(ordered) - 1)))))
+        return ordered[idx]
+
+    def as_summary(self) -> dict[str, float]:
+        if not self.samples_ms:
+            return {"count": 0, "p50": 0.0, "p95": 0.0, "p99": 0.0, "max": 0.0}
+        return {
+            "count": self.count,
+            "p50": self.percentile(50),
+            "p95": self.percentile(95),
+            "p99": self.percentile(99),
+            "max": max(self.samples_ms),
+        }
+
+
+@dataclass
+class CodexWsStats:
+    opened: int = 0
+    response_completed: int = 0
+    errors: dict[str, int] = field(default_factory=dict)
+
+    def record_error(self, kind: str) -> None:
+        self.errors[kind] = self.errors.get(kind, 0) + 1
+
+
+@dataclass
+class AnthropicHttpStats:
+    attempted: int = 0
+    ok_2xx: int = 0
+    non_2xx: int = 0
+    timed_out: int = 0
+    errors: int = 0
+    first_byte_latency_ms: list[float] = field(default_factory=list)
+
+    @property
+    def avg_first_byte_ms(self) -> float:
+        if not self.first_byte_latency_ms:
+            return 0.0
+        return statistics.mean(self.first_byte_latency_ms)
+
+
+# ---------------------------------------------------------------------------
+# URL helpers
+# ---------------------------------------------------------------------------
+
+
+def _http_to_ws_url(http_url: str, path: str) -> str:
+    parsed = urlparse(http_url)
+    scheme = "wss" if parsed.scheme == "https" else "ws"
+    # Preserve host:port; override scheme + path. Ensure path starts with a single "/".
+    normalized_path = "/" + path.lstrip("/") if path else ""
+    return urlunparse((scheme, parsed.netloc, normalized_path, "", "", ""))
+
+
+# ---------------------------------------------------------------------------
+# Warmup phase
+# ---------------------------------------------------------------------------
+
+
+async def warmup_probe(
+    url: str,
+    ws_frame: dict[str, Any],
+    timeout_s: float = 10.0,
+) -> tuple[bool, float, str]:
+    """Open one WS, send the response.create frame, wait briefly.
+
+    Returns ``(success, elapsed_ms, note)``. Success is true if the proxy
+    accepts the WebSocket handshake and the initial frame is sent without
+    error. We do not require ``response.completed`` — upstream auth will
+    often fail locally and the proxy will send an error frame; the important
+    signal is that the handshake + relay wiring worked.
+    """
+    ws_url = _http_to_ws_url(url, "/v1/responses")
+    start = time.perf_counter()
+    handshake_ok = False
+    frame_sent = False
+    note = "unknown"
+    try:
+        async with _asyncio_timeout(timeout_s):
+            async with ws_connect(
+                ws_url,
+                additional_headers={"Authorization": "Bearer repro-harness"},
+            ) as ws:
+                handshake_ok = True
+                try:
+                    await ws.send(json.dumps(ws_frame))
+                    frame_sent = True
+                except (ConnectionClosed, WebSocketException) as exc:
+                    note = f"send_failed:{type(exc).__name__}"
+                    return False, (time.perf_counter() - start) * 1000.0, note
+                # Drain until terminal event, close, or deadline.
+                deadline_local = time.perf_counter() + timeout_s
+                while time.perf_counter() < deadline_local:
+                    remaining = deadline_local - time.perf_counter()
+                    if remaining <= 0:
+                        break
+                    try:
+                        raw = await asyncio.wait_for(ws.recv(), timeout=remaining)
+                    except asyncio.TimeoutError:
+                        note = "timeout_waiting_for_completion"
+                        break
+                    except ConnectionClosed:
+                        # Upstream (or mock) closed cleanly after accepting our
+                        # frame — this still counts as a working pre-upstream
+                        # path; the warmup's purpose is handshake + send, not
+                        # demanding real upstream output.
+                        note = "upstream_closed_after_send"
+                        break
+                    try:
+                        evt = json.loads(raw) if isinstance(raw, str) else {}
+                    except json.JSONDecodeError:
+                        continue
+                    etype = evt.get("type", "")
+                    if etype == "response.completed":
+                        note = "response.completed"
+                        break
+                    if etype in {"error", "response.failed"}:
+                        note = f"terminal:{etype}"
+                        break
+        elapsed_ms = (time.perf_counter() - start) * 1000.0
+        return frame_sent, elapsed_ms, note
+    except (OSError, WebSocketException, InvalidStatus, asyncio.TimeoutError) as exc:
+        elapsed_ms = (time.perf_counter() - start) * 1000.0
+        # If the handshake completed and we sent the frame, a subsequent close
+        # error from the context manager is not a warmup failure.
+        if frame_sent:
+            return True, elapsed_ms, f"post_send_close:{type(exc).__name__}"
+        if handshake_ok:
+            return False, elapsed_ms, f"post_handshake_error:{type(exc).__name__}: {exc}"
+        return False, elapsed_ms, f"{type(exc).__name__}: {exc}"
+
+
+# ---------------------------------------------------------------------------
+# Storm phase
+# ---------------------------------------------------------------------------
+
+
+async def _ws_client(
+    idx: int,
+    url: str,
+    ws_frame: dict[str, Any],
+    deadline: float,
+    stats: CodexWsStats,
+) -> None:
+    ws_url = _http_to_ws_url(url, "/v1/responses")
+    try:
+        async with ws_connect(
+            ws_url,
+            additional_headers={"Authorization": f"Bearer repro-harness-ws-{idx}"},
+            open_timeout=10,
+        ) as ws:
+            stats.opened += 1
+            try:
+                await ws.send(json.dumps(ws_frame))
+            except (ConnectionClosed, WebSocketException) as exc:
+                stats.record_error(f"send:{type(exc).__name__}")
+                return
+            # Hold the session open, draining events until deadline or close.
+            while True:
+                remaining = deadline - time.perf_counter()
+                if remaining <= 0:
+                    break
+                try:
+                    raw = await asyncio.wait_for(ws.recv(), timeout=min(remaining, 5.0))
+                except asyncio.TimeoutError:
+                    continue
+                except ConnectionClosed:
+                    break
+                try:
+                    evt = json.loads(raw) if isinstance(raw, str) else {}
+                except json.JSONDecodeError:
+                    continue
+                etype = evt.get("type", "")
+                if etype == "response.completed":
+                    stats.response_completed += 1
+                    break
+                if etype in {"error", "response.failed"}:
+                    stats.record_error(f"upstream:{etype}")
+                    break
+    except (OSError, InvalidStatus) as exc:
+        stats.record_error(f"connect:{type(exc).__name__}")
+    except WebSocketException as exc:
+        stats.record_error(f"ws:{type(exc).__name__}")
+    except Exception as exc:  # pragma: no cover - defensive
+        stats.record_error(f"unexpected:{type(exc).__name__}")
+
+
+async def _anthropic_client(
+    idx: int,
+    url: str,
+    body: dict[str, Any],
+    deadline: float,
+    stats: AnthropicHttpStats,
+) -> None:
+    endpoint = url.rstrip("/") + "/v1/messages?beta=true"
+    headers = {
+        "Content-Type": "application/json",
+        "Authorization": f"Bearer repro-harness-anthropic-{idx}",
+        "anthropic-version": "2023-06-01",
+        "x-api-key": f"repro-harness-anthropic-{idx}",
+    }
+    retry_cutoff = min(deadline, time.perf_counter() + 60.0)
+    attempt = 0
+    async with httpx.AsyncClient(timeout=httpx.Timeout(30.0, connect=5.0)) as client:
+        while time.perf_counter() < deadline:
+            attempt += 1
+            stats.attempted += 1
+            start = time.perf_counter()
+            try:
+                resp = await client.post(endpoint, headers=headers, json=body)
+                first_byte_ms = (time.perf_counter() - start) * 1000.0
+                stats.first_byte_latency_ms.append(first_byte_ms)
+                # Drain body so the connection is released.
+                with suppress(Exception):
+                    _ = resp.content
+                if 200 <= resp.status_code < 300:
+                    stats.ok_2xx += 1
+                    return
+                stats.non_2xx += 1
+                if resp.status_code < 500:
+                    # Client error — no retry, agent would surface this.
+                    return
+            except httpx.TimeoutException:
+                stats.timed_out += 1
+            except (httpx.HTTPError, OSError):
+                stats.errors += 1
+            # Retry loop — mimic agent behavior with bounded wall clock.
+            if time.perf_counter() >= retry_cutoff:
+                return
+            # Exponential backoff with 50-150% jitter, mirroring the
+            # ``jitter_delay_ms(base_ms=250, max_ms=5000, attempt=n)``
+            # helper used inside the proxy. Keeping the formula inline
+            # so this script stays dependency-free of the proxy package.
+            _base_ms = 250
+            _max_ms = 5000
+            _delay_ms = min(_base_ms * (2 ** (attempt - 1)), _max_ms) * (0.5 + random.random())
+            await asyncio.sleep(_delay_ms / 1000.0)
+
+
+async def _livez_prober(
+    url: str,
+    histogram: LatencyHistogram,
+    deadline: float,
+    interval_ms: int = 250,
+) -> None:
+    endpoint = url.rstrip("/") + "/livez"
+    async with httpx.AsyncClient(timeout=httpx.Timeout(5.0, connect=2.0)) as client:
+        while time.perf_counter() < deadline:
+            start = time.perf_counter()
+            try:
+                resp = await client.get(endpoint)
+                elapsed_ms = (time.perf_counter() - start) * 1000.0
+                if resp.status_code == 200:
+                    histogram.record(elapsed_ms)
+                else:
+                    # Treat non-200 as worst-case for our threshold.
+                    histogram.record(max(elapsed_ms, 5000.0))
+            except (httpx.HTTPError, OSError):
+                histogram.record(5000.0)
+            # Tick every interval_ms regardless of probe latency.
+            sleep_for = interval_ms / 1000.0
+            next_wake = time.perf_counter() + sleep_for
+            remaining = max(0.0, next_wake - time.perf_counter())
+            if remaining > 0:
+                await asyncio.sleep(remaining)
+
+
+# ---------------------------------------------------------------------------
+# Orchestration
+# ---------------------------------------------------------------------------
+
+
+async def _check_reachable(url: str, timeout_s: float = 5.0) -> tuple[bool, str]:
+    endpoint = url.rstrip("/") + "/livez"
+    try:
+        async with httpx.AsyncClient(timeout=httpx.Timeout(timeout_s, connect=timeout_s)) as client:
+            resp = await client.get(endpoint)
+            return (True, f"HTTP {resp.status_code}")
+    except (httpx.ConnectError, ConnectionRefusedError, OSError) as exc:
+        return (False, f"{type(exc).__name__}: {exc}")
+    except httpx.HTTPError as exc:
+        return (True, f"reachable-but-error: {type(exc).__name__}: {exc}")
+
+
+async def run_harness(args: argparse.Namespace) -> dict[str, Any]:
+    ws_frame = _load_json(Path(args.ws_frame_fixture))
+    anthropic_body = _load_json(Path(args.anthropic_body_fixture))
+
+    # Reachability gate — quick, clear failure if the proxy is not up.
+    reachable, reach_note = await _check_reachable(args.url, timeout_s=5.0)
+    if not reachable:
+        return {
+            "ok": False,
+            "reason": "proxy_unreachable",
+            "detail": reach_note,
+            "url": args.url,
+        }
+
+    # Phase 1: warmup
+    warmup_result: dict[str, Any]
+    if args.no_warmup:
+        warmup_result = {"skipped": True}
+    else:
+        success, elapsed_ms, note = await warmup_probe(
+            args.url, ws_frame, timeout_s=args.warmup_timeout
+        )
+        warmup_result = {
+            "skipped": False,
+            "success": success,
+            "elapsed_ms": round(elapsed_ms, 2),
+            "note": note,
+        }
+
+    # Phase 2: storm
+    livez_hist = LatencyHistogram()
+    ws_stats = CodexWsStats()
+    http_stats = AnthropicHttpStats()
+
+    storm_start = time.perf_counter()
+    deadline = storm_start + args.duration
+
+    livez_task = asyncio.create_task(
+        _livez_prober(args.url, livez_hist, deadline, interval_ms=args.livez_interval_ms),
+        name="repro-livez-prober",
+    )
+    ws_tasks = [
+        asyncio.create_task(
+            _ws_client(i, args.url, ws_frame, deadline, ws_stats),
+            name=f"repro-ws-{i}",
+        )
+        for i in range(args.ws_clients)
+    ]
+    http_tasks = [
+        asyncio.create_task(
+            _anthropic_client(i, args.url, anthropic_body, deadline, http_stats),
+            name=f"repro-http-{i}",
+        )
+        for i in range(args.anthropic_clients)
+    ]
+
+    # Let the storm run. Gather storm tasks but keep probing /livez for the
+    # full requested duration regardless of when the clients exit — the whole
+    # point is to observe event-loop health across the window.
+    storm_tasks = ws_tasks + http_tasks
+    try:
+        await asyncio.wait_for(
+            asyncio.gather(*storm_tasks, return_exceptions=True),
+            timeout=args.duration + 10.0,
+        )
+    except asyncio.TimeoutError:
+        for t in storm_tasks:
+            if not t.done():
+                t.cancel()
+        await asyncio.gather(*storm_tasks, return_exceptions=True)
+
+    # Let the /livez prober run to its own deadline so the histogram covers
+    # the full window, not just the interval where storm tasks were alive.
+    remaining = deadline - time.perf_counter()
+    if remaining > 0:
+        with suppress(asyncio.TimeoutError):
+            await asyncio.wait_for(asyncio.shield(livez_task), timeout=remaining + 2.0)
+    if not livez_task.done():
+        livez_task.cancel()
+    with suppress(asyncio.CancelledError, Exception):
+        await livez_task
+
+    storm_duration_s = time.perf_counter() - storm_start
+    livez_summary = livez_hist.as_summary()
+
+    # Soft assertion: livez p99 under threshold.
+    threshold_ms = args.livez_threshold_ms
+    livez_ok = livez_summary["p99"] <= threshold_ms
+
+    return {
+        "ok": (warmup_result.get("skipped", False) or warmup_result.get("success", False))
+        and livez_ok,
+        "warmup": warmup_result,
+        "storm": {
+            "ws_clients": args.ws_clients,
+            "anthropic_clients": args.anthropic_clients,
+            "requested_duration_s": args.duration,
+            "actual_duration_s": round(storm_duration_s, 3),
+        },
+        "livez": {
+            **livez_summary,
+            "threshold_ms": threshold_ms,
+            "threshold_ok": livez_ok,
+        },
+        "codex_ws": {
+            "opened": ws_stats.opened,
+            "response_completed": ws_stats.response_completed,
+            "errors": dict(ws_stats.errors),
+        },
+        "anthropic_http": {
+            "attempted": http_stats.attempted,
+            "ok_2xx": http_stats.ok_2xx,
+            "non_2xx": http_stats.non_2xx,
+            "timed_out": http_stats.timed_out,
+            "errors": http_stats.errors,
+            "avg_first_byte_ms": round(http_stats.avg_first_byte_ms, 2),
+        },
+    }
+
+
+# ---------------------------------------------------------------------------
+# Printing
+# ---------------------------------------------------------------------------
+
+
+def format_summary(result: dict[str, Any]) -> str:
+    if result.get("reason") == "proxy_unreachable":
+        return (
+            "Proxy unreachable at {url}.\n"
+            "  {detail}\n"
+            "  Hint: start the proxy with `headroom proxy` and retry."
+        ).format(**result)
+
+    lines: list[str] = []
+    lines.append("=" * 72)
+    lines.append("Codex proxy reconnect-storm repro harness — summary")
+    lines.append("=" * 72)
+    warm = result.get("warmup", {})
+    if warm.get("skipped"):
+        lines.append("Warmup:         skipped")
+    else:
+        lines.append(
+            "Warmup:         success={success} elapsed_ms={elapsed_ms} note={note}".format(**warm)
+        )
+    storm = result.get("storm", {})
+    lines.append(
+        "Storm:          ws_clients={ws_clients} anthropic_clients={anthropic_clients} "
+        "requested={requested_duration_s}s actual={actual_duration_s}s".format(**storm)
+    )
+    livez = result.get("livez", {})
+    lines.append(
+        "/livez:         count={count} p50={p50:.2f}ms p95={p95:.2f}ms "
+        "p99={p99:.2f}ms max={max:.2f}ms (threshold={threshold_ms}ms, "
+        "ok={threshold_ok})".format(**livez)
+    )
+    ws = result.get("codex_ws", {})
+    lines.append(
+        "Codex WS:       opened={opened} response.completed={response_completed} "
+        "errors={errors}".format(**ws)
+    )
+    http = result.get("anthropic_http", {})
+    lines.append(
+        "Anthropic HTTP: attempted={attempted} ok_2xx={ok_2xx} non_2xx={non_2xx} "
+        "timed_out={timed_out} errors={errors} avg_first_byte_ms={avg_first_byte_ms}".format(**http)
+    )
+    lines.append("=" * 72)
+    lines.append("RESULT: {}".format("OK" if result.get("ok") else "FAIL"))
+    lines.append("=" * 72)
+    return "\n".join(lines)
+
+
+# ---------------------------------------------------------------------------
+# CLI
+# ---------------------------------------------------------------------------
+
+
+def build_parser() -> argparse.ArgumentParser:
+    parser = argparse.ArgumentParser(
+        description="Reproducibly exercise the multi-agent Codex reconnect/retry "
+        "storm against a local Headroom proxy.",
+        formatter_class=argparse.ArgumentDefaultsHelpFormatter,
+    )
+    parser.add_argument(
+        "--url",
+        default="http://127.0.0.1:8787",
+        help="Base URL of the running proxy.",
+    )
+    parser.add_argument(
+        "--ws-clients",
+        type=int,
+        default=8,
+        help="Number of concurrent Codex WS connections to open during the storm.",
+    )
+    parser.add_argument(
+        "--anthropic-clients",
+        type=int,
+        default=4,
+        help="Number of concurrent Anthropic /v1/messages POST clients.",
+    )
+    parser.add_argument(
+        "--duration",
+        type=float,
+        default=30.0,
+        help="Total storm phase length, in seconds.",
+    )
+    parser.add_argument(
+        "--livez-threshold-ms",
+        type=float,
+        default=500.0,
+        help="Soft assertion threshold for /livez p99 (ms). Exit non-zero if exceeded.",
+    )
+    parser.add_argument(
+        "--livez-interval-ms",
+        type=int,
+        default=250,
+        help="Interval between /livez probes, in milliseconds.",
+    )
+    parser.add_argument(
+        "--warmup-timeout",
+        type=float,
+        default=10.0,
+        help="Max seconds to wait for the warmup WS session to complete.",
+    )
+    parser.add_argument(
+        "--no-warmup",
+        action="store_true",
+        help="Skip the warmup probe phase.",
+    )
+    parser.add_argument(
+        "--ws-frame-fixture",
+        default=str(DEFAULT_WS_FRAME_FIXTURE),
+        help="Path to the Codex response.create frame JSON fixture.",
+    )
+    parser.add_argument(
+        "--anthropic-body-fixture",
+        default=str(DEFAULT_ANTHROPIC_BODY_FIXTURE),
+        help="Path to the Anthropic /v1/messages request body JSON fixture.",
+    )
+    parser.add_argument(
+        "--json",
+        action="store_true",
+        help=(
+            "Emit the full summary as JSON on stdout; the human-readable "
+            "summary is routed to stderr. Without --json, the human summary "
+            "is on stdout and nothing goes to JSON."
+        ),
+    )
+    return parser
+
+
+# Exit codes. Keep POSIX-style — 0 means success, non-zero categorises
+# the failure so callers (CI, smoke tests, shell wrappers) can branch.
+EXIT_OK = 0
+EXIT_CRASH = 1
+EXIT_PROXY_UNREACHABLE = 2
+EXIT_LIVEZ_THRESHOLD = 3
+EXIT_WARMUP_FAILED = 4
+EXIT_SIGINT = 130
+
+
+def _classify_exit(result: dict[str, Any]) -> int:
+    """Map a harness result dict onto one of the exit constants above."""
+    if result.get("reason") == "proxy_unreachable":
+        return EXIT_PROXY_UNREACHABLE
+    warm = result.get("warmup", {})
+    if not warm.get("skipped", False) and not warm.get("success", True):
+        return EXIT_WARMUP_FAILED
+    livez = result.get("livez", {})
+    if livez and not livez.get("threshold_ok", True):
+        return EXIT_LIVEZ_THRESHOLD
+    return EXIT_OK if result.get("ok") else EXIT_CRASH
+
+
+def main(argv: list[str] | None = None) -> int:
+    parser = build_parser()
+    args = parser.parse_args(argv)
+    try:
+        result = asyncio.run(run_harness(args))
+    except KeyboardInterrupt:
+        print("\nInterrupted by user.", file=sys.stderr)
+        return EXIT_SIGINT
+    except Exception as exc:  # noqa: BLE001 - top-level guard
+        print(f"Harness crashed: {type(exc).__name__}: {exc}", file=sys.stderr)
+        return EXIT_CRASH
+
+    # Output routing: when --json is set, JSON on stdout (machine-readable)
+    # and human summary on stderr (so operators still see what happened
+    # without polluting stdout). Without --json, human summary on stdout.
+    if args.json:
+        print(format_summary(result), file=sys.stderr)
+        print(json.dumps(result, indent=2, sort_keys=True))
+    else:
+        print(format_summary(result))
+
+    return _classify_exit(result)
+
+
+if __name__ == "__main__":
+    sys.exit(main())

scripts/smoke_issue_327.py

@@ -0,0 +1,233 @@
+"""Issue #327 — live API smoke test.
+
+Drives a 10-turn multi-turn conversation against `api.anthropic.com` directly
+(NO proxy in the loop) and against the local Headroom proxy, and asserts:
+
+  1. Both shapes of `tool_result` content (string and list-of-blocks) are
+     accepted by the upstream API for both streaming and non-streaming.
+  2. When proxied through Headroom in token mode, at least one post-warmup
+     turn has `transforms_applied != []` AND `cache_read_input_tokens > 0`
+     on turns 2+ (proves prefix cache is intact AND compression resumed).
+  3. No cache busts (bust_count stays at 0 across the session).
+
+GUARDS
+
+* Skipped unless `RUN_LIVE_API=1` to keep CI hermetic.
+* Requires `ANTHROPIC_API_KEY` in env (read from `.env` if not exported).
+* Costs a few cents per run (small messages × 10 turns × 2 modes).
+
+USAGE
+
+    RUN_LIVE_API=1 .venv/bin/python scripts/smoke_issue_327.py
+
+    # With local proxy running on :8787:
+    RUN_LIVE_API=1 HEADROOM_PROXY_URL=http://localhost:8787 \
+        .venv/bin/python scripts/smoke_issue_327.py
+"""
+
+from __future__ import annotations
+
+import json
+import os
+import sys
+from pathlib import Path
+
+if os.environ.get("RUN_LIVE_API") != "1":
+    print("Set RUN_LIVE_API=1 to run this smoke test.")
+    sys.exit(0)
+
+# Hydrate .env if needed
+_env = Path(__file__).resolve().parent.parent / ".env"
+if _env.exists():
+    for line in _env.read_text().splitlines():
+        line = line.strip()
+        if not line or line.startswith("#") or "=" not in line:
+            continue
+        k, _, v = line.partition("=")
+        os.environ.setdefault(k, v.strip().strip('"').strip("'"))
+
+KEY = os.environ.get("ANTHROPIC_API_KEY")
+if not KEY:
+    print("ANTHROPIC_API_KEY not set — cannot run live smoke")
+    sys.exit(1)
+
+import anthropic  # noqa: E402
+
+PROXY_URL = os.environ.get("HEADROOM_PROXY_URL")  # optional
+TOOLS = [
+    {
+        "name": "get_lines",
+        "description": "Return N lines of synthetic test output for compression testing",
+        "input_schema": {
+            "type": "object",
+            "properties": {"n": {"type": "integer"}},
+            "required": ["n"],
+        },
+    }
+]
+LARGE_OUTPUT = "\n".join(
+    f"line {i:04d}: synthetic content for compression smoke test" for i in range(120)
+)
+
+
+def _make_string_tool_result(tool_use_id: str) -> dict:
+    return {"type": "tool_result", "tool_use_id": tool_use_id, "content": LARGE_OUTPUT}
+
+
+def _make_list_tool_result(tool_use_id: str) -> dict:
+    return {
+        "type": "tool_result",
+        "tool_use_id": tool_use_id,
+        "content": [{"type": "text", "text": LARGE_OUTPUT}],
+    }
+
+
+def _drive_conversation(client: anthropic.Anthropic, *, shape: str, stream: bool) -> dict:
+    """Drive a 10-turn conversation, return aggregate stats."""
+    print(f"\n=== shape={shape} stream={stream} ===")
+    messages: list = []
+    cache_reads: list[int] = []
+    cache_writes: list[int] = []
+    turn_results: list[dict] = []
+
+    for turn in range(10):
+        if turn == 0:
+            messages.append({"role": "user", "content": "Use get_lines with n=120"})
+        else:
+            messages.append({"role": "user", "content": f"continue {turn}, run get_lines again"})
+
+        try:
+            kwargs = {
+                "model": "claude-haiku-4-5-20251001",
+                "max_tokens": 128,
+                "tools": TOOLS,
+                "messages": messages,
+            }
+            if stream:
+                with client.messages.stream(**kwargs) as s:
+                    for _ in s:
+                        pass
+                    final = s.get_final_message()
+            else:
+                final = client.messages.create(**kwargs)
+
+            usage = getattr(final, "usage", None)
+            cache_read = getattr(usage, "cache_read_input_tokens", 0) or 0
+            cache_write = getattr(usage, "cache_creation_input_tokens", 0) or 0
+            cache_reads.append(cache_read)
+            cache_writes.append(cache_write)
+
+            # Append the assistant turn
+            messages.append({"role": "assistant", "content": final.content})
+
+            # If model called the tool, send back a tool_result in the chosen shape
+            tool_use_block = None
+            for blk in final.content:
+                if getattr(blk, "type", None) == "tool_use":
+                    tool_use_block = blk
+                    break
+            if tool_use_block is not None:
+                tool_use_id = tool_use_block.id
+                if shape == "string":
+                    tr = _make_string_tool_result(tool_use_id)
+                else:
+                    tr = _make_list_tool_result(tool_use_id)
+                messages.append({"role": "user", "content": [tr]})
+
+            turn_results.append(
+                {
+                    "turn": turn,
+                    "stop_reason": final.stop_reason,
+                    "cache_read_input_tokens": cache_read,
+                    "cache_creation_input_tokens": cache_write,
+                }
+            )
+            print(
+                f"  turn {turn}: stop={final.stop_reason} "
+                f"cache_read={cache_read} cache_write={cache_write}"
+            )
+        except anthropic.APIError as e:
+            print(f"  turn {turn} FAIL: {type(e).__name__}: {e}")
+            return {"ok": False, "error": str(e), "turn_results": turn_results}
+
+    return {
+        "ok": True,
+        "turn_results": turn_results,
+        "cache_reads": cache_reads,
+        "cache_writes": cache_writes,
+    }
+
+
+def main() -> int:
+    base_url_kwarg = {}
+    if PROXY_URL:
+        base_url_kwarg["base_url"] = PROXY_URL
+        print(f"Routing through proxy: {PROXY_URL}")
+    else:
+        print("Direct to api.anthropic.com (no proxy)")
+
+    client = anthropic.Anthropic(api_key=KEY, **base_url_kwarg)
+
+    overall_ok = True
+    matrix = [
+        ("string", False),
+        ("string", True),
+        ("list_of_blocks", False),
+        ("list_of_blocks", True),
+    ]
+
+    summary: dict = {}
+    for shape, stream in matrix:
+        r = _drive_conversation(client, shape=shape, stream=stream)
+        summary[(shape, stream)] = r
+        if not r["ok"]:
+            overall_ok = False
+            continue
+
+        # Assert: at least one turn after turn 0 reports cache_read > 0 (prefix
+        # cache is being used).
+        post_turn0_cache_reads = [t["cache_read_input_tokens"] for t in r["turn_results"][1:]]
+        if not any(c > 0 for c in post_turn0_cache_reads):
+            print(
+                f"  WARN: shape={shape} stream={stream} — no post-warmup turn had "
+                f"cache_read > 0. cache_reads={post_turn0_cache_reads}"
+            )
+
+    print("\n=== SUMMARY ===")
+    for k, v in summary.items():
+        print(f"  {k}: ok={v['ok']}")
+
+    print("\n--- Per-turn cache_read tokens ---")
+    for k, v in summary.items():
+        if v["ok"]:
+            print(f"  {k}: {v['cache_reads']}")
+
+    print(f"\noverall_ok={overall_ok}")
+    if PROXY_URL:
+        try:
+            import httpx
+
+            stats = httpx.get(f"{PROXY_URL}/stats", timeout=5).json()
+            print("\n=== Proxy /stats key fields ===")
+            print(
+                json.dumps(
+                    {
+                        "summary": stats.get("summary"),
+                        "compressions_by_strategy": stats.get("compressions_by_strategy"),
+                        "tokens_saved_by_strategy": stats.get("tokens_saved_by_strategy"),
+                        "compression_cache": stats.get("compression_cache"),
+                        "prefix_cache_busts": (
+                            stats.get("prefix_cache", {}).get("totals", {}).get("bust_count")
+                        ),
+                    },
+                    indent=2,
+                )
+            )
+        except Exception as e:  # pragma: no cover
+            print(f"  proxy stats fetch failed: {e}")
+
+    return 0 if overall_ok else 1
+
+
+if __name__ == "__main__":
+    sys.exit(main())

scripts/sync-plugin-versions.py

@@ -0,0 +1,116 @@
+"""Sync plugin manifest versions to the repo's computed release semver.
+
+Branch-aware: by default this script is a NO-OP on feature branches.
+Pre-this-fix it ran on every commit and bumped the manifests to the
+PREDICTED next release version, which polluted every PR with version-
+bump noise (the prediction advanced as commits landed; each PR ended
+up carrying the bump as collateral).
+
+Sync now only runs when EITHER:
+  * We're on the ``main`` branch, OR
+  * ``HEADROOM_SYNC_VERSIONS=1`` is set explicitly (release workflow)
+
+Result: feature-branch PRs no longer carry manifest bumps; the
+release workflow still gets a canonical sync at publish time.
+"""
+
+from __future__ import annotations
+
+import os
+import subprocess
+import sys
+from pathlib import Path
+
+ROOT = Path(__file__).resolve().parent.parent
+if str(ROOT) not in sys.path:
+    sys.path.insert(0, str(ROOT))
+
+try:
+    from headroom.release_version import (  # noqa: E402
+        compute_release_version,
+        determine_bump_level,
+        find_latest_release_tag,
+        get_canonical_version,
+        list_release_commits,
+        list_release_tags,
+    )
+except ImportError:
+    print("skip: headroom deps not installed (run from a dev venv to enable)")
+    sys.exit(0)
+
+
+def compute_repo_semver(root: Path) -> str:
+    """Return the npm-style semver for the repo's next release."""
+    tags = list_release_tags(root)
+    previous_tag = find_latest_release_tag(tags) or ""
+    level = determine_bump_level(list_release_commits(root, previous_tag))
+    info = compute_release_version(
+        canonical_version=get_canonical_version(root),
+        level=level,
+        tags=tags,
+    )
+    return info.npm_version
+
+
+def _current_branch(root: Path) -> str | None:
+    """Return the current git branch name, or None if git isn't usable."""
+    try:
+        result = subprocess.run(
+            ["git", "rev-parse", "--abbrev-ref", "HEAD"],
+            cwd=root,
+            capture_output=True,
+            text=True,
+            check=False,
+        )
+    except (FileNotFoundError, OSError):
+        return None
+    if result.returncode != 0:
+        return None
+    return result.stdout.strip() or None
+
+
+def _should_sync(root: Path) -> bool:
+    """Decide whether to actually run the sync.
+
+    Release workflow opts in via ``HEADROOM_SYNC_VERSIONS=1``; otherwise
+    we only sync on ``main`` (where the next-release prediction
+    legitimately lives). On feature branches we no-op — the prediction
+    would just create PR-level noise.
+    """
+    if os.environ.get("HEADROOM_SYNC_VERSIONS") == "1":
+        return True
+    branch = _current_branch(root)
+    if branch is None:
+        # Git unavailable or detached HEAD — safest default is no-op.
+        return False
+    return branch == "main"
+
+
+def main() -> None:
+    root = ROOT
+    if not _should_sync(root):
+        # Quiet no-op on feature branches. Print a single line so
+        # pre-commit users see the reason if they look.
+        branch = _current_branch(root) or "<unknown>"
+        print(
+            f"sync-plugin-versions: skipping on branch '{branch}' (set HEADROOM_SYNC_VERSIONS=1 to force)"
+        )
+        return
+    version = compute_repo_semver(root)
+    subprocess.run(
+        [
+            sys.executable,
+            str(root / "scripts" / "version-sync.py"),
+            "--root",
+            str(root),
+            "--version",
+            version,
+            "--plugin-manifests-only",
+        ],
+        cwd=root,
+        check=True,
+    )
+
+
+if __name__ == "__main__":
+    main()

scripts/tests/test_changelog_gen.py

@@ -0,0 +1,302 @@
+"""Tests for changelog-gen.py."""
+
+from __future__ import annotations
+
+import importlib.util
+from pathlib import Path
+
+import pytest
+
+ROOT = Path(__file__).resolve().parent.parent.parent
+
+# Load changelog_gen module from scripts directory (filename has hyphen)
+_spec = importlib.util.spec_from_file_location(
+    "changelog_gen", ROOT / "scripts" / "changelog-gen.py"
+)
+if _spec is None:
+    raise ImportError("Could not load changelog_gen module")
+_changelog_gen = importlib.util.module_from_spec(_spec)
+if _spec.loader is None:
+    raise ImportError("Could not load changelog_gen module")
+_spec.loader.exec_module(_changelog_gen)
+
+COMMIT_PATTERN = _changelog_gen.COMMIT_PATTERN
+BREAKING_CHANGE_PATTERN = _changelog_gen.BREAKING_CHANGE_PATTERN
+FIELD_SEP = _changelog_gen.FIELD_SEP
+RECORD_SEP = _changelog_gen.RECORD_SEP
+ParsedCommit = _changelog_gen.ParsedCommit
+generate_changelog = _changelog_gen.generate_changelog
+iter_commit_entries = _changelog_gen.iter_commit_entries
+parse_commits = _changelog_gen.parse_commits
+
+
+def make_log_entry(subject: str, commit_hash: str, body: str = "") -> str:
+    return f"{subject}{FIELD_SEP}{body}{FIELD_SEP}{commit_hash}{RECORD_SEP}"
+
+
+class TestParseCommits:
+    """Tests for parse_commits function."""
+
+    def test_parses_feat_commit(self) -> None:
+        log_output = make_log_entry("feat(core): add feature", "abc1234")
+        commits = parse_commits(log_output)
+        assert len(commits) == 1
+        assert commits[0].type == "feat"
+        assert commits[0].scope == "core"
+        assert commits[0].message == "add feature"
+        assert commits[0].hash == "abc1234"
+        assert commits[0].breaking is False
+
+    def test_parses_fix_commit(self) -> None:
+        log_output = make_log_entry("fix(ui): fix bug", "def5678")
+        commits = parse_commits(log_output)
+        assert len(commits) == 1
+        assert commits[0].type == "fix"
+        assert commits[0].scope == "ui"
+        assert commits[0].message == "fix bug"
+        assert commits[0].hash == "def5678"
+
+    def test_parses_ci_commit(self) -> None:
+        log_output = make_log_entry("ci: update github actions", "xyz789")
+        commits = parse_commits(log_output)
+        assert len(commits) == 1
+        assert commits[0].type == "ci"
+        assert commits[0].scope is None
+        assert commits[0].message == "update github actions"
+        assert commits[0].hash == "xyz789"
+
+    def test_parses_chore_commit(self) -> None:
+        log_output = make_log_entry("chore: cleanup", "xyz999")
+        commits = parse_commits(log_output)
+        assert len(commits) == 1
+        assert commits[0].type == "chore"
+        assert commits[0].scope is None
+
+    def test_parses_perf_commit(self) -> None:
+        log_output = make_log_entry("perf(dashboard): improve performance", "abc111")
+        commits = parse_commits(log_output)
+        assert len(commits) == 1
+        assert commits[0].type == "perf"
+        assert commits[0].scope == "dashboard"
+
+    def test_parses_refactor_commit(self) -> None:
+        log_output = make_log_entry("refactor(api): refactor endpoint", "abc222")
+        commits = parse_commits(log_output)
+        assert len(commits) == 1
+        assert commits[0].type == "refactor"
+        assert commits[0].scope == "api"
+
+    def test_parses_docs_commit(self) -> None:
+        log_output = make_log_entry("docs: update readme", "abc333")
+        commits = parse_commits(log_output)
+        assert len(commits) == 1
+        assert commits[0].type == "docs"
+
+    def test_parses_style_commit(self) -> None:
+        log_output = make_log_entry("style: format code", "abc444")
+        commits = parse_commits(log_output)
+        assert len(commits) == 1
+        assert commits[0].type == "style"
+
+    def test_parses_test_commit(self) -> None:
+        log_output = make_log_entry("test: add tests for feature", "abc555")
+        commits = parse_commits(log_output)
+        assert len(commits) == 1
+        assert commits[0].type == "test"
+
+    def test_detects_breaking_change_in_body(self) -> None:
+        log_output = make_log_entry(
+            "feat(core): add feature",
+            "abc666",
+            "BREAKING CHANGE: api changed",
+        )
+        commits = parse_commits(log_output)
+        assert len(commits) == 1
+        assert commits[0].breaking is True
+
+    def test_detects_breaking_change_exclamation(self) -> None:
+        log_output = make_log_entry("feat(core)!: api changed", "abc777")
+        commits = parse_commits(log_output)
+        assert len(commits) == 1
+        assert commits[0].breaking is True
+
+    def test_no_scope_no_problem(self) -> None:
+        log_output = make_log_entry("feat: simple feature", "abc888")
+        commits = parse_commits(log_output)
+        assert len(commits) == 1
+        assert commits[0].scope is None
+        assert commits[0].message == "simple feature"
+
+    def test_uses_pr_title_from_merge_commit_body(self) -> None:
+        log_output = make_log_entry(
+            "Merge pull request #173 from JerrettDavis/fix/pipeline-permissions-and-docs",
+            "73f6673",
+            "fix: repair release and docs pipelines",
+        )
+        commits = parse_commits(log_output)
+        assert len(commits) == 1
+        assert commits[0].type == "fix"
+        assert commits[0].message == "repair release and docs pipelines"
+
+    def test_falls_back_to_other_changes_for_non_conventional_merge(self) -> None:
+        log_output = make_log_entry(
+            "Merge pull request #186 from skorokithakis/patch-1",
+            "1e80ee3",
+            "Add support for custom Anthropic API URL",
+        )
+        commits = parse_commits(log_output)
+        assert len(commits) == 1
+        assert commits[0].type == "other"
+        assert commits[0].message == "Add support for custom Anthropic API URL"
+
+    def test_iter_commit_entries_parses_real_git_log_delimiters(self) -> None:
+        log_output = make_log_entry(
+            "fix: patch release flow", "abc1234", "BREAKING CHANGE: no"
+        ) + make_log_entry("docs: update readme", "def5678")
+        assert iter_commit_entries(log_output) == [
+            ("fix: patch release flow", "BREAKING CHANGE: no", "abc1234"),
+            ("docs: update readme", "", "def5678"),
+        ]
+
+    def test_iter_commit_entries_keeps_field_separator_inside_body(self) -> None:
+        body = f"line one{FIELD_SEP}line two"
+        log_output = make_log_entry("fix: patch release flow", "abc1234", body)
+        assert iter_commit_entries(log_output) == [
+            ("fix: patch release flow", body, "abc1234"),
+        ]
+
+
+class TestGenerateChangelog:
+    """Tests for generate_changelog function."""
+
+    def test_generates_version_header(self) -> None:
+        commits = [
+            ParsedCommit(type="feat", scope=None, breaking=False, message="test", hash="abc123")
+        ]
+        result = generate_changelog("0.6.0", commits)
+        assert "## [0.6.0]" in result
+
+    def test_includes_date(self) -> None:
+        commits = []
+        result = generate_changelog("0.6.0", commits)
+        import re
+
+        date_match = re.search(r"\d{4}-\d{2}-\d{2}", result)
+        assert date_match is not None
+
+    def test_groups_by_type(self) -> None:
+        commits = [
+            ParsedCommit(
+                type="feat", scope=None, breaking=False, message="add feature", hash="abc123"
+            ),
+            ParsedCommit(type="fix", scope=None, breaking=False, message="fix bug", hash="def456"),
+        ]
+        result = generate_changelog("0.6.0", commits)
+        assert "### Features" in result
+        assert "### Bug Fixes" in result
+        assert "- add feature (abc123)" in result
+        assert "- fix bug (def456)" in result
+
+    def test_includes_scope_in_bullet(self) -> None:
+        commits = [
+            ParsedCommit(
+                type="feat", scope="core", breaking=False, message="add feature", hash="abc123"
+            )
+        ]
+        result = generate_changelog("0.6.0", commits)
+        assert "- **core**: add feature (abc123)" in result
+
+    def test_breaking_change_section_when_present(self) -> None:
+        commits = [
+            ParsedCommit(
+                type="feat", scope="core", breaking=True, message="api changed", hash="abc123"
+            )
+        ]
+        result = generate_changelog("0.6.0", commits)
+        assert "### Breaking Changes" in result
+        assert "**core**" in result
+
+    def test_no_breaking_change_section_when_none(self) -> None:
+        commits = [
+            ParsedCommit(
+                type="feat", scope=None, breaking=False, message="add feature", hash="abc123"
+            )
+        ]
+        result = generate_changelog("0.6.0", commits)
+        assert "Breaking Changes" not in result
+
+    def test_includes_other_changes_section(self) -> None:
+        commits = [
+            ParsedCommit(
+                type="other",
+                scope=None,
+                breaking=False,
+                message="Add support for custom Anthropic API URL",
+                hash="abc123",
+            )
+        ]
+        result = generate_changelog("0.6.0", commits)
+        assert "### Other Changes" in result
+        assert "- Add support for custom Anthropic API URL (abc123)" in result
+
+
+class TestIntegrationWithMock:
+    """Integration tests with mocked subprocess.run."""
+
+    def test_full_flow_with_mocked_git(self) -> None:
+        log_output = (
+            make_log_entry("feat(core): add new feature", "abc1234")
+            + make_log_entry("fix(ui): fix bug", "def5678")
+            + make_log_entry("ci: update github actions", "xyz789")
+            + make_log_entry(
+                "feat(outer): breaking change", "bbb111", "BREAKING CHANGE: this is breaking"
+            )
+            + make_log_entry("chore: cleanup", "yyy999")
+        )
+        commits = parse_commits(log_output)
+
+        assert len(commits) == 5
+        assert any(c.type == "feat" and c.scope == "core" for c in commits)
+        assert any(c.type == "fix" and c.scope == "ui" for c in commits)
+        assert any(c.type == "ci" for c in commits)
+        assert any(c.type == "feat" and c.breaking for c in commits)
+        assert any(c.type == "chore" for c in commits)
+
+        changelog = generate_changelog("0.7.0", commits)
+        assert "## [0.7.0]" in changelog
+        assert "### Features" in changelog
+        assert "### Bug Fixes" in changelog
+        assert "### CI/CD" in changelog
+        assert "### Breaking Changes" in changelog
+
+
+class TestCommitPattern:
+    """Tests for the commit regex pattern."""
+
+    def test_feat_with_scope(self) -> None:
+        match = COMMIT_PATTERN.match("feat(core): add feature")
+        assert match is not None
+        assert match.group(1) == "feat"
+        assert match.group(2) == "(core)"
+        assert match.group(4) == "add feature"
+
+    def test_fix_without_scope(self) -> None:
+        match = COMMIT_PATTERN.match("fix: fix bug")
+        assert match is not None
+        assert match.group(1) == "fix"
+        assert match.group(2) is None
+        assert match.group(4) == "fix bug"
+
+    def test_with_exclamation(self) -> None:
+        match = COMMIT_PATTERN.match("feat(core)!: api changed")
+        assert match is not None
+        assert match.group(3) == "!"
+
+    def test_without_exclamation(self) -> None:
+        match = COMMIT_PATTERN.match("feat(core): add feature")
+        assert match is not None
+        assert match.group(3) is None
+
+
+if __name__ == "__main__":
+    pytest.main([__file__, "-v"])

scripts/tests/test_ci_workflow.py

@@ -0,0 +1,16 @@
+"""Tests for CI workflow hardening contracts."""
+
+from __future__ import annotations
+
+from pathlib import Path
+
+
+def test_sharded_ci_verifies_offline_huggingface_cache_before_pytest() -> None:
+    workflow = Path(".github/workflows/ci.yml").read_text(encoding="utf-8")
+
+    verify_step = "Verify offline HuggingFace model cache"
+    pytest_step = "Run test shard ${{ matrix.shard }}/4"
+
+    assert verify_step in workflow
+    assert "python scripts/ci/verify_hf_model_cache.py" in workflow
+    assert workflow.index(verify_step) < workflow.index(pytest_step)

scripts/tests/test_pr_governance.py

@@ -0,0 +1,214 @@
+"""Tests for pr-governance.py."""
+
+from __future__ import annotations
+
+import importlib.util
+import json
+import sys
+from pathlib import Path
+
+
+def _load_module():
+    script = Path(__file__).parent.parent / "pr-governance.py"
+    spec = importlib.util.spec_from_file_location("pr_governance", script)
+    assert spec is not None
+    assert spec.loader is not None
+    module = importlib.util.module_from_spec(spec)
+    sys.modules[spec.name] = module
+    spec.loader.exec_module(module)
+    return module
+
+
+def _event(body: str, *, draft: bool = False, login: str = "octocat") -> dict[str, object]:
+    return {
+        "pull_request": {
+            "number": 42,
+            "draft": draft,
+            "body": body,
+            "user": {"login": login},
+        }
+    }
+
+
+VALID_BODY = """## Description
+
+Add a required PR-governance gate for template validation and review readiness.
+
+Closes #123
+
+## Type of Change
+
+- [x] New feature (non-breaking change that adds functionality)
+- [ ] Documentation update
+
+## Changes Made
+
+- Added a workflow-backed PR template validator.
+- Added local commit message linting in the commit-msg hook.
+
+## Testing
+
+- [x] Unit tests pass (`pytest`)
+- [x] Manual testing performed
+
+### Test Output
+
+```text
+pytest scripts/tests/test_pr_governance.py -q
+```
+
+## Real Behavior Proof
+
+- Environment: Ubuntu runner, Python 3.12
+- Exact command / steps: Open a PR, remove the ready checkbox, re-run the workflow.
+- Observed result: The governance check fails and the PR gets a needs-author-action label.
+- Not tested: Automatic Copilot review rulesets in repository settings.
+
+## Review Readiness
+
+- [x] I have performed a self-review
+- [x] This PR is ready for human review
+
+## Additional Notes
+
+- Maintainers can optionally enable Copilot code review from repository rulesets.
+"""
+
+
+def test_validate_pull_request_marks_ready_pr_valid() -> None:
+    module = _load_module()
+
+    report = module.validate_pull_request(_event(VALID_BODY))
+
+    assert report.valid is True
+    assert report.ready_for_review is True
+    assert report.needs_author_action is False
+    assert report.problems == []
+    assert report.labels_to_add == [module.READY_LABEL]
+    assert module.AUTHOR_ACTION_LABEL in report.labels_to_remove
+
+
+def test_validate_pull_request_accepts_crlf_test_output_code_block() -> None:
+    module = _load_module()
+    body = VALID_BODY.replace("\n", "\r\n")
+
+    report = module.validate_pull_request(_event(body))
+
+    assert report.valid is True
+    assert report.problems == []
+
+
+def test_validate_pull_request_body_override_uses_live_body() -> None:
+    module = _load_module()
+    stale_event_body = ""
+
+    report = module.validate_pull_request_body(_event(stale_event_body), VALID_BODY)
+
+    assert report.valid is True
+    assert report.ready_for_review is True
+    assert report.problems == []
+
+
+def test_cli_body_file_override_uses_live_body(tmp_path: Path, monkeypatch) -> None:
+    module = _load_module()
+    event_path = tmp_path / "event.json"
+    body_path = tmp_path / "body.md"
+    report_path = tmp_path / "report.json"
+    event_path.write_text(
+        json.dumps(_event("")),
+        encoding="utf-8",
+    )
+    body_path.write_text(VALID_BODY, encoding="utf-8")
+    monkeypatch.delenv("GITHUB_OUTPUT", raising=False)
+
+    exit_code = module.main(
+        [
+            "--event",
+            str(event_path),
+            "--body-file",
+            str(body_path),
+            "--report",
+            str(report_path),
+        ]
+    )
+
+    assert exit_code == 0
+    report = json.loads(report_path.read_text(encoding="utf-8"))
+    assert report["valid"] is True
+    assert report["ready_for_review"] is True
+
+
+def test_validate_pull_request_allows_draft_without_ready_checkboxes() -> None:
+    module = _load_module()
+    body = VALID_BODY.replace(
+        "- [x] I have performed a self-review", "- [ ] I have performed a self-review"
+    )
+    body = body.replace(
+        "- [x] This PR is ready for human review",
+        "- [ ] This PR is ready for human review",
+    )
+
+    report = module.validate_pull_request(_event(body, draft=True))
+
+    assert report.valid is True
+    assert report.ready_for_review is False
+    assert report.needs_author_action is False
+    assert report.labels_to_add == []
+    assert module.READY_LABEL in report.labels_to_remove
+
+
+def test_validate_pull_request_fails_on_missing_required_content() -> None:
+    module = _load_module()
+    body = """## Description
+
+Fixes #123
+
+## Type of Change
+
+- [ ] New feature (non-breaking change that adds functionality)
+
+## Changes Made
+
+- Change 1
+
+## Testing
+
+### Test Output
+
+```text
+# Paste relevant command output or artifact links here
+```
+
+## Real Behavior Proof
+
+- Environment:
+- Exact command / steps:
+- Observed result:
+- Not tested:
+
+## Review Readiness
+
+- [ ] I have performed a self-review
+- [ ] This PR is ready for human review
+"""
+
+    report = module.validate_pull_request(_event(body))
+
+    assert report.valid is False
+    assert report.needs_author_action is True
+    assert module.AUTHOR_ACTION_LABEL in report.labels_to_add
+    assert any("Description" in problem for problem in report.problems)
+    assert any("Type of Change" in problem for problem in report.problems)
+    assert any("Test Output" in problem for problem in report.problems)
+    assert any("Real Behavior Proof" in problem for problem in report.problems)
+
+
+def test_validate_pull_request_skips_bot_authored_prs() -> None:
+    module = _load_module()
+
+    report = module.validate_pull_request(_event("", login="dependabot[bot]"))
+
+    assert report.valid is True
+    assert report.is_bot_pr is True
+    assert report.needs_author_action is False
+    assert report.labels_to_add == []

scripts/tests/test_pr_health_labels.py

@@ -0,0 +1,94 @@
+"""Tests for pr-health-labels.py."""
+
+from __future__ import annotations
+
+import importlib.util
+import sys
+from pathlib import Path
+
+
+def _load_module():
+    script = Path(__file__).parents[2] / ".github" / "scripts" / "pr-health-labels.py"
+    spec = importlib.util.spec_from_file_location("pr_health_labels", script)
+    assert spec is not None
+    assert spec.loader is not None
+    module = importlib.util.module_from_spec(spec)
+    sys.modules[spec.name] = module
+    spec.loader.exec_module(module)
+    return module
+
+
+def test_check_state_ignores_historical_failures_when_latest_attempt_passed() -> None:
+    module = _load_module()
+    payload = {
+        "statusCheckRollup": [
+            {
+                "__typename": "CheckRun",
+                "workflowName": "PR Governance",
+                "name": "template",
+                "conclusion": "FAILURE",
+                "startedAt": "2026-06-14T13:38:35Z",
+                "completedAt": "2026-06-14T13:38:46Z",
+            },
+            {
+                "__typename": "CheckRun",
+                "workflowName": "PR Governance",
+                "name": "template",
+                "conclusion": "SUCCESS",
+                "startedAt": "2026-06-14T13:43:26Z",
+                "completedAt": "2026-06-14T13:43:35Z",
+            },
+            {
+                "__typename": "CheckRun",
+                "workflowName": "PR Governance",
+                "name": "label",
+                "conclusion": "CANCELLED",
+                "startedAt": "2026-06-14T13:43:18Z",
+                "completedAt": "2026-06-14T13:43:24Z",
+            },
+            {
+                "__typename": "CheckRun",
+                "workflowName": "PR Governance",
+                "name": "label",
+                "conclusion": "SUCCESS",
+                "startedAt": "2026-06-14T13:43:26Z",
+                "completedAt": "2026-06-14T13:43:35Z",
+            },
+            {
+                "__typename": "CheckRun",
+                "workflowName": "",
+                "name": "GitGuardian Security Checks",
+                "conclusion": "SUCCESS",
+                "startedAt": "2026-06-14T13:38:32Z",
+                "completedAt": "2026-06-14T13:39:04Z",
+            },
+        ]
+    }
+
+    assert module.check_state(payload) == "passing"
+
+
+def test_check_state_fails_when_latest_attempt_failed() -> None:
+    module = _load_module()
+    payload = {
+        "statusCheckRollup": [
+            {
+                "__typename": "CheckRun",
+                "workflowName": "PR Governance",
+                "name": "template",
+                "conclusion": "SUCCESS",
+                "startedAt": "2026-06-14T13:38:35Z",
+                "completedAt": "2026-06-14T13:38:46Z",
+            },
+            {
+                "__typename": "CheckRun",
+                "workflowName": "PR Governance",
+                "name": "template",
+                "conclusion": "FAILURE",
+                "startedAt": "2026-06-14T13:43:26Z",
+                "completedAt": "2026-06-14T13:43:35Z",
+            },
+        ]
+    }
+
+    assert module.check_state(payload) == "failing"

scripts/tests/test_pr_health_workflow.py

@@ -0,0 +1,32 @@
+"""Tests for the PR governance workflow contract."""
+
+from __future__ import annotations
+
+from pathlib import Path
+
+
+def test_incomplete_pr_template_is_reported_without_failing_job() -> None:
+    workflow = Path(".github/workflows/pr-health.yml").read_text(encoding="utf-8")
+
+    assert "Fetch current PR body" in workflow
+    assert "--body-file .pr-body.md" in workflow
+    assert "Report incomplete PR body" in workflow
+    assert "PR template validation found missing fields" in workflow
+    assert "Fail when the PR body is incomplete" not in workflow
+    assert 'echo "PR template validation failed' not in workflow
+
+
+def test_ready_for_review_label_is_removed_when_changes_are_requested() -> None:
+    workflow = Path(".github/workflows/pr-health.yml").read_text(encoding="utf-8")
+
+    assert "reviewDecision" in workflow
+    assert 'review_decision="$(jq -r \'.reviewDecision // ""\'' in workflow
+    assert '$review_decision" == "CHANGES_REQUESTED"' in workflow
+
+
+def test_merge_state_unknown_does_not_clear_conflict_or_rebase_labels() -> None:
+    workflow = Path(".github/workflows/pr-health.yml").read_text(encoding="utf-8")
+
+    assert 'elif [[ "$merge_state" != "UNKNOWN" ]]; then' in workflow
+    assert 'gh pr edit "$pr" --repo "$REPO" --remove-label "status: needs rebase"' in workflow
+    assert 'gh pr edit "$pr" --repo "$REPO" --remove-label "status: has conflicts"' in workflow

scripts/tests/test_sync_plugin_versions.py

@@ -0,0 +1,131 @@
+"""Tests for sync-plugin-versions.py."""
+
+from __future__ import annotations
+
+import importlib.util
+from pathlib import Path
+
+
+def _load_module():
+    script = Path(__file__).parent.parent / "sync-plugin-versions.py"
+    spec = importlib.util.spec_from_file_location("sync_plugin_versions", script)
+    assert spec is not None
+    assert spec.loader is not None
+    module = importlib.util.module_from_spec(spec)
+    spec.loader.exec_module(module)
+    return module
+
+
+def test_compute_repo_semver_uses_release_helpers(monkeypatch) -> None:
+    module = _load_module()
+    calls: dict[str, object] = {}
+
+    monkeypatch.setattr(module, "list_release_tags", lambda root: ["v0.9.0"])
+    monkeypatch.setattr(module, "find_latest_release_tag", lambda tags: "v0.9.0")
+    monkeypatch.setattr(module, "list_release_commits", lambda root, tag: ["feat: add init"])
+    monkeypatch.setattr(module, "determine_bump_level", lambda commits: "minor")
+    monkeypatch.setattr(module, "get_canonical_version", lambda root: "0.5.25")
+
+    def fake_compute_release_version(*, canonical_version: str, level: str, tags: list[str]):
+        calls["canonical_version"] = canonical_version
+        calls["level"] = level
+        calls["tags"] = tags
+        return type("Info", (), {"npm_version": "0.10.0"})()
+
+    monkeypatch.setattr(module, "compute_release_version", fake_compute_release_version)
+
+    assert module.compute_repo_semver(Path("repo")) == "0.10.0"
+    assert calls == {
+        "canonical_version": "0.5.25",
+        "level": "minor",
+        "tags": ["v0.9.0"],
+    }
+
+
+def test_main_runs_plugin_only_version_sync(monkeypatch) -> None:
+    """Locks the sync-execution path. ``_should_sync`` is forced True so
+    we don't depend on the test machine's git branch context."""
+    module = _load_module()
+    commands: list[list[str]] = []
+
+    monkeypatch.setattr(module, "compute_repo_semver", lambda root: "0.10.0")
+    monkeypatch.setattr(module, "_should_sync", lambda root: True)
+    monkeypatch.setattr(
+        module.subprocess,
+        "run",
+        lambda command, cwd, check: commands.append(command),
+    )
+
+    module.main()
+
+    assert commands == [
+        [
+            module.sys.executable,
+            str(module.ROOT / "scripts" / "version-sync.py"),
+            "--root",
+            str(module.ROOT),
+            "--version",
+            "0.10.0",
+            "--plugin-manifests-only",
+        ]
+    ]
+
+
+def test_main_is_noop_on_feature_branch(monkeypatch, capsys) -> None:
+    """Locks the branch-aware contract: when ``_should_sync`` returns
+    False (feature branch, no HEADROOM_SYNC_VERSIONS opt-in), main()
+    prints a skip line and returns without invoking the version-sync
+    subprocess. Pre-this-fix the hook bumped manifests on every commit
+    regardless of branch — leaking version-bump noise into every PR."""
+    module = _load_module()
+    commands: list[list[str]] = []
+
+    monkeypatch.setattr(module, "_should_sync", lambda root: False)
+    monkeypatch.setattr(module, "_current_branch", lambda root: "feature/foo")
+    monkeypatch.setattr(
+        module.subprocess,
+        "run",
+        lambda *args, **kwargs: commands.append(args),
+    )
+
+    module.main()
+
+    assert commands == [], "Subprocess must not run when _should_sync returns False"
+    captured = capsys.readouterr()
+    assert "skipping on branch 'feature/foo'" in captured.out
+
+
+def test_should_sync_honours_env_override(monkeypatch) -> None:
+    """``HEADROOM_SYNC_VERSIONS=1`` forces a sync even on feature
+    branches — the release workflow uses this opt-in so the canonical
+    manifest sync still happens at publish time."""
+    module = _load_module()
+    monkeypatch.setenv("HEADROOM_SYNC_VERSIONS", "1")
+    # Even on a "feature" branch, env override wins.
+    monkeypatch.setattr(module, "_current_branch", lambda root: "feature/foo")
+    assert module._should_sync(Path("ignored")) is True
+
+
+def test_should_sync_main_branch_runs(monkeypatch) -> None:
+    """On ``main``, sync runs without needing the env var."""
+    module = _load_module()
+    monkeypatch.delenv("HEADROOM_SYNC_VERSIONS", raising=False)
+    monkeypatch.setattr(module, "_current_branch", lambda root: "main")
+    assert module._should_sync(Path("ignored")) is True
+
+
+def test_should_sync_feature_branch_is_skip(monkeypatch) -> None:
+    """On any non-main branch without the env var, sync is a no-op."""
+    module = _load_module()
+    monkeypatch.delenv("HEADROOM_SYNC_VERSIONS", raising=False)
+    monkeypatch.setattr(module, "_current_branch", lambda root: "fix/some-bug")
+    assert module._should_sync(Path("ignored")) is False
+
+
+def test_should_sync_returns_false_when_git_unavailable(monkeypatch) -> None:
+    """Defensive: if ``_current_branch`` returns None (git not on
+    PATH, detached HEAD, etc.) the safe default is no-op."""
+    module = _load_module()
+    monkeypatch.delenv("HEADROOM_SYNC_VERSIONS", raising=False)
+    monkeypatch.setattr(module, "_current_branch", lambda root: None)
+    assert module._should_sync(Path("ignored")) is False

scripts/tests/test_version_sync.py

@@ -0,0 +1,291 @@
+"""Tests for version-sync.py."""
+
+import json
+import subprocess
+import sys
+from pathlib import Path
+
+import pytest
+
+
+@pytest.fixture
+def temp_project(tmp_path: Path) -> dict[str, Path]:
+    """Create a temporary project with all versioned files."""
+    # Create directory structure
+    root = tmp_path / "project"
+    headroom = root / "headroom"
+    headroom.mkdir(parents=True)
+    repo_claude_plugin = root / ".claude-plugin"
+    repo_claude_plugin.mkdir(parents=True)
+    repo_github_plugin = root / ".github" / "plugin"
+    repo_github_plugin.mkdir(parents=True)
+    plugins = root / "plugins"
+    openclaw = plugins / "openclaw"
+    openclaw.mkdir(parents=True)
+    agent_hooks_claude = plugins / "headroom-agent-hooks" / ".claude-plugin"
+    agent_hooks_claude.mkdir(parents=True)
+    agent_hooks_github = plugins / "headroom-agent-hooks" / ".github" / "plugin"
+    agent_hooks_github.mkdir(parents=True)
+    sdk = root / "sdk"
+    typescript = sdk / "typescript"
+    typescript.mkdir(parents=True)
+
+    # pyproject.toml
+    pyproject = root / "pyproject.toml"
+    pyproject.write_text('[project]\nversion = "0.5.25"\n')
+
+    # headroom/_version.py is runtime-derived and must not be rewritten by version-sync.
+    version_py = headroom / "_version.py"
+    version_py.write_text('"""Package version metadata."""\n\n__version__ = "0.5.25"\n')
+
+    # plugins/openclaw/package.json
+    openclaw_pkg = openclaw / "package.json"
+    openclaw_pkg.write_text(json.dumps({"name": "test", "version": "0.5.25"}))
+
+    repo_claude_marketplace = repo_claude_plugin / "marketplace.json"
+    repo_claude_marketplace.write_text(
+        json.dumps(
+            {
+                "metadata": {"name": "claude-marketplace", "version": "0.1.0"},
+                "plugins": [{"name": "headroom-agent-hooks", "version": "0.1.0"}],
+            }
+        )
+    )
+
+    repo_github_marketplace = repo_github_plugin / "marketplace.json"
+    repo_github_marketplace.write_text(
+        json.dumps(
+            {
+                "metadata": {"name": "copilot-marketplace", "version": "0.1.0"},
+                "plugins": [{"name": "headroom-agent-hooks", "version": "0.1.0"}],
+            }
+        )
+    )
+
+    claude_plugin = agent_hooks_claude / "plugin.json"
+    claude_plugin.write_text(json.dumps({"name": "headroom-agent-hooks", "version": "0.1.0"}))
+
+    github_plugin = agent_hooks_github / "plugin.json"
+    github_plugin.write_text(json.dumps({"name": "headroom-agent-hooks", "version": "0.1.0"}))
+
+    # sdk/typescript/package.json
+    typescript_pkg = typescript / "package.json"
+    typescript_pkg.write_text(json.dumps({"name": "test", "version": "0.5.25"}))
+
+    return {
+        "root": root,
+        "pyproject": pyproject,
+        "version_py": version_py,
+        "openclaw_pkg": openclaw_pkg,
+        "repo_claude_marketplace": repo_claude_marketplace,
+        "repo_github_marketplace": repo_github_marketplace,
+        "claude_plugin": claude_plugin,
+        "github_plugin": github_plugin,
+        "typescript_pkg": typescript_pkg,
+    }
+
+
+def test_version_sync_explicit_version(temp_project: dict[str, Path]) -> None:
+    """Test --version flag updates all files."""
+    root = temp_project["root"]
+    script = Path(__file__).parent.parent / "version-sync.py"
+
+    result = subprocess.run(
+        [sys.executable, str(script), "--root", str(root), "--version", "0.7.0"],
+        capture_output=True,
+        text=True,
+    )
+
+    assert result.returncode == 0, f"Script failed: {result.stderr}"
+
+    # Verify pyproject.toml
+    pyproject_content = temp_project["pyproject"].read_text()
+    assert 'version = "0.7.0"' in pyproject_content
+
+    # Verify headroom/_version.py is not a synced manifest.
+    version_py_content = temp_project["version_py"].read_text()
+    assert '__version__ = "0.5.25"' in version_py_content
+
+    # Verify plugins/openclaw/package.json
+    openclaw_pkg = json.loads(temp_project["openclaw_pkg"].read_text())
+    assert openclaw_pkg["version"] == "0.7.0"
+
+    # Verify sdk/typescript/package.json
+    typescript_pkg = json.loads(temp_project["typescript_pkg"].read_text())
+    assert typescript_pkg["version"] == "0.7.0"
+
+    repo_claude_marketplace = json.loads(temp_project["repo_claude_marketplace"].read_text())
+    assert repo_claude_marketplace["metadata"]["version"] == "0.7.0"
+    assert repo_claude_marketplace["plugins"][0]["version"] == "0.7.0"
+
+    repo_github_marketplace = json.loads(temp_project["repo_github_marketplace"].read_text())
+    assert repo_github_marketplace["metadata"]["version"] == "0.7.0"
+    assert repo_github_marketplace["plugins"][0]["version"] == "0.7.0"
+
+    claude_plugin = json.loads(temp_project["claude_plugin"].read_text())
+    assert claude_plugin["version"] == "0.7.0"
+
+    github_plugin = json.loads(temp_project["github_plugin"].read_text())
+    assert github_plugin["version"] == "0.7.0"
+
+    # Verify .releasemetadata was created
+    release_metadata = root / ".releasemetadata"
+    assert release_metadata.exists()
+    metadata = json.loads(release_metadata.read_text())
+    assert metadata["version"] == "0.7.0"
+    assert metadata["packages"]["pypi"] == "0.7.0"
+    assert metadata["packages"]["npm-sdk"] == "0.7.0"
+    assert metadata["packages"]["npm-openclaw"] == "0.7.0"
+    assert metadata["packages"]["agent-hooks-plugin"] == "0.7.0"
+
+
+def test_bump_patch(temp_project: dict[str, Path]) -> None:
+    """Test --bump patch bumps 0.5.25 to 0.5.26."""
+    root = temp_project["root"]
+    script = Path(__file__).parent.parent / "version-sync.py"
+
+    result = subprocess.run(
+        [sys.executable, str(script), "--root", str(root), "--bump", "patch"],
+        capture_output=True,
+        text=True,
+    )
+
+    assert result.returncode == 0, f"Script failed: {result.stderr}"
+
+    # Verify all files updated to 0.5.26
+    pyproject_content = temp_project["pyproject"].read_text()
+    assert 'version = "0.5.26"' in pyproject_content
+
+    version_py_content = temp_project["version_py"].read_text()
+    assert '__version__ = "0.5.25"' in version_py_content
+
+    openclaw_pkg = json.loads(temp_project["openclaw_pkg"].read_text())
+    assert openclaw_pkg["version"] == "0.5.26"
+
+    typescript_pkg = json.loads(temp_project["typescript_pkg"].read_text())
+    assert typescript_pkg["version"] == "0.5.26"
+
+    claude_plugin = json.loads(temp_project["claude_plugin"].read_text())
+    assert claude_plugin["version"] == "0.5.26"
+
+
+def test_bump_minor(temp_project: dict[str, Path]) -> None:
+    """Test --bump minor bumps 0.5.25 to 0.6.0."""
+    root = temp_project["root"]
+    script = Path(__file__).parent.parent / "version-sync.py"
+
+    result = subprocess.run(
+        [sys.executable, str(script), "--root", str(root), "--bump", "minor"],
+        capture_output=True,
+        text=True,
+    )
+
+    assert result.returncode == 0, f"Script failed: {result.stderr}"
+
+    # Verify all files updated to 0.6.0
+    pyproject_content = temp_project["pyproject"].read_text()
+    assert 'version = "0.6.0"' in pyproject_content
+
+    version_py_content = temp_project["version_py"].read_text()
+    assert '__version__ = "0.5.25"' in version_py_content
+
+    openclaw_pkg = json.loads(temp_project["openclaw_pkg"].read_text())
+    assert openclaw_pkg["version"] == "0.6.0"
+
+    typescript_pkg = json.loads(temp_project["typescript_pkg"].read_text())
+    assert typescript_pkg["version"] == "0.6.0"
+
+    github_plugin = json.loads(temp_project["github_plugin"].read_text())
+    assert github_plugin["version"] == "0.6.0"
+
+
+def test_bump_major(temp_project: dict[str, Path]) -> None:
+    """Test --bump major bumps 0.5.25 to 1.0.0."""
+    root = temp_project["root"]
+    script = Path(__file__).parent.parent / "version-sync.py"
+
+    result = subprocess.run(
+        [sys.executable, str(script), "--root", str(root), "--bump", "major"],
+        capture_output=True,
+        text=True,
+    )
+
+    assert result.returncode == 0, f"Script failed: {result.stderr}"
+
+    # Verify all files updated to 1.0.0
+    pyproject_content = temp_project["pyproject"].read_text()
+    assert 'version = "1.0.0"' in pyproject_content
+
+    version_py_content = temp_project["version_py"].read_text()
+    assert '__version__ = "0.5.25"' in version_py_content
+
+    openclaw_pkg = json.loads(temp_project["openclaw_pkg"].read_text())
+    assert openclaw_pkg["version"] == "1.0.0"
+
+    typescript_pkg = json.loads(temp_project["typescript_pkg"].read_text())
+    assert typescript_pkg["version"] == "1.0.0"
+
+    repo_claude_marketplace = json.loads(temp_project["repo_claude_marketplace"].read_text())
+    assert repo_claude_marketplace["metadata"]["version"] == "1.0.0"
+
+
+def test_release_metadata_written(temp_project: dict[str, Path]) -> None:
+    """Test .releasemetadata is written correctly."""
+    root = temp_project["root"]
+    script = Path(__file__).parent.parent / "version-sync.py"
+
+    result = subprocess.run(
+        [sys.executable, str(script), "--root", str(root), "--version", "0.6.0"],
+        capture_output=True,
+        text=True,
+    )
+
+    assert result.returncode == 0, f"Script failed: {result.stderr}"
+
+    release_metadata = root / ".releasemetadata"
+    assert release_metadata.exists()
+
+    metadata = json.loads(release_metadata.read_text())
+    assert metadata == {
+        "version": "0.6.0",
+        "packages": {
+            "pypi": "0.6.0",
+            "npm-sdk": "0.6.0",
+            "npm-openclaw": "0.6.0",
+            "agent-hooks-plugin": "0.6.0",
+        },
+    }
+
+
+def test_plugin_manifests_only_leaves_package_versions_unchanged(
+    temp_project: dict[str, Path],
+) -> None:
+    """Test plugin-only sync leaves canonical package versions alone."""
+    root = temp_project["root"]
+    script = Path(__file__).parent.parent / "version-sync.py"
+
+    result = subprocess.run(
+        [
+            sys.executable,
+            str(script),
+            "--root",
+            str(root),
+            "--version",
+            "0.8.0",
+            "--plugin-manifests-only",
+        ],
+        capture_output=True,
+        text=True,
+    )
+
+    assert result.returncode == 0, f"Script failed: {result.stderr}"
+    assert 'version = "0.5.25"' in temp_project["pyproject"].read_text()
+    assert '__version__ = "0.5.25"' in temp_project["version_py"].read_text()
+    assert json.loads(temp_project["openclaw_pkg"].read_text())["version"] == "0.5.25"
+    assert json.loads(temp_project["typescript_pkg"].read_text())["version"] == "0.5.25"
+    assert json.loads(temp_project["claude_plugin"].read_text())["version"] == "0.8.0"
+    assert (
+        json.loads(temp_project["repo_github_marketplace"].read_text())["metadata"]["version"]
+        == "0.8.0"
+    )
+    assert not (root / ".releasemetadata").exists()

scripts/verify-versions.py

@@ -0,0 +1,66 @@
+#!/usr/bin/env python3
+"""Verify all package manifest versions are in sync before publishing."""
+
+import json
+from pathlib import Path
+
+try:
+    import tomllib
+except ImportError:  # pragma: no cover - Python 3.10 fallback
+    import tomli as tomllib
+
+ROOT = Path(__file__).parent.parent
+
+
+def _read_json_version(path: Path) -> str:
+    with open(path, encoding="utf-8") as f:
+        return str(json.load(f)["version"])
+
+
+def _read_marketplace_versions(path: Path) -> dict[str, str]:
+    with open(path, encoding="utf-8") as f:
+        payload = json.load(f)
+
+    versions: dict[str, str] = {}
+    metadata = payload.get("metadata")
+    if isinstance(metadata, dict):
+        versions[f"{path}:metadata"] = str(metadata.get("version"))
+    plugins = payload.get("plugins")
+    if isinstance(plugins, list):
+        for index, plugin in enumerate(plugins):
+            if isinstance(plugin, dict):
+                versions[f"{path}:plugins[{index}]"] = str(plugin.get("version"))
+    return versions
+
+
+def main() -> None:
+    with open(ROOT / "pyproject.toml", "rb") as f:
+        py_ver = tomllib.load(f)["project"]["version"]
+
+    versions = {
+        "pyproject.toml": py_ver,
+        "plugins/openclaw/package.json": _read_json_version(ROOT / "plugins/openclaw/package.json"),
+        "sdk/typescript/package.json": _read_json_version(ROOT / "sdk/typescript/package.json"),
+        "plugins/headroom-agent-hooks/.claude-plugin/plugin.json": _read_json_version(
+            ROOT / "plugins/headroom-agent-hooks/.claude-plugin/plugin.json"
+        ),
+        "plugins/headroom-agent-hooks/.github/plugin/plugin.json": _read_json_version(
+            ROOT / "plugins/headroom-agent-hooks/.github/plugin/plugin.json"
+        ),
+    }
+    versions.update(_read_marketplace_versions(ROOT / ".claude-plugin/marketplace.json"))
+    versions.update(_read_marketplace_versions(ROOT / ".github/plugin/marketplace.json"))
+
+    if not all(v == py_ver for v in versions.values()):
+        print("Version mismatch detected:")
+        for file, ver in versions.items():
+            print(f"  {file}: {ver}")
+        print(f"Expected all to be: {py_ver}")
+        raise SystemExit(1)
+
+    print(f"All versions aligned at {py_ver}")
+    print("Packages:", ", ".join(versions.keys()))
+
+
+if __name__ == "__main__":
+    main()

scripts/version-sync.py

@@ -0,0 +1,180 @@
+#!/usr/bin/env python3
+"""Synchronize version across all headroom packages."""
+
+from __future__ import annotations
+
+import argparse
+import json
+import re
+from pathlib import Path
+
+try:
+    import tomllib
+except ImportError:  # pragma: no cover - Python 3.10 fallback
+    import tomli as tomllib
+
+
+def get_version_from_pyproject(root: Path) -> str:
+    """Read version from pyproject.toml."""
+    pyproject_path = root / "pyproject.toml"
+    with open(pyproject_path, "rb") as f:
+        data = tomllib.load(f)
+    return data["project"]["version"]
+
+
+def bump_version(version: str, bump_type: str) -> str:
+    """Bump version according to bump_type (major, minor, patch)."""
+    major, minor, patch = map(int, version.split("."))
+    if bump_type == "major":
+        major += 1
+        minor = 0
+        patch = 0
+    elif bump_type == "minor":
+        minor += 1
+        patch = 0
+    elif bump_type == "patch":
+        patch += 1
+    return f"{major}.{minor}.{patch}"
+
+
+def update_package_json(file_path: Path, version: str) -> None:
+    """Update a package.json version field."""
+    with open(file_path, encoding="utf-8") as f:
+        data = json.load(f)
+    data["version"] = version
+    with open(file_path, "w", encoding="utf-8") as f:
+        json.dump(data, f, indent=2)
+        f.write("\n")
+
+
+def update_plugin_manifest(file_path: Path, version: str) -> None:
+    """Update a plugin.json version field."""
+    with open(file_path, encoding="utf-8") as f:
+        data = json.load(f)
+    data["version"] = version
+    with open(file_path, "w", encoding="utf-8") as f:
+        json.dump(data, f, indent=2)
+        f.write("\n")
+
+
+def update_marketplace_manifest(file_path: Path, version: str) -> None:
+    """Update marketplace metadata and plugin entry versions."""
+    with open(file_path, encoding="utf-8") as f:
+        data = json.load(f)
+    metadata = data.get("metadata")
+    if isinstance(metadata, dict):
+        metadata["version"] = version
+    plugins = data.get("plugins")
+    if isinstance(plugins, list):
+        for plugin in plugins:
+            if isinstance(plugin, dict):
+                plugin["version"] = version
+    with open(file_path, "w", encoding="utf-8") as f:
+        json.dump(data, f, indent=2)
+        f.write("\n")
+
+
+def update_plugin_versions(root: Path, version: str) -> None:
+    """Update marketplace and plugin manifest versions."""
+    update_marketplace_manifest(root / ".claude-plugin" / "marketplace.json", version)
+    update_marketplace_manifest(root / ".github" / "plugin" / "marketplace.json", version)
+    update_plugin_manifest(
+        root / "plugins" / "headroom-agent-hooks" / ".claude-plugin" / "plugin.json", version
+    )
+    update_plugin_manifest(
+        root / "plugins" / "headroom-agent-hooks" / ".github" / "plugin" / "plugin.json",
+        version,
+    )
+
+
+def update_openclaw_package_json(file_path: Path, version: str, sdk_version: str) -> None:
+    """Update openclaw package.json version and headroom-ai dependency range."""
+    with open(file_path, encoding="utf-8") as f:
+        data = json.load(f)
+    data["version"] = version
+    if "dependencies" in data and "headroom-ai" in data["dependencies"]:
+        data["dependencies"]["headroom-ai"] = f"^{sdk_version}"
+    with open(file_path, "w", encoding="utf-8") as f:
+        json.dump(data, f, indent=2)
+        f.write("\n")
+
+
+def update_pyproject_version(root: Path, version: str) -> None:
+    """Update pyproject.toml version."""
+    pyproject_path = root / "pyproject.toml"
+    content = pyproject_path.read_text(encoding="utf-8")
+    updated = re.sub(
+        r'^version = "[^"]+"',
+        f'version = "{version}"',
+        content,
+        flags=re.MULTILINE,
+    )
+    pyproject_path.write_text(updated, encoding="utf-8")
+
+
+def write_release_metadata(root: Path, version: str) -> None:
+    """Write .releasemetadata JSON file."""
+    metadata = {
+        "version": version,
+        "packages": {
+            "pypi": version,
+            "npm-sdk": version,
+            "npm-openclaw": version,
+            "agent-hooks-plugin": version,
+        },
+    }
+    metadata_path = root / ".releasemetadata"
+    with open(metadata_path, "w", encoding="utf-8") as f:
+        json.dump(metadata, f, indent=2)
+        f.write("\n")
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser(description="Synchronize version across headroom packages")
+    parser.add_argument(
+        "--root",
+        type=Path,
+        default=Path(__file__).parent.parent,
+        help="Root directory of the project",
+    )
+    group = parser.add_mutually_exclusive_group()
+    group.add_argument("--version", help="Explicit version to set (e.g., 0.6.0)")
+    group.add_argument(
+        "--bump",
+        choices=["major", "minor", "patch"],
+        help="Bump version from pyproject.toml",
+    )
+    parser.add_argument(
+        "--plugin-manifests-only",
+        action="store_true",
+        help="Only update marketplace/plugin manifest versions",
+    )
+    args = parser.parse_args()
+
+    if args.version:
+        version = args.version
+    elif args.bump:
+        base_version = get_version_from_pyproject(args.root)
+        version = bump_version(base_version, args.bump)
+    else:
+        version = get_version_from_pyproject(args.root)
+
+    if args.plugin_manifests_only:
+        update_plugin_versions(args.root, version)
+        print(f"Plugin versions synchronized to {version}")
+        return
+
+    # Update all versioned files
+    update_pyproject_version(args.root, version)
+    update_openclaw_package_json(
+        args.root / "plugins" / "openclaw" / "package.json", version, version
+    )
+    update_package_json(args.root / "sdk" / "typescript" / "package.json", version)
+    update_plugin_versions(args.root, version)
+    write_release_metadata(args.root, version)
+
+    print(f"Version synchronized to {version}")
+
+
+if __name__ == "__main__":
+    main()